The testing suite

I’ve already talked about the testing suite here, but this thing saved me countless hours of bug chasing, so I think it deserves one more article.

As you may probably know by now, this year’s Magic the Gathering core set, called Magic 2010, comes with a bunch of rules changes, that we wanted to implement in Wagic. Some of these changes were easy to code, such as removing the manaburn effect, others were a real pain in the a## such as the new blockers ordering thing.

Overall the new rules took me a few weeks to implement. Last week I also reworked the parsing engine so that it gets a bit more generic and less painful to work with. The code in MTGAbility.cpp and in AllAbilities.h is still way too big though, as these two files together have around 6500 lines of code. These two files handle the cards specific code, so since Wagic handles more than 3000 cards, you can say that each card requires an average of 2 lines of C++ code. Of course there’s no such thing as an “average” card, but rather hardcoded cards (that can take up to 50 lines of code), and “soft” coded cards, that the parser can handle.

Anyways I digress. I was saying that I had to rewrite the parser entirely last week. To make it clear, I rewrote and reorganized something like 2000 lines of code in a few days. This represents around 10% of the entire Wagic code, and since it’s the code that handles the way the cards act in the game, it is a very important part of Wagic. If I hadn’t the test suite with me, this would mean a painful phase of intense play-testing: create decks with the cards I want to test, hope for the exact situation to happen, confirm the card works, change the code and try again if it doesn’t, etc…

Here’s an example: Imagine I change the code for the First Strike ability, for some reason. How do I test that I didn’t break it? Well, yes, I could create a deck with a bunch of black knights, start a game against the AI (or a 2 players game against myself), and hope that my opponent will put a */2 creature with no first strike into play, then cross fingers and hope my black knight kills it without getting killed himself. If it doesn’t work, I have to fix the code, and test again…each test takes between 2 and 5 minutes, so if I’m really not good, or if the problem’s tougher than I expected, it could easily take me one hour just to make sure first strike works correctly.
Of course, when I changed first strike, maybe I broke double strike as well… So I should test that as well, and so on…
Did I mention Wagic handles more than 3000 cards? Manual test became a nightmare when we reached 300 cards, so I couldn’t even imagine what it would be today…

Enters the testing suite. The Testing suite basically takes a bunch of test files, runs them, and checks if the tests pass or if they don’t.
A test file is a simple text file made of 3 parts: An initial State, a series of actions, and a final state.

INIT

When it loads a test file, the Testing Suite initializes the game with the data from the initial state. So for example:

[INIT]
FIRSTMAIN
[PLAYER1]
hand:Ancestor’s chosen
graveyard:swamp,grizzly bears,dragon engine
manapool:{5}{W}{W}
[PLAYER2]
graveyard:black knight

The above init state is loaded by the testing suite. The Game engine is “forced” to go to the “first main phase” of player 1. Player 1 has
{5}{W}{W} in his manapool, Ancestor’s chosen in his hand, and 3 cards in his graveyard. Player 2 has 1 card in his graveyard.
(Obviously what we want to test here is Ancestor’s chosen ability: When Ancestor’s Chosen comes into play, you gain 1 life for each card in your graveyard)
The obvious advantage is that I don’t have to create a specific deck with Ancestor’s chosen an playtest it in order to to test its ability.

DO

As I said in my other article, the testing suite is like a CPU player, except it is very dumb: Instead of computing the best moves for its cards, it just takes orders from the test file. This is the second part of the test file for Ancestor’s chosen:

[DO]
Ancestor’s chosen

It looks simple? Well it is. What I tell the “dumb AI player” here is: just click on the Ancestor’s chosen card.
And that’s it. Since player 1 has the mana in its mana pool, and it’s his first main phase, clicking on the ancestor’s chosen card will put it into play.

ASSERT

Now that we’ve done the actions needed in the test, what we want is to compare the new state of the Game, with what we expect to happen. We write what we expect to happen in the third part of the test file:

[ASSERT]
FIRSTMAIN
[PLAYER1]
inplay:Ancestor’s chosen
life:23
graveyard:swamp,grizzly bears,dragon engine
manapool:{0}
[PLAYER2]
graveyard:black knight
[END]

So, if you compare this to the initial state we described above, the differences are the following: Ancestor’s chosen is in play, player 1 has no more left mana in his manapool, and he has 23 life (initial life of player 1 wasn’t described in the initial state. In that case the game uses the default value of 20).
Now, this is what the test expects to happen. The testing suite will of course compare this to the actual state of the game. If the expected and the actual states are not equivalent, then the test fails and the program lets me know about the error.

Here is the full test file for ancestor’s chosen:

#Testing Ancestors chosen
[INIT]
FIRSTMAIN
[PLAYER1]
hand:Ancestor’s chosen
graveyard:swamp,grizzly bears,dragon engine
manapool:{5}{W}{W}
[PLAYER2]
graveyard:black knight
[DO]
Ancestor’s chosen
[ASSERT]
FIRSTMAIN
[PLAYER1]
inplay:Ancestor’s chosen
life:23
graveyard:swamp,grizzly bears,dragon engine
manapool:{0}
[PLAYER2]
graveyard:black knight
[END]

you can find the other test files in the SVN repository.

Writing a test takes a few minutes. It is then added to the list of tests, and whenever I change some code, I can make sure that no important card functionality is broken.
Of course, even with a lot of tests, we still test only a very low percentage of the game situations, but the Testing suite allows us to test around 300 cards in 5 minutes. Doing the same thing manually in the game would take a few days for a very dedicated tester!

Last week, I rewrote the parser methods (the piece of code that reads the “_cards.dat” files and converts a text such as “lord(merfolk +1/+1″ into  actual C++ code). The Testing suite helped me catch 59 bugs and 3 memory leaks. I wouldn’t even have considered rewriting the parser if I hadn’t had the testing suite with me!

  1. John Rohan’s avatar

    You’re the man! I am astounded by the amount of effort you’ve put into this program.

    Reply

  2. Hellfish’s avatar

    Video not available due to a copyright claim by Wizards of the Coast, Inc.

    Bad news for wagic? :(

    Reply

  3. Dr. Solomat’s avatar

    video has been removed. ;(

    Reply

  4. NyghtcrawleR’s avatar

    As always I cant completely comprehend it since I really dont know how to code, but I can very well appreciate it and understand how vital it has been.

    Again, you do some great work and Wagic shows it.

    Reply

  5. wololo’s avatar

    Don’t worry about the video, it had actually been removed when I got the C&D 6 months ago, but has I am the owner I can still see it, so I didn’t realize it wasn’t available. I’ll upload a new one.

    Reply

  6. SeekingTruth’s avatar

    Oooh! I’m not much of a coder myself but I used to be, and that’s a very neat trick you’re doing with the testing suite! Kudos Wololo, you’re the man, and Wagic is fabulous! It’s great being able to play outside with friends when it’s too windy for real cards. :)

    Reply

  7. BlackMamba (moxdev)’s avatar

    Excellent, I love automated tests :) They also give me courage to add new features!

    Reply

  8. Lord Raptor’s avatar

    Just a little comment to say that Wagic has resurrected my PSP. I spend 1 or 2 hours every day playing it, and I enjoy it alot !

    Very good work !

    Thank you for all the time you spend and all efforts you make on this awesome project !

    Reply

  9. Xuelynom’s avatar

    Hi

    Is your card file available somewhere ? the file with the “lord (merfolk +1/+1″ ?

    Reply

  10. wololo’s avatar

    @Xuelynom, sure, check out the SVN:
    this is 10E, there’s one file for each expansion
    http://code.google.com/p/wagic/source/browse/trunk/projects/mtg/bin/Res/sets/10E/_cards.dat

    Reply

  11. Xuelynom’s avatar

    Thanks, I was wondering what abilities are supported by your engine, and how you write them. It seems pretty complete.
    What are the abilities you can’t program ?
    How do you handle triggered abilities ? do you save all of them somewhere or do you watch each permanent each time something happens to see if an ability is triggered ?
    The same for global static continuous effects such as “all your soldiers gains +1/+1″, do you save all those effects in a single list or do you watch each permanent each time you need to check something?
    My “thaum” class (the base magic effect) is getting more and more complex, first it had just a cost and an effect, then a boolean if the effect was targeted, then a list of targets instead of just one target, then a boolean if its “you may”, then effect becomes a list of effects to handle “+3/+3 AND trample”…

    Reply

Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>