I’ve already talked about the testing suite here, but this thing saved me countless hours of bug chasing, so I think it deserves one more article.

As you may probably know by now, this year’s Magic the Gathering core set, called Magic 2010, comes with a bunch of rules changes, that we wanted to implement in Wagic. Some of these changes were easy to code, such as removing the manaburn effect, others were a real pain in the a## such as the new blockers ordering thing.

Overall the new rules took me a few weeks to implement. Last week I also reworked the parsing engine so that it gets a bit more generic and less painful to work with. The code in MTGAbility.cpp and in AllAbilities.h is still way too big though, as these two files together have around 6500 lines of code. These two files handle the cards specific code, so since Wagic handles more than 3000 cards, you can say that each card requires an average of 2 lines of C++ code. Of course there’s no such thing as an “average” card, but rather hardcoded cards (that can take up to 50 lines of code), and “soft” coded cards, that the parser can handle.

Anyways I digress. I was saying that I had to rewrite the parser entirely last week. To make it clear, I rewrote and reorganized something like 2000 lines of code in a few days. This represents around 10% of the entire Wagic code, and since it’s the code that handles the way the cards act in the game, it is a very important part of Wagic. If I hadn’t the test suite with me, this would mean a painful phase of intense play-testing: create decks with the cards I want to test, hope for the exact situation to happen, confirm the card works, change the code and try again if it doesn’t, etc…

Here’s an example: Imagine I change the code for the First Strike ability, for some reason. How do I test that I didn’t break it? Well, yes, I could create a deck with a bunch of black knights, start a game against the AI (or a 2 players game against myself), and hope that my opponent will put a */2 creature with no first strike into play, then cross fingers and hope my black knight kills it without getting killed himself. If it doesn’t work, I have to fix the code, and test again…each test takes between 2 and 5 minutes, so if I’m really not good, or if the problem’s tougher than I expected, it could easily take me one hour just to make sure first strike works correctly.
Of course, when I changed first strike, maybe I broke double strike as well… So I should test that as well, and so on…
Did I mention Wagic handles more than 3000 cards? Manual test became a nightmare when we reached 300 cards, so I couldn’t even imagine what it would be today…

Enters the testing suite. The Testing suite basically takes a bunch of test files, runs them, and checks if the tests pass or if they don’t.
A test file is a simple text file made of 3 parts: An initial State, a series of actions, and a final state.

INIT

When it loads a test file, the Testing Suite initializes the game with the data from the initial state. So for example:



[INIT]
FIRSTMAIN
[PLAYER1]
hand:Ancestor’s chosen
graveyard:swamp,grizzly bears,dragon engine
manapool:{5}{W}{W}
[PLAYER2]
graveyard:black knight

---

---

The above init state is loaded by the testing suite. The Game engine is “forced” to go to the “first main phase” of player 1. Player 1 has
{5}{W}{W} in his manapool, Ancestor’s chosen in his hand, and 3 cards in his graveyard. Player 2 has 1 card in his graveyard.
(Obviously what we want to test here is Ancestor’s chosen ability: When Ancestor’s Chosen comes into play, you gain 1 life for each card in your graveyard)
The obvious advantage is that I don’t have to create a specific deck with Ancestor’s chosen an playtest it in order to to test its ability.

DO

As I said in my other article, the testing suite is like a CPU player, except it is very dumb: Instead of computing the best moves for its cards, it just takes orders from the test file. This is the second part of the test file for Ancestor’s chosen:



[DO]
Ancestor’s chosen

It looks simple? Well it is. What I tell the “dumb AI player” here is: just click on the Ancestor’s chosen card.
And that’s it. Since player 1 has the mana in its mana pool, and it’s his first main phase, clicking on the ancestor’s chosen card will put it into play.

ASSERT

Now that we’ve done the actions needed in the test, what we want is to compare the new state of the Game, with what we expect to happen. We write what we expect to happen in the third part of the test file:



[ASSERT]
FIRSTMAIN
[PLAYER1]
inplay:Ancestor’s chosen
life:23
graveyard:swamp,grizzly bears,dragon engine
manapool:{0}
[PLAYER2]
graveyard:black knight
[END]

So, if you compare this to the initial state we described above, the differences are the following: Ancestor’s chosen is in play, player 1 has no more left mana in his manapool, and he has 23 life (initial life of player 1 wasn’t described in the initial state. In that case the game uses the default value of 20).
Now, this is what the test expects to happen. The testing suite will of course compare this to the actual state of the game. If the expected and the actual states are not equivalent, then the test fails and the program lets me know about the error.

Here is the full test file for ancestor’s chosen:



#Testing Ancestors chosen
[INIT]
FIRSTMAIN
[PLAYER1]
hand:Ancestor’s chosen
graveyard:swamp,grizzly bears,dragon engine
manapool:{5}{W}{W}
[PLAYER2]
graveyard:black knight
[DO]
Ancestor’s chosen
[ASSERT]
FIRSTMAIN
[PLAYER1]
inplay:Ancestor’s chosen
life:23
graveyard:swamp,grizzly bears,dragon engine
manapool:{0}
[PLAYER2]
graveyard:black knight
[END]

you can find the other test files in the SVN repository.

Writing a test takes a few minutes. It is then added to the list of tests, and whenever I change some code, I can make sure that no important card functionality is broken.
Of course, even with a lot of tests, we still test only a very low percentage of the game situations, but the Testing suite allows us to test around 300 cards in 5 minutes. Doing the same thing manually in the game would take a few days for a very dedicated tester!

Last week, I rewrote the parser methods (the piece of code that reads the “_cards.dat” files and converts a text such as “lord(merfolk +1/+1” into actual C++ code). The Testing suite helped me catch 59 bugs and 3 memory leaks. I wouldn’t even have considered rewriting the parser if I hadn’t had the testing suite with me!