Page 1 of 1

Test suite question

Posted: Sat Jun 01, 2013 3:53 pm
by dhannum
Hi,

I'm new here. Been playing Wagic for about a year. I do have a programming background so I would like to help. I built the code on Windows and ran the test suite. It got around 40-50 failures, but the exact failures would vary with each run. For example the simple "cockatrice.txt" and "creature_bond" tests both failed and succeeded when I'd run them over and over. In the big group they'd fail, but then running just one test would pass.

I'm using the latest svn, but I'm wondering if I'm doing something wrong. Unless the test involves coin flips, shouldn't it be consistent?

I was going to fix broken tests or add new tests for bugs that I have found, but I'm stumped by the randomness of the test suite.

Thanks for the help, and great game!

Dan

Re: Test suite question

Posted: Sun Jun 02, 2013 1:41 am
by dhannum
I did see something here in the forums mentioning a race condition in the test suite, but I have a list of tests that failed 3 times in a row. I'd post the html output, but I can't :(

I'll wait to see what the experts think before I make a big deal of it.

Re: Test suite question

Posted: Sun Jun 30, 2013 11:51 pm
by Zethfox
though there are race condition fails in test suite when it is happening on a large scale, with many cards involved it usually points to an actual bug introduced with that code.

Re: Test suite question

Posted: Sun Jun 30, 2013 11:52 pm
by Zethfox
do you have a patch you can provide?

Re: Test suite question

Posted: Tue Jul 09, 2013 1:07 pm
by wololo
dhannum wrote:I did see something here in the forums mentioning a race condition in the test suite, but I have a list of tests that failed 3 times in a row. I'd post the html output, but I can't :(

I'll wait to see what the experts think before I make a big deal of it.
in TestSuiteAI.cpp, line 21:
http://code.google.com/p/wagic/source/b ... eAI.cpp#21

Try to replace the value "#define FAST_TEST 6" with something bigger. Try maybe 40, for a start. This will make the test suite slower, but that should remove the "race condition" possible cause. If you still see the tests fails when the suite runs slowly, then the problem is probably somewhere else.

that value is the number of frames the test waits between 2 actions. If your machine is fast, it is possible that 6 frames is not enough.

Note: the race conditions ARE bugs because of the way Wagic implemented some of its logic based on elapsed time instead of a proper event system. By increasing the value in the test suite you do not get rid of the bugs, you just hide them for the purpose of your tests. The goal being to hide that one "big" bug to allow you to test for the ones you really want to test.

Re: Test suite question

Posted: Thu Jul 18, 2013 1:08 am
by dhannum
Sorry! Got busy and never saw the reply. Let me give that a shot. I've discovered a number of small bugs (I think), and I'd love to provide a patch (I did C++ back in the day)

Re: Test suite question

Posted: Thu Jul 18, 2013 1:32 am
by wololo
don't apologize, I replied to your question very late too, sorry for that :oops:

Re: Test suite question

Posted: Thu Jul 18, 2013 11:22 am
by dhannum
No luck, I tried 40, 100, and even 400 but still about 20 tests fail, and when I spot check a few separately, they work.

That said, I'm not going to delve too deeply into that right now. I'd like to do something constructive rather than fight with the test suite. I'll circle back as time permits.