Test suite question

If you have a question about the git repository and the code, or if you have code to propose then please post in this forum.
Do not post request here.
Post Reply
dhannum
Posts: 5
Joined: Sat Jun 01, 2013 3:47 pm

Test suite question

Post by dhannum » Sat Jun 01, 2013 3:53 pm

Hi,

I'm new here. Been playing Wagic for about a year. I do have a programming background so I would like to help. I built the code on Windows and ran the test suite. It got around 40-50 failures, but the exact failures would vary with each run. For example the simple "cockatrice.txt" and "creature_bond" tests both failed and succeeded when I'd run them over and over. In the big group they'd fail, but then running just one test would pass.

I'm using the latest svn, but I'm wondering if I'm doing something wrong. Unless the test involves coin flips, shouldn't it be consistent?

I was going to fix broken tests or add new tests for bugs that I have found, but I'm stumped by the randomness of the test suite.

Thanks for the help, and great game!

Dan

dhannum
Posts: 5
Joined: Sat Jun 01, 2013 3:47 pm

Re: Test suite question

Post by dhannum » Sun Jun 02, 2013 1:41 am

I did see something here in the forums mentioning a race condition in the test suite, but I have a list of tests that failed 3 times in a row. I'd post the html output, but I can't :(

I'll wait to see what the experts think before I make a big deal of it.

Zethfox
Posts: 3022
Joined: Thu Jun 10, 2010 11:28 pm

Re: Test suite question

Post by Zethfox » Sun Jun 30, 2013 11:51 pm

though there are race condition fails in test suite when it is happening on a large scale, with many cards involved it usually points to an actual bug introduced with that code.

Zethfox
Posts: 3022
Joined: Thu Jun 10, 2010 11:28 pm

Re: Test suite question

Post by Zethfox » Sun Jun 30, 2013 11:52 pm

do you have a patch you can provide?

wololo
Site Admin
Posts: 3727
Joined: Wed Oct 15, 2008 12:42 am
Location: Japan

Re: Test suite question

Post by wololo » Tue Jul 09, 2013 1:07 pm

dhannum wrote:I did see something here in the forums mentioning a race condition in the test suite, but I have a list of tests that failed 3 times in a row. I'd post the html output, but I can't :(

I'll wait to see what the experts think before I make a big deal of it.
in TestSuiteAI.cpp, line 21:
http://code.google.com/p/wagic/source/b ... eAI.cpp#21

Try to replace the value "#define FAST_TEST 6" with something bigger. Try maybe 40, for a start. This will make the test suite slower, but that should remove the "race condition" possible cause. If you still see the tests fails when the suite runs slowly, then the problem is probably somewhere else.

that value is the number of frames the test waits between 2 actions. If your machine is fast, it is possible that 6 frames is not enough.

Note: the race conditions ARE bugs because of the way Wagic implemented some of its logic based on elapsed time instead of a proper event system. By increasing the value in the test suite you do not get rid of the bugs, you just hide them for the purpose of your tests. The goal being to hide that one "big" bug to allow you to test for the ones you really want to test.

dhannum
Posts: 5
Joined: Sat Jun 01, 2013 3:47 pm

Re: Test suite question

Post by dhannum » Thu Jul 18, 2013 1:08 am

Sorry! Got busy and never saw the reply. Let me give that a shot. I've discovered a number of small bugs (I think), and I'd love to provide a patch (I did C++ back in the day)

wololo
Site Admin
Posts: 3727
Joined: Wed Oct 15, 2008 12:42 am
Location: Japan

Re: Test suite question

Post by wololo » Thu Jul 18, 2013 1:32 am

don't apologize, I replied to your question very late too, sorry for that :oops:

dhannum
Posts: 5
Joined: Sat Jun 01, 2013 3:47 pm

Re: Test suite question

Post by dhannum » Thu Jul 18, 2013 11:22 am

No luck, I tried 40, 100, and even 400 but still about 20 tests fail, and when I spot check a few separately, they work.

That said, I'm not going to delve too deeply into that right now. I'd like to do something constructive rather than fight with the test suite. I'll circle back as time permits.

Post Reply