[hatari-devel] Code-review, unit tests and bug-tracker

[Deniz Turkoglu]

On Mon, May 9, 2011 at 10:41 PM, Eero Tamminen <eerot at users.berlios.de> wrote:
> Hi,
>
> On lauantai 07 toukokuu 2011, Deniz Turkoglu wrote:
>> I have been following the list for sometime and now I have the time to
>> contribute more, at least I hope so :) I have a few
>> questions/suggestions, mainly based on the flow we use at work;
>>
>> - We use ReviewBoard http://www.reviewboard.org/ at work. I think this
>> is nice for a few reasons. It makes discussing on changes easier and
>> you can associate changes with bugs and end results tends to be
>> better. I think this would speed up the discussion that goes on
>> patches?
>
> If BerliOS will support reviewboard, why not.

It doesn't seem to have it.

http://www.berlios.de/developers.php.en

>
>
>> - I tried the bug tracker to have a go at things but seems it is not
>> as active as the mailing list, maybe we need another bug tracker, like
>> bugzilla or trac?
>
> You can file bugs/tickets for BerliOS bug tracker itself too I assume.
>
>
> Or were you proposing on hosting reviewboard and bug tracking services
> for Hatari yourself?

The bug tracker and reviewboard can be hosted elsewhere of course. I
have shared hosting and can give it a go, it is not a fancy hosting
company though, so maybe setting up a home server is a better option.

>
>> - There is already a tests dir with some coverage, I am now looking at
>> C unit tests frameworks and I stumbled upon
>> http://check.sourceforge.net/ , any ideas on this?
>
> It's OK as C unit-test framework (I have some collegues at work who use it),
> but it's not very applicable for Hatari.
>
> The test/ directory contains two kind of things:
> - Code testing Hatari debugger
> - Code to run on real HW and inside the emulator
>  (output should be same on both)
>
> For former I prefer something that outputs extra debugging output while
> it runs, and which finishes all checks within a test for particular area,
> instead of breaking on first failure.  I also want it to be easy (automatic)
> to run code under Valgrind & GCC Mudflap and the current Makefile already
> does that.  Check is overkill for that.
>
> For latter it's not unit test framework isn't what's needed but manual work
> with real HW for comparison.
>
> Main bulk of Hatari testing is running different (hundreds of) Atari ST, STE
> and Falcon programs and just checking whether their output looks OK without
> (much) need to compare it to real HW though.  And checking that the output
> hasn't regressed.
>
> If you want to look into that, you should try "hconsole" python program in
> tools/hconsole/ and try the Hatari remote API on how you automate[1] running
> of the tests (ST/STE/Falcon demos).  Through that you could e.g. run demos
> and record sound output & get screenshots[1] from them.
>
> Comparing the sound files could be a bit inconvenient because 1-1 file
> compare probably isn't the best thing (you'd catch changes, not whether
> they're regressions).  I think for screenshots 1-1 checking should be fine
> for catching regressions though.
>
>
> [1]  Automating other things than demos would probably require adding
>    support for positioning Atari mouse on some specific co-ordinate
>    on Atari screen.
>
>    To get them screenshots etc at suitable time in the demo, adding some
>    kind of "pause at VBL n from start/now" functionality to Hatari might be
>    needed.  Or "dump screenshot at exit" coupled with "--run-vbls" option,
>    if one screenshot per demo is enough.
>
>   Somehow detecting bus errors and other emulated Atari exceptions
>   through the remote API could also be useful.

Perfect, I have done some work on this area, it turned out to be
harder than it looked at first sight :)

Screenshots: we hash the buffer and do a compare on host (i.e: hashes
are sent to hatari and if it matches any it is a pass, else we
compress to png and send it over the wire. This is probably not very
useful for hatari as is since timing issues with frames are always a
problem, esp. considering different OS and devices. The idea to add
pause functionality sounds like a very good idea to solve this
bottleneck.

Mouse/keyboard interaction: there are 2 options with this one, we can
either fake it within hatari or we can directly do it within OS level,
both has its ups and downs, if we fake it we can not be sure if the
cursor really moved, was visible on screen etc.. If we do it natively
it requires to port to every single OS that we run on. Also one neat
part about doing it hatari level is that it can compile and run tests
and you can still work on the computer at the same time, will have a
look at this and how feasible it is.

Audio: This one is rather tricky and I haven't done it before, 1-1
compare using the hash method can help with the regression, for
instance if we have a blacklist of hashes (regressions) we can find
out the root cause easier. Comments on this are more than welcome :)

Exceptions and errors: These can be sent as events over the wire?

-deniz
>
>
>        - Eero
> _______________________________________________
> hatari-devel mailing list
> hatari-devel at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/hatari-devel
>