[hatari-devel] Code-review, unit tests and bug-tracker

[Eero Tamminen]

Hi,

On lauantai 07 toukokuu 2011, Deniz Turkoglu wrote:
> I have been following the list for sometime and now I have the time to
> contribute more, at least I hope so :) I have a few
> questions/suggestions, mainly based on the flow we use at work;
> 
> - We use ReviewBoard http://www.reviewboard.org/ at work. I think this
> is nice for a few reasons. It makes discussing on changes easier and
> you can associate changes with bugs and end results tends to be
> better. I think this would speed up the discussion that goes on
> patches?

If BerliOS will support reviewboard, why not.


> - I tried the bug tracker to have a go at things but seems it is not
> as active as the mailing list, maybe we need another bug tracker, like
> bugzilla or trac?

You can file bugs/tickets for BerliOS bug tracker itself too I assume.


Or were you proposing on hosting reviewboard and bug tracking services
for Hatari yourself?


> - There is already a tests dir with some coverage, I am now looking at
> C unit tests frameworks and I stumbled upon
> http://check.sourceforge.net/ , any ideas on this?

It's OK as C unit-test framework (I have some collegues at work who use it),
but it's not very applicable for Hatari.

The test/ directory contains two kind of things:
- Code testing Hatari debugger
- Code to run on real HW and inside the emulator
  (output should be same on both)

For former I prefer something that outputs extra debugging output while
it runs, and which finishes all checks within a test for particular area,
instead of breaking on first failure.  I also want it to be easy (automatic)
to run code under Valgrind & GCC Mudflap and the current Makefile already
does that.  Check is overkill for that.

For latter it's not unit test framework isn't what's needed but manual work
with real HW for comparison.


Main bulk of Hatari testing is running different (hundreds of) Atari ST, STE
and Falcon programs and just checking whether their output looks OK without
(much) need to compare it to real HW though.  And checking that the output
hasn't regressed.

If you want to look into that, you should try "hconsole" python program in
tools/hconsole/ and try the Hatari remote API on how you automate[1] running
of the tests (ST/STE/Falcon demos).  Through that you could e.g. run demos
and record sound output & get screenshots[1] from them.

Comparing the sound files could be a bit inconvenient because 1-1 file 
compare probably isn't the best thing (you'd catch changes, not whether
they're regressions).  I think for screenshots 1-1 checking should be fine
for catching regressions though.


[1]  Automating other things than demos would probably require adding
    support for positioning Atari mouse on some specific co-ordinate
    on Atari screen.

    To get them screenshots etc at suitable time in the demo, adding some
    kind of "pause at VBL n from start/now" functionality to Hatari might be
    needed.  Or "dump screenshot at exit" coupled with "--run-vbls" option,
    if one screenshot per demo is enough.

   Somehow detecting bus errors and other emulated Atari exceptions
   through the remote API could also be useful.


	- Eero