[contestms] Another contest post-mortem

[Here's hoping 2012 is the year of unbroken HTML-enabled posts. Apologies to anybody who is offended by or otherwise unable to read this].

Another month, another contest. 35ish contestants, 3 questions, 4 hours.

This one went much more smoothly than last time, with no queue explosions. Although the work queue was quite full at times (up to 10 jobs waiting), I never observed any duplicates in the queue. Our test data was also much less demanding, leading to faster evaluations, so this may have contributed to not hitting any bugs in the queue too! :)

Setup was forked off giomasce/master on 7 Mar (b23c40f8...), with some local changes applied. https://github.com/bblackham/cms/commits/fario2012 has the full list of changes and is what we ran for the contest. Thanks Stefano for some speedy debugging too!

We used the USACO-style timing to start the contest upon login. All tasks were standard batch tasks, with one of them having a custom comparator (required some changes to the web interface - fd415e28).

One of the most useful changes for us was some modified ScoreTypes for using sub-tasks (f7257f56 and d44c7cb5). We changed the score parameters to ScoreTypeGroupMin to specify: (a) a name for a subtask (b) a weight (c) a list of test-case ids to include. This meant we could re-use test cases between subtasks, and display to the user some meaningful case names too, making it look far more polished. e.g.:

The only drawback of this was the fragile nature of the score parameter's syntax - there's lots of room for user error. And errors aren't handled cleanly at the moment, which results in python exceptions from anything that computes the scores, including the admin page on which you'd modify this. So if you get a syntax error, you have to fix it in the DB. This probably wants two separate fixes: (1) support for a scoring type to return scoring failure, which postpones scoring for that submission, but allows ScoringService to keep scoring others, and (2) support for checking the score parameters before they get committed to the DB.

408698ed adds a command-line utility to submit solutions via HTTP. I used this for some rudimentary testing. The commit also changes the stress-testing tool to not require a patched python-mechanize (works with the python-mechanize 0.2.5). Next useful feature on that would be to poll for the results of the submission too!

For posterity, more details: hardware was a dual-core 3 GHz Xeon, running Ubuntu 11.10 in a chroot on a 2.6.26 Debian kernel.

The ignore feature on questions was really useful - thanks DavidG!

Some other random TODOs for me are below. I may get to them eventually.

Bernard.