How to setup and run your own programming contest

It’s been a couple of weeks since my last post as there’s been less going on with the programming teams lately due to exams. This post is about how to run your own programming contest, but first, I thought I should mention that all the UVic team members were recognized and awarded Outstanding Leadership Awards last night at the computer science department Christmas party. There’s also been talk about establishing some kind of yearly high school programming competition in BC. It would potentially be great for recruitment, be a good way to get young people interested in computer science, and hopefully keep the top BC students in BC.

Ok, now onto the topic: how to setup and run your own programming contest.

Before I get into the details, I’ll just mention a few bad experiences I have had as a competitor. You may not think organizing a contest would be that difficult, but it can be quite a disaster if there isn’t much care put into it. I use to compete in the Northeast North America regional. This is one of the few regions in North America that has preliminary rounds. Between all the contest sites, there is no consistency in the problem writing style or the submission software used. Year to year, the contest submission software changes and not always for the better.

In one particular preliminary contest, every team had to use a Windows system, but telnet into a Unix box, and use a single non-resizable terminal for coding and submitting. Even more surprising was that the problems were not printed for the teams before the contest and teams had to read the problems directly on the machine. We were told that we could print screen the problems piece by piece if we needed print outs.

Since only one terminal was supported, you would of course be forced to switch between your code editor back to the single terminal for compiling, running, and submitting. The submission software consisted of shell scripts, solution judgements were written into a file, which you would need to periodically check.

Many of the input files had been created on a Windows machine and ported to Unix, which led to strange end of line and end of file characters. Finally, there was a number of errors in both the judge’s test cases and the problem specifications. Not surprisingly, over half the teams solved nothing, and we came in second with a grand total of three problems solved.

One of the other issues with the preliminary round was the problem writing inconsistency. This may have been due to lack of experience from the problem writters or lack of care from the organizers, but each problem would usually be formatted completely different. Some looked as if they were typed in Notepad, with no input and output specified, sometimes without any sample test cases. Others may be a PDF file, written in Latex. Usually if there was some description of the input and output, the constraints of the problem would not be specified.

Now that the horror stories are complete, as for running your own contest, there are several options and it’s not really that complicated, but is perhaps a little daunting for those unfamiliar with competing. To start with my own experience, I’ve written problems for, acted as judge, organized, and ran about half a dozen contests, both at UVic and the University of New Brunswick (UNB). Probably the most difficult part, besides getting students interested in coming out for such an event, is coming up with the problem set, judge’s inputs and outputs, all without major errors.

The goal with most ACM-style contests is to create a problem set such that every student/team solves at least one problem, no student/team solves all the problems, but all problems are solved by someone. This is very tricky to pull off. If the problems are too simple, then you may end up with a situation like this year’s Greater NY Regional where nine teams solved all the problems. This is usually a sign of a poorly designed and potentially unfair contest. It potentially rewards the fastest coders for simple problems rather than the best problem solvers.

One of the major issues I’ve struggled with in any contest that I’ve run is the large variance in skill level between the few exceptional students I had access to and the less experienced students. You don’t want the more advanced students solving everything in the first two hours, but you don’t want to make the problem set so difficult that the majority of the students leave frustrated and embarrassed. I’ve been both successful and unsuccessful at achieving this balance.

For example, in the last contest I ran at UNB, the top two students solved 6/7 problems, two students solved 2, two solved 1, and everyone else solved nothing. I felt quite bad judging this contest, and many of the students left frustrated. However, the first UVic contest that I organized, had a much better balance (scoreboard).

In individual contests, I have usually tried to have between 6 and 7 problems, with at least one very simple problem to get everyone started on the right foot. I also usually organize the problems roughly from easiest to hardest, as a lot of inexperienced people will start with the first problem regardless of its difficulty. I also usually try to have the second or third easiest problem be something that is easy to understand conceptually, but may take some time to implement. My goal here is to give less experienced students something to work on and potentially solved, but it will keep them busy for a while. Finally, I try to mix in some shorter to code problems that require a bit of insight.

If you are simply trying to set up a contest environment for team practice, then you can probably skip the problem writing stage. If you go to the regional page on the ACM ICPC website and follow the links to various regionals, many will include their problem set, judges input and output, and solutions. You can use these files to create your own practice contest environment.

In all the contests that I’ve organized, I’ve used the PC^2 software for problem submission and judging. I recommend using version 9, as it’s much better with firewall issues. PC^2 is quite simple to set up. The software is written in Java, so it runs on most platforms. It has several different programs, a server, scoreboard, judge, admin, and client. You start by running the server, and all other programs connect through the server. Configuration is as simple as modifying a config file to point to the server location. The administration control panel is used for creating accounts, inputting problems, and configuring how the judgements will run. Judging can be done automatically or semi-automatically.

I always create a judge’s set of input and output, and configure PC^2 with that information for each problem. I also use automatic judging as a guide. If the automatic judge gives me an accept, I usually take it. If there’s a different answer, I’ll do a side by side comparison between the submitted program output and my judge’s output just to make sure.

The client software is what the students or teams use to submit their solutions. They can select the problem to submit under, the language of the source code, and the source file. Submissions are put into a queue on the judge’s side. The client software also supports clarification submissions, which are questions about the problem set that you can send to the judge.

The scoreboard is very simple, it writes the current scores into different html files. The summary.html is the best view. The admin software allows you to configure when the scoreboard should be frozen, typically for the last hour or half hour in a contest.

There are other contest systems out there. CodeChef, which I wrote about in my last post, will host contests for you. One of the advantages of using PC^2 is that it is the system used both at the world finals and at a lot of the regional level contests so it exposes your team to the system early. It is very simple to use, so it’s probably not a huge gain, but it may help a bit :-).