To: GA Members at IOI2002 This message presents some important topics for discussion at IOI2002 concerning future IOI competitions. Feel free to skip the introduction. Introduction ------------ All Delegation Leaders and Deputies invited for participation in IOI2002 will form the General Assembly (GA), which is "a temporary, short-term committee during IOI2002" and "the owner of the IOI". They have had an opportunity to read the rewritten IOI Regulations and to provide feedback. If you have done so, then you will have noticed that the rewritten IOI Regulations have remained very close to the previous version, in WHAT is captured by the rules. The improvement consists for a large part in HOW the rules are expressed. In particular, the (rewritten) IOI Regulations leave a lot of "freedom" concerning the preparation and execution of the IOI competition. Over the past thirteen years, the IOI has built up a TRADITION, which is anchored in the shared memory of the GA, of the International Committee (IC) and, since 1999, also of the IOI Scientific Committee (ISC). The role of the ISC can be described in many ways. For the purpose of this message, the following view is appropriate. The preparation and execution of an IOI competition involve two major parties: * The participating countries, that is, the GA and their contestants. * The Organizing Committee of the organizing Host On various issues, these parties have diverging interests. The Organizing Committee has limited resources (time, people, budget, ...) and a big responsibility. The GA has high expectations about the prestigious competition that the IOI has become, and for which they prepare their contestants. For example, the Present Host wants the best deal on the 400+ computers they need to provide, and this requires some freedom in the time domain. The GA want to know as early as possible what computers will be used, so they can better prepare their contestants. The role of the ISC is to help in BALANCING the interests of these two parties concerning the IOI competition. For that reason, the ISC has four members that represent the Past Host (1x), Present Host (2x), and Future Host (1x), and also three members elected from the GA at large. Throughout the year, there is close contact between the Present Host, especially its (local) Scientific Committee, and the ISC. The contact between the ISC and the GA is more limited, for several reasons: * The GA "exists" only during the IOI, or maybe from the moment of registration for the IOI (although this is not clear from the IOI Regulations) * The GA is large, with 80 participating countries it counts 160 members. * The GA is diverse, with various cultural and educational backgrounds. * The GA has limited time and many "distractions", such as full-time jobs and national competitions. In this message, I wish to raise some issues concerning the IOI competition, on which the ISC would like to hear the opinion of the GA. This opinion is intended to "complement" the IOI Regulations. I hope that we will be able to make clear decisions on (a number of) these items. It is NOT our aim to criticize the IOI Regulations or to write extra rules, but we do want to get feedback and guidance from the GA. Discussion Items Concerning the IOI Competition ---------- ----- ---------- --- --- ----------- I have grouped the items into four categories: * Format * Competition tasks * Scores for submissions * Grading process You may find that some important items are missing. This may be because the ISC already has a clear opinion about it, or it may be an oversight. Just let us know. In the following, the references to the IOI Regulations refer to the proposed rewritten version. Format of the IOI Competition ------ -- --- --- ----------- The IOI Regulations prescribe "two Competition Days both of which are directly preceded and followed by a non-Competition Day" (S6.1). Furthermore, "each Contestant must work independently ... with an appropriate computer ..." (S6.5). Tradition has added to this: * Each competition day offers THREE competition tasks to be solved in FIVE hours. * The competition tasks cover varying problem domains, solution techniques, and difficulty levels. and more recently also * A Practice Session before the first competition day How "strict" are these traditions? What process is appropriate to discuss/make changes with respect to these traditions? Would it be acceptable by both parties if changes to these traditions are discussed and approved at the IOI preceding the IOI where the changes would apply? These questions also apply to the following points. Competition Tasks ----------- ----- The IOI Regulations do not really prescribe anything concerning the content of the competition tasks, other than that the IOI is an "informatics competition" (S1.1), where informatics is explained in (S1.6) as "computer science, computing science and information technology, but not the domain computer engineering". Tradition has given us Competition Rules (written for each IOI separately) that state that * the competition tasks "are designed to be algorithmic in nature". Less explicit is the tradition that * the tasks focus on PROBLEM SOLVING, and that the tasks are solvable by ELEMENTARY methods from informatics. The meaning of "elementary" is open for discussion, and in the past has ruled out, or was felt to must rule out, such things as * numerical (floating-point) computations * parallel computations * max-flow network computations * various advanced heuristics The GA may also find it desirable that * the organizers have at least one solution with accompanying scientific arguments that their solution scores 100%, and that this solution will be published immediately after the competition. In fact, the abstract solution must be implemented in all allowed programming languages and must indeed have been evaluated at a score of 100% by the grading system as used in the competition. though this has been less obvious in the past, and it does rule out some interesting ideas. Tradition has also given a limited set of task types: * batch-style (predetermined, secret input; evaluated on output) * reactive-style (dialogue involving interdependent input and output) * output-only (or non-secret input) Scores for Submissions at the IOI Competition ------ --- ----------- -- --- --- ----------- Again, the IOI Regulations do not really say anything about scores, other than that S6.11 and S6.12 mention "number of points achieved by contestants" (e.g. as basis for the awarding of medals). Tradition has fluctuated somewhat, but now has converged on * scores of 0 to 100 INTEGER points per task, * scores obtained by ADDING the scores of a sufficient number of separate, unweighed test cases, * (in case of program submissions) black-box test runs with predetermined, secret inputs, which are the same for all contestants, * an opportunity for PARTIAL CREDIT per test case for some, but not necessarily all, tasks on an IOI. Grading Process ------- ------- By the grading process, I mean the organizational and technical aspects of determining and approving scores for all submissions. The IOI Regulations, again, say very little about this, other than requiring that the Host prepares Judging Procedures for the purpose of defining the grading process (S5.7, S6.9), and that the outcome is "registered on Evaluation Forms" (A6.9). Since IOI94, the tradition is that * the determination of scores is largely automated. Difficult issues remain * the measurement of resource usage by submitted programs (compile time, run time, memory, disk, ...) * the contestants' interface to the grading system (simple to use under contest conditions, understandable messages, help avoid silly mistakes, ...) Concerning the approval of the scores, the tradition is that * after each competition round, the delegation leaders receive, for each of their contestants, a printed Evaluation Form, with total score, and score break down (per test case and subtask, including details of the run, such as run time and explanation of the score), TOGETHER WITH additional material to help understand and inspect the results. What additional material exactly is provided fluctuates from year to year, but the following items (per contestant) have often been asked: - All submissions (source and exe) - All output/log files produced - All test data (input, and correct output if applicable) - Input validity checkers (source and exe) - File comparator (exe) - Output checkers used for grading, if applicable (source and exe) - All libraries and other support software (source and exe) - A computer to perform additional experiments - Grading appeal forms Note that this material is not only necessary for inspecting the results, but also for a PEDAGOGICAL reason, namely to educate the contestants on the basis of their mistakes. Closing Remark ------- ------ At IOI2002, we intend to make some specific proposals concerning the IOI2003 competition. Your input will be appreciated. Best regards, Tom Verhoeff Chair of ISC