Task Approval Procedure

(Approval Procedure from IOI'2002)


At the GA meeting immediately preceding translation, the competition tasks are presented for approval. The task approval procedure has evolved over the years, because the IOI competition and its preparation has changed. Before explaining the proposed task approval procedure, it is good to take a closer look at the preparation process.

Task preparation is a delicate process, where a large number of factors is relevant. The 'life cycle' of a competition task consists of the following phases:

  1. Call for Tasks, requesting for suitable competition tasks; submissions must contain a task description, at least one documented solution, and a grading plan
  2. Collection, typically aiming at about 30 tasks
  3. Selection, typically in two rounds, each preceded by rework where necessary, resulting in eight to ten tasks, incl. several 'backup' tasks
  4. Completion and polishing, incl. setting of various limits, preparation of grading material (data, handout), and development of support software such as libraries and checkers
  5. Translation into native languages of all contestants
  6. Use in competition, incl. reproduction and distribution
  7. Grading, incl. collection of submissions by contestants, execution of submissions (if applicable), scoring of submissions, inspection of results by leaders

For obvious reasons, step 5 (translation) must be carried out by the leaders and deputies at the IOI. Translation must be done very carefully, and in complete isolation from the contestants. It requires from about one hour per task to several hours per task for some languages.

It is understandable that the GA wants to be involved in Step 3 (selection). However, proper assessment of competition tasks for the IOI cannot be done in the limited period provided by the current format of the IOI. Furthermore, it is practically impossible to separate leaders and deputies from contestants for the required time at the IOI.

Note that at the International Mathematical Olympiad (IMO), the leaders arrive three or four days before the deputies and contestants. During that period they select and translate all six competition problems (for two competition days) from a shortlist of about 30 problems. This IMO shortlist fits on four pages, whereas a similar list for the IOI would cover some 60 pages! Even though the IMO spends a considerable amount of time (and hence money!) on selection, many IMO leaders are not satisfied with the result and are not convinced that it is cost effective.

There is another compelling reason why it is not possible to do the selection so close to the IOI competition. Step 4 (completion) is time consuming. Tasks must be completed well in advance of the competition, to make sure that everything will work properly during the competition and subsequent automated grading. Therefore, we want to carry out step 4 only for tasks really needed for the competition.

Compare this again to the IMO. After the selection phase at the IMO, the formulation is first polished by an English language group, presented to the leaders, and approved. Next, official French, German, Russian, and Spanish translations are prepared, presented, and approved. Finally all other translations are done, posted on the wall, and approved. On the first competition day, a group of over 50 so-called coordinators from the Host prepare a marking scheme for each problem. IMO grading is basically a binary process. The contestant solved the problem or not, plus or minus a few points. Grading is done manually by leaders and coordinators on the basis of the marking schemes in two or three days after both competition days. It would be impossible to complete the grading material for even three IOI competition tasks in one day.

For all these reasons, the Host Scientific Committee does the selection, in close cooperation with the ISC. At the ISC review meeting last May in Korea, the Host SC presented 16 competition tasks, from which 10 were selected for further development. This selection was carried out in one week.

A disadvantage of doing the selection with a smaller group, is that certain issues may have been overlooked. That is why the selected tasks are presented to the GA for approval. During this approval process there is room for minor improvements. In case of an emergency, it is even possible to use a backup task. But it must be understood that backup tasks are only to handle true emergencies. The quality of the competition is bound to suffer from using a backup task.

Requirements for Competition Tasks

The following aspects are taken into account when assessing individual tasks:

  1. Appropriateness for the IOI competition, depending on such things as:
    1. Novelty
    2. Task type
    3. Task domain
    4. Solution methods
    5. Grading method
  2. Difficulty level, where we distinguish:
    1. Comprehension and analysis of the task, incl. discovering the set of concepts and relations involved in a formal model
    2. Design of solutions, incl. choosing algorithms and data structures
    3. Implementation of designs in the allowed programming languages, incl. typing in the program text
  3. Grading:
    1. To what extent black-box testing can make the relevant distinctions
    2. Partial credit
  4. Formulation, including such things as:
    1. English (spelling, grammar, punctuation, word choice, keep English simple to facilitate translation by non-native speakers)
    2. Formatting (fonts, headings, layout)
    3. Style (short sentences, consistent terminology, clarity, relevance)
    4. Suitability of the 'story' and/or 'metaphor'
    5. Illustrations and diagrams
    6. Constraints (on values of variables and relationships between variables, resource limits)
    7. Input and output formats (if applicable)
    8. Library interface, interaction protocol (if applicable)
    9. Example input and output data
    10. Overall length, and amount of text to be translated

Concerning the whole set of competition tasks, we consider:

  1. Balance in difficulty levels for both competition days
  2. Diversity in task domains and solution methods, also across IOIs
  3. Likeliness to generate a good range of scores
  4. Uniformity of presentation
  5. Availability of appropriate backup tasks in case of emergency

Note that the current regulations state very few requirements on the competition tasks. The lists above do not have an official status, and are not intended to be complete. The ISC has written a document titled 'Guidelines for IOI Competitions', which presents further details on competition tasks.

Ideas behind Task Approval Procedure

Based on feedback about the task approval procedure used at IOI2001, we propose a slightly modified procedure for IOI2002. The basic aims of the approval procedure are:

  1. Allow for major concerns to be discussed and taken into account.
  2. Allow for minor improvements to be incorporated.
  3. Respond to clarification requests to facilitate translation.

It is felt that at this point it is not appropriate to provide details about solutions and grading. For one thing, this would take much extra time to present and digest, and for another thing, there is the fear that this information could "influence" the translations. General background information about the tasks, expected solutions, and their grading will be presented orally, if so desired by the GA.

At IOI2001, the communication of minor remarks in writing went very well. In fact, most major objections were also communicated in writing, though this had not been explicitly requested. Therefore, we would like to handle both major and minor comments in writing at IOI2002.

Proposed Task Approval Procedure

First, two definitions:

Major objection: An objection that might lead to withdrawal of at least one task in the task set. A major objection concerns an issue that, with a high probability, cannot be corrected within the time planned for task selection. Examples are: The task has already been used in another competition, the task needs too many changes, the topic is inappropriate, etc.
Minor remark: A remark that does not classify as a major objection, that is, which involves no more than a small correction. Minor remarks often concern the formulation of the task, such as ambiguities, inconsistencies, missing details, etc. They could also request for advice about translation.

We propose the following task approval procedure for IOI2002:

  1. SC introduces task set for upcoming competition day.
  2. GA receives whole task set.
    1. GA gets 10 minutes for reading per task.
    2. GA submits major objections and minor remarks in writing to SC.
    3. SC prepares responses to major objections.
  3. For each task, and at end for whole set:
    1. GA Chair handles each submitted major objection as follows:
      1. SC can ask GA to vote on classification as major/minor;
        if minor, then postponed; otherwise, treated as major.
      2. SC responds to major objection.
      3. GA can debate the issue.
    2. GA Chair summarizes situation.
    3. GA votes on acceptance, modulo minor changes.
  4. If task or task set rejected in step 3,
    then SC introduces an alternative task,
    and preceding steps are applied.
  5. After approving whole task set,
    1. GA submits further minor remarks in writing.
    2. SC provides responses to GA on minor remarks.
  6. Final versions of tasks are prepared and distributed for translation.