1 Title : Guidelines for IOI Competitions 2 Author : IOI Scientific Committee 3 Editor : Tom Verhoeff (Chair of IOI SC) 4 Status : Draft for further review, and for use by IOI2000 SC 5 Date : January/February 2000 6 Version: 0.3 7 History: Incorporated suggestions by reviewers of version 0.2 8 9 NOTES 10 ----- 11 The application of these Guidelines to IOI2000 is an experiment. The 12 IOI2000 organizers have agreed to participate in this experiment. 13 Therefore, these Guidelines currently have the status of recommendations 14 only, even though they are phrased in a more obligatory way. These 15 Guidelines will be applied as much as possible for IOI2000, and deviations 16 will be reported to the IOI SC. 17 18 Currently, no procedures have been established for approving and 19 changing these Guidelines, and for deciding on the mandatory use of 20 these Guidelines after IOI2000. 21 22 The following three words are used with a special meaning in these 23 Guidelines: 24 * "must" expresses that something is obligatory, and that deviation 25 is in violation of these Guidelines 26 * "should" expresses that something is strongly recommended, and 27 that a deviation must be reported and explained 28 * "may" expresses that something is optional, and that deviations 29 are acceptable and require no further commenting. 30 31 Aim 32 --- 33 These Guidelines set a standard for the organization of the COMPETITION 34 at the International Olympiad in Informatics (IOI). Each IOI provides, 35 among other things, a Computer Science competition for talented 36 high-school students, selected through competitive national olympiads. 37 From the first IOI in 1989, the focus of these competitions has been on 38 algorithmic computer programming tasks. 39 40 Organizing an IOI competition is a big challenge. Using a set of 41 written guidelines and refining these over the years, will help ensure a 42 high-quality competition that can develop and improve continuously. 43 44 These Guidelines are based on experience gained at past IOI 45 competitions and similar events, and on best practices in Computing 46 Science and Software Engineering. 47 48 These Guidelines focus on the competition part of the IOI. They do not 49 address such issues as sleeping accommodation, meals, transportation, 50 social events, political issues, etc. The latter are a concern of 51 the International Committee (IC) of the IOI. 52 53 The intended readership consists of all IOI Scientific Committee (SC) 54 members, all Host SC members, and Host Team managers. Delegation 55 leaders may also find some of the information useful when preparing for 56 an IOI. 57 58 Also consult the IOI Regulations. 59 60 List of Abbreviations 61 ---- -- ------------- 62 CC Competition Committee 63 CI Configuration Item (see Configuration Management) 64 CM Competition Manager 65 GA General Assembly 66 IC International Committee 67 ICPC International Collegiate Programming Contest (by ACM) 68 IMO International Mathematics Olympiad 69 IOI International Olympiad in Informatics 70 IOIn The particular IOI being organized 71 PGP Pretty Good Privacy (http://www.pgpinternational.com/) 72 SC Scientific Committee 73 74 Glossary 75 -------- 76 CENTRAL COMPETITION DESK: The location where competition-related 77 material, such as Clarification Request Forms and Grading Appeal Forms, 78 are submitted and from there distributed to the appropriate destination. 79 The actual desk location may depend on the activity that is taking place 80 (translation, competition, grading, ...). 81 82 COMPETITION: Several interpretations will be distinguished: 83 * All the events and activities at IOIn (incl. two Competition Rounds 84 and the awarding of medals) concerning the IOIn Competition. This 85 broadest interpretation is denoted in these Guidelines by "IOIn 86 Competition". 87 * A single Competition Round at IOIn (starting with the presentation 88 of the Competition Tasks to the GA, and ending with the finalization 89 of the results by the GA). This interpretation is denoted in these 90 Guidelines by "Competition Round". 91 * A period of five hours when the competitors work on Competition 92 Tasks. This narrowest interpretation is denoted in these Guidelines 93 by "competition proper", or simply "competition" when confusion is 94 unlikely. 95 96 COMPETITION DAY: A day at IOIn when a competition proper takes place, 97 that is, when the competitors compete by working on Competition Tasks. 98 99 COMPETITION MANAGER (CM): The person who is responsible for managing 100 the Competition Committee, in particular, its organizational aspects. 101 In most cases, the chair of the IOIn Host SC will act as IOIn CM. 102 103 COMPETITION MATERIAL: Several interpretations will be distinguished: 104 * All the information relevant to the IOIn Competition, whether 105 eventually published or not. 106 * All the information concerning the IOIn Competition that will be 107 made publicly available, in particular to the GA prior to a 108 competition proper. This includes background information and 109 grading details about the Competition Tasks, which must not be given 110 to the competitors before the competition proper has ended. 111 * All the information to be given to the competitors at a competition 112 proper. This includes the Competition Tasks, but also such items as 113 a Task Overview, sample files, helper programs, etc. 114 The intended interpretation should be obvious from the context in these 115 Guidelines. 116 117 COMPETITION OFFICIAL: Any person involved in executing the IOIn 118 Competition, and recognizable as such (special badge, clothing). 119 There are various roles for officials, for example: 120 * patrol the Competition Site during a competition proper 121 * attend to the Central Competition Desk 122 * manage the technical equipment (servers, printers) 123 * deal with issues related to Competition Tasks (answering 124 Clarification Requests, for example) 125 * carry out the grading 126 127 COMPETITION ROUND: One of the two Competition Rounds that take place at 128 IOIn. The Competition Rounds have the same structure: 129 1. IOIn CC presents Competition Tasks to GA (in a GA Meeting) 130 2. GA translates Competition Tasks 131 3. Competitors work on Competition Tasks for five hours 132 4. IOIn CC grades the work of the competitors and reports to GA 133 5. GA verifies grading results and may file appeals 134 6. IOIn CC formulates responses to appeals 135 7. GA finalizes the results (in a GA Meeting) 136 These activities are typically spread out over several days. 137 138 Confusingly, a term like "First Competition Day" is sometimes (but not 139 in these Guidelines) used to denote just the First Competition proper 140 and on other occasions to denote the entire First Competition Round. 141 However, the competition proper is only one of several events that may 142 take place on a Competition Day; traditionally, grading in a Competition 143 Round is done on the same day, but (since IOI'96) translation has been 144 done on the preceding day. On the other hand, as noted above, a 145 Competition Round usually spans more that one day. 146 147 COMPETITION SITE: The location with workplaces at IOIn where each 148 competition proper takes place, that is, where competitors compete by 149 working on the Competition Tasks. The Competition Site includes rooms 150 with support equipment (such as servers, printers, ...). 151 152 COMPETITION TASK: A task in the IOIn Competition that the competitors 153 have to complete. The work submitted by the competitors for the tasks 154 is graded and the result is used to award medals. Each Competition 155 Round typically involves three independent and self-contained 156 Competition Tasks. 157 158 GRADING: The process of evaluating the work submitted by the competitors 159 and then awarding points, with the intent to rank the competitors for 160 awarding medals. 161 162 INPUT VALIDATOR: A program that determines whether a file or set of 163 files is a valid input instance (satisfies the precondition) of a 164 Competition Task. There are some generic aspects to this validation, 165 which apply to all Competition Tasks (e.g. the restriction to lines of 166 characters with ASCII values in the range 32..126). Complete validation 167 however always involves task-specific aspects. Some of the checks are 168 merely syntactic (e.g. "the first line contains three numbers"), others 169 are semantic (bounds on numbers, special relationships between data 170 items). Many (and sometimes all) of these task-specific conditions can 171 be handled by a "scriptable" validator, where the script captures the 172 restrictions to be checked. For some tasks, the input validator is more 173 complex than a solution to the task. 174 175 IOIn COMPETITION: All the activities and events that take place at IOIn 176 concerning the competition part, including: 177 1. Introduction (1st GA Meeting, practicing, questions about 178 the Competition Rules) 179 2. First Competition Round 180 3. Second Competition Round 181 4. Medal decisions and awarding 182 183 IOIn COMPETITION COMMITTEE (CC): The group of persons who are organizing 184 an IOIn Competition. The IOIn CC includes persons from the IOI SC and 185 IOIn Host SC, as well as others that have been specifically invited to 186 contribute. Not all persons have to be involved in all phases 187 (preparation, execution, conclusion). 188 189 LANGUAGE BARRIER: The impossibility of communicating with all IOI 190 competitors using a single natural language. The language barrier needs 191 to be crossed in BOTH directions: 192 * TO the competitors, when presenting the competition material and 193 results; 194 * FROM the competitors, when grading the submitted work. 195 All competition material is prepared in English. The GA members help in 196 crossing the language barrier by acting as translators and interpreters. 197 The language barrier FROM the competitors is partly circumvented by 198 having them submit their work in machine interpretable form. 199 200 OUTPUT CHECKER: A program that determines whether a file or set of files 201 is correct output for (satisfies the postcondition of) a Competition 202 Task and one input instance (consisting of one or more input files). 203 204 If the task is deterministic, that is, the input uniquely determines the 205 output, then an output checker can simply compare the file(s) to be 206 checked with the correct output file(s) for the given input instance. 207 It may be necessary to filter files before comparison to eliminate extra 208 spaces and empty lines. 209 210 In case of nondeterministic tasks, the output checker will be task 211 specific. Many (and sometimes all) of the task-specific conditions can 212 be handled by a "scriptable" checker, where the script captures the 213 output restrictions to checked. Examples of script constructs are 214 * the next item must appear in the following set 215 * the next list of items must be a permutation of the following set 216 For some tasks, a special-purpose checker may be needed. 217 218 Overview 219 -------- 220 First, the general format of the IOI Competition is described. Next, a 221 number of organizational topics are addressed in subsequent sections. 222 These topics are: 223 224 Project Plan 225 Progress Reports 226 Evaluation Report 227 Staff 228 Deliverables 229 Competition Rules 230 Schedule 231 Configuration Management 232 Confidentiality 233 Competition Accommodations 234 235 In the Appendix, there are some checklists and templates. Stylistic 236 conventions in these Guidelines are summarized at the very end. 237 238 Format of the Competition at IOIn 239 ------ -- --- ----------- -- ---- 240 The IOI Regulations provide some information about the IOI Competition. 241 In particular, the following sections of the Statutes are relevant: 242 243 * S4.7 about the Host Scientific Committee 244 * S4.8 about the General Assembly (GA) 245 * S4.9 about the GA meetings at IOIn 246 * S4.12 about the IOI Scientific Committee (added at IOI'99) 247 * S6.4 about the IOIn Host's obligation to produce Competition 248 Procedures and Judging Procedures 249 * S7 (i.e. S7.1 to S7.14) about Competition, Judging and Awarding 250 251 However, this information is rather limited and, in many cases, leaves 252 too much to the imagination. The Competition Procedures and Judging 253 Procedures mentioned in S6.4 are intended to provide the missing 254 details. In other cases, the IOI Regulations may prove to be too 255 restrictive. For instance, S4.9 requires a GA meeting "at the beginning 256 of the First Competition Day" and S7.11 requires that, for the purpose 257 of grading, various people "meet at the desk and computer of the 258 Contestant". In this section, we summarize, extend, and update this 259 information. 260 261 The following description is not intended to define the IOI Competition 262 format once and for all. The possibility for change remains 263 desirable. A described format will help focus the discussion about 264 such change. 265 266 According to the IOI Regulations, the IOIn Host Scientific Committee is 267 a temporary committee carrying the responsibilities for preparing and 268 executing the IOIn Competition. At IOI'99, the General Assembly 269 established the IOI Scientific Committee. The IOI SC is a long-term, 270 standing committee responsible for supervising the development, 271 preparation, and execution of the IOI Competitions. 272 273 In these Guidelines, we will speak of the group of persons who are 274 actually organizing the IOIn Competition as the "IOIn CC" (CC= 275 Competition Committee). The IOIn CC includes persons from the IOI SC 276 and IOIn Host SC, as well as others that have been specifically invited 277 to contribute. 278 279 In chronological order, the main events that concern the participating 280 delegations of the IOIn Competition are as follows: 281 282 * Announcement of the TYPES of Competition Tasks for IOIn, especially 283 new types (5 months prior to IOIn) 284 * Publication of the Competition Rules for IOIn, especially defining 285 the computing environment and the grading process (4 months prior 286 to IOIn) 287 * Questions about the task types and Competition Rules can be posed by 288 e-mail 289 290 * Arrival at the IOIn venue 291 * 1st GA meeting: Introduction 292 * Practice Competition: with one or two very simple tasks, which 293 preferably can be translated at home before IOIn 294 * 2nd GA meeting: Questions & Answers 295 * Other practicing opportunities 296 297 * 3rd GA meeting: Competition Tasks for the First Competition 298 are presented and subsequently translated 299 * First Competition of five hours, followed by the grading of all 300 submissions, opportunity to inspect results, and to file appeals 301 * 4th GA meeting: Finalize results of First Competition 302 303 * 5th GA meeting: Competition Tasks for the Second Competition 304 are presented and subsequently translated 305 * Second Competition of five hours, followed by the grading of all 306 submissions, opportunity to inspect results, and to file appeals 307 * 6th GA meeting: Finalize results of Second Competition 308 309 * 7th GA meeting: Finalize combined results of both competitions, 310 decide on medal distribution 311 * IOIn Evaluation Questionnaire 312 * Awards Ceremony, published results and solutions 313 * Departure from IOIn venue 314 315 Here is a summary: Detailed information is disclosed prior to IOIn. At 316 IOIn there is some introductory activity, followed by two identically 317 structured Competition Rounds. Everything is closed off by distributing 318 awards and solutions. 319 320 Some of these items are elaborated below. These descriptions are 321 necessarily brief. Therefore, it is recommended that IOIn CC members 322 have also gained experience by direct involvement in the preparation or 323 execution of previous IOIs. Reading reports of past IOIs may also be 324 helpful. 325 326 TYPES OF COMPETITION TASKS: Traditionally, the IOI Competition Tasks 327 have been programming tasks of an algorithmic nature, where the focus is 328 on designing correct and efficient algorithms. Input and output are 329 kept as simple as possible. As an aid to grading, algorithms must be 330 programmed so that machine execution is possible. Also see the 331 Competition Tasks of previous IOIs. 332 333 Currently, four types of programming tasks can be distinguished: 334 335 * Classic Batch: All input for one instance resides on disk in one 336 text file; the input is small enough to fit completely into some 337 global variables; the output is to be written to one text file on 338 disk. Established at IOI'94. 339 340 * Modern Batch: As classic batch, but input is from one or more text 341 files that are possibly too large to fit completely into global 342 variables; output may also involve multiple text files. Introduced 343 at IOI'95. 344 345 * Standard I/O Reactive: Input and output are through standard I/O 346 text streams, where input may depend on prior output, that is, 347 there is a dialog between the program and an environment. 348 Introduced at IOI'95. 349 350 * Library Reactive: Input and output occur through a well-defined 351 interface of routines in a special-purpose library; these routines 352 may involve files that are not directly visible to the competitors. 353 Introduced at IOI'96. 354 355 Traditionally, each run concerns a single problem instance. However, as 356 pointed out below under grading, it may be necessary to have programs 357 deal with multiple instances per run in the future. 358 359 The tasks should be such that clever competitors can decide by reasoning 360 that their submissions fulfill the stated requirements. This excludes 361 tasks that require submission of programs involving heuristics or hidden 362 features of the actual input data used for grading. 363 364 Here are four suggestions for new task types: 365 366 * Non-Secret Input: All competitors receive the same set of input cases 367 (on disk in text files) and have to deliver the corresponding output 368 (on disk in text files) as specified in the task description. The 369 input cases are so complex or large that their processing will 370 require the use of a computer. 371 372 The task may involve heuristics, and it may be necessary to develop 373 several programs, first to gather some statistics about the input and 374 decide on a suitable approach, and then a program for each approach. 375 The developed programs are not submitted for grading. 376 377 * Testing: The task is to develop a program T that tests another 378 program P, by confronting P with a set of input cases and 379 subsequently checking the output. The task description specifies 380 the program P under test. P can be given as a routine in a 381 library, with I/O done through parameters. Grading is done by 382 letting the competitor's program T test a number of reference 383 programs P, each with zero or more known errors. 384 385 * Resource Limited: The task involves additional constraints (apart 386 from those "naturally" imposed by the equipment), such as very little 387 memory, several memories with varying access costs, or restricted 388 operators (e.g. no multiplication or no more than a give number). 389 This can be simulated with a library, but may also involve special 390 pre-processing (to enforce the constraints before execution) or a 391 special environment in which to run programs (to monitor constraints 392 during execution). 393 394 * Tournament: The task involves the development of a reactive program 395 that plays a game. During grading, all programs play against each 396 other (and possibly some reference programs) in a tournament. 397 The score (also) depends on the game outcomes. 398 399 COMPETITION RULES: These are dealt with separately. Also consult 400 Competition Rules of past IOIs. 401 402 1ST GA MEETING (INTRODUCTION): Delegations will have received the 403 description of the (new) task types and the Competition Rules prior to 404 arriving at the IOIn venue. They have even had an opportunity to pose 405 questions about them. Nevertheless, it is good to provide a brief 406 introduction into the whole IOIn Competition. Questions whose 407 answers are needed for practicing must be handled in this meeting. 408 Other questions should be postponed until the 2nd GA Meeting after 409 the first practicing has taken place. 410 411 PRACTICE COMPETITION: Past IOIs have usually offered some opportunity 412 for the competitors to practice with the competition equipment, especially 413 the computer, its operating system, and the programming environment. 414 The First Competition Round has always been the first time that everything 415 comes together. Consequently, the First Competition Round has often been 416 hampered by various troubles, mostly occurring in the competition proper 417 and the subsequent grading. 418 419 Each IOIn is a unique, one-time event. Almost all the 420 * equipment (rooms, furniture, computers, printers, network), 421 * software (operating system, compilers, editors, support tools, 422 libraries, grading system), 423 * officials (from Host SC members to assistants doing errands), and 424 * delegations (from all over the world) 425 will be new at IOIn. They certainly have never before operated together 426 in this mix and on such a grand scale as required by IOIn. The First 427 Competition Round has often revealed a number of deficiencies, most of 428 which are then repaired before the Second Competition Round. There is no 429 reason to believe that a first large-scale run can be made to work 430 flawlessly. 431 432 At the ACM International Collegiate Programming Contests (ICPC), at 433 both the preliminary Regional Contests and the World Finals, there is 434 always a practice contest prior to the real contest. The practice 435 contest resembles the real contest as much as possible, except that 436 the contest problems are far fewer and simpler, and that it takes 437 less time. 438 439 It is strongly recommended that a Practice Competition is conducted at 440 IOIn. The intent is to run the entire system through all critical 441 phases and in that way help eliminate troubles before they can do any 442 harm. The most critical phases are the competition proper and its 443 grading. Whether translation should also be covered in the Practice 444 Competition remains to be considered. 445 446 A Practice Competition also helps build confidence. The Competition 447 Tasks at the Practice Competition must be feasible with minimal 448 intellectual effort. It remains desirable that, besides the Practice 449 Competition, the schedule also allows time for "free format" practicing. 450 451 There should be sufficient time between practicing and the 2nd GA 452 Meeting, so that any issues that have arisen can be assessed and their 453 solutions formulated. 454 455 2ND GA MEETING (QUESTIONS): After practicing, the IOIn CC may want to 456 make last-minute announcements, and the GA may wish to ask questions 457 about the IOIn Competition. After this meeting, delegations must be 458 given the opportunity to provide final instructions to their competitors 459 before the First Competition proper (i.e. before the 3rd GA Meeting, 460 because that puts them out of touch with their competitors). 461 462 3RD GA MEETING (PRE-TRANSLATION 1): All IOIn Competition Material has 463 been prepared in English. The IOIn CC presents the material for the 464 First Competition to the GA. This includes the information exactly as 465 it is intended to be offered to the competitors and which needs to be 466 translated to the native languages of the competitors. It also 467 includes additional information (not to be offered to the competitors 468 until after the competition), such as background information on the 469 tasks and grading details. The GA has the opportunity to reject a task, 470 but only for compelling reasons (such as prior use). Once an approved 471 set of Competition Tasks and accompanying information has been 472 established, this material is subsequently translated by the GA. The 473 translations are published, for example, by posting the printouts on the 474 walls of the GA Meeting Room. 475 476 Attendants of the 3rd GA Meeting promise not to communicate to the 477 competitors until after the First Competition proper is over. The 478 IOIn CC should take appropriate measures to minimize opportunity for 479 any such communication. 480 481 FIRST COMPETITION PROPER: The Competition Site is prepared for the First 482 Competition proper (reset equipment, distribute Competition Material, 483 position officials, ...). Competitors enter the Competition Site (check 484 badges) and are seated at their workplaces, where they find a sealed 485 envelope with the Competition Material (including English originals and 486 a translated copy if supplied by the delegation). Additional 487 competition material is accessible through their computer. When the 488 start signal for the competition has been given, they may open the 489 envelopes and start to work, in particular, access their computer. In 490 the future, it may be possible to offer all the Competition Material in 491 an electronic form (when more advanced display technology is available). 492 493 Since IOI'94, each of the two competitions at IOIn has had THREE 494 independent self-contained tasks. The three tasks each present a 495 different kind of challenge and there is some variety in the level of 496 difficulty. The First Competition is typically somewhat easier than the 497 Second Competition. The example data given in the task descriptions is 498 also available in electronic form on the computers. There possibly are 499 additional aids, such as more example data and helper programs. 500 501 How difficult a competition will turn out to be for the competitors is 502 hard to assess and control. The level of difficulty needs to be tuned 503 carefully, and this is one of the toughest jobs for the organizers. If 504 the competition is too easy, then too many competitors will end up with 505 the same (high) score, thereby complicating the task of awarding medals. 506 If it is too difficult, then this has a negative impact on the image of 507 the IOI, and the effort of the organizers is wasted. The level of 508 difficulty depends on 509 * the number of tasks 510 * dependencies among tasks 511 * the difficulty of each task, which in turn depends on how hard 512 and how much work it is 513 - to comprehend the task 514 - to isolate the "real" issues 515 - to tackle these issues 516 - to design a solution 517 - to implement the design 518 - to verify the implementation 519 In view of the limited duration (five hours), it seems unreasonable to 520 include more than three challenging tasks per competition, keeping in 521 mind that context switches are a heavy burden on the competitors. With 522 fewer than three tasks per competition one cannot ensure sufficient 523 diversity. 524 525 In the first hour of the competition, written questions about the 526 Competition Tasks can be submitted on a Clarification Request Form to 527 any of the designated Competition Officials. Each form accommodates a 528 single question that must be answerable by Yes, No, or No Comment. The 529 competitor can pose the question in their native language. The official 530 takes the form to the Central Competition Desk where it is registered 531 (given a time stamp and tracking number), and delivered to an 532 appropriate member of the IOIn CC. The IOIn CC may approach an 533 appropriate GA member for translating a question. For this 534 purpose, GA members need to be available during the first hour of the 535 competition. The IOIn CC marks the answer on the form (the answer is 536 NOT translated). The form is returned to the competitor, after being 537 photocopied. Note that non-task-related issues, such as equipment 538 malfunction, can be reported directly and at any time (not only in the 539 first hour) to any competition official. The end of the first hour is 540 announced. 541 542 Competitors may only use the material that the IOIn CC provides or has 543 explicitly approved. They are not allowed to bring such things as 544 books, calculators, or communication equipment. Competitors are not 545 allowed to communicate with each other in any way, and they may not 546 cooperate or interfere with the work of others. Competitors 547 can get printouts of their work, and there is a mechanism for making 548 backups (e.g. by providing removable disks). During the competition, 549 refreshments are offered and restroom facilities are available. 550 551 Near the end of the competition proper, the time is announced, 552 typically at fifteen, five, and one minute before the end. The 553 competitors submit their work in electronic form, usually by leaving it 554 on their computer in a designated location. The competitors do not 555 take a copy of their submitted work when leaving the Competition Site. 556 The delegations will receive copies of the submitted work later. 557 558 GRADING OF FIRST COMPETITION: After the First Competition, the work of 559 the competitors is graded. If grading requires competitor involvement, 560 then there will first be a one-hour break. The work submitted by the 561 competitors is evaluated according to the prepared grading process, and 562 points are awarded for each task. These Guidelines strongly recommend 563 that for programming tasks, the competitors submit SOURCE CODE, instead 564 of executables. Here are some reasons to do so: 565 * Scientifically, the quality of software is judged from its design 566 and its source code, and not (solely) from the executable code. 567 * Source code is more platform independent. This makes it easier 568 to prepare competitors for an IOI Competition and to evolve the IOI 569 Competition's computing platform. 570 * Source code provides more grading options, including the 571 enforcement of more varied resource restrictions. 572 * The ACM ICPC and various Internet programming competitions 573 are successfully based on source code submission. 574 575 Traditionally, the grading of programming tasks is based solely on 576 execution of the programs, and (since IOI'94) it does not involve 577 human inspection of the program's source code. At IOI'99, a petition 578 supported by many delegations pointed out that the past grading practice 579 is too often not able to recognize and reward scientifically sound work. 580 The petition also requested that the grading would be reconsidered and, 581 if at all possible, would be made more "humane" and "realistic". The 582 spirit of IOI grading is that good work is given proper credit (as 583 opposed to the spirit of ICPC grading, where every mistake, however 584 small, is punished severely). This is a difficult issue. The least 585 that can be done is designing Competition Tasks such that partial credit 586 can be awarded reflecting partial accomplishment of the task. In the 587 future, it may be necessary to introduce other forms of source code 588 inspection besides compiling and executing. 589 590 The grading results are presented to the delegations in printed reports 591 (one per competitor). The delegations also receive all relevant 592 material (see checklist in Appendix) to help verify the results. A 593 delegation can submit motivated objections to the results on a Grading 594 Appeal Form at the Central Competition Desk. These appeals are handled 595 by the IOIn CC, which prepares responses in a closed meeting and 596 distributes them to the delegations (also see 4th GA Meeting). 597 598 At some other international science olympiads (in particular the IMO), 599 the complete preliminary results of the First Competition Round are made 600 available to all GA members. Though this is not yet common practice at 601 the IOI, it is worth considering, because it helps locate grading issues 602 quickly. 603 604 4TH GA MEETING (POST-GRADING 1): The IOIn CC explains their responses to 605 all the submitted grading appeals in this GA Meeting. If the need 606 arises, the GA can vote on decisions and actions to be taken. In this 607 GA Meeting the final results of the First Competition should be 608 established. 609 610 5TH GA MEETING (PRE-TRANSLATION 2): Identical in structure to 3rd GA 611 Meeting. 612 613 SECOND COMPETITION PROPER: Identical in structure to First Competition 614 proper. 615 616 GRADING OF SECOND COMPETITION: Identical in structure to grading of First 617 Competition. 618 619 6TH GA MEETING (POST-GRADING 2): Identical in structure to 4th GA Meeting. 620 621 7TH GA MEETING (AWARD DECISION): Once the results of both Competition 622 Rounds have been approved, the medal distribution needs to be established. 623 Although the medal distribution is almost completely determined by 624 the final scores of all the competitors, it is an IOI tradition that 625 the IOIn CC makes a proposal and the GA has the final say. 626 627 It is tempting to merge the 6th and 7th GA Meeting. These Guidelines do 628 not recommend this. There are also other matters that need to be 629 addressed at the GA Meetings, and the primary purpose of the 6th and 7th 630 GA Meeting are best kept separated. 631 632 MISCELLANEOUS: Each of the two Competition Days is preceded and followed 633 by at least one day without competition (see IOI Regulations). 634 635 At IOIn there needs to be ample opportunity for the GA to deal with 636 the development of the IOI Competition. In particular, the following 637 issues need attention: 638 * the format of the competition (how many competition days, how many 639 tasks per day, ...) 640 * the types of competition tasks 641 * the computing platform (operating system, programming languages, ...) 642 * the grading process 643 * the medal distribution process 644 These issues could be addressed at the 1st, 4th, 6th, and 7th GA meeting. 645 646 There can always arise unforeseen circumstances. The IOI Regulations 647 (S4.8) rule that the GA can "take decisions if the Regulations do not 648 anticipate a situation". If these Guidelines do not anticipate a 649 situation, the Contest Manager must make a proposal to the GA for 650 resolving the issue. 651 652 REMARKS: Instead of the mandatory six private GA meetings according to 653 the IOI Regulations, these Guidelines define (at least) seven such 654 meetings. Furthermore, a Practice Competition is strongly recommended 655 (possibly to be made mandatory in a later version), and grading of 656 programs is based on source code submission (instead of executables). 657 658 Project Plan 659 ------- ---- 660 The IOIn Competition Committee (CC) must first PLAN its work. For that 661 purpose, the COMPETITION MANAGER (CM) must produce a PROJECT PLAN. This 662 plan must 663 664 * list all DELIVERABLES, giving a brief description of each item 665 (document, form, program, file, etc.) to be delivered. Such items 666 can be composite (e.g. the bundle of all Competition Tasks); the 667 parts of composite deliverables must also be described, but this 668 may be done elsewhere. 669 670 * specify a STAFFING PLAN, listing all key people involved in the 671 organization of the IOIn Competition, with their name, role(s), 672 contact information, and to whom they report 673 674 * list all major ACTIVITIES, other than "produce deliverable X" 675 (e.g. development of a special timer device) 676 677 * define a SCHEDULE, giving for each (class of) deliverable: 678 - the (one!) person responsible for delivery, 679 - the target audience, 680 - the starting date, 681 - the date(s) and method for review, and 682 - the date and method for final delivery to the target audience 683 684 * identify and assess the (probability and impact of) major RISKS to 685 the project and indicate how to prevent (reduce probability of) and 686 correct (reduce impact of) the occurrence of these risks 687 688 The Project Plan is submitted for approval to the IOI SC at least nine 689 months prior to IOIn. The IOI SC reviews the plan in a timely manner 690 and provides feedback. 691 692 It is recommended that the approved Project Plan is also made available to 693 the IOI IC and to the Host Team executive board. 694 695 The Project Plan may EVOLVE over time, e.g. by adding new parts, by 696 changing existing parts, or by filling in details to existing parts 697 that were not known earlier. However, once approved, any changes to 698 the Project Plan require separate approval by the IOI SC. This is then 699 brought up in a Progress Report (see below). 700 701 The IOIn Competition Committee must EXECUTE the approved plan. 702 703 The IOI Scientific Committee MONITORS the execution of the plan. 704 705 Progress Reports 706 -------- ------- 707 As part of the monitoring process, the CM presents a monthly PROGRESS 708 REPORT to the Chair of the IOI SC. Each Progress Report must include 709 the following items: 710 711 * Status of deliverables (e.g. not yet started, p% completed, ready 712 for review, under review, approved, ...), highlighting the status 713 changes that occurred in the reporting period 714 * Forecast of status changes for the next reporting period 715 716 and whenever applicable also the following items: 717 718 * Corrective actions taken for deliverables whose current status 719 deviates from the scheduled status according to the Project Plan 720 * Risks that have materialized during the reporting period, and 721 the corrective action taken 722 * Risks that are expected to materialize in the next reporting period, 723 and the preventive action taken 724 * Changes in the Project Plan 725 726 The first reporting period starts nine months prior to IOIn, regardless 727 of whether the Project Plan has already been approved by that time. 728 That is, the first Progress Report is due eight months prior to IOIn. 729 730 Evaluation Report 731 ---------- ------ 732 Within one month after IOIn, the Chair of the IOI SC must write an 733 EVALUATION REPORT. This Evaluation Report summarizes the IOIn 734 Competition (preparation AND execution). It is intended to help 735 fine-tune and improve the organization process. Material from the 736 Project Plan, the Progress Reports, and other deliverables, as well as 737 material produced at IOIn (the competitors' work, grading appeals, etc.) 738 will be used to produce the Evaluation Report. The Competition Manager 739 may be requested to provide additional material for the Evaluation 740 Report. 741 742 It is recommended that the Chair of the IOI SC drafts the Evaluation 743 Report while still at the IOIn site and discusses it in person with the 744 CM or a knowledgeable representative. 745 746 The Evaluation Report is first submitted to the IOI SC and Host SC 747 for feedback. Upon approval it is delivered to the IOI SC and Host 748 Team executive board, and subsequently published at the IOI 749 Secretariat. 750 751 The Evaluation Report may include recommendations for changing 752 these Guidelines. (PROCEDURES FOR CHANGING AND APPROVING THESE 753 GUIDELINES STILL NEED TO BE ESTABLISHED.) 754 755 Requirements on the content of the Evaluation Report may need to be 756 established. It is expected to contain various STATISTICS, such as the 757 use of programming languages, number and kind of clarification requests, 758 results per task, etc. 759 760 Staff 761 ----- 762 Important roles to be fulfilled in organizing the IOIn Competition are: 763 764 * Competition Manager (CM) 765 * Chair of Competition Task Creators and Reviewers 766 * Competition Task Creators 767 * Competition Task Reviewers 768 * Chair of Technical Committee (dealing with hardware, operating system, 769 compilers) 770 * Chair of Grading Committee 771 * Chair of Competition Accommodations 772 * Assistants 773 774 A person may fulfill several roles. For each person involved in the 775 organization, it must be clear what role(s) this person fulfills. Each 776 role reports to exactly one other role as follows: 777 778 Competition Manager 779 | | | | 780 +-----------------+ | | +-----------------+ 781 | +-----+ +-----+ | 782 | | | | 783 Chair of Chair of Chair of Chair of 784 Competition Grading Technical Competition 785 Tasks Committee Committee Committee Accommodations 786 || || || || || 787 || || || || || 788 || || Assistants Assistants Assistants 789 Creators Reviewers 790 || 791 || 792 Assistants 793 794 The Competition Manager reports to the IOIn Host Team Executive Board 795 (mostly on organizational issues) and to the Chair of the IOI SC 796 (mostly on technical issues). 797 798 The Technical Committee and the Competition Accommodations Committee 799 are expected to consist mostly of persons from the IOIn Host Team, 800 familiar with the local situation of the IOIn venue. 801 802 Deliverables 803 ------------ 804 The following deliverables are mandatory. Mentioned above are: 805 806 * Project Plan 807 * Monthly Progress Reports 808 * Evaluation Report 809 810 Detailed below are: 811 812 * Competition Rules 813 * Competition Tasks, incl. practice and reserve 814 * Grading System (procedure and software) 815 * Final Results 816 * List of Delegations (with contact information) 817 * Various templates and forms: 818 - Competition task description template (for task authors) 819 - Competition task comment form (for use during GA meeting) 820 - Translation Submission Form (for use during translation) 821 - Clarification Request Form (for use during competition) 822 - Grading Appeal Form (for use after grading) 823 - IOIn Evaluation Form (for feedback at the end of IOIn) 824 * Various scripts and agendas 825 826 Each deliverable must have exactly ONE main author, who is responsible 827 for delivering the content. 828 829 Each deliverable should be authored by at least two persons, one 830 of them being the main author. This ensures that each deliverable 831 is intimately known by more than one person and, hence, that the loss 832 of an author will not jeopardize the timely delivery. 833 834 No person should be main author of more than two Competition Tasks. 835 This reduces the impact of losing an author, and also helps ensure 836 diversity. For instance, it makes it impossible that all three tasks 837 in a Competition Round are authored by one and the same person. 838 839 Persons cannot act as a reviewer for the tasks they author. A reviewer 840 needs to start from a fresh perspective. 841 842 Competition Rules 843 ----------- ----- 844 The IOIn Competition Rules must provide a complete description of all 845 those general aspects of the competition at IOIn that are relevant for 846 the participating delegations and that are not already explained in the 847 IOI Regulations. The term "general aspects" is intended to cover those 848 aspects that are independent of the specific competition tasks posed at 849 IOIn. A checklist for the contents appears in the Appendix. 850 851 Examples of questions that need to be answered in the Competition Rules 852 are the following: Which programming languages are available for the 853 programming tasks, and through which compilers? What other tools are 854 available? What is the interface to the grading system? What needs to 855 be submitted for a task, and in what format? How is the grading done? 856 857 Schedule 858 -------- 859 The following MILESTONES must be incorporated into the schedule. 860 861 Deliver description and examples of new TYPES of tasks 862 6 months prior to IOIn: for review by IOI SC (via e-mail) and 863 IC (during site visit) 864 5 months prior to IOIn: for delegations (via web & IOI mailing list) 865 866 Deliver Competition Rules 867 5 months prior to IOIn: for review by IOI SC (via e-mail) 868 4 months prior to IOIn: for delegations (via web & IOI mailing list) 869 870 Deliver Competition Tasks 871 3 months prior to IOIn: for review by IOI SC (preferably during site 872 visit, or via e-mail) 873 in GA meeting at IOIn: for review by GA 874 875 Deliver Grading System and Grading Support Material 876 3 months prior to IOIn: for review by IOI SC (preferably during site 877 visit, or via e-mail) 878 879 Deliver Final Results and List of Delegations 880 Before departure day of IOIn: for all delegations and officials 881 882 Deliver Evaluation Report 883 1 month after IOIn: for review to IOI SC and IOIn Host Team 884 2 months after IOIn: for public (via web and IOI mailing list) 885 886 Configuration Management 887 ------------- ---------- 888 The organization of an IOI competition involves a large number of 889 documents, forms, files, software programs, etc. Borrowing a term 890 from Software Engineering, each such item is referred to as a 891 CONFIGURATION ITEM (CI). Configuration management concerns the 892 identification, storage, change, and release of configuration items. 893 894 The Project Plan, Progress Reports, and all other deliverables are CIs 895 and must be subjected to configuration management. They must be 896 identified and stored in such a way that their status is clear and that 897 all approved versions, including older ones, can be retrieved later. 898 899 Approved CIs are under CHANGE CONTROL. This means that changes can only 900 be made when the changes have been reviewed and approved, and when 901 afterwards it can be verified that the changes have indeed been made as 902 agreed. 903 904 Approved CIs must be released to their intended audience according to 905 the schedule. 906 907 Appropriate backup measures must be taken to ensure that CIs are never 908 lost. This applies not only to approved CIs, but also to CIs still under 909 development. Approved CIs must also be stored by the Chair of the IOI SC. 910 911 Confidentiality 912 --------------- 913 The Project Plan, Progress Reports, and other deliverables are 914 CONFIDENTIAL documents up to the moment of their public release. 915 Confidential documents, whether printed or in electronic form, must be 916 handled, stored, and transmitted in a secure manner to ensure that 917 unauthorized persons have no access to their contents. 918 919 During the life-time of a document, the set of authorized persons can 920 change. For example, the Competition Tasks for round X are first 921 developed in a small group, then released to the IOI SC for review, then 922 released to the GA members at IOIn prior to the competition proper of round 923 X, and, finally, they are released to the competitors at the start of 924 the competition on day X and made publicly available to guests 925 and other interested persons, e.g. back home (in printed form and 926 via the IOIn web site). 927 928 For electronic storage of confidential documents on multi-user or 929 networked systems, it suffices to use appropriate read/write permissions 930 of the file system in a password-protected user account. Encryption is 931 not required for storage, as long as limited access is guaranteed. 932 933 For electronic transmission, confidential documents should be encrypted. 934 This can, for instance, be done through PGP. For printed documents, 935 delivery by regular mail directly to an authorized recipient is 936 acceptable. 937 938 Competition Accommodations 939 ----------- -------------- 940 There are a number of requirements on the competition accommodations. 941 These are listed in the Appendix. Their supervision is mostly a 942 concern of the IOI IC. 943 944 The three major competition-related facilities are: 945 946 * Competition Site, with individual workplaces for each competitor 947 * GA meeting room, with at least two seats per delegation, and 948 seats for the various committee members 949 * GA computer room, with one computerized workplace per delegation 950 and for various other individuals 951 952 953 APPENDICES 954 955 Checklists 956 ========== 957 958 Task Type Descriptions 959 ---- ---- ------------ 960 * A general description of the type of task, including grading 961 information 962 * At least one complete example of a task description for this type 963 of task 964 * Complete documented solutions for the examples 965 966 Competition Rules 967 ----------- ----- 968 * Dates and times for practicing and for the Practice, First, and 969 Second Competition 970 * Dates and times for the competition-related GA Meetings, especially 971 - an Introduction Meeting, before the Practice Competition, where the 972 entire competition procedure is (again) briefly explained; 973 questions must be restricted to those that are urgent before the 974 Practice Competition takes place; other questions should be posed 975 at the following Questions Meeting 976 - a Questions Meeting, after the Practice Competition; there must be 977 time after this meeting for delegations to report to the 978 competitors before the First Competition 979 - a meeting before the First Competition, where the tasks are 980 presented and afterwards translated; the delegation members in 981 this meeting must not communicate with the competitors until 982 the First Competition is over 983 - a meeting where the results of the First Competition are 984 finalized; there must be time after this meeting for delegations 985 to report to the competitors before the Second Competition 986 - a meeting before the Second Competition, where the tasks are 987 presented and afterwards translated; the delegation members in 988 this meeting must not communicate with the competitors until 989 the Second Competition is over 990 - a meeting where the results of the Second Competition are 991 finalized 992 - a meeting where the final results of both competitions are 993 approved and where medals are assigned 994 - at least two meetings with ample opportunity to present, discuss, 995 and approve changes in the IOI Competition 996 997 * Hardware specifications: 998 - processor(s) type and speed, RAM size, HD size, monitor, 999 keyboard, pointing device 1000 * Software specifications: 1001 - Operating System, compilers, editors, etc. (exact versions) 1002 * Directory structure and other environment/setup info 1003 * Backup measures (e.g. do-it-yourself on diskette) 1004 * Printing facility 1005 * Task description general info, such as I/O format restrictions 1006 (also see below) 1007 * Auxiliary material 1008 - What is supplied, e.g. paper, diskette, special helper progs 1009 - What competitors may bring (pen), what not (calculators, books) 1010 * Outline of each Competition Round: 1011 - presentation of the Competition Tasks at the GA meeting 1012 - translation 1013 - preparation: competitors enter competition room(s), go to their 1014 assigned workplace without touching the competition 1015 material or competition equipment 1016 - start (competitors open envelope with competition material) 1017 - clarification requests (restricted to first hour; competitors 1018 may ask one questions per form; if this question is not understood 1019 by the organizing committee, then the competitor's delegation 1020 leader is requested to help interpret the question; the ONLY 1021 answers are YES, NO, and NO COMMENT) 1022 - what to do in case of systems trouble, restrooms, eating/drinking, 1023 - 15 minutes, 5 minutes, and 1 minute before-end announcements 1024 - end (stop working, power off) 1025 - grading of competitors' work 1026 - delegations have the opportunity to verify the results 1027 - appeals (through Grading Appeal Form) 1028 - finalization at GA meeting 1029 * Interface to grading system, additional details 1030 1031 Competition Tasks 1032 ----------- ----- 1033 * General requirements: 1034 the IOI Competition 1035 - must focus on problem solving in Computer Science 1036 - must emphasize fundamental, long-lasting concepts 1037 - must aim to interest and challenge all competitors 1038 - must offer a sufficiently diverse set of tasks, 1039 avoiding a one-sided view on Computer Science; 1040 not only should there be diversity within a single IOI, 1041 but also from one IOI to the next 1042 - must not require special skills or knowledge outside 1043 the approved areas of Computer Science (currently, 1044 computer programming, algorithms, data structures, but 1045 NOT speed typing or familiarity with computer architecture) 1046 - must have an effective way to overcome the language 1047 barrier, both to and from the competitors 1048 - must allow objective grading of the competitors' work; 1049 the grading process shall be developed together with the tasks 1050 (and not as an afterthought) 1051 - must be politically, culturally, and gender neutral; 1052 it must not favor any competitor 1053 - must present high-quality solutions for all Competition Tasks 1054 before the end of IOIn 1055 1056 * Task-specific general requirements: 1057 - Competition Tasks 1058 o must be new, that is, must not have appeared in earlier 1059 competitions or in the published literature 1060 o should contain some innovative element 1061 o must have been completely solved by the IOIn CC: 1062 * preferably in all allowed programming languages; 1063 * knowing the time and space complexity of the solutions 1064 * with a proof that the solutions meet the specified 1065 resource limits (time, memory) for all allowed inputs 1066 o must be effectively and efficiently gradable 1067 o must allow for partial credit to be awarded, that is, 1068 each task allows a, preferably wide, range of resulting 1069 scores; in particular, it is not 0% or 100% only. 1070 There are various ways to accomplish this, such as multiple 1071 test cases or the distinction of subtasks, each of which 1072 can be credited independently. 1073 - Task descriptions 1074 o must be written in correct and simple English, 1075 keeping in mind that the text needs to be translated 1076 o must be presented in a consistent layout and style 1077 o must be precise, complete, and unambiguous, 1078 paying special attention to 1079 * constraints (bounds) on input parameters 1080 (precondition) 1081 * input-output relationship (postcondition) 1082 * illustrations 1083 * input and output format (syntax) 1084 * resource limits (time, memory, or others) 1085 * file names and locations 1086 * library routine headers and parameter meanings 1087 * example(s) 1088 * grading details (what the scoring function is, e.g. 1089 the number of test cases and the number of points per 1090 test case for each subtask) 1091 o should be no longer than two printed pages at 12 point 1092 including illustrations, and preferably fit on ONE page 1093 o must include at least one non-trivial example 1094 o must be available in electronic form during translation 1095 - Additional task material accompanying the task description: 1096 o example input and output data in electronic form, 1097 must include all examples in the task description and 1098 also some larger cases 1099 o output format checker utility 1100 1101 * The set of Competition Tasks: 1102 - must contain a sufficient number of tasks: 1103 o Practice Competition (one or two) 1104 o Two Competition Rounds (typically, three each) 1105 o reserve (at least one per round) 1106 - must be of an appropriate level of difficulty: 1107 o the First Competition is somewhat easier than the Second 1108 o the set contains tasks of various difficulty levels, 1109 taking into account the considerable variation in skills 1110 of the competitors and ensuring more spread in the total 1111 score 1112 - must be accompanied by a Task Overview sheet, summarizing 1113 the key information of each task in the set, including 1114 o Task number, title, input file names, output file 1115 names, maximum total number of points 1116 1117 * Background info (as given to GA when deciding on tasks): 1118 - topic of task (in abstract Computing Science terms) 1119 - intended challenge (what are the major difficulties to overcome) 1120 - how will imperfect submissions score (receive partial credit); 1121 details of the test data need not be presented, but it should 1122 be explained what the design criteria of the test data were 1123 1124 * Grading material (to be used for grading) 1125 - test data: 1126 o with motivation (what is the purpose of each test case) 1127 o with scoring details (how is score derived from the test 1128 result) 1129 o sufficient number of test cases to ensure effective grading 1130 o reasonable distribution of size and difficulty, covering 1131 the entire spectrum allowed by the task description 1132 o testing all requirements (functionality and performance) 1133 - input validators: 1134 o that reads the test input data and checks that it agrees 1135 with the task description 1136 - output checkers: 1137 o for tasks where the correct output is uniquely determined 1138 by the input, a kind of file comparison utility can be used 1139 o for tasks where the correct output is not uniquely 1140 determined by the input, it will be necessary to develop 1141 a special-purpose output checker that reads the input and 1142 the captured output, and reports the result 1143 - grading scripts 1144 1145 * Documented solutions (to be disclosed to delegations after the start 1146 of the competition proper) incl. common imperfect and known-incorrect 1147 programs and how they would score 1148 1149 * Input-output shall be kept as simple as possible. 1150 - When represented as text, input data consists of a sequence of 1151 items. Each item is either a number or a string: 1152 o Numbers are represented in the usual decimal format, 1153 so that they can be read in by a standard input routine. 1154 Unless specifically stated otherwise, number values are 1155 within the range supported by the programming languages. 1156 o A string is a non-empty sequence of printable characters 1157 (ASCII values in the range 32..126). 1158 - Items on the same line are separated by a single space character. 1159 There are no space characters at the beginning or end of a line. 1160 Strings only appear at the end of a line; consequently, there is 1161 at most one string per line. 1162 - The structure of input text files is always such that end-of-line 1163 and end-of-file recognition is not an issue. There are three ways 1164 to accomplish this: 1165 o The number of items (or lines) is the same for all instances 1166 (e.g. "the first line contains the three numbers a, b, and c") 1167 o The number of items (or lines) is given earlier in the input 1168 (e.g. "the first line gives the number n of nodes; 1169 the following n lines each describe one node") 1170 o The number of items (or lines) is implicitly defined, by 1171 including a special terminating item 1172 (e.g. "each line contains a sequence of positive numbers 1173 followed by 0") 1174 In particular, this excludes a description like "the i-th line 1175 contains the neighbors of node i" when the number of neighbors 1176 is not known in advance, because in that case, determining the 1177 number of neighbors requires the recognition of end-of-line. 1178 - All input data used in grading must agree with the task 1179 description. The competitor programs never need to validate 1180 the input data. 1181 - Before checking, output files are first filtered to 1182 o remove space characters at the beginning and end of a line 1183 o replace multiple space characters between items by a single 1184 space character 1185 o remove empty lines 1186 1187 Grading 1188 ------- 1189 * General grading requirements for programming tasks: 1190 - the grading process must implement the following three 1191 (separate) phases: 1192 1st phase (execution): running the competitor's programs 1193 for all test cases, capturing the output; this requires 1194 compilation, when programs have been submitted in source form. 1195 2nd phase (evaluation): checking the captured output, and 1196 awarding points 1197 3rd phase (reporting): generating the reports 1198 - the 1st phase 1199 o must provide external timing 1200 o should provide automated abort of competitor's program 1201 when a resource limit (e.g. time) is exceeded 1202 o should detect competition rule violation 1203 o must detect and report run-time errors 1204 o must also capture standard output generated during execution 1205 o must provide a secure execution environment, especially 1206 for reactive tasks 1207 o must capture the dialog between the program and its 1208 environment for reactive tasks 1209 - the test cases (input, and how to check the output) must be 1210 developed in close cooperation with the task creator 1211 * Automated grading system (preferably defined by IOI Software Team) 1212 * Well-defined interfaces to 1213 - competitors, explaining 1214 o what source code restrictions apply (e.g. which 1215 compiler directives are permitted) 1216 o whether data files prepared during the competition can be 1217 used by the program when it is executed for grading, and 1218 if so, what restrictions apply 1219 o whether the program can read other (e.g. system) files 1220 when it is executed for grading 1221 o whether temporary files can be created and used by the 1222 program when it is executed for grading, and if so, 1223 what restrictions apply 1224 o what environment conditions hold during execution of 1225 the programs 1226 - task creators, explaining 1227 o in what format the correct and known-incorrect 1228 solutions are to be supplied 1229 o in what format the test cases are to be supplied 1230 (input files, input validators, output files or output 1231 checkers) 1232 o in what format the scoring details are to be supplied 1233 - system operators, explaining 1234 o how to set configuration files (to fill in details of 1235 each competition round) 1236 o how to operate the grading system 1237 * Grading schedule (if necessary: who is graded where at what time) 1238 * Appeal mechanism 1239 * After grading a competition, each delegation leader must receive: 1240 - A printed grading report, with total score and score break down 1241 (per test case and subtask), for each of their competitors 1242 - All submissions of each of their competitors (source and exe) 1243 - All output/log files produced by each of their competitors 1244 - All test data (input, and correct output if applicable) 1245 - Input validity checkers (source and exe) 1246 - File comparator (exe) 1247 - Output checkers used for grading, if applicable (source and exe) 1248 - All libraries and other support software (source and exe) 1249 - A grading appeal form, one for each of their competitors 1250 1251 NOTE: The availability of ALL relevant material (in source and exe 1252 form) is necessary to help locate errors quickly and to provide 1253 confidence. 1254 1255 Scripts and agendas 1256 ------- --- ------- 1257 All competition activities, especially those involving many officials, 1258 should be scripted. Such scripts describe, in chronological order, all 1259 steps involved in an activity. Each entry in a MASTER SCRIPT describes 1260 who is supposed to do what at which time and where. From the master 1261 script, one can extract scripts for specific roles or locations, only 1262 listing their activities. 1263 1264 Recommended are at least: 1265 * script for the entire competition proper, including 1266 - preparation of Competition Site before starting the competition: 1267 o reset equipment 1268 o test key functionality (printing, backup, ...) 1269 o distribute Competition Material 1270 o position officials (to supervise entry and to assist) 1271 - competitors assemble near Competition Site 1272 - competitors enter Competition Site, while officials check their 1273 badges 1274 - competitors locate their designated workplace and sit down 1275 - issue start signal (note exact time or start master stopwatch) 1276 - handling of Clarification Requests 1277 - signal to end period for Clarification Requests 1278 - regularly check and replenish refreshment supplies 1279 - issue before-end-signals (15, 5, 1 minute before end) 1280 - issue end signal and shut off equipment 1281 - competitors exit Competition Site and take refreshments 1282 * script for grading 1283 1284 It is also recommended that agendas are set for all the GA Meetings, 1285 even if they are not distributed. The agenda lists, in chronological 1286 order, the topics that are addressed in the meeting. 1287 1288 CC Room 1289 -- ---- 1290 * some networked (also Internet) computers for editing, etc. 1291 * some competition computers for tests 1292 * fast high-quality printer(s) 1293 * fast high-quality, high-volume photocopier(s) 1294 * blackboards or equivalent 1295 * communication lines to other rooms 1296 * office supplies 1297 1298 Competition Site 1299 ----------- ---- 1300 * separate secure area for technical committee (servers, printers, ...) 1301 * one workplace per competitor, including a chair, a flat 1302 table surface (minimum area 0.80m deep x 1.20m wide), and 1303 a computer with adjustable monitor, standard U.S. keyboard, and 1304 standard pointing device 1305 * some spare workplaces, for quick recovery when losing a workplace 1306 * all workplaces are as similar as possible 1307 * workplaces equally accommodate left- and right-handed competitors 1308 * minimal interference between workplaces 1309 * workplaces are clearly labeled with some kind of identification; 1310 a map is convenient 1311 * proper control over temperature, humidity, airflow, light, noise, 1312 and other distractions 1313 * could be in one large hall, possibly partitioned, or distributed 1314 over several laboratories (these should be reasonably close together) 1315 * ergonomic setup of computer and work space 1316 * facility to get printouts of files 1317 * backup mechanism available (either automatic to server, or by 1318 competitor to removable disk) 1319 * visible clock(s), synchronized in case of multiple clocks 1320 * system for voice announcements (such as start and stop signal, 1321 end-of-first-hour and before-end announcements), preferably with 1322 loudspeakers 1323 * no-smoking 1324 * opportunity to consume food and drinks 1325 * sufficient restrooms nearby 1326 * fire code requirements (emergency exits, wide aisles, nonflammable 1327 decoration, etc.) 1328 * concerns about assignment of competitors to workplaces: 1329 - competitors with special requirements 1330 - keep competitors from the same delegation separate 1331 - keep competitors with similar native languages separate 1332 * opportunity for delegations to take pictures 1333 * signs to (emergency) exit, refreshments, restrooms 1334 * competition officials to 1335 - assist with non-task-related issues (e.g. equipment) 1336 - pick up clarification requests 1337 - deliver printouts 1338 - keep an eye on obeyance of the rules 1339 * central competition desk to receive and track clarification requests, 1340 and other issue reports (near exit/entrance), with communication 1341 line to CC Room and technical area of Competition Site 1342 1343 GA Meeting Room 1344 -- ------- ---- 1345 * Single room 1346 * Seats, preferably with writing facility, on main floor facing podium 1347 - two per delegation, plus extra registered delegation members 1348 - one per IOI IC member 1349 - one per IOI SC member 1350 - one for IOI Secretariat 1351 * Podium with seats/table for competition organization (panel setup) 1352 - with name/role signs 1353 * Voice amplification system (microphones, loudspeakers) 1354 * Large Delegation Identification Signs (also used for voting) 1355 * Overhead projector 1356 * Computer projection (e.g. from laptop) 1357 * Communication line to CC Room 1358 * Clock 1359 1360 GA Computer Room(s) 1361 -- -------- ------- 1362 * Preferably a single room, but multiple rooms close together OK 1363 * Provide one workplace for 1364 - each delegation, 1365 - each IOI IC member, 1366 - each IOI SC member, and 1367 - the IOI Secretariat 1368 * Each workplace includes a computer with 1369 - adjustable monitor, standard U.S. keyboard, standard pointing 1370 device (preferably identical to competition computers) 1371 - appropriate software tools to 1372 o help produce translations for the task descriptions 1373 o rerun tests (after grading) 1374 - access to 1375 o a network printer 1376 o the Internet; Internet access may be interrupted during 1377 the period when the GA deals with confidential competition 1378 material 1379 * Clock 1380 1381 Practice Competition 1382 -------- ----------- 1383 * In format and procedure, as similar as possible, to First and 1384 Second Competition 1385 * One very simple task per task TYPE, especially for new types 1386 * How about translation? There is less need to keep these tasks 1387 secret. They could be released to the team leaders before IOIn. 1388 The team leaders can translate them before IOIn, without disclosing 1389 them to the competitors. However, that way, the translation process 1390 itself is not practiced (tested). 1391 1392 1393 Templates and Forms 1394 ========= === ===== 1395 1396 All officially distributed IOIn competition documents (incl. forms) must 1397 carry an IOIn identification (preferably with IOIn logo) and date 1398 or date range. The IOIn location may also be mentioned. 1399 1400 All pages of multipage documents must be numbered and the total number 1401 of pages must appear near the page number, such as "page n of t" or 1402 "n/t". 1403 1404 Competition Task Description Template (for task creators) 1405 ----------- ---- ----------- -------- 1406 { Practice | First | Second } 1407 { preceding items may be supplied later } 1408 { one or two simple short words } 1409 <General part> 1410 <Input format> 1411 <Output format> 1412 <Example> 1413 <Resource constraints> { time limit, etc. } 1414 <Grading information> 1415 1416 Competition Task Overview Template 1417 ----------- ---- -------- -------- 1418 To be supplied 1419 1420 Competition Task Comment Form (for GA members) 1421 ----------- ---- ------- ---- 1422 Delegation Code: ... (filled in by delegation leader) 1423 Topic: General/Task 1/2/3/4/5/6 (circled by delegation leader) 1424 Task Version: a/b/c/... 1425 1426 Comments: ..... (filled in by delegation leader) 1427 1428 Response: ..... (optionally filled in by IOIn CC) 1429 1430 Translation Submission Form (for delegations, format to be defined) 1431 ----------- ---------- ---- 1432 Concerns: Task 1/2/3/4/5/6 1433 Competitor Code(s): ... 1434 Remarks: ... 1435 1436 Clarification Request Form (for competitors, format to be defined) 1437 ------------- ------- ---- 1438 Time Received: ..... (filled in at central competition desk) 1439 Tracking Number: ..... (filled in at central competition desk) 1440 1441 Competitor Code: ... (filled in by competitor or official) 1442 Topic: General/Task 1/2/3/4/5/6 (circled by competitor) 1443 1444 Question: .......... (filled in by competitor) 1445 1446 Translation: ......... (optionally filled in by GA member) 1447 1448 Answer: [ ] Yes (on three lines to avoid ambiguity) 1449 [ ] No (marked by IOIn CC) 1450 [ ] No Comment 1451 1452 Before returning, archive a photocopy! 1453 1454 Grading Appeal Form (format to be defined) 1455 ------- ------ ---- 1456 Time Received: ..... (filled in at central competition desk) 1457 Tracking Number: ..... (filled in at central competition desk) 1458 1459 Competitor Code: ... (filled in by delegation leader) 1460 Topic: General/Task 1/2/3/4/5/6 (circled by delegation leader) 1461 1462 Appeal Text: ......... (filled in by delegation leader) 1463 1464 Response: ............ (optionally filled in by IOIn CC) 1465 1466 IOIn Evaluation Questionnaire 1467 ---- ---------- ------------- 1468 To be supplied 1469 1470 1471 KNOWN MISSING ITEMS 1472 ===== ======= ===== 1473 More details on requirements for Competition Tasks, especially the task 1474 types, task descriptions, and grading. Options: Require additional test data 1475 (e.g. two larger test cases), require specific helper programs (to display 1476 text files with all spaces and end-of-lines clearly visible; to do a timed 1477 test run; to check output format; ...). 1478 1479 Identification of Competition Tasks: Task 1, 2, 3 for First 1480 Competition Round, Task 4, 5, 6 for Second Competition Round; I do not 1481 like the combination Day x/Task y; Names are also used, but then it is not 1482 so clear to what round they belong. 1483 1484 Grading Software 1485 1486 and more ... (?) 1487 1488 1489 CONVENTIONS (in these Guidelines) 1490 =========== 1491 1492 American English spelling and style are used. 1493 1494 For ease of reference, lines are numbered in distributed versions. 1495 1496 Sets of items are presented in unnumbered lists, where each item is 1497 decorated with "*", "-", "o", depending on nesting depth. 1498 1499 Sequences of items are presented in numbered lists. 1500 1501 IOI'xy is short for IOI19xy. 1502 1503 The passive voice is avoided as much as possible, because it encourages 1504 vagueness.