From gvcormac@uwaterloo.ca Mon May 2 13:25:21 2005 Date: Mon, 2 May 2005 13:25:21 -0400 From: "Gordon V. Cormack" To: trecspam@nist.gov Subject: Test Environment - Fedora/Windows I'd like to outline for participants (i.e. people who will actually be submitting systems to be tested in the TREC runs) the test setup. I would appreciate two kinds of feedback: i. (Urgent!) Please let me know if you require Windows, or if anything stated below would preclude your participation. ii. General comments. --- 1. Primary (Linux) Operating Environment: The primary environment will be Fedora Core 3 (full installation) running on AMD 64 systems. [Addendum: 32-bit Linux configuration will be used] Participants must notify the coordinators in advance if any other software packages are required. [Addendum: list of Fedora packages is here: http://plg.uwaterloo.ca/~gvcormac/spam/rpms.txt ] At least 1GB of RAM and 1GB of disk space will be available in the current directory in which the filter is run. Cumulative and per-message time limits will be enforced: approx. 2 seconds cumulative and 15 seconds for each particular message. [Addendum: the 15 second per-command limit will have to be reconsidered so as to allow occasional batch reorganization.] If the filter exceeds these limits or crashes, a "ham" classification will be assumed. All scripts (including initialize and finalize) will be run as a non-privileged user. No network access will be permitted. [Addendum: TCP/IP will be configured, but non-local routes will be disabled.] Except for the enforcement of these limits, the behaviour of the test environment will be identical to that of the published evaluation kit. Some minor revisions to the kit are pending, and will be available by May 15. 2. Alternate (Windows) Operating Environment: Participants are encouraged to use the Primary Linux Operating Environment; however, a Windows XP [Addendum: 32-bit] environment will be available to participants who notify the coordinators in adavance. The Windows XP environment will contain a complete Cygwin environment, and the filter commands will be invoked from this environment. Resource limits will be enforced; however, the enforcement may be more arbitrary owing to limitations in the Windows environment. [Addendum: IIS will be installed] Additional software required must be supplied by the participant in advance. 3. Chronology May 15 - Evaluation kit finalized, sample data available. Deadlines finalized. **** NOTE **** Pilot filter submission deadline has been changed. July 5 - Pilot filter submission. Participants should submit a filter implementation for evaluation in the test environment. Participants should also submit the expected results of running the pilot filter on the sample data. The purpose of this implementation is to test the setup, not to measure the effectiveness of the filter. The coordinators will run the filter on the sample data and verify that it matches the submitted results. Timings and results will be returned to the submitting participant. July 15 - Results of pilot runs returned to participants. July 28 - Final filter submission. Each participant submits one or more versions of their filter for evaluation. Filters will be run by the coordinators on the private data and by the participant on the public data (to be released after filter submission). Participants who submit muliple filter versions should rank them by priority for evaluation. The coordinators will evaluate as many versions as possible with the resources available. It is our intent to evaluate at least two versions per participant. Until the number of submissions is known, we cannot anticipate how many versions will actually be evaluated. July 29 - Public test data released. September 6 - Results on public data submitted. Each participant must submit the raw result files from running their filter on the public data. -- Gordon V. Cormack CS Dept, University of Waterloo, Canada N2L 3G1 gvcormack@uwaterloo.ca http://cormack.uwaterloo.ca/cormack