From gvcormac@uwaterloo.ca Mon May 2 13:25:21 2005
Date: Mon, 2 May 2005 13:25:21 -0400
From: "Gordon V. Cormack" <gvcormac@uwaterloo.ca>
To: trecspam@nist.gov
Subject: Test Environment - Fedora/Windows
I'd like to outline for participants (i.e. people who will actually be
submitting systems to be tested in the TREC runs) the test setup.
I would appreciate two kinds of feedback:
i. (Urgent!) Please let me know if you require Windows, or if anything
stated below would preclude your participation.
ii. General comments.
---
1. Primary (Linux) Operating Environment:
The primary environment will be Fedora Core 3 (full installation)
running on AMD 64 systems. [Addendum: 32-bit Linux configuration
will be used] Participants must notify the
coordinators in advance if any other software packages are
required.
[Addendum: list of Fedora packages is here:
http://plg.uwaterloo.ca/~gvcormac/spam/rpms.txt ]
At least 1GB of RAM and 1GB of disk space will be available in
the current directory in which the filter is run.
Cumulative and per-message time limits will be enforced: approx.
2 seconds cumulative and 15 seconds for each particular message.
[Addendum: the 15 second per-command limit will have to be
reconsidered so as to allow occasional batch reorganization.]
If the filter exceeds these limits or crashes, a "ham" classification
will be assumed.
All scripts (including initialize and finalize) will be run as a
non-privileged user.
No network access will be permitted. [Addendum: TCP/IP will be
configured, but non-local routes will be disabled.]
Except for the enforcement of these limits, the behaviour of the
test environment will be identical to that of the published
evaluation kit. Some minor revisions to the kit are pending, and
will be available by May 15.
2. Alternate (Windows) Operating Environment:
Participants are encouraged to use the Primary Linux Operating
Environment; however, a Windows XP [Addendum: 32-bit] environment will
be available to participants who notify the coordinators in adavance.
The Windows XP environment will contain a complete Cygwin environment,
and the filter commands will be invoked from this environment.
Resource limits will be enforced; however, the enforcement may be
more arbitrary owing to limitations in the Windows environment.
[Addendum: IIS will be installed]
Additional software required must be supplied by the participant
in advance.
3. Chronology
May 15 - Evaluation kit finalized, sample data available. Deadlines
finalized.
**** NOTE **** Pilot filter submission deadline has been changed.
July 5 - Pilot filter submission. Participants should submit a
filter implementation for evaluation in the test environment.
Participants should also submit the expected results of
running the pilot filter on the sample data.
The purpose of this implementation is to test the setup, not
to measure the effectiveness of the filter. The coordinators
will run the filter on the sample data and verify that it
matches the submitted results. Timings and results will be
returned to the submitting participant.
July 15 - Results of pilot runs returned to participants.
July 28 - Final filter submission. Each participant submits one or
more versions of their filter for evaluation. Filters will
be run by the coordinators on the private data and by the
participant on the public data (to be released after filter
submission).
Participants who submit muliple filter versions should rank
them by priority for evaluation. The coordinators will
evaluate as many versions as possible with the resources
available. It is our intent to evaluate at least two versions
per participant. Until the number of submissions is known, we
cannot anticipate how many versions will actually be evaluated.
July 29 - Public test data released.
September 6 - Results on public data submitted. Each participant must
submit the raw result files from running their filter on
the public data.
--
Gordon V. Cormack CS Dept, University of Waterloo, Canada N2L 3G1
gvcormack@uwaterloo.ca http://cormack.uwaterloo.ca/cormack