Re Zdziarski's Factual Errors
We shall not respond to Mr. Zdziarski's attacks, except to identify
the most outstanding factual errors and to note that ad hominem
arguments are irrelevant in assessing the validity of our work.
We encourage interested parties to read our paper.
1. Gold Standard. The bottom line is that for every case of disagreement
between any filter and the Gold Standard, X re-adjudicated the message.
That means, for example, that DSPAM's 116 ham misclassifications (false
positives) and 791 spam misclassifications (false negatives) were all
examined and verified by X to be misclassifications.
Further, X did examine every message at least once in constructing the
original Gold Standard. Any remaining errors in the Gold Standard would
be to the advantage of the subject systems.
Our paper states:
All subsequent disagreements between the gold standard and later
runs were also manually adjudicated, and all runs were repeated with
the updated gold standard. The results presented here are based on
this revised standard, in which all cases of disagreement have been
2. Learning configuration. DSPAM was configured exactly as specified in
the documentation supplied with v 2.8.3, and discussed with Zdziarski
in numerous emails. One change was made as the result of this
correspondence: we used the output from DSPAM as input to our
"filtertrain" procedure, as opposed to using the original message.
v 2.8.3 has no flags for "train on error" and our report as to DSPAM's
internal training behaviour was based on our understanding from our
correspondence of April 23 with Zdziarski. Based on our very recent
correspondence with Zdziarski, we now understand that DSPAM internally
trains on everything, and we will note this in the paper. In
the meantime we have placed errata on our web page.
This descriptive characterization of DSPAM's internal behaviour has
no bearing on the test setup. We used DSPAM "out of the box" as
documented. More precisely, we implemented Algorithm 1 in the
for each email (in arrival order)
dspam --stdout --deliver-spam -d < email > dspamout
if (email is ham and dspam reports spam)
dspam --falsepositive < dspamout
if (email is spam and dspam reports ham)
dspam --addspam < dspamout
As Zdziarski point out, a misunderstanding as to version number
probably accounts for our miscommunication in this matter.
3. Our paper says "Zdziarski reports 99.95% to 99.991% accuracy for DSPAM
based on an unspecified methodology." Part of the lack of specification
is the version and configuration of DSPAM used in achieving these
results. Similar claims were reported at the time of the Slashdot
article "Two Spam Filters 10 Times As Accurate As Humans." Although
no configuration was stated at this time either, it is our understanding
that v2.8.3 was the current stable release.
We believe it is scientifically appropriate to select a set of filters
and freeze them prior to the collection of results; our pilot evaluations
began in February.
4. The conclusion of this paper is that supervised statistical filters
greatly improve on the filtering capabilities of Spamassassin's static
At no point does the paper suggest that Spamassassin's static rules
are better than statistical filters.
Only the statistical filtering component of Spamassassin (with no
static rules) was compared against the other statistical filters,
5. We measured initial error rates and final error rates, and plotted
piecewise and regression-based estimates of the error rates as a
functions of the number of messages processed. So, for example,
if the reader would like to see the performance after 10,000 messages
of training, this information may be determined from the figures.
As Zdziarski pointed out in an earlier draft, DSPAM's discontinuous
learning process can be observed in figure 16.
June 24, 2004
Zdziarski's comments have been revised many times since June 24.
We thank Zdziarski for removing or qualifying some of the most egregious
ad hominem statements.
In response to point 1 above, Zdziarski qualifies his assertions
about the manner in which we constructed our gold standard. We stand by our
original description, which is amplified above.
With regard to the configuration used for DSPAM, we state that the
description given in point 2 accurately reflects the setup we used for
DSPAM. This setup has been invariant since April 23, and is the setup we
used to evaluate DSPAM's performance. The DSPAM distribution that we
used is reproduced here. This version does not have the training parameters
"TOE, TEFT, and TUM". Its only training parameter, --enable-test-conditional,
is not recommended and was not used.
Zdziarski appears to have removed reference to his unsubstantiated
accuracy claims for DSPAM, which are noted in point 3.
We re-assert points 4 and 5.
June 27, 2004
DSPAM Version and Configuration Information
As stated in our paper, we used DSPAM 2.8.3.
The configuration log is reproduced here.