[sanesecurity] Re: ham testing
- From: Michael Orlitzky <michael@xxxxxxxxxxxx>
- To: sanesecurity@xxxxxxxxxxxxx
- Date: Tue, 14 Jul 2009 13:27:59 -0400
Tom Shaw wrote:
Steve and Bill,
Personally I think "ham" testing will not add as much "safety" as being
asserted.
1st your ham and my ham are vastly different as are others on the list.
Further, ham for Europe is different than ham for an Asian than ham for
a South America user, etc.
OK, ham testing theoretically could have detected "acebook.com" but I
have friends and clients who do not have facebook.com in their ham
because they wash their ham every 14 days and so would have never
detected the problem prior to a facebook message appearing. FUrther, I
expect that the next FP to happen will not be in whatever "ham" set you
are testing against which might make ham testing intrinsically problematic.
I'm going to have to disagree here, too. Estimates of "safety" aside,
testing the signatures against a known-good corpus undoubtedly provides
/some/ safety, and I would argue that it does so at little to no extra cost.
In general, I'm much more concerned about false positives than I am
false negatives. Testing against known ham provides a safety net against
false positives, at only a comparable risk of false negatives (in the
case where we ignore some signatures temporarily). Given that I weigh
the false positives much more heavily, I see that as an overall benefit.
True, users will differ with regards to what they consider "good" mail,
but I don't think that can be used as an argument against testing here:
if your users don't have any mail that would trigger "acebook.com," then
the "acebook.com" signature is less likely to be a false positive for
your users. Not impossible, but less likely. And in that case, you're
right back where you started; no harm done. But for others (for whom
"acebook.com" is a false positive), damage would have been prevented.
IMHO you are better off checking for small sigs (which would have
detected the "com" problem) and washing against large whitelists (which
we do already do).
I can provide you and Bill my script to check signatures against URIbl
whitelists as well as bondedsender and many others and cache results if
you want. It currently is in PHP but could easily be used as is or recoded.
These queryable DB's are much more comprehensive than someone's (or a
group of someones') ham. Further, the shear effort to maintain a
comprehensive, world aware ham database seems like a tall order.
If admins are alerted when the "ham test" fails, they will be able to
report false positives quicker, improving the overall quality of the
database as a result.
I don't think anyone is suggesting we maintain a world-wide database of
known-good mail. On the contrary, as you mentioned, there would be less
benefit to me testing against someone else's mail. In this respect, I
think that the ability to test one's own mail (e.g. via the update
script) could be more effective than pre-screening the signatures before
they are entered in to the database.
Of course, having to do the work oneself significantly reduces the
number of people who are willing to put forth the effort. Fortunately,
the two are not mutually exclusive.
I would suggest trying filtering on small sigs and checking these world
aware whitelists as a first start before taking on the task of ham
maintenance. Our experience in adding these checks to the winnow dynamic
process makes a big difference.
If after that more checking seems to be in order, someone can start to
build a comprehensive ham DB.
I would also like to query how many folks are using these dynamic sigs
without scoring. The reason I ask is it has been reiterated over and
over again to use them as part of scoring. We score and did not
experience rejections of com nor acebook.com. Maybe the solution is to
ask for scoring or have users reconsider there scores - after all this
is what you have to do in any scoring based system.
These are all good ideas. However, they aren't mutually exclusive
either. It is entirely possible that maximum accuracy will be attained
by some combination of the methods being discussed. For example, if
someone is using the signatures without scoring; sure, he can improve
his accuracy by implementing scoring. But, could he improve it further
by implementing scoring *and* the "ham test?" Probably.
Other related posts:
