Forum Post
Index > Scribe > Bayesian filter query

Author/Date Bayesian filter query
Mike Green
09/09/2004 4:28pm
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

I'd like a bit of clarification on the way that the Bayesian
filtering works in InScribe. I've read the help and understand
that the SPAM list is updated when initiating re-scanning and
that this requires that all spam be left in the spam folder.
Fine. It sounds, however, as if the same applies to HAM. i.e.
the HAM wdb is built up from scratch each time the word list
update function is run. Is this correct?

If this IS the case, and it seems like it from a couple of
experiments I've done, then this seems to be a bit of an issue
it that I personally tend to delete most mail pretty much
straight away (maybe that's abnormal or aberrant behaviour?!).
This isn't to say it's spam, it just means that I don't want to
retain it for any reason. Net result is that the HAM file is
very small, and rather prone to frequent, on-going change.

I don't claim to understand Bayesian filtering all that well,
but doesn't it rely on checking for positive words as well as
recognising 'negative', spam-like patterns? If so then my
failure to keep much HAM will weaken the filter - but is that a
significant problem?

Just seeking clarification here really!

Thanks,

Mike

-----BEGIN PGP SIGNATURE-----
Version: PGP 8.0.3 - not licensed for commercial use: www.pgp.com

iQA/AwUBQUCEsPVnEmFoUW40EQJLKwCgwiezoOTSv1z0l8KMYyNBMjWw/U8AnjJc
9+6gPajUhVlSlPZVv85NZVr1
=D8lU
-----END PGP SIGNATURE-----
fReT
09/09/2004 9:55pm
You are right the ham must remain for the filter to work correctly. I'm working on a incremental version of the algorithm so that you can delete mail and still retain the word counts.
Mike Green
09/09/2004 10:08pm
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

So does the ham have a significant effect on the efficiency of
the filter? ie. if there isn't much of it will the filter skew,
once trained with lots of spam, towards false positives rather
more than it would if I didn't delete just about everything
rather than retain it? (My ham db is anyway mostly meaningless
strings since most of my mail is encrypted and that's what the
ham db is picking up.)

Excellent program by the way. Far more elegant than most mail
programs and the fact that I can install it on a USB drive
along with encryption software is a very major plus point. This
query is largely out of interest, not something I'd regard as
important. Now the other one, regarding attributed quoting,
would be a really nice to have, it's the one thing 'The Bat!'
does which could, but probably won't, lure me towards its use.
I'll keep my fingers crossed that it'll be easy to program :-)

Thanks for the response.

Mike

-----BEGIN PGP SIGNATURE-----
Version: PGP 8.0.3 - not licensed for commercial use: www.pgp.com

iQA/AwUBQUDUYPVnEmFoUW40EQIIRwCgvIy893OoHYohmbDN+bsxs3H8LbEAn2xs
ofdpu+Jue1iupYH96Cv6xvud
=TuE8
-----END PGP SIGNATURE-----
Justin Heiner
17/09/2004 7:54am
I use the ham folder for basically a whitelist. Move all the e-mails that are wrongly marked as spam into ham and all e-mails from then on from that sender are correctly placed.

Probably ham's intended usage, i'm guessing :)
Reply