1SA-LEARN(1)           User Contributed Perl Documentation          SA-LEARN(1)
2
3
4

NAME

6       sa-learn - train SpamAssassin's Bayesian classifier
7

SYNOPSIS

9       sa-learn [options] [file]...
10
11       sa-learn [options] --dump [ all | data | magic ]
12
13       Options:
14
15        --ham                 Learn messages as ham (non-spam)
16        --spam                Learn messages as spam
17        --forget              Forget a message
18        --use-ignores         Use bayes_ignore_from and bayes_ignore_to
19        --sync                Synchronize the database and the journal if needed
20        --force-expire        Force a database sync and expiry run
21        --dbpath <path>       Allows commandline override (in bayes_path form)
22                              for where to read the Bayes DB from
23        --dump [all|data|magic]  Display the contents of the Bayes database
24                              Takes optional argument for what to display
25         --regexp <re>        For dump only, specifies which tokens to
26                              dump based on a regular expression.
27        -f file, --folders=file  Read list of files/directories from file
28        --dir                 Ignored; historical compatibility
29        --file                Ignored; historical compatibility
30        --mbox                Input sources are in mbox format
31        --mbx                 Input sources are in mbx format
32        --showdots            Show progress using dots
33        --progress            Show progress using progress bar
34        --no-sync             Skip synchronizing the database and journal
35                              after learning
36        -L, --local           Operate locally, no network accesses
37        --import              Migrate data from older version/non DB_File
38                              based databases
39        --clear               Wipe out existing database
40        --backup              Backup, to STDOUT, existing database
41        --restore <filename>  Restore a database from filename
42        -u username, --username=username
43                              Override username taken from the runtime
44                              environment, used with SQL
45        -C path, --configpath=path, --config-file=path
46                              Path to standard configuration dir
47        -p prefs, --prefspath=file, --prefs-file=file
48                              Set user preferences file
49        --siteconfigpath=path Path for site configs
50                              (default: /etc/mail/spamassassin)
51        --cf='config line'    Additional line of configuration
52        -D, --debug [area=n,...]  Print debugging messages
53        -V, --version         Print version
54        -h, --help            Print usage message
55

DESCRIPTION

57       Given a typical selection of your incoming mail classified as spam or
58       ham (non-spam), this tool will feed each mail to SpamAssassin, allowing
59       it to 'learn' what signs are likely to mean spam, and which are likely
60       to mean ham.
61
62       Simply run this command once for each of your mail folders, and it will
63       ''learn'' from the mail therein.
64
65       Note that csh-style globbing in the mail folder names is supported; in
66       other words, listing a folder name as "*" will scan every folder that
67       matches.  See "Mail::SpamAssassin::ArchiveIterator" for more details.
68
69       SpamAssassin remembers which mail messages it has learnt already, and
70       will not re-learn those messages again, unless you use the --forget
71       option. Messages learnt as spam will have SpamAssassin markup removed,
72       on the fly.
73
74       If you make a mistake and scan a mail as ham when it is spam, or vice
75       versa, simply rerun this command with the correct classification, and
76       the mistake will be corrected.  SpamAssassin will automatically
77       'forget' the previous indications.
78
79       Users of "spamd" who wish to perform training remotely, over a network,
80       should investigate the "spamc -L" switch.
81

OPTIONS

83       --ham
84           Learn the input message(s) as ham.   If you have previously learnt
85           any of the messages as spam, SpamAssassin will forget them first,
86           then re-learn them as ham.  Alternatively, if you have previously
87           learnt them as ham, it'll skip them this time around.  If the
88           messages have already been filtered through SpamAssassin, the
89           learner will ignore any modifications SpamAssassin may have made.
90
91       --spam
92           Learn the input message(s) as spam.   If you have previously learnt
93           any of the messages as ham, SpamAssassin will forget them first,
94           then re-learn them as spam.  Alternatively, if you have previously
95           learnt them as spam, it'll skip them this time around.  If the
96           messages have already been filtered through SpamAssassin, the
97           learner will ignore any modifications SpamAssassin may have made.
98
99       --folders=filename, -f filename
100           sa-learn will read in the list of folders from the specified file,
101           one folder per line in the file.  If the folder is prefixed with
102           "ham:type:" or "spam:type:", sa-learn will learn that folder
103           appropriately, otherwise the folders will be assumed to be of the
104           type specified by --ham or --spam.
105
106           "type" above is optional, but is the same as the standard for
107           ArchiveIterator: mbox, mbx, dir, file, or detect (the default if
108           not specified).
109
110       --mbox
111           sa-learn will read in the file(s) containing the emails to be
112           learned, and will process them in mbox format (one or more emails
113           per file).
114
115       --mbx
116           sa-learn will read in the file(s) containing the emails to be
117           learned, and will process them in mbx format (one or more emails
118           per file).
119
120       --use-ignores
121           Don't learn the message if a from address matches configuration
122           file item "bayes_ignore_from" or a to address matches
123           "bayes_ignore_to".  The option might be used when learning from a
124           large file of messages from which the hammy spam messages or spammy
125           ham messages have not been removed.
126
127       --sync
128           Synchronize the journal and databases.  Upon successfully syncing
129           the database with the entries in the journal, the journal file is
130           removed.
131
132       --force-expire
133           Forces an expiry attempt, regardless of whether it may be necessary
134           or not.  Note: This doesn't mean any tokens will actually expire.
135           Please see the EXPIRATION section below.
136
137           Note: "--force-expire" also causes the journal data to be
138           synchronized into the Bayes databases.
139
140       --forget
141           Forget a given message previously learnt.
142
143       --dbpath
144           Allows a commandline override of the bayes_path configuration
145           option.
146
147       --dump option
148           Display the contents of the Bayes database.  Without an option or
149           with the all option, all magic tokens and data tokens will be
150           displayed.  magic will only display magic tokens, and data will
151           only display the data tokens.
152
153           Can also use the --regexp RE option to specify which tokens to
154           display based on a regular expression.
155
156       --clear
157           Clear an existing Bayes database by removing all traces of the
158           database.
159
160           WARNING: This is destructive and should be used with care.
161
162       --backup
163           Performs a dump of the Bayes database in machine/human readable
164           format.
165
166           The dump will include token and seen data.  It is suitable for
167           input back into the --restore command.
168
169       --restore=filename
170           Performs a restore of the Bayes database defined by filename.
171
172           WARNING: This is a destructive operation, previous Bayes data will
173           be wiped out.
174
175       -h, --help
176           Print help message and exit.
177
178       -u username, --username=username
179           If specified this username will override the username taken from
180           the runtime environment.  You can use this option to specify users
181           in a virtual user configuration when using SQL as the Bayes
182           backend.
183
184           NOTE: This option will not change to the given username, it will
185           only attempt to act on behalf of that user.  Because of this you
186           will need to have proper permissions to be able to change files
187           owned by username.  In the case of SQL this generally is not a
188           problem.
189
190       -C path, --configpath=path, --config-file=path
191           Use the specified path for locating the distributed configuration
192           files.  Ignore the default directories (usually
193           "/usr/share/spamassassin" or similar).
194
195       --siteconfigpath=path
196           Use the specified path for locating site-specific configuration
197           files.  Ignore the default directories (usually
198           "/etc/mail/spamassassin" or similar).
199
200       --cf='config line'
201           Add additional lines of configuration directly from the command-
202           line, parsed after the configuration files are read.   Multiple
203           --cf arguments can be used, and each will be considered a separate
204           line of configuration.
205
206       -p prefs, --prefspath=prefs, --prefs-file=prefs
207           Read user score preferences from prefs (usually
208           "$HOME/.spamassassin/user_prefs").
209
210       --progress
211           Prints a progress bar (to STDERR) showing the current progress.  In
212           the case where no valid terminal is found this option will behave
213           very much like the --showdots option.
214
215       -D [area,...], --debug [area,...]
216           Produce debugging output. If no areas are listed, all debugging
217           information is printed. Diagnostic output can also be enabled for
218           each area individually; area is the area of the code to instrument.
219           For example, to produce diagnostic output on bayes, learn, and dns,
220           use:
221
222                   spamassassin -D bayes,learn,dns
223
224           For more information about which areas (also known as channels) are
225           available, please see the documentation at:
226
227                   C<http://wiki.apache.org/spamassassin/DebugChannels>
228
229           Higher priority informational messages that are suitable for
230           logging in normal circumstances are available with an area of
231           "info".
232
233       --no-sync
234           Skip the slow synchronization step which normally takes place after
235           changing database entries.  If you plan to learn from many folders
236           in a batch, or to learn many individual messages one-by-one, it is
237           faster to use this switch and run "sa-learn --sync" once all the
238           folders have been scanned.
239
240           Clarification: The state of --no-sync overrides the
241           bayes_learn_to_journal configuration option.  If not specified, sa-
242           learn will learn to the database directly.  If specified, sa-learn
243           will learn to the journal file.
244
245           Note: --sync and --no-sync can be specified on the same
246           commandline, which is slightly confusing.  In this case, the
247           --no-sync option is ignored since there is no learn operation.
248
249       -L, --local
250           Do not perform any network accesses while learning details about
251           the mail messages.  This will speed up the learning process, but
252           may result in a slightly lower accuracy.
253
254           Note that this is currently ignored, as current versions of
255           SpamAssassin will not perform network access while learning; but
256           future versions may.
257
258       --import
259           If you previously used SpamAssassin's Bayesian learner without the
260           "DB_File" module installed, it will have created files in other
261           formats, such as "GDBM_File", "NDBM_File", or "SDBM_File".  This
262           switch allows you to migrate that old data into the "DB_File"
263           format.  It will overwrite any data currently in the "DB_File".
264
265           Can also be used with the --dbpath path option to specify the
266           location of the Bayes files to use.
267

MIGRATION

269       There are now multiple backend storage modules available for storing
270       user's bayesian data. As such you might want to migrate from one
271       backend to another. Here is a simple procedure for migrating from one
272       backend to another.
273
274       Note that if you have individual user databases you will have to
275       perform a similar procedure for each one of them.
276
277       sa-learn --sync
278           This will sync any outstanding journal entries
279
280       sa-learn --backup > backup.txt
281           This will save all your Bayes data to a plain text file.
282
283       sa-learn --clear
284           This is optional, but good to do to clear out the old database.
285
286       Repeat!
287           At this point, if you have multiple databases, you should perform
288           the procedure above for each of them. (i.e. each user's database
289           needs to be backed up before continuing.)
290
291       Switch backends
292           Once you have backed up all databases you can update your
293           configuration for the new database backend. This will involve at
294           least the bayes_store_module config option and may involve some
295           additional config options depending on what is required by the
296           module. (For example, you may need to configure an SQL database.)
297
298       sa-learn --restore backup.txt
299           Again, you need to do this for every database.
300
301       If you are migrating to SQL you can make use of the -u <username>
302       option in sa-learn to populate each user's database. Otherwise, you
303       must run sa-learn as the user who database you are restoring.
304

INTRODUCTION TO BAYESIAN FILTERING

306       (Thanks to Michael Bell for this section!)
307
308       For a more lengthy description of how this works, go to
309       http://www.paulgraham.com/ and see "A Plan for Spam". It's reasonably
310       readable, even if statistics make me break out in hives.
311
312       The short semi-inaccurate version: Given training, a spam heuristics
313       engine can take the most "spammy" and "hammy" words and apply
314       probabilistic analysis. Furthermore, once given a basis for the
315       analysis, the engine can continue to learn iteratively by applying both
316       the non-Bayesian and Bayesian rulesets together to create evolving
317       "intelligence".
318
319       SpamAssassin 2.50 and later supports Bayesian spam analysis, in the
320       form of the BAYES rules. This is a new feature, quite powerful, and is
321       disabled until enough messages have been learnt.
322
323       The pros of Bayesian spam analysis:
324
325       Can greatly reduce false positives and false negatives.
326           It learns from your mail, so it is tailored to your unique e-mail
327           flow.
328
329       Once it starts learning, it can continue to learn from SpamAssassin and
330       improve over time.
331
332       And the cons:
333
334       A decent number of messages are required before results are useful for
335       ham/spam determination.
336       It's hard to explain why a message is or isn't marked as spam.
337           i.e.: a straightforward rule, that matches, say, "VIAGRA" is easy
338           to understand. If it generates a false positive or false negative,
339           it is fairly easy to understand why.
340
341           With Bayesian analysis, it's all probabilities - "because the past
342           says it is likely as this falls into a probabilistic distribution
343           common to past spam in your systems". Tell that to your users!
344           Tell that to the client when he asks "what can I do to change
345           this". (By the way, the answer in this case is "use whitelisting".)
346
347       It will take disk space and memory.
348           The databases it maintains take quite a lot of resources to store
349           and use.
350

GETTING STARTED

352       Still interested? Ok, here's the guidelines for getting this working.
353
354       First a high-level overview:
355
356       Build a significant sample of both ham and spam.
357           I suggest several thousand of each, placed in SPAM and HAM
358           directories or mailboxes.  Yes, you MUST hand-sort this - otherwise
359           the results won't be much better than SpamAssassin on its own.
360           Verify the spamminess/haminess of EVERY message.  You're urged to
361           avoid using a publicly available corpus (sample) - this must be
362           taken from YOUR mail server, if it is to be statistically useful.
363           Otherwise, the results may be pretty skewed.
364
365       Use this tool to teach SpamAssassin about these samples, like so:
366                   sa-learn --spam /path/to/spam/folder
367                   sa-learn --ham /path/to/ham/folder
368                   ...
369
370           Let SpamAssassin proceed, learning stuff. When it finds ham and
371           spam it will add the "interesting tokens" to the database.
372
373       If you need SpamAssassin to forget about specific messages, use the
374       --forget option.
375           This can be applied to either ham or spam that has run through the
376           sa-learn processes. It's a bit of a hammer, really, lowering the
377           weighting of the specific tokens in that message (only if that
378           message has been processed before).
379
380       Learning from single messages uses a command like this:
381                   sa-learn --ham --no-sync mailmessage
382
383           This is handy for binding to a key in your mail user agent.  It's
384           very fast, as all the time-consuming stuff is deferred until you
385           run with the "--sync" option.
386
387       Autolearning is enabled by default
388           If you don't have a corpus of mail saved to learn, you can let
389           SpamAssassin automatically learn the mail that you receive.  If you
390           are autolearning from scratch, the amount of mail you receive will
391           determine how long until the BAYES_* rules are activated.
392

EFFECTIVE TRAINING

394       Learning filters require training to be effective.  If you don't train
395       them, they won't work.  In addition, you need to train them with new
396       messages regularly to keep them up-to-date, or their data will become
397       stale and impact accuracy.
398
399       You need to train with both spam and ham mails.  One type of mail alone
400       will not have any effect.
401
402       Note that if your mail folders contain things like forwarded spam,
403       discussions of spam-catching rules, etc., this will cause trouble.  You
404       should avoid scanning those messages if possible.  (An easy way to do
405       this is to move them aside, into a folder which is not scanned.)
406
407       If the messages you are learning from have already been filtered
408       through SpamAssassin, the learner will compensate for this.  In effect,
409       it learns what each message would look like if you had run
410       "spamassassin -d" over it in advance.
411
412       Another thing to be aware of, is that typically you should aim to train
413       with at least 1000 messages of spam, and 1000 ham messages, if
414       possible.  More is better, but anything over about 5000 messages does
415       not improve accuracy significantly in our tests.
416
417       Be careful that you train from the same source -- for example, if you
418       train on old spam, but new ham mail, then the classifier will think
419       that a mail with an old date stamp is likely to be spam.
420
421       It's also worth noting that training with a very small quantity of ham,
422       will produce atrocious results.  You should aim to train with at least
423       the same amount (or more if possible!) of ham data than spam.
424
425       On an on-going basis, it is best to keep training the filter to make
426       sure it has fresh data to work from.  There are various ways to do
427       this:
428
429       1. Supervised learning
430           This means keeping a copy of all or most of your mail, separated
431           into spam and ham piles, and periodically re-training using those.
432           It produces the best results, but requires more work from you, the
433           user.
434
435           (An easy way to do this, by the way, is to create a new folder for
436           'deleted' messages, and instead of deleting them from other
437           folders, simply move them in there instead.  Then keep all spam in
438           a separate folder and never delete it.  As long as you remember to
439           move misclassified mails into the correct folder set, it is easy
440           enough to keep up to date.)
441
442       2. Unsupervised learning from Bayesian classification
443           Another way to train is to chain the results of the Bayesian
444           classifier back into the training, so it reinforces its own
445           decisions.  This is only safe if you then retrain it based on any
446           errors you discover.
447
448           SpamAssassin does not support this method, due to experimental
449           results which strongly indicate that it does not work well, and
450           since Bayes is only one part of the resulting score presented to
451           the user (while Bayes may have made the wrong decision about a
452           mail, it may have been overridden by another system).
453
454       3. Unsupervised learning from SpamAssassin rules
455           Also called 'auto-learning' in SpamAssassin.  Based on statistical
456           analysis of the SpamAssassin success rates, we can automatically
457           train the Bayesian database with a certain degree of confidence
458           that our training data is accurate.
459
460           It should be supplemented with some supervised training in
461           addition, if possible.
462
463           This is the default, but can be turned off by setting the
464           SpamAssassin configuration parameter "bayes_auto_learn" to 0.
465
466       4. Mistake-based training
467           This means training on a small number of mails, then only training
468           on messages that SpamAssassin classifies incorrectly.  This works,
469           but it takes longer to get it right than a full training session
470           would.
471

FILES

473       sa-learn and the other parts of SpamAssassin's Bayesian learner, use a
474       set of persistent database files to store the learnt tokens, as
475       follows.
476
477       bayes_toks
478           The database of tokens, containing the tokens learnt, their count
479           of occurrences in ham and spam, and the timestamp when the token
480           was last seen in a message.
481
482           This database also contains some 'magic' tokens, as follows: the
483           version number of the database, the number of ham and spam messages
484           learnt, the number of tokens in the database, and timestamps of:
485           the last journal sync, the last expiry run, the last expiry token
486           reduction count, the last expiry timestamp delta, the oldest token
487           timestamp in the database, and the newest token timestamp in the
488           database.
489
490           This is a database file, using "DB_File".  The database 'version
491           number' is 0 for databases from 2.5x, 1 for databases from certain
492           2.6x development releases, 2 for 2.6x, and 3 for 3.0 and later
493           releases.
494
495       bayes_seen
496           A map of Message-Id and some data from headers and body to what
497           that message was learnt as. This is used so that SpamAssassin can
498           avoid re-learning a message it has already seen, and so it can
499           reverse the training if you later decide that message was learnt
500           incorrectly.
501
502           This is a database file, using "DB_File".
503
504       bayes_journal
505           While SpamAssassin is scanning mails, it needs to track which
506           tokens it uses in its calculations.  To avoid the contention of
507           having each SpamAssassin process attempting to gain write access to
508           the Bayes DB, the token timestamps are written to a 'journal' file
509           which will later (either automatically or via "sa-learn --sync") be
510           used to synchronize the Bayes DB.
511
512           Also, through the use of "bayes_learn_to_journal", or when using
513           the "--no-sync" option with sa-learn, the actual learning data will
514           take be placed into the journal for later synchronization.  This is
515           typically useful for high-traffic sites to avoid the same
516           contention as stated above.
517

EXPIRATION

519       Since SpamAssassin can auto-learn messages, the Bayes database files
520       could increase perpetually until they fill your disk.  To control this,
521       SpamAssassin performs journal synchronization and bayes expiration
522       periodically when certain criteria (listed below) are met.
523
524       SpamAssassin can sync the journal and expire the DB tokens either
525       manually or opportunistically.  A journal sync is due if --sync is
526       passed to sa-learn (manual), or if the following is true
527       (opportunistic):
528
529       - bayes_journal_max_size does not equal 0 (means don't sync)
530       - the journal file exists
531
532       and either:
533
534       - the journal file has a size greater than bayes_journal_max_size
535
536       or
537
538       - a journal sync has previously occurred, and at least 1 day has passed
539       since that sync
540
541       Expiry is due if --force-expire is passed to sa-learn (manual), or if
542       all of the following are true (opportunistic):
543
544       - the last expire was attempted at least 12hrs ago
545       - bayes_auto_expire does not equal 0
546       - the number of tokens in the DB is > 100,000
547       - the number of tokens in the DB is > bayes_expiry_max_db_size
548       - there is at least a 12 hr difference between the oldest and newest
549       token atimes
550
551   EXPIRE LOGIC
552       If either the manual or opportunistic method causes an expire run to
553       start, here is the logic that is used:
554
555       - figure out how many tokens to keep.  take the larger of either
556       bayes_expiry_max_db_size * 75% or 100,000 tokens.  therefore, the goal
557       reduction is number of tokens - number of tokens to keep.
558       - if the reduction number is < 1000 tokens, abort (not worth the
559       effort).
560       - if an expire has been done before, guesstimate the new atime delta
561       based on the old atime delta.  (new_atime_delta = old_atime_delta *
562       old_reduction_count / goal)
563       - if no expire has been done before, or the last expire looks "weird",
564       do an estimation pass.  The definition of "weird" is:
565           - last expire over 30 days ago
566           - last atime delta was < 12 hrs
567           - last reduction count was < 1000 tokens
568           - estimated new atime delta is < 12 hrs
569           - the difference between the last reduction count and the goal
570           reduction count is > 50%
571
572   ESTIMATION PASS LOGIC
573       Go through each of the DB's tokens.  Starting at 12hrs, calculate
574       whether or not the token would be expired (based on the difference
575       between the token's atime and the db's newest token atime) and keep the
576       count.  Work out from 12hrs exponentially by powers of 2.  ie: 12hrs *
577       1, 12hrs * 2, 12hrs * 4, 12hrs * 8, and so on, up to 12hrs * 512
578       (6144hrs, or 256 days).
579
580       The larger the delta, the smaller the number of tokens that will be
581       expired.  Conversely, the number of tokens goes up as the delta gets
582       smaller.  So starting at the largest atime delta, figure out which
583       delta will expire the most tokens without going above the goal
584       expiration count.  Use this to choose the atime delta to use, unless
585       one of the following occurs:
586
587       - the largest atime (smallest reduction count) would expire too many
588       tokens.  this means the learned tokens are mostly old and there needs
589       to be new tokens learned before an expire can occur.
590       - all of the atime choices result in 0 tokens being removed. this means
591       the tokens are all newer than 12 hours and there needs to be new tokens
592       learned before an expire can occur.
593       - the number of tokens that would be removed is < 1000.  the benefit
594       isn't worth the effort.  more tokens need to be learned.
595
596       If the expire run gets past this point, it will continue to the end.  A
597       new DB is created since the majority of DB libraries don't shrink the
598       DB file when tokens are removed.  So we do the "create new, migrate old
599       to new, remove old, rename new" shuffle.
600
601   EXPIRY RELATED CONFIGURATION SETTINGS
602       "bayes_auto_expire" is used to specify whether or not SpamAssassin
603       ought to opportunistically attempt to expire the Bayes database. The
604       default is 1 (yes).
605       "bayes_expiry_max_db_size" specifies both the auto-expire token count
606       point, as well as the resulting number of tokens after expiry as
607       described above.  The default value is 150,000, which is roughly
608       equivalent to a 6Mb database file if you're using DB_File.
609       "bayes_journal_max_size" specifies how large the Bayes journal will
610       grow before it is opportunistically synced.  The default value is
611       102400.
612

INSTALLATION

614       The sa-learn command is part of the Mail::SpamAssassin Perl module.
615       Install this as a normal Perl module, using "perl -MCPAN -e shell", or
616       by hand.
617

SEE ALSO

619       spamassassin(1) spamc(1) Mail::SpamAssassin(3)
620       Mail::SpamAssassin::ArchiveIterator(3)
621
622       <http://www.paulgraham.com/> Paul Graham's "A Plan For Spam" paper
623
624       <http://www.linuxjournal.com/article/6467> Gary Robinson's f(x) and
625       combining algorithms, as used in SpamAssassin
626
627       <http://www.bgl.nu/~glouis/bogofilter/> 'Training on error' page.  A
628       discussion of various Bayes training regimes, including 'train on
629       error' and unsupervised training.
630

PREREQUISITES

632       "Mail::SpamAssassin"
633

AUTHORS

635       The SpamAssassin(tm) Project <http://spamassassin.apache.org/>
636
637
638
639perl v5.12.4                      2011-09-13                       SA-LEARN(1)
Impressum