1SA-LEARN(1)           User Contributed Perl Documentation          SA-LEARN(1)
2
3
4

NAME

6       sa-learn - train SpamAssassin's Bayesian classifier
7

SYNOPSIS

9       sa-learn [options] [file]...
10
11       sa-learn [options] --dump [ all | data | magic ]
12
13       Options:
14
15        --ham                 Learn messages as ham (non-spam)
16        --spam                Learn messages as spam
17        --forget              Forget a message
18        --use-ignores         Use bayes_ignore_from and bayes_ignore_to
19        --sync                Synchronize the database and the journal if needed
20        --force-expire        Force a database sync and expiry run
21        --dbpath <path>       Allows commandline override (in bayes_path form)
22                              for where to read the Bayes DB from
23        --dump [all|data|magic]  Display the contents of the Bayes database
24                              Takes optional argument for what to display
25         --regexp <re>        For dump only, specifies which tokens to
26                              dump based on a regular expression.
27        -f file, --folders=file  Read list of files/directories from file
28        --dir                 Ignored; historical compatibility
29        --file                Ignored; historical compatibility
30        --mbox                Input sources are in mbox format
31        --mbx                 Input sources are in mbx format
32        --max-size <b>        Skip messages larger than b bytes;
33                              defaults to 256 KB, 0 implies no limit
34        --showdots            Show progress using dots
35        --progress            Show progress using progress bar
36        --no-sync             Skip synchronizing the database and journal
37                              after learning
38        -L, --local           Operate locally, no network accesses
39        --import              Migrate data from older version/non DB_File
40                              based databases
41        --clear               Wipe out existing database
42        --backup              Backup, to STDOUT, existing database
43        --restore <filename>  Restore a database from filename
44        -u username, --username=username
45                              Override username taken from the runtime
46                              environment, used with SQL
47        -C path, --configpath=path, --config-file=path
48                              Path to standard configuration dir
49        -p prefs, --prefspath=file, --prefs-file=file
50                              Set user preferences file
51        --siteconfigpath=path Path for site configs
52                              (default:  /usr/etc/mail/spamassassin)
53        --cf='config line'    Additional line of configuration
54        -D, --debug [area=n,...]  Print debugging messages
55        -V, --version         Print version
56        -h, --help            Print usage message
57

DESCRIPTION

59       Given a typical selection of your incoming mail classified as spam or
60       ham (non-spam), this tool will feed each mail to SpamAssassin, allowing
61       it to 'learn' what signs are likely to mean spam, and which are likely
62       to mean ham.
63
64       Simply run this command once for each of your mail folders, and it will
65       ''learn'' from the mail therein.
66
67       Note that csh-style globbing in the mail folder names is supported; in
68       other words, listing a folder name as "*" will scan every folder that
69       matches.  See "Mail::SpamAssassin::ArchiveIterator" for more details.
70
71       If you are using mail boxes in format other than maildir you should use
72       the --mbox or --mbx parameters.
73
74       SpamAssassin remembers which mail messages it has learnt already, and
75       will not re-learn those messages again, unless you use the --forget
76       option. Messages learnt as spam will have SpamAssassin markup removed,
77       on the fly.
78
79       If you make a mistake and scan a mail as ham when it is spam, or vice
80       versa, simply rerun this command with the correct classification, and
81       the mistake will be corrected.  SpamAssassin will automatically
82       'forget' the previous indications.
83
84       Users of "spamd" who wish to perform training remotely, over a network,
85       should investigate the "spamc -L" switch.
86

OPTIONS

88       --ham
89           Learn the input message(s) as ham.   If you have previously learnt
90           any of the messages as spam, SpamAssassin will forget them first,
91           then re-learn them as ham.  Alternatively, if you have previously
92           learnt them as ham, it'll skip them this time around.  If the
93           messages have already been filtered through SpamAssassin, the
94           learner will ignore any modifications SpamAssassin may have made.
95
96       --spam
97           Learn the input message(s) as spam.   If you have previously learnt
98           any of the messages as ham, SpamAssassin will forget them first,
99           then re-learn them as spam.  Alternatively, if you have previously
100           learnt them as spam, it'll skip them this time around.  If the
101           messages have already been filtered through SpamAssassin, the
102           learner will ignore any modifications SpamAssassin may have made.
103
104       --folders=filename, -f filename
105           sa-learn will read in the list of folders from the specified file,
106           one folder per line in the file.  If the folder is prefixed with
107           "ham:type:" or "spam:type:", sa-learn will learn that folder
108           appropriately, otherwise the folders will be assumed to be of the
109           type specified by --ham or --spam.
110
111           "type" above is optional, but is the same as the standard for
112           ArchiveIterator: mbox, mbx, dir, file, or detect (the default if
113           not specified).
114
115       --mbox
116           sa-learn will read in the file(s) containing the emails to be
117           learned, and will process them in mbox format (one or more emails
118           per file).
119
120       --mbx
121           sa-learn will read in the file(s) containing the emails to be
122           learned, and will process them in mbx format (one or more emails
123           per file).
124
125       --use-ignores
126           Don't learn the message if a from address matches configuration
127           file item "bayes_ignore_from" or a to address matches
128           "bayes_ignore_to".  The option might be used when learning from a
129           large file of messages from which the hammy spam messages or spammy
130           ham messages have not been removed.
131
132       --sync
133           Synchronize the journal and databases.  Upon successfully syncing
134           the database with the entries in the journal, the journal file is
135           removed.
136
137       --force-expire
138           Forces an expiry attempt, regardless of whether it may be necessary
139           or not.  Note: This doesn't mean any tokens will actually expire.
140           Please see the EXPIRATION section below.
141
142           Note: "--force-expire" also causes the journal data to be
143           synchronized into the Bayes databases.
144
145       --forget
146           Forget a given message previously learnt.
147
148       --dbpath
149           Allows a commandline override of the bayes_path configuration
150           option.
151
152       --dump option
153           Display the contents of the Bayes database.  Without an option or
154           with the all option, all magic tokens and data tokens will be
155           displayed.  magic will only display magic tokens, and data will
156           only display the data tokens.
157
158           Can also use the --regexp RE option to specify which tokens to
159           display based on a regular expression.
160
161       --clear
162           Clear an existing Bayes database by removing all traces of the
163           database.
164
165           WARNING: This is destructive and should be used with care.
166
167       --backup
168           Performs a dump of the Bayes database in machine/human readable
169           format.
170
171           The dump will include token and seen data.  It is suitable for
172           input back into the --restore command.
173
174       --restore=filename
175           Performs a restore of the Bayes database defined by filename.
176
177           WARNING: This is a destructive operation, previous Bayes data will
178           be wiped out.
179
180       -h, --help
181           Print help message and exit.
182
183       -u username, --username=username
184           If specified this username will override the username taken from
185           the runtime environment.  You can use this option to specify users
186           in a virtual user configuration when using SQL as the Bayes
187           backend.
188
189           NOTE: This option will not change to the given username, it will
190           only attempt to act on behalf of that user.  Because of this you
191           will need to have proper permissions to be able to change files
192           owned by username.  In the case of SQL this generally is not a
193           problem.
194
195       -C path, --configpath=path, --config-file=path
196           Use the specified path for locating the distributed configuration
197           files.  Ignore the default directories (usually
198           "/usr/share/spamassassin" or similar).
199
200       --siteconfigpath=path
201           Use the specified path for locating site-specific configuration
202           files.  Ignore the default directories (usually
203           "/etc/mail/spamassassin" or similar).
204
205       --cf='config line'
206           Add additional lines of configuration directly from the command-
207           line, parsed after the configuration files are read.   Multiple
208           --cf arguments can be used, and each will be considered a separate
209           line of configuration.
210
211       -p prefs, --prefspath=prefs, --prefs-file=prefs
212           Read user score preferences from prefs (usually
213           "$HOME/.spamassassin/user_prefs").
214
215       --progress
216           Prints a progress bar (to STDERR) showing the current progress.  In
217           the case where no valid terminal is found this option will behave
218           very much like the --showdots option.
219
220       -D [area,...], --debug [area,...]
221           Produce debugging output. If no areas are listed, all debugging
222           information is printed. Diagnostic output can also be enabled for
223           each area individually; area is the area of the code to instrument.
224           For example, to produce diagnostic output on bayes, learn, and dns,
225           use:
226
227                   spamassassin -D bayes,learn,dns
228
229           For more information about which areas (also known as channels) are
230           available, please see the documentation at:
231
232                   C<http://wiki.apache.org/spamassassin/DebugChannels>
233
234           Higher priority informational messages that are suitable for
235           logging in normal circumstances are available with an area of
236           "info".
237
238       --no-sync
239           Skip the slow synchronization step which normally takes place after
240           changing database entries.  If you plan to learn from many folders
241           in a batch, or to learn many individual messages one-by-one, it is
242           faster to use this switch and run "sa-learn --sync" once all the
243           folders have been scanned.
244
245           Clarification: The state of --no-sync overrides the
246           bayes_learn_to_journal configuration option.  If not specified, sa-
247           learn will learn to the database directly.  If specified, sa-learn
248           will learn to the journal file.
249
250           Note: --sync and --no-sync can be specified on the same
251           commandline, which is slightly confusing.  In this case, the
252           --no-sync option is ignored since there is no learn operation.
253
254       -L, --local
255           Do not perform any network accesses while learning details about
256           the mail messages.  This will speed up the learning process, but
257           may result in a slightly lower accuracy.
258
259           Note that this is currently ignored, as current versions of
260           SpamAssassin will not perform network access while learning; but
261           future versions may.
262
263       --import
264           If you previously used SpamAssassin's Bayesian learner without the
265           "DB_File" module installed, it will have created files in other
266           formats, such as "GDBM_File", "NDBM_File", or "SDBM_File".  This
267           switch allows you to migrate that old data into the "DB_File"
268           format.  It will overwrite any data currently in the "DB_File".
269
270           Can also be used with the --dbpath path option to specify the
271           location of the Bayes files to use.
272

MIGRATION

274       There are now multiple backend storage modules available for storing
275       user's bayesian data. As such you might want to migrate from one
276       backend to another. Here is a simple procedure for migrating from one
277       backend to another.
278
279       Note that if you have individual user databases you will have to
280       perform a similar procedure for each one of them.
281
282       sa-learn --sync
283           This will sync any outstanding journal entries
284
285       sa-learn --backup > backup.txt
286           This will save all your Bayes data to a plain text file.
287
288       sa-learn --clear
289           This is optional, but good to do to clear out the old database.
290
291       Repeat!
292           At this point, if you have multiple databases, you should perform
293           the procedure above for each of them. (i.e. each user's database
294           needs to be backed up before continuing.)
295
296       Switch backends
297           Once you have backed up all databases you can update your
298           configuration for the new database backend. This will involve at
299           least the bayes_store_module config option and may involve some
300           additional config options depending on what is required by the
301           module. (For example, you may need to configure an SQL database.)
302
303       sa-learn --restore backup.txt
304           Again, you need to do this for every database.
305
306       If you are migrating to SQL you can make use of the -u <username>
307       option in sa-learn to populate each user's database. Otherwise, you
308       must run sa-learn as the user who database you are restoring.
309

INTRODUCTION TO BAYESIAN FILTERING

311       (Thanks to Michael Bell for this section!)
312
313       For a more lengthy description of how this works, go to
314       http://www.paulgraham.com/ and see "A Plan for Spam". It's reasonably
315       readable, even if statistics make me break out in hives.
316
317       The short semi-inaccurate version: Given training, a spam heuristics
318       engine can take the most "spammy" and "hammy" words and apply
319       probabilistic analysis. Furthermore, once given a basis for the
320       analysis, the engine can continue to learn iteratively by applying both
321       the non-Bayesian and Bayesian rulesets together to create evolving
322       "intelligence".
323
324       SpamAssassin 2.50 and later supports Bayesian spam analysis, in the
325       form of the BAYES rules. This is a new feature, quite powerful, and is
326       disabled until enough messages have been learnt.
327
328       The pros of Bayesian spam analysis:
329
330       Can greatly reduce false positives and false negatives.
331           It learns from your mail, so it is tailored to your unique e-mail
332           flow.
333
334       Once it starts learning, it can continue to learn from SpamAssassin and
335       improve over time.
336
337       And the cons:
338
339       A decent number of messages are required before results are useful for
340       ham/spam determination.
341       It's hard to explain why a message is or isn't marked as spam.
342           i.e.: a straightforward rule, that matches, say, "VIAGRA" is easy
343           to understand. If it generates a false positive or false negative,
344           it is fairly easy to understand why.
345
346           With Bayesian analysis, it's all probabilities - "because the past
347           says it is likely as this falls into a probabilistic distribution
348           common to past spam in your systems". Tell that to your users!
349           Tell that to the client when he asks "what can I do to change
350           this". (By the way, the answer in this case is "use whitelisting".)
351
352       It will take disk space and memory.
353           The databases it maintains take quite a lot of resources to store
354           and use.
355

GETTING STARTED

357       Still interested? Ok, here's the guidelines for getting this working.
358
359       First a high-level overview:
360
361       Build a significant sample of both ham and spam.
362           I suggest several thousand of each, placed in SPAM and HAM
363           directories or mailboxes.  Yes, you MUST hand-sort this - otherwise
364           the results won't be much better than SpamAssassin on its own.
365           Verify the spamminess/haminess of EVERY message.  You're urged to
366           avoid using a publicly available corpus (sample) - this must be
367           taken from YOUR mail server, if it is to be statistically useful.
368           Otherwise, the results may be pretty skewed.
369
370       Use this tool to teach SpamAssassin about these samples, like so:
371                   sa-learn --spam /path/to/spam/folder
372                   sa-learn --ham /path/to/ham/folder
373                   ...
374
375           Let SpamAssassin proceed, learning stuff. When it finds ham and
376           spam it will add the "interesting tokens" to the database.
377
378       If you need SpamAssassin to forget about specific messages, use the
379       --forget option.
380           This can be applied to either ham or spam that has run through the
381           sa-learn processes. It's a bit of a hammer, really, lowering the
382           weighting of the specific tokens in that message (only if that
383           message has been processed before).
384
385       Learning from single messages uses a command like this:
386                   sa-learn --ham --no-sync mailmessage
387
388           This is handy for binding to a key in your mail user agent.  It's
389           very fast, as all the time-consuming stuff is deferred until you
390           run with the "--sync" option.
391
392       Autolearning is enabled by default
393           If you don't have a corpus of mail saved to learn, you can let
394           SpamAssassin automatically learn the mail that you receive.  If you
395           are autolearning from scratch, the amount of mail you receive will
396           determine how long until the BAYES_* rules are activated.
397

EFFECTIVE TRAINING

399       Learning filters require training to be effective.  If you don't train
400       them, they won't work.  In addition, you need to train them with new
401       messages regularly to keep them up-to-date, or their data will become
402       stale and impact accuracy.
403
404       You need to train with both spam and ham mails.  One type of mail alone
405       will not have any effect.
406
407       Note that if your mail folders contain things like forwarded spam,
408       discussions of spam-catching rules, etc., this will cause trouble.  You
409       should avoid scanning those messages if possible.  (An easy way to do
410       this is to move them aside, into a folder which is not scanned.)
411
412       If the messages you are learning from have already been filtered
413       through SpamAssassin, the learner will compensate for this.  In effect,
414       it learns what each message would look like if you had run
415       "spamassassin -d" over it in advance.
416
417       Another thing to be aware of, is that typically you should aim to train
418       with at least 1000 messages of spam, and 1000 ham messages, if
419       possible.  More is better, but anything over about 5000 messages does
420       not improve accuracy significantly in our tests.
421
422       Be careful that you train from the same source -- for example, if you
423       train on old spam, but new ham mail, then the classifier will think
424       that a mail with an old date stamp is likely to be spam.
425
426       It's also worth noting that training with a very small quantity of ham,
427       will produce atrocious results.  You should aim to train with at least
428       the same amount (or more if possible!) of ham data than spam.
429
430       On an on-going basis, it is best to keep training the filter to make
431       sure it has fresh data to work from.  There are various ways to do
432       this:
433
434       1. Supervised learning
435           This means keeping a copy of all or most of your mail, separated
436           into spam and ham piles, and periodically re-training using those.
437           It produces the best results, but requires more work from you, the
438           user.
439
440           (An easy way to do this, by the way, is to create a new folder for
441           'deleted' messages, and instead of deleting them from other
442           folders, simply move them in there instead.  Then keep all spam in
443           a separate folder and never delete it.  As long as you remember to
444           move misclassified mails into the correct folder set, it is easy
445           enough to keep up to date.)
446
447       2. Unsupervised learning from Bayesian classification
448           Another way to train is to chain the results of the Bayesian
449           classifier back into the training, so it reinforces its own
450           decisions.  This is only safe if you then retrain it based on any
451           errors you discover.
452
453           SpamAssassin does not support this method, due to experimental
454           results which strongly indicate that it does not work well, and
455           since Bayes is only one part of the resulting score presented to
456           the user (while Bayes may have made the wrong decision about a
457           mail, it may have been overridden by another system).
458
459       3. Unsupervised learning from SpamAssassin rules
460           Also called 'auto-learning' in SpamAssassin.  Based on statistical
461           analysis of the SpamAssassin success rates, we can automatically
462           train the Bayesian database with a certain degree of confidence
463           that our training data is accurate.
464
465           It should be supplemented with some supervised training in
466           addition, if possible.
467
468           This is the default, but can be turned off by setting the
469           SpamAssassin configuration parameter "bayes_auto_learn" to 0.
470
471       4. Mistake-based training
472           This means training on a small number of mails, then only training
473           on messages that SpamAssassin classifies incorrectly.  This works,
474           but it takes longer to get it right than a full training session
475           would.
476

FILES

478       sa-learn and the other parts of SpamAssassin's Bayesian learner, use a
479       set of persistent database files to store the learnt tokens, as
480       follows.
481
482       bayes_toks
483           The database of tokens, containing the tokens learnt, their count
484           of occurrences in ham and spam, and the timestamp when the token
485           was last seen in a message.
486
487           This database also contains some 'magic' tokens, as follows: the
488           version number of the database, the number of ham and spam messages
489           learnt, the number of tokens in the database, and timestamps of:
490           the last journal sync, the last expiry run, the last expiry token
491           reduction count, the last expiry timestamp delta, the oldest token
492           timestamp in the database, and the newest token timestamp in the
493           database.
494
495           This is a database file, using "DB_File".  The database 'version
496           number' is 0 for databases from 2.5x, 1 for databases from certain
497           2.6x development releases, 2 for 2.6x, and 3 for 3.0 and later
498           releases.
499
500       bayes_seen
501           A map of Message-Id and some data from headers and body to what
502           that message was learnt as. This is used so that SpamAssassin can
503           avoid re-learning a message it has already seen, and so it can
504           reverse the training if you later decide that message was learnt
505           incorrectly.
506
507           This is a database file, using "DB_File".
508
509       bayes_journal
510           While SpamAssassin is scanning mails, it needs to track which
511           tokens it uses in its calculations.  To avoid the contention of
512           having each SpamAssassin process attempting to gain write access to
513           the Bayes DB, the token timestamps are written to a 'journal' file
514           which will later (either automatically or via "sa-learn --sync") be
515           used to synchronize the Bayes DB.
516
517           Also, through the use of "bayes_learn_to_journal", or when using
518           the "--no-sync" option with sa-learn, the actual learning data will
519           take be placed into the journal for later synchronization.  This is
520           typically useful for high-traffic sites to avoid the same
521           contention as stated above.
522

EXPIRATION

524       Since SpamAssassin can auto-learn messages, the Bayes database files
525       could increase perpetually until they fill your disk.  To control this,
526       SpamAssassin performs journal synchronization and bayes expiration
527       periodically when certain criteria (listed below) are met.
528
529       SpamAssassin can sync the journal and expire the DB tokens either
530       manually or opportunistically.  A journal sync is due if --sync is
531       passed to sa-learn (manual), or if the following is true
532       (opportunistic):
533
534       - bayes_journal_max_size does not equal 0 (means don't sync)
535       - the journal file exists
536
537       and either:
538
539       - the journal file has a size greater than bayes_journal_max_size
540
541       or
542
543       - a journal sync has previously occurred, and at least 1 day has passed
544       since that sync
545
546       Expiry is due if --force-expire is passed to sa-learn (manual), or if
547       all of the following are true (opportunistic):
548
549       - the last expire was attempted at least 12hrs ago
550       - bayes_auto_expire does not equal 0
551       - the number of tokens in the DB is > 100,000
552       - the number of tokens in the DB is > bayes_expiry_max_db_size
553       - there is at least a 12 hr difference between the oldest and newest
554       token atimes
555
556   EXPIRE LOGIC
557       If either the manual or opportunistic method causes an expire run to
558       start, here is the logic that is used:
559
560       - figure out how many tokens to keep.  take the larger of either
561       bayes_expiry_max_db_size * 75% or 100,000 tokens.  therefore, the goal
562       reduction is number of tokens - number of tokens to keep.
563       - if the reduction number is < 1000 tokens, abort (not worth the
564       effort).
565       - if an expire has been done before, guesstimate the new atime delta
566       based on the old atime delta.  (new_atime_delta = old_atime_delta *
567       old_reduction_count / goal)
568       - if no expire has been done before, or the last expire looks "weird",
569       do an estimation pass.  The definition of "weird" is:
570           - last expire over 30 days ago
571           - last atime delta was < 12 hrs
572           - last reduction count was < 1000 tokens
573           - estimated new atime delta is < 12 hrs
574           - the difference between the last reduction count and the goal
575           reduction count is > 50%
576
577   ESTIMATION PASS LOGIC
578       Go through each of the DB's tokens.  Starting at 12hrs, calculate
579       whether or not the token would be expired (based on the difference
580       between the token's atime and the db's newest token atime) and keep the
581       count.  Work out from 12hrs exponentially by powers of 2.  ie: 12hrs *
582       1, 12hrs * 2, 12hrs * 4, 12hrs * 8, and so on, up to 12hrs * 512
583       (6144hrs, or 256 days).
584
585       The larger the delta, the smaller the number of tokens that will be
586       expired.  Conversely, the number of tokens goes up as the delta gets
587       smaller.  So starting at the largest atime delta, figure out which
588       delta will expire the most tokens without going above the goal
589       expiration count.  Use this to choose the atime delta to use, unless
590       one of the following occurs:
591
592       - the largest atime (smallest reduction count) would expire too many
593       tokens.  this means the learned tokens are mostly old and there needs
594       to be new tokens learned before an expire can occur.
595       - all of the atime choices result in 0 tokens being removed. this means
596       the tokens are all newer than 12 hours and there needs to be new tokens
597       learned before an expire can occur.
598       - the number of tokens that would be removed is < 1000.  the benefit
599       isn't worth the effort.  more tokens need to be learned.
600
601       If the expire run gets past this point, it will continue to the end.  A
602       new DB is created since the majority of DB libraries don't shrink the
603       DB file when tokens are removed.  So we do the "create new, migrate old
604       to new, remove old, rename new" shuffle.
605
606   EXPIRY RELATED CONFIGURATION SETTINGS
607       "bayes_auto_expire" is used to specify whether or not SpamAssassin
608       ought to opportunistically attempt to expire the Bayes database. The
609       default is 1 (yes).
610       "bayes_expiry_max_db_size" specifies both the auto-expire token count
611       point, as well as the resulting number of tokens after expiry as
612       described above.  The default value is 150,000, which is roughly
613       equivalent to a 6Mb database file if you're using DB_File.
614       "bayes_journal_max_size" specifies how large the Bayes journal will
615       grow before it is opportunistically synced.  The default value is
616       102400.
617

INSTALLATION

619       The sa-learn command is part of the Mail::SpamAssassin Perl module.
620       Install this as a normal Perl module, using "perl -MCPAN -e shell", or
621       by hand.
622

SEE ALSO

624       spamassassin(1) spamc(1) Mail::SpamAssassin(3)
625       Mail::SpamAssassin::ArchiveIterator(3)
626
627       <http://www.paulgraham.com/> Paul Graham's "A Plan For Spam" paper
628
629       <http://www.linuxjournal.com/article/6467> Gary Robinson's f(x) and
630       combining algorithms, as used in SpamAssassin
631
632       <http://www.bgl.nu/~glouis/bogofilter/> 'Training on error' page.  A
633       discussion of various Bayes training regimes, including 'train on
634       error' and unsupervised training.
635

PREREQUISITES

637       "Mail::SpamAssassin"
638

AUTHORS

640       The SpamAssassin(tm) Project <https://spamassassin.apache.org/>
641
642
643
644perl v5.32.1                      2021-04-14                       SA-LEARN(1)
Impressum