1SA-LEARN(1) User Contributed Perl Documentation SA-LEARN(1)
2
6 sa-learn - train SpamAssassin's Bayesian classifier
7
9 sa-learn [options] [file]...
10 sa-learn [options] --dump [ all | data | magic ]
12 Options:
14 --ham Learn messages as ham (non-spam)
16 --spam Learn messages as spam
17 --forget Forget a message
18 --use-ignores Use bayes_ignore_from and bayes_ignore_to
19 --sync Synchronize the database and the journal if needed
20 --force-expire Force a database sync and expiry run
21 --dbpath <path> Allows commandline override (in bayes_path form)
22 for where to read the Bayes DB from
23 --dump [all|data|magic] Display the contents of the Bayes database
24 Takes optional argument for what to display
25 --regexp <re> For dump only, specifies which tokens to
26 dump based on a regular expression.
27 -f file, --folders=file Read list of files/directories from file
28 --dir Ignored; historical compatibility
29 --file Ignored; historical compatibility
30 --mbox Input sources are in mbox format
31 --mbx Input sources are in mbx format
32 --max-size <b> Skip messages larger than b bytes;
33 defaults to 256 KB, 0 implies no limit
34 --showdots Show progress using dots
35 --progress Show progress using progress bar
36 --no-sync Skip synchronizing the database and journal
37 after learning
38 -L, --local Operate locally, no network accesses
39 --import Migrate data from older version/non DB_File
40 based databases
41 --clear Wipe out existing database
42 --backup Backup, to STDOUT, existing database
43 --restore <filename> Restore a database from filename
44 -u username, --username=username
45 Override username taken from the runtime
46 environment, used with SQL
47 -C path, --configpath=path, --config-file=path
48 Path to standard configuration dir
49 -p prefs, --prefspath=file, --prefs-file=file
50 Set user preferences file
51 --siteconfigpath=path Path for site configs
52 (default: /usr/etc/mail/spamassassin)
53 --cf='config line' Additional line of configuration
54 -D, --debug [area=n,...] Print debugging messages
55 -V, --version Print version
56 -h, --help Print usage message
57
59 Given a typical selection of your incoming mail classified as spam or
60 ham (non-spam), this tool will feed each mail to SpamAssassin, allowing
61 it to 'learn' what signs are likely to mean spam, and which are likely
62 to mean ham.
63 Simply run this command once for each of your mail folders, and it will
65 ''learn'' from the mail therein.
66 Note that csh-style globbing in the mail folder names is supported; in
68 other words, listing a folder name as "*" will scan every folder that
69 matches. See "Mail::SpamAssassin::ArchiveIterator" for more details.
70 If you are using mail boxes in format other than maildir you should use
72 the --mbox or --mbx parameters.
73 SpamAssassin remembers which mail messages it has learnt already, and
75 will not re-learn those messages again, unless you use the --forget
76 option. Messages learnt as spam will have SpamAssassin markup removed,
77 on the fly.
78 If you make a mistake and scan a mail as ham when it is spam, or vice
80 versa, simply rerun this command with the correct classification, and
81 the mistake will be corrected. SpamAssassin will automatically
82 'forget' the previous indications.
83 Users of "spamd" who wish to perform training remotely, over a network,
85 should investigate the "spamc -L" switch.
86
88 --ham
89 Learn the input message(s) as ham. If you have previously learnt
90 any of the messages as spam, SpamAssassin will forget them first,
91 then re-learn them as ham. Alternatively, if you have previously
92 learnt them as ham, it'll skip them this time around. If the
93 messages have already been filtered through SpamAssassin, the
94 learner will ignore any modifications SpamAssassin may have made.
95 --spam
97 Learn the input message(s) as spam. If you have previously learnt
98 any of the messages as ham, SpamAssassin will forget them first,
99 then re-learn them as spam. Alternatively, if you have previously
100 learnt them as spam, it'll skip them this time around. If the
101 messages have already been filtered through SpamAssassin, the
102 learner will ignore any modifications SpamAssassin may have made.
103 --folders=filename, -f filename
105 sa-learn will read in the list of folders from the specified file,
106 one folder per line in the file. If the folder is prefixed with
107 "ham:type:" or "spam:type:", sa-learn will learn that folder
108 appropriately, otherwise the folders will be assumed to be of the
109 type specified by --ham or --spam.
110 "type" above is optional, but is the same as the standard for
112 ArchiveIterator: mbox, mbx, dir, file, or detect (the default if
113 not specified).
114 --mbox
116 sa-learn will read in the file(s) containing the emails to be
117 learned, and will process them in mbox format (one or more emails
118 per file).
119 --mbx
121 sa-learn will read in the file(s) containing the emails to be
122 learned, and will process them in mbx format (one or more emails
123 per file).
124 --use-ignores
126 Don't learn the message if a from address matches configuration
127 file item "bayes_ignore_from" or a to address matches
128 "bayes_ignore_to". The option might be used when learning from a
129 large file of messages from which the hammy spam messages or spammy
130 ham messages have not been removed.
131 --sync
133 Synchronize the journal and databases. Upon successfully syncing
134 the database with the entries in the journal, the journal file is
135 removed.
136 --force-expire
138 Forces an expiry attempt, regardless of whether it may be necessary
139 or not. Note: This doesn't mean any tokens will actually expire.
140 Please see the EXPIRATION section below.
141 Note: "--force-expire" also causes the journal data to be
143 synchronized into the Bayes databases.
144 --forget
146 Forget a given message previously learnt.
147 --dbpath
149 Allows a commandline override of the bayes_path configuration
150 option.
151 --dump option
153 Display the contents of the Bayes database. Without an option or
154 with the all option, all magic tokens and data tokens will be
155 displayed. magic will only display magic tokens, and data will
156 only display the data tokens.
157 Can also use the --regexp RE option to specify which tokens to
159 display based on a regular expression.
160 --clear
162 Clear an existing Bayes database by removing all traces of the
163 database.
164 WARNING: This is destructive and should be used with care.
166 --backup
168 Performs a dump of the Bayes database in machine/human readable
169 format.
170 The dump will include token and seen data. It is suitable for
172 input back into the --restore command.
173 --restore=filename
175 Performs a restore of the Bayes database defined by filename.
176 WARNING: This is a destructive operation, previous Bayes data will
178 be wiped out.
179 -h, --help
181 Print help message and exit.
182 -u username, --username=username
184 If specified this username will override the username taken from
185 the runtime environment. You can use this option to specify users
186 in a virtual user configuration when using SQL as the Bayes
187 backend.
188 NOTE: This option will not change to the given username, it will
190 only attempt to act on behalf of that user. Because of this you
191 will need to have proper permissions to be able to change files
192 owned by username. In the case of SQL this generally is not a
193 problem.
194 -C path, --configpath=path, --config-file=path
196 Use the specified path for locating the distributed configuration
197 files. Ignore the default directories (usually
198 "/usr/share/spamassassin" or similar).
199 --siteconfigpath=path
201 Use the specified path for locating site-specific configuration
202 files. Ignore the default directories (usually
203 "/etc/mail/spamassassin" or similar).
204 --cf='config line'
206 Add additional lines of configuration directly from the command-
207 line, parsed after the configuration files are read. Multiple
208 --cf arguments can be used, and each will be considered a separate
209 line of configuration.
210 -p prefs, --prefspath=prefs, --prefs-file=prefs
212 Read user score preferences from prefs (usually
213 "$HOME/.spamassassin/user_prefs").
214 --progress
216 Prints a progress bar (to STDERR) showing the current progress. In
217 the case where no valid terminal is found this option will behave
218 very much like the --showdots option.
219 -D [area,...], --debug [area,...]
221 Produce debugging output. If no areas are listed, all debugging
222 information is printed. Diagnostic output can also be enabled for
223 each area individually; area is the area of the code to instrument.
224 For example, to produce diagnostic output on bayes, learn, and dns,
225 use:
226 spamassassin -D bayes,learn,dns
228 For more information about which areas (also known as channels) are
230 available, please see the documentation at:
231 C<http://wiki.apache.org/spamassassin/DebugChannels>
233 Higher priority informational messages that are suitable for
235 logging in normal circumstances are available with an area of
236 "info".
237 --no-sync
239 Skip the slow synchronization step which normally takes place after
240 changing database entries. If you plan to learn from many folders
241 in a batch, or to learn many individual messages one-by-one, it is
242 faster to use this switch and run "sa-learn --sync" once all the
243 folders have been scanned.
244 Clarification: The state of --no-sync overrides the
246 bayes_learn_to_journal configuration option. If not specified, sa-
247 learn will learn to the database directly. If specified, sa-learn
248 will learn to the journal file.
249 Note: --sync and --no-sync can be specified on the same
251 commandline, which is slightly confusing. In this case, the
252 --no-sync option is ignored since there is no learn operation.
253 -L, --local
255 Do not perform any network accesses while learning details about
256 the mail messages. This will speed up the learning process, but
257 may result in a slightly lower accuracy.
258 Note that this is currently ignored, as current versions of
260 SpamAssassin will not perform network access while learning; but
261 future versions may.
262 --import
264 If you previously used SpamAssassin's Bayesian learner without the
265 "DB_File" module installed, it will have created files in other
266 formats, such as "GDBM_File", "NDBM_File", or "SDBM_File". This
267 switch allows you to migrate that old data into the "DB_File"
268 format. It will overwrite any data currently in the "DB_File".
269 Can also be used with the --dbpath path option to specify the
271 location of the Bayes files to use.
272
274 There are now multiple backend storage modules available for storing
275 user's bayesian data. As such you might want to migrate from one
276 backend to another. Here is a simple procedure for migrating from one
277 backend to another.
278 Note that if you have individual user databases you will have to
280 perform a similar procedure for each one of them.
281 sa-learn --sync
283 This will sync any outstanding journal entries
284 sa-learn --backup > backup.txt
286 This will save all your Bayes data to a plain text file.
287 sa-learn --clear
289 This is optional, but good to do to clear out the old database.
290 Repeat!
292 At this point, if you have multiple databases, you should perform
293 the procedure above for each of them. (i.e. each user's database
294 needs to be backed up before continuing.)
295 Switch backends
297 Once you have backed up all databases you can update your
298 configuration for the new database backend. This will involve at
299 least the bayes_store_module config option and may involve some
300 additional config options depending on what is required by the
301 module. (For example, you may need to configure an SQL database.)
302 sa-learn --restore backup.txt
304 Again, you need to do this for every database.
305 If you are migrating to SQL you can make use of the -u <username>
307 option in sa-learn to populate each user's database. Otherwise, you
308 must run sa-learn as the user who database you are restoring.
309
311 (Thanks to Michael Bell for this section!)
312 For a more lengthy description of how this works, go to
314 http://www.paulgraham.com/ and see "A Plan for Spam". It's reasonably
315 readable, even if statistics make me break out in hives.
316 The short semi-inaccurate version: Given training, a spam heuristics
318 engine can take the most "spammy" and "hammy" words and apply
319 probabilistic analysis. Furthermore, once given a basis for the
320 analysis, the engine can continue to learn iteratively by applying both
321 the non-Bayesian and Bayesian rulesets together to create evolving
322 "intelligence".
323 SpamAssassin 2.50 and later supports Bayesian spam analysis, in the
325 form of the BAYES rules. This is a new feature, quite powerful, and is
326 disabled until enough messages have been learnt.
327 The pros of Bayesian spam analysis:
329 Can greatly reduce false positives and false negatives.
331 It learns from your mail, so it is tailored to your unique e-mail
332 flow.
333 Once it starts learning, it can continue to learn from SpamAssassin and
335 improve over time.
336 And the cons:
338 A decent number of messages are required before results are useful for
340 ham/spam determination.
341 It's hard to explain why a message is or isn't marked as spam.
342 i.e.: a straightforward rule, that matches, say, "VIAGRA" is easy
343 to understand. If it generates a false positive or false negative,
344 it is fairly easy to understand why.
345 With Bayesian analysis, it's all probabilities - "because the past
347 says it is likely as this falls into a probabilistic distribution
348 common to past spam in your systems". Tell that to your users!
349 Tell that to the client when he asks "what can I do to change
350 this". (By the way, the answer in this case is "use whitelisting".)
351 It will take disk space and memory.
353 The databases it maintains take quite a lot of resources to store
354 and use.
355
357 Still interested? Ok, here's the guidelines for getting this working.
358 First a high-level overview:
360 Build a significant sample of both ham and spam.
362 I suggest several thousand of each, placed in SPAM and HAM
363 directories or mailboxes. Yes, you MUST hand-sort this - otherwise
364 the results won't be much better than SpamAssassin on its own.
365 Verify the spamminess/haminess of EVERY message. You're urged to
366 avoid using a publicly available corpus (sample) - this must be
367 taken from YOUR mail server, if it is to be statistically useful.
368 Otherwise, the results may be pretty skewed.
369 Use this tool to teach SpamAssassin about these samples, like so:
371 sa-learn --spam /path/to/spam/folder
372 sa-learn --ham /path/to/ham/folder
373 ...
374 Let SpamAssassin proceed, learning stuff. When it finds ham and
376 spam it will add the "interesting tokens" to the database.
377 If you need SpamAssassin to forget about specific messages, use the
379 --forget option.
380 This can be applied to either ham or spam that has run through the
381 sa-learn processes. It's a bit of a hammer, really, lowering the
382 weighting of the specific tokens in that message (only if that
383 message has been processed before).
384 Learning from single messages uses a command like this:
386 sa-learn --ham --no-sync mailmessage
387 This is handy for binding to a key in your mail user agent. It's
389 very fast, as all the time-consuming stuff is deferred until you
390 run with the "--sync" option.
391 Autolearning is enabled by default
393 If you don't have a corpus of mail saved to learn, you can let
394 SpamAssassin automatically learn the mail that you receive. If you
395 are autolearning from scratch, the amount of mail you receive will
396 determine how long until the BAYES_* rules are activated.
397
399 Learning filters require training to be effective. If you don't train
400 them, they won't work. In addition, you need to train them with new
401 messages regularly to keep them up-to-date, or their data will become
402 stale and impact accuracy.
403 You need to train with both spam and ham mails. One type of mail alone
405 will not have any effect.
406 Note that if your mail folders contain things like forwarded spam,
408 discussions of spam-catching rules, etc., this will cause trouble. You
409 should avoid scanning those messages if possible. (An easy way to do
410 this is to move them aside, into a folder which is not scanned.)
411 If the messages you are learning from have already been filtered
413 through SpamAssassin, the learner will compensate for this. In effect,
414 it learns what each message would look like if you had run
415 "spamassassin -d" over it in advance.
416 Another thing to be aware of, is that typically you should aim to train
418 with at least 1000 messages of spam, and 1000 ham messages, if
419 possible. More is better, but anything over about 5000 messages does
420 not improve accuracy significantly in our tests.
421 Be careful that you train from the same source -- for example, if you
423 train on old spam, but new ham mail, then the classifier will think
424 that a mail with an old date stamp is likely to be spam.
425 It's also worth noting that training with a very small quantity of ham,
427 will produce atrocious results. You should aim to train with at least
428 the same amount (or more if possible!) of ham data than spam.
429 On an on-going basis, it is best to keep training the filter to make
431 sure it has fresh data to work from. There are various ways to do
432 this:
433 1. Supervised learning
435 This means keeping a copy of all or most of your mail, separated
436 into spam and ham piles, and periodically re-training using those.
437 It produces the best results, but requires more work from you, the
438 user.
439 (An easy way to do this, by the way, is to create a new folder for
441 'deleted' messages, and instead of deleting them from other
442 folders, simply move them in there instead. Then keep all spam in
443 a separate folder and never delete it. As long as you remember to
444 move misclassified mails into the correct folder set, it is easy
445 enough to keep up to date.)
446 2. Unsupervised learning from Bayesian classification
448 Another way to train is to chain the results of the Bayesian
449 classifier back into the training, so it reinforces its own
450 decisions. This is only safe if you then retrain it based on any
451 errors you discover.
452 SpamAssassin does not support this method, due to experimental
454 results which strongly indicate that it does not work well, and
455 since Bayes is only one part of the resulting score presented to
456 the user (while Bayes may have made the wrong decision about a
457 mail, it may have been overridden by another system).
458 3. Unsupervised learning from SpamAssassin rules
460 Also called 'auto-learning' in SpamAssassin. Based on statistical
461 analysis of the SpamAssassin success rates, we can automatically
462 train the Bayesian database with a certain degree of confidence
463 that our training data is accurate.
464 It should be supplemented with some supervised training in
466 addition, if possible.
467 This is the default, but can be turned off by setting the
469 SpamAssassin configuration parameter "bayes_auto_learn" to 0.
470 4. Mistake-based training
472 This means training on a small number of mails, then only training
473 on messages that SpamAssassin classifies incorrectly. This works,
474 but it takes longer to get it right than a full training session
475 would.
476
478 sa-learn and the other parts of SpamAssassin's Bayesian learner, use a
479 set of persistent database files to store the learnt tokens, as
480 follows.
481 bayes_toks
483 The database of tokens, containing the tokens learnt, their count
484 of occurrences in ham and spam, and the timestamp when the token
485 was last seen in a message.
486 This database also contains some 'magic' tokens, as follows: the
488 version number of the database, the number of ham and spam messages
489 learnt, the number of tokens in the database, and timestamps of:
490 the last journal sync, the last expiry run, the last expiry token
491 reduction count, the last expiry timestamp delta, the oldest token
492 timestamp in the database, and the newest token timestamp in the
493 database.
494 This is a database file, using "DB_File". The database 'version
496 number' is 0 for databases from 2.5x, 1 for databases from certain
497 2.6x development releases, 2 for 2.6x, and 3 for 3.0 and later
498 releases.
499 bayes_seen
501 A map of Message-Id and some data from headers and body to what
502 that message was learnt as. This is used so that SpamAssassin can
503 avoid re-learning a message it has already seen, and so it can
504 reverse the training if you later decide that message was learnt
505 incorrectly.
506 This is a database file, using "DB_File".
508 bayes_journal
510 While SpamAssassin is scanning mails, it needs to track which
511 tokens it uses in its calculations. To avoid the contention of
512 having each SpamAssassin process attempting to gain write access to
513 the Bayes DB, the token timestamps are written to a 'journal' file
514 which will later (either automatically or via "sa-learn --sync") be
515 used to synchronize the Bayes DB.
516 Also, through the use of "bayes_learn_to_journal", or when using
518 the "--no-sync" option with sa-learn, the actual learning data will
519 take be placed into the journal for later synchronization. This is
520 typically useful for high-traffic sites to avoid the same
521 contention as stated above.
522
524 Since SpamAssassin can auto-learn messages, the Bayes database files
525 could increase perpetually until they fill your disk. To control this,
526 SpamAssassin performs journal synchronization and bayes expiration
527 periodically when certain criteria (listed below) are met.
528 SpamAssassin can sync the journal and expire the DB tokens either
530 manually or opportunistically. A journal sync is due if --sync is
531 passed to sa-learn (manual), or if the following is true
532 (opportunistic):
533 - bayes_journal_max_size does not equal 0 (means don't sync)
535 - the journal file exists
536 and either:
538 - the journal file has a size greater than bayes_journal_max_size
540 or
542 - a journal sync has previously occurred, and at least 1 day has passed
544 since that sync
545 Expiry is due if --force-expire is passed to sa-learn (manual), or if
547 all of the following are true (opportunistic):
548 - the last expire was attempted at least 12hrs ago
550 - bayes_auto_expire does not equal 0
551 - the number of tokens in the DB is > 100,000
552 - the number of tokens in the DB is > bayes_expiry_max_db_size
553 - there is at least a 12 hr difference between the oldest and newest
554 token atimes
555 EXPIRE LOGIC
557 If either the manual or opportunistic method causes an expire run to
558 start, here is the logic that is used:
559 - figure out how many tokens to keep. take the larger of either
561 bayes_expiry_max_db_size * 75% or 100,000 tokens. therefore, the goal
562 reduction is number of tokens - number of tokens to keep.
563 - if the reduction number is < 1000 tokens, abort (not worth the
564 effort).
565 - if an expire has been done before, guesstimate the new atime delta
566 based on the old atime delta. (new_atime_delta = old_atime_delta *
567 old_reduction_count / goal)
568 - if no expire has been done before, or the last expire looks "weird",
569 do an estimation pass. The definition of "weird" is:
570 - last expire over 30 days ago
571 - last atime delta was < 12 hrs
572 - last reduction count was < 1000 tokens
573 - estimated new atime delta is < 12 hrs
574 - the difference between the last reduction count and the goal
575 reduction count is > 50%
576 ESTIMATION PASS LOGIC
578 Go through each of the DB's tokens. Starting at 12hrs, calculate
579 whether or not the token would be expired (based on the difference
580 between the token's atime and the db's newest token atime) and keep the
581 count. Work out from 12hrs exponentially by powers of 2. ie: 12hrs *
582 1, 12hrs * 2, 12hrs * 4, 12hrs * 8, and so on, up to 12hrs * 512
583 (6144hrs, or 256 days).
584 The larger the delta, the smaller the number of tokens that will be
586 expired. Conversely, the number of tokens goes up as the delta gets
587 smaller. So starting at the largest atime delta, figure out which
588 delta will expire the most tokens without going above the goal
589 expiration count. Use this to choose the atime delta to use, unless
590 one of the following occurs:
591 - the largest atime (smallest reduction count) would expire too many
593 tokens. this means the learned tokens are mostly old and there needs
594 to be new tokens learned before an expire can occur.
595 - all of the atime choices result in 0 tokens being removed. this means
596 the tokens are all newer than 12 hours and there needs to be new tokens
597 learned before an expire can occur.
598 - the number of tokens that would be removed is < 1000. the benefit
599 isn't worth the effort. more tokens need to be learned.
600 If the expire run gets past this point, it will continue to the end. A
602 new DB is created since the majority of DB libraries don't shrink the
603 DB file when tokens are removed. So we do the "create new, migrate old
604 to new, remove old, rename new" shuffle.
605 EXPIRY RELATED CONFIGURATION SETTINGS
607 "bayes_auto_expire" is used to specify whether or not SpamAssassin
608 ought to opportunistically attempt to expire the Bayes database. The
609 default is 1 (yes).
610 "bayes_expiry_max_db_size" specifies both the auto-expire token count
611 point, as well as the resulting number of tokens after expiry as
612 described above. The default value is 150,000, which is roughly
613 equivalent to a 6Mb database file if you're using DB_File.
614 "bayes_journal_max_size" specifies how large the Bayes journal will
615 grow before it is opportunistically synced. The default value is
616 102400.
617
619 The sa-learn command is part of the Mail::SpamAssassin Perl module.
620 Install this as a normal Perl module, using "perl -MCPAN -e shell", or
621 by hand.
622
624 spamassassin(1) spamc(1) Mail::SpamAssassin(3)
625 Mail::SpamAssassin::ArchiveIterator(3)
626 <http://www.paulgraham.com/> Paul Graham's "A Plan For Spam" paper
628 <http://www.linuxjournal.com/article/6467> Gary Robinson's f(x) and
630 combining algorithms, as used in SpamAssassin
631 <http://www.bgl.nu/~glouis/bogofilter/> 'Training on error' page. A
633 discussion of various Bayes training regimes, including 'train on
634 error' and unsupervised training.
635
637 "Mail::SpamAssassin"
638
640 The SpamAssassin(tm) Project <http://spamassassin.apache.org/>
641perl v5.28.0 2018-09-20 SA-LEARN(1)