1SA-LEARN(1) User Contributed Perl Documentation SA-LEARN(1)
2
6 sa-learn - train SpamAssassin's Bayesian classifier
7
9 sa-learn [options] [file]...
10 sa-learn [options] --dump [ all | data | magic ]
12 Options:
14 --ham Learn the following messages as ham (non-spam)
16 --spam Learn the following messages as spam
17 --forget Forget the following messages
18 --use-ignores Use bayes_ignore_from and bayes_ignore_to
19 --sync Synchronize the database and the journal if needed
20 --force-expire Force a database sync and expiry run
21 --dbpath <path> Allows commandline override (in bayes_path form)
22 for where to read the Bayes DB from
23 --dump [all|data|magic] Display the contents of the Bayes database
24 Takes optional argument for what to display
25 --regexp <re> For dump only, specifies which tokens to
26 dump based on a regular expression.
27 -f file, --folders=file Read list of files/directories from file
28 --dir Ignored; historical compatibility
29 --file Ignored; historical compatibility
30 --mbox Input sources are in mbox format
31 --mbx Input sources are in mbx format
32 --max-size <b> Skip messages larger than b bytes;
33 defaults to 500 KB, 0 implies no limit
34 --showdots Show progress using dots
35 --progress Show progress using progress bar
36 --no-sync Skip synchronizing the database and journal
37 after learning
38 -L, --local Operate locally, no network accesses. Use
39 of this is recommended, see documentation.
40 --import Migrate data from older version/non DB_File
41 based databases
42 --clear Wipe out existing database
43 --backup Backup, to STDOUT, existing database
44 --restore <filename> Restore a database from filename
45 -u username, --username=username
46 Override username taken from the runtime
47 environment, used with SQL
48 -C path, --configpath=path, --config-file=path
49 Path to standard configuration dir
50 -p prefs, --prefspath=file, --prefs-file=file
51 Set user preferences file
52 --siteconfigpath=path Path for site configs
53 (default: /usr/etc/mail/spamassassin)
54 --cf='config line' Additional line of configuration
55 -D, --debug [area,...] Print debugging messages
56 -V, --version Print version
57 -h, --help Print usage message
58
60 Given a typical selection of your incoming mail classified as spam or
61 ham (non-spam), this tool will feed each mail to SpamAssassin, allowing
62 it to 'learn' what signs are likely to mean spam, and which are likely
63 to mean ham.
64 Simply run this command once for each of your mail folders, and it will
66 ''learn'' from the mail therein.
67 Note that csh-style globbing in the mail folder names is supported; in
69 other words, listing a folder name as "*" will scan every folder that
70 matches. See "Mail::SpamAssassin::ArchiveIterator" for more details.
71 If you are using mail boxes in format other than maildir you should use
73 the --mbox or --mbx parameters.
74 Files compressed with gzip/bzip2/xz/lz4/lzip/lzo are uncompressed
76 automatically. See "Mail::SpamAssassin::ArchiveIterator" for more
77 details.
78 SpamAssassin remembers which mail messages it has learnt already, and
80 will not re-learn those messages again, unless you use the --forget
81 option. Messages learnt as spam will have SpamAssassin markup removed,
82 on the fly.
83 If you make a mistake and scan a mail as ham when it is spam, or vice
85 versa, simply rerun this command with the correct classification, and
86 the mistake will be corrected. SpamAssassin will automatically
87 'forget' the previous indications.
88 Users of "spamd" who wish to perform training remotely, over a network,
90 should investigate the "spamc -L" switch.
91
93 --ham
94 Learn the input message(s) in the files following the option as
95 ham. If you have previously learnt any of the messages as spam,
96 SpamAssassin will forget them first, then re-learn them as ham.
97 Alternatively, if you have previously learnt them as ham, it'll
98 skip them this time around. If the messages have already been
99 filtered through SpamAssassin, the learner will ignore any
100 modifications SpamAssassin may have made.
101 --spam
103 Learn the input message(s) in the files following the option as
104 spam. If you have previously learnt any of the messages as ham,
105 SpamAssassin will forget them first, then re-learn them as spam.
106 Alternatively, if you have previously learnt them as spam, it'll
107 skip them this time around. If the messages have already been
108 filtered through SpamAssassin, the learner will ignore any
109 modifications SpamAssassin may havemmade.
110 --folders=filename, -f filename
112 sa-learn will read in the list of folders from the specified file,
113 one folder per line in the file. If the folder is prefixed with
114 "ham:type:" or "spam:type:", sa-learn will learn that folder
115 appropriately, otherwise the folders will be assumed to be of the
116 type specified by --ham or --spam.
117 "type" above is optional, but is the same as the standard for
119 ArchiveIterator: mbox, mbx, dir, file, or detect (the default if
120 not specified).
121 --mbox
123 sa-learn will read in the file(s) containing the emails to be
124 learned, and will process them in mbox format (one or more emails
125 per file).
126 --mbx
128 sa-learn will read in the file(s) containing the emails to be
129 learned, and will process them in mbx format (one or more emails
130 per file).
131 --use-ignores
133 Don't learn the message if a from address matches configuration
134 file item "bayes_ignore_from" or a to address matches
135 "bayes_ignore_to". The option might be used when learning from a
136 large file of messages from which the hammy spam messages or spammy
137 ham messages have not been removed.
138 --sync
140 Synchronize the journal and databases. Upon successfully syncing
141 the database with the entries in the journal, the journal file is
142 removed.
143 --force-expire
145 Forces an expiry attempt, regardless of whether it may be necessary
146 or not. Note: This doesn't mean any tokens will actually expire.
147 Please see the EXPIRATION section below.
148 Note: "--force-expire" also causes the journal data to be
150 synchronized into the Bayes databases.
151 --forget
153 Forget the input message(s) in the files following the option as
154 previously learnt.
155 --dbpath
157 Allows a commandline override of the bayes_path configuration
158 option.
159 --dump option
161 Display the contents of the Bayes database. Without an option or
162 with the all option, all magic tokens and data tokens will be
163 displayed. magic will only display magic tokens, and data will
164 only display the data tokens.
165 Can also use the --regexp RE option to specify which tokens to
167 display based on a regular expression.
168 --clear
170 Clear an existing Bayes database by removing all traces of the
171 database.
172 WARNING: This is destructive and should be used with care.
174 --backup
176 Performs a dump of the Bayes database in machine/human readable
177 format.
178 The dump will include token and seen data. It is suitable for
180 input back into the --restore command.
181 --restore=filename
183 Performs a restore of the Bayes database defined by filename.
184 WARNING: This is a destructive operation, previous Bayes data will
186 be wiped out.
187 -h, --help
189 Print help message and exit.
190 -u username, --username=username
192 If specified this username will override the username taken from
193 the runtime environment. You can use this option to specify users
194 in a virtual user configuration when using SQL as the Bayes
195 backend.
196 NOTE: This option will not change to the given username, it will
198 only attempt to act on behalf of that user. Because of this you
199 will need to have proper permissions to be able to change files
200 owned by username. In the case of SQL this generally is not a
201 problem.
202 -C path, --configpath=path, --config-file=path
204 Use the specified path for locating the distributed configuration
205 files. Ignore the default directories (usually
206 "/usr/share/spamassassin" or similar).
207 --siteconfigpath=path
209 Use the specified path for locating site-specific configuration
210 files. Ignore the default directories (usually
211 "/etc/mail/spamassassin" or similar).
212 --cf='config line'
214 Add additional lines of configuration directly from the command-
215 line, parsed after the configuration files are read. Multiple
216 --cf arguments can be used, and each will be considered a separate
217 line of configuration.
218 -p prefs, --prefspath=prefs, --prefs-file=prefs
220 Read user score preferences from prefs (usually
221 "$HOME/.spamassassin/user_prefs").
222 --progress
224 Prints a progress bar (to STDERR) showing the current progress. In
225 the case where no valid terminal is found this option will behave
226 very much like the --showdots option.
227 -D [area,...], --debug [area,...]
229 Produce debugging output. If no areas are listed, all debugging
230 information is printed. Diagnostic output can also be enabled for
231 each area individually; area is the area of the code to instrument.
232 For example, to produce diagnostic output on bayes, learn, and dns,
233 use:
234 spamassassin -D bayes,learn,dns
236 Use an empty string (-D '') to indicate no areas when the next item
238 on the command line is a path, to prevent the path from being
239 parsed as an area.
240 For more information about which areas (also known as channels) are
242 available, please see the documentation at:
243 C<https://wiki.apache.org/spamassassin/DebugChannels>
245 Higher priority informational messages that are suitable for
247 logging in normal circumstances are available with an area of
248 "info".
249 --no-sync
251 Skip the slow synchronization step which normally takes place after
252 changing database entries. If you plan to learn from many folders
253 in a batch, or to learn many individual messages one-by-one, it is
254 faster to use this switch and run "sa-learn --sync" once all the
255 folders have been scanned.
256 Clarification: The state of --no-sync overrides the
258 bayes_learn_to_journal configuration option. If not specified, sa-
259 learn will learn to the database directly. If specified, sa-learn
260 will learn to the journal file.
261 Note: --sync and --no-sync can be specified on the same
263 commandline, which is slightly confusing. In this case, the
264 --no-sync option is ignored since there is no learn operation.
265 -L, --local
267 Do not perform any network accesses while learning details about
268 the mail messages. This should be normally used, as there really
269 isn't anything Bayes can learn from network lookup results.
270 Official SpamAssassin plugins do not currently do any network
271 lookups when learning, but it's possible that third party ones
272 might.
273 --import
275 If you previously used SpamAssassin's Bayesian learner without the
276 "DB_File" module installed, it will have created files in other
277 formats, such as "GDBM_File", "NDBM_File", or "SDBM_File". This
278 switch allows you to migrate that old data into the "DB_File"
279 format. It will overwrite any data currently in the "DB_File".
280 Can also be used with the --dbpath path option to specify the
282 location of the Bayes files to use.
283
285 There are now multiple backend storage modules available for storing
286 user's bayesian data. As such you might want to migrate from one
287 backend to another. Here is a simple procedure for migrating from one
288 backend to another.
289 Note that if you have individual user databases you will have to
291 perform a similar procedure for each one of them.
292 sa-learn --sync
294 This will sync any outstanding journal entries
295 sa-learn --backup > backup.txt
297 This will save all your Bayes data to a plain text file.
298 sa-learn --clear
300 This is optional, but good to do to clear out the old database.
301 Repeat!
303 At this point, if you have multiple databases, you should perform
304 the procedure above for each of them. (i.e. each user's database
305 needs to be backed up before continuing.)
306 Switch backends
308 Once you have backed up all databases you can update your
309 configuration for the new database backend. This will involve at
310 least the bayes_store_module config option and may involve some
311 additional config options depending on what is required by the
312 module. (For example, you may need to configure an SQL database.)
313 sa-learn --restore backup.txt
315 Again, you need to do this for every database.
316 If you are migrating to SQL you can make use of the -u <username>
318 option in sa-learn to populate each user's database. Otherwise, you
319 must run sa-learn as the user who database you are restoring.
320
322 (Thanks to Michael Bell for this section!)
323 For a more lengthy description of how this works, go to
325 http://www.paulgraham.com/ and see "A Plan for Spam". It's reasonably
326 readable, even if statistics make me break out in hives.
327 The short semi-inaccurate version: Given training, a spam heuristics
329 engine can take the most "spammy" and "hammy" words and apply
330 probabilistic analysis. Furthermore, once given a basis for the
331 analysis, the engine can continue to learn iteratively by applying both
332 the non-Bayesian and Bayesian rulesets together to create evolving
333 "intelligence".
334 SpamAssassin 2.50 and later supports Bayesian spam analysis, in the
336 form of the BAYES rules. This is a new feature, quite powerful, and is
337 disabled until enough messages have been learnt.
338 The pros of Bayesian spam analysis:
340 Can greatly reduce false positives and false negatives.
342 It learns from your mail, so it is tailored to your unique e-mail
343 flow.
344 Once it starts learning, it can continue to learn from SpamAssassin and
346 improve over time.
347 And the cons:
349 A decent number of messages are required before results are useful for
351 ham/spam determination.
352 It's hard to explain why a message is or isn't marked as spam.
353 i.e.: a straightforward rule, that matches, say, "VIAGRA" is easy
354 to understand. If it generates a false positive or false negative,
355 it is fairly easy to understand why.
356 With Bayesian analysis, it's all probabilities - "because the past
358 says it is likely as this falls into a probabilistic distribution
359 common to past spam in your systems". Tell that to your users!
360 Tell that to the client when he asks "what can I do to change
361 this". (By the way, the answer in this case is "use
362 welcomelisting".)
363 It will take disk space and memory.
365 The databases it maintains take quite a lot of resources to store
366 and use.
367
369 Still interested? Ok, here's the guidelines for getting this working.
370 First a high-level overview:
372 Build a significant sample of both ham and spam.
374 I suggest several thousand of each, placed in SPAM and HAM
375 directories or mailboxes. Yes, you MUST hand-sort this - otherwise
376 the results won't be much better than SpamAssassin on its own.
377 Verify the spamminess/haminess of EVERY message. You're urged to
378 avoid using a publicly available corpus (sample) - this must be
379 taken from YOUR mail server, if it is to be statistically useful.
380 Otherwise, the results may be pretty skewed.
381 Use this tool to teach SpamAssassin about these samples, like so:
383 sa-learn --spam /path/to/spam/folder
384 sa-learn --ham /path/to/ham/folder
385 sa-learn --ham hampath1 hampath2 --spam spampath1 spampath2
386 ...
387 Let SpamAssassin proceed, learning stuff. When it finds ham and
389 spam it will add the "interesting tokens" to the database.
390 If you need SpamAssassin to forget about specific messages, use the
392 --forget option.
393 This can be applied to either ham or spam that has run through the
394 sa-learn processes. It's a bit of a hammer, really, lowering the
395 weighting of the specific tokens in that message (only if that
396 message has been processed before).
397 Learning from single messages uses a command like this:
399 sa-learn --ham --no-sync mailmessage
400 This is handy for binding to a key in your mail user agent. It's
402 very fast, as all the time-consuming stuff is deferred until you
403 run with the "--sync" option.
404 Autolearning is enabled by default
406 If you don't have a corpus of mail saved to learn, you can let
407 SpamAssassin automatically learn the mail that you receive. If you
408 are autolearning from scratch, the amount of mail you receive will
409 determine how long until the BAYES_* rules are activated.
410
412 Learning filters require training to be effective. If you don't train
413 them, they won't work. In addition, you need to train them with new
414 messages regularly to keep them up-to-date, or their data will become
415 stale and impact accuracy.
416 You need to train with both spam and ham mails. One type of mail alone
418 will not have any effect.
419 Note that if your mail folders contain things like forwarded spam,
421 discussions of spam-catching rules, etc., this will cause trouble. You
422 should avoid scanning those messages if possible. (An easy way to do
423 this is to move them aside, into a folder which is not scanned.)
424 If the messages you are learning from have already been filtered
426 through SpamAssassin, the learner will compensate for this. In effect,
427 it learns what each message would look like if you had run
428 "spamassassin -d" over it in advance.
429 Another thing to be aware of, is that typically you should aim to train
431 with at least 1000 messages of spam, and 1000 ham messages, if
432 possible. More is better, but anything over about 5000 messages does
433 not improve accuracy significantly in our tests.
434 Be careful that you train from the same source -- for example, if you
436 train on old spam, but new ham mail, then the classifier will think
437 that a mail with an old date stamp is likely to be spam.
438 It's also worth noting that training with a very small quantity of ham,
440 will produce atrocious results. You should aim to train with at least
441 the same amount (or more if possible!) of ham data than spam.
442 On an on-going basis, it is best to keep training the filter to make
444 sure it has fresh data to work from. There are various ways to do
445 this:
446 1. Supervised learning
448 This means keeping a copy of all or most of your mail, separated
449 into spam and ham piles, and periodically re-training using those.
450 It produces the best results, but requires more work from you, the
451 user.
452 (An easy way to do this, by the way, is to create a new folder for
454 'deleted' messages, and instead of deleting them from other
455 folders, simply move them in there instead. Then keep all spam in
456 a separate folder and never delete it. As long as you remember to
457 move misclassified mails into the correct folder set, it is easy
458 enough to keep up to date.)
459 2. Unsupervised learning from Bayesian classification
461 Another way to train is to chain the results of the Bayesian
462 classifier back into the training, so it reinforces its own
463 decisions. This is only safe if you then retrain it based on any
464 errors you discover.
465 SpamAssassin does not support this method, due to experimental
467 results which strongly indicate that it does not work well, and
468 since Bayes is only one part of the resulting score presented to
469 the user (while Bayes may have made the wrong decision about a
470 mail, it may have been overridden by another system).
471 3. Unsupervised learning from SpamAssassin rules
473 Also called 'auto-learning' in SpamAssassin. Based on statistical
474 analysis of the SpamAssassin success rates, we can automatically
475 train the Bayesian database with a certain degree of confidence
476 that our training data is accurate.
477 It should be supplemented with some supervised training in
479 addition, if possible.
480 This is the default, but can be turned off by setting the
482 SpamAssassin configuration parameter "bayes_auto_learn" to 0.
483 4. Mistake-based training
485 This means training on a small number of mails, then only training
486 on messages that SpamAssassin classifies incorrectly. This works,
487 but it takes longer to get it right than a full training session
488 would.
489
491 sa-learn and the other parts of SpamAssassin's Bayesian learner, use a
492 set of persistent database files to store the learnt tokens, as
493 follows.
494 bayes_toks
496 The database of tokens, containing the tokens learnt, their count
497 of occurrences in ham and spam, and the timestamp when the token
498 was last seen in a message.
499 This database also contains some 'magic' tokens, as follows: the
501 version number of the database, the number of ham and spam messages
502 learnt, the number of tokens in the database, and timestamps of:
503 the last journal sync, the last expiry run, the last expiry token
504 reduction count, the last expiry timestamp delta, the oldest token
505 timestamp in the database, and the newest token timestamp in the
506 database.
507 This is a database file, using "DB_File". The database 'version
509 number' is 0 for databases from 2.5x, 1 for databases from certain
510 2.6x development releases, 2 for 2.6x, and 3 for 3.0 and later
511 releases.
512 bayes_seen
514 A map of Message-Id and some data from headers and body to what
515 that message was learnt as. This is used so that SpamAssassin can
516 avoid re-learning a message it has already seen, and so it can
517 reverse the training if you later decide that message was learnt
518 incorrectly.
519 This is a database file, using "DB_File".
521 bayes_journal
523 While SpamAssassin is scanning mails, it needs to track which
524 tokens it uses in its calculations. To avoid the contention of
525 having each SpamAssassin process attempting to gain write access to
526 the Bayes DB, the token timestamps are written to a 'journal' file
527 which will later (either automatically or via "sa-learn --sync") be
528 used to synchronize the Bayes DB.
529 Also, through the use of "bayes_learn_to_journal", or when using
531 the "--no-sync" option with sa-learn, the actual learning data will
532 take be placed into the journal for later synchronization. This is
533 typically useful for high-traffic sites to avoid the same
534 contention as stated above.
535
537 Since SpamAssassin can auto-learn messages, the Bayes database files
538 could increase perpetually until they fill your disk. To control this,
539 SpamAssassin performs journal synchronization and bayes expiration
540 periodically when certain criteria (listed below) are met.
541 SpamAssassin can sync the journal and expire the DB tokens either
543 manually or opportunistically. A journal sync is due if --sync is
544 passed to sa-learn (manual), or if the following is true
545 (opportunistic):
546 - bayes_journal_max_size does not equal 0 (means don't sync)
548 - the journal file exists
549 and either:
551 - the journal file has a size greater than bayes_journal_max_size
553 or
555 - a journal sync has previously occurred, and at least 1 day has passed
557 since that sync
558 Expiry is due if --force-expire is passed to sa-learn (manual), or if
560 all of the following are true (opportunistic):
561 - the last expire was attempted at least 12hrs ago
563 - bayes_auto_expire does not equal 0
564 - the number of tokens in the DB is > 100,000
565 - the number of tokens in the DB is > bayes_expiry_max_db_size
566 - there is at least a 12 hr difference between the oldest and newest
567 token atimes
568 EXPIRE LOGIC
570 If either the manual or opportunistic method causes an expire run to
571 start, here is the logic that is used:
572 - figure out how many tokens to keep. take the larger of either
574 bayes_expiry_max_db_size * 75% or 100,000 tokens. therefore, the goal
575 reduction is number of tokens - number of tokens to keep.
576 - if the reduction number is < 1000 tokens, abort (not worth the
577 effort).
578 - if an expire has been done before, guesstimate the new atime delta
579 based on the old atime delta. (new_atime_delta = old_atime_delta *
580 old_reduction_count / goal)
581 - if no expire has been done before, or the last expire looks "weird",
582 do an estimation pass. The definition of "weird" is:
583 - last expire over 30 days ago
584 - last atime delta was < 12 hrs
585 - last reduction count was < 1000 tokens
586 - estimated new atime delta is < 12 hrs
587 - the difference between the last reduction count and the goal
588 reduction count is > 50%
589 ESTIMATION PASS LOGIC
591 Go through each of the DB's tokens. Starting at 12hrs, calculate
592 whether or not the token would be expired (based on the difference
593 between the token's atime and the db's newest token atime) and keep the
594 count. Work out from 12hrs exponentially by powers of 2. ie: 12hrs *
595 1, 12hrs * 2, 12hrs * 4, 12hrs * 8, and so on, up to 12hrs * 512
596 (6144hrs, or 256 days).
597 The larger the delta, the smaller the number of tokens that will be
599 expired. Conversely, the number of tokens goes up as the delta gets
600 smaller. So starting at the largest atime delta, figure out which
601 delta will expire the most tokens without going above the goal
602 expiration count. Use this to choose the atime delta to use, unless
603 one of the following occurs:
604 - the largest atime (smallest reduction count) would expire too many
606 tokens. this means the learned tokens are mostly old and there needs
607 to be new tokens learned before an expire can occur.
608 - all of the atime choices result in 0 tokens being removed. this means
609 the tokens are all newer than 12 hours and there needs to be new tokens
610 learned before an expire can occur.
611 - the number of tokens that would be removed is < 1000. the benefit
612 isn't worth the effort. more tokens need to be learned.
613 If the expire run gets past this point, it will continue to the end. A
615 new DB is created since the majority of DB libraries don't shrink the
616 DB file when tokens are removed. So we do the "create new, migrate old
617 to new, remove old, rename new" shuffle.
618 EXPIRY RELATED CONFIGURATION SETTINGS
620 "bayes_auto_expire" is used to specify whether or not SpamAssassin
621 ought to opportunistically attempt to expire the Bayes database. The
622 default is 1 (yes).
623 "bayes_expiry_max_db_size" specifies both the auto-expire token count
624 point, as well as the resulting number of tokens after expiry as
625 described above. The default value is 150,000, which is roughly
626 equivalent to a 6Mb database file if you're using DB_File.
627 "bayes_journal_max_size" specifies how large the Bayes journal will
628 grow before it is opportunistically synced. The default value is
629 102400.
630
632 The sa-learn command is part of the Mail::SpamAssassin Perl module.
633 Install this as a normal Perl module, using "perl -MCPAN -e shell", or
634 by hand.
635
637 spamassassin(1) spamc(1) Mail::SpamAssassin(3)
638 Mail::SpamAssassin::ArchiveIterator(3)
639 <http://www.paulgraham.com/> Paul Graham's "A Plan For Spam" paper
641 <http://www.linuxjournal.com/article/6467> Gary Robinson's f(x) and
643 combining algorithms, as used in SpamAssassin
644 <http://web.archive.org/web/20120512230723/http://www.bgl.nu/~glouis/bogofilter/>
646 'Training on error' page. A discussion of various Bayes training
647 regimes, including 'train on error' and unsupervised training.
648
650 "Mail::SpamAssassin"
651
653 The SpamAssassin(tm) Project <https://spamassassin.apache.org/>
654perl v5.36.0 2023-01-21 SA-LEARN(1)