1T-PROT(1) General Commands Manual T-PROT(1)
2
6 t-prot - TOFU Protection - Display Filter for RFC 5322 messages
7
9 t-prot [OPTIONS]...
10
12 This program is a filter to improve the readability of internet mes‐
13 sages (emails and usenet posts) by *hiding* some annoying parts, e.g.
14 mailing list footers, signatures, and TOFU (see definition below), as
15 well as squeezing sequences of blank lines or punctuation. The program
16 also detects TOFU or a high quoting ratio in a message (so you may take
17 appropriate action, e.g. when submitting messages to a mailing list or
18 a news server).
19 The filter is written in Perl and relies on input to be a single mes‐
20 sage conforming to RFC 822 or its successors, RFC 2822 and RFC 5322.
21 In messages conforming to MIME (RFCs 2045-2049) t-prot handles
22 text/plain parts, others are not touched.
23 Already reformatted messages are handled well: the script was initially
25 designed to cope with the output of the MUA mutt (which is the reason
26 for not using standard CPAN modules for handling messages).
27 T-prot offers example configuration files for mutt and its fork
29 mutt-kz, Heirloom mailx and metamail. Also coming with the t-prot pack‐
30 age is the example S-Lang macro t-prot.sl for using t-prot from within
31 slrn. There is a proof-of-concept filter for INN2, which you will have
32 to adapt to the needs of the news site you host. For use with send‐
33 mail's alias(5) file, please see below (the option -p provides an exam‐
34 ple line).
35
37 If you do not specify any options, t-prot does ... nothing. Every fea‐
38 ture you want must be turned on explicitly. Admittedly, we have quite
39 a number of options for t-prot. To limit confusion they are grouped
40 into five sections: Input/Output Options, Advertisement And Mailing
41 List Footers, Filtering Options, Detection Options, and Other Options.
42 While the others should be quite clear, filtering and detection might
43 deserve a word (or two).
44 If you want to tune the appearance of your mail from within your MUA
46 (or news messages from within your NUA), then go for the Filtering
47 Options section.
48 If you want to use t-prot to check on mails before they are submitted
50 to mailing lists, fed to your news server, or delivered by your MDA,
51 then have a peek at the Detection Options section. You may accept or
52 reject/bounce messages depending on t-prot's result.
53
55 -i FILE
56 Defines an input file; default is '-' i.e. STDIN.
57 -o FILE
59 Defines the output file; default is STDOUT.
60 --body Input consists just of the message's body. There are no RFC 5322
62 header lines.
63 NOTE: This does not work with --pgp-short, and multipart mes‐
65 sages will not be detected due to missing headers.
66 --lax-security
68 Allow insecure writing method. DO NOT USE UNLESS YOU REALLY KNOW
69 WHAT YOU ARE DOING. (This ugly workaround is needed for some
70 early mutt versions and should NEVER be used as a default, oth‐
71 erwise it will probably turn into a security issue.)
72 You can use this option safely to enable -o /dev/null (or other
74 files which cannot be changed with the user's privileges).
75 --max-lines=x
77 Maximum number of lines a message may count (with headers). If
78 the message is longer than x lines, the message will not be pro‐
79 cessed but printed unmodified. Exit status will be EX_DATAERR
80 except when called with -Mmutt or -Mmutt-kz.
81
83 -a "commercial signature": Hides "footers" (signatures) from com‐
84 mercial email providers.
85 This option compares the last lines of the message body with any
86 footer file found in the directory specified with -A DIRECTORY
87 (which is mandatory for this option). The comparison is done by
88 perl's index() function (please try perldoc -f index for
89 details).
90 NOTE: This option is not needed if --ftr-ad is specified.
92 --ftr-ad
94 "enable aggressive ad footer matching": With this option
95 enabled, t-prot makes footer detection really greedy: We assume
96 that commercial email providers aren't even frightened to append
97 changing texts *under* their ads which are appended to the mes‐
98 sage body. Because these texts even have changing lengths we
99 simply detect the lines of the footer *anywhere* in the body of
100 the message and assume that everything below belongs to the
101 footer. (Man, if life where always that easy! ;)
102 With this option even GMX ads should be easy to hide -- you buy
103 this with a slight performance hit (which is the reason this
104 option is disabled by default), and with the possibility that
105 sometimes the algorithm is just a little *too* greedy.
106 NOTE: This requires a directory with footer files to be given
108 with option -A DIRECTORY.
109 -A DIRECTORY
111 "ad footer directory": Defines the directory which contains the
112 advertisement list footers (one footer per file) which are to be
113 tested when removing them with options -a or --ftr-ad.
114 This option is also needed if you do not want signature lengths
115 to be counted wrong or fullquotes get undetected when an ad
116 footer is appended at the bottom of the message (especially when
117 using -S or -t).
118 -l "list signature": Hides "footers" (signatures) from mailing
120 lists. Footer detection works like the -a option.
121 NOTE: This requires a directory with footer files to be given
123 with option -L DIRECTORY. -l is not needed if --ftr-ml is spec‐
124 ified.
125 --ftr-ml
127 "enable aggressive mailing list footer matching": With this
128 option enabled t-prot makes footer detection really greedy:
129 Should be helpful with broken list servers, or even if your
130 email provider munges the bodies of your messages.
131 Works similar to --ftr-ad, just that it is intended for mailing
132 list footers.
133 NOTE: This requires a directory with footer files to be given
135 with option -L DIRECTORY.
136 -L DIRECTORY
138 "list footer directory": Defines the directory which contains
139 the mailing list footers (one footer per file) which are to be
140 tested when removing them with the options -l or --ftr-ml.
141 This option is also needed if you do not want signature lengths
142 to be counted wrong or fullquotes get undetected when a mailing
143 list footer is appended at the bottom of the message (especially
144 when using -S or -t).
145
147 --bigq[=n[,x]]
148 "shrink big quotes": Blocks of quotes with more than n lines
149 will be shrunk to x lines. Defaults are 30 for n and 10 for x.
150 -c[n] "compress": Squeezes a sequence of blank lines to just n blank
152 lines. n defaults to 2.
153 --diff Tolerate unified diff (see diff(1) and patch(1)) appended after
155 the signature (which usually makes the signature too long to be
156 valid).
157 Also, protect diff standard output from hiding (which would oth‐
159 erwise be easy prey for -t).
160 -e "ellipsis": Squeezes a sequence of four or more dots, exclama‐
162 tion marks, or question marks to only three dots or marks,
163 respectively.
164 --fixind
166 Fix broken quotes to adhere to RFC 3676 by removing spaces
167 between quote characters and adding a space after them.
168 NOTE: This may produce false positives if spaces in between
169 quote characters are intended (thus changing the quoting level,
170 see RFC 3676 for details).
171 --groupwise
173 Hides TOFU as produced by Novell Groupwise.
174 -k "anti Kammquote": Tries (not too aggressively) to fix those bro‐
176 ken zig-zag-shaped lines wrapped around by some MUAs which are
177 known as "Kammquoting" in German.
178 NOTE: This option is considered stable by now. However, some‐
180 times Kammquotes should have been removed but weren't. Please
181 send a bug report if this happens to you (after carefully read‐
182 ing the BUGS and REPORTING BUGS section of this man page, that
183 is).
184 Please also note that enabling this option is quite a perfor‐
186 mance hit.
187 --kdiff=n
189 Minimum length difference between two lines for wrapped line
190 detection on Kammquotes. For details, please see the source
191 code.
192 Anyway, lower values make the algorithm more aggressive, higher
193 values make Kammquotes harder to detect. Default is 20.
194 Requires -k.
196 --kmaxl=n
198 Maximum line length for wrapped line detection on Kammquotes.
199 For details, please see the source code.
200 Anyway, higher values make the algorithm more aggressive, lower
201 values make Kammquotes harder to detect. Default is 80.
202 Requires -k.
204 --kminl=n
206 Minimum line length for wrapped line detection on Kammquotes.
207 For details, please see the source code.
208 Anyway, lower values make the algorithm more aggressive, higher
209 values make Kammquotes harder to detect. Default is 65.
210 Requires -k.
212 --locale=LOCALE
214 Specify which locale to use for correct parsing of your MUA's
215 formatting of the displayed message (usually it is the locale
216 your MUA uses). Right now this option is only used when -Mmutt
217 or -Mmutt-kz is specified, but this may change in future. You
218 need the Perl module Locale::gettext for this feature.
219 NOTE: If you use mutt, mutt-kz or gnupg with locales, t-prot
221 will only work correctly if you specify the corresponding locale
222 string. Alternatively, you can use the environment variables
223 LC_ALL, LC_MESSAGES, or LANG to specify the locale string.
224 NOTE also: You also have to make sure you are running t-prot
226 with matching gnupg and mutt / mutt-kz versions. T-prot detects
227 gnupg and mutt / mutt-kz locales of the recent stable versions
228 of those programs, earlier versions might not work well with a
229 recent version of t-prot.
230 -M, --muaMUA
232 "mail user agent": Turn on special treatment for some mail user
233 agents. (Right now only mutt(1) and mutt-kz(1) are supported,
234 but more might be added in future.) Caveat: If your MUA is sup‐
235 ported by this feature you must ensure t-prot makes use of it
236 when called from within your MUA to work as desired.
237 -m "Microsoft TOFU": Hides TOFU as given by some Microsoft mailers.
239 (You all surely know these fullquotes beginning with
240 "----- Original Message -----"
241 and some header lines...)
242 --ms-smart
244 Burn CPU cycles trying to be smart with MS style TOFU: If there
245 are PGP signed parts inside the TOFU, the text still might con‐
246 ceal other message parts and therefore should not be deleted.
247 Please note that this is probably just a waste of time because
249 most MS Outlook users who do produce this kind of TOFU won't
250 care about making their messages the least bit readable or even
251 predictable. So this option will probably just be interesting
252 for mutt message hooks (to activate it on demand when you know
253 the sender tries to write legible messages).
254 Requires -Mmutt / -Mmutt-kz and -m.
256 --pgp-move
258 Move PGP and SSL verification output to bottom; requires -Mmutt
259 / -Mmutt-kz.
260 --pgp-move-vrf
262 Move PGP and SSL verification output to bottom only if verifica‐
263 tion shows a good signature and the signature could be verified
264 as authentic (using a trust path). If there is any problem with
265 the signature, the PGP output should not be moved so the user is
266 more likely to notice. Requires -Mmutt / -Mmutt-kz.
267 NOTE: If gpg is terminated before finished (e.g. hitting Ctrl-C,
269 or using kill(1)), we cannot always detect if the check was
270 interrupted. Though t-prot tries to be smart, there will be
271 false positives.
272 --pgp-short
274 Hide non-relevant PGP key uids; requires -Mmutt / -Mmutt-kz.
275 -r "rip header off": Hides all mail header lines.
277 --reply
279 Subject lines with multiple reply prefixes (Re: and translations
280 into other languages) get squeezed to only one prefix.
281 -S[n] "suppression of overlong signatures": Signatures are to be n
283 lines (not including the one containing dash-dash-space) or
284 less. If there are more, it is probably not that spirited after
285 all. So with this option you trade it for a truely nice line.
286 If no n is given, default is 4. (We do not recommend using a
287 value other than 4. Consider this old-fashioned, but we actually
288 do *like* RFC conformance.)
289 NOTE: The line containing "-- " ist not counted when testing for
291 an overlong signature, but it is included when displaying how
292 many lines were deleted.
293 -s "signature deletion": Hides signatures, i.e. all lines after a
295 "signature dashes" line, i.e. a line with three characters:
296 dash-dash-space (no more, no less).
297 --sani Sanitize headers "To:", "From:" and "Subject:": Quoted-printable
299 gets fixed to the corresponding chars. German Umlauts are trans‐
300 lated to their "ae", "oe", "ue" pendants.
301 Useful e.g. for searching by subject within MUAs like Berkeley
302 mailx.
303 --sigsmax[=n]
305 "maximum number of tolerated signatures": Here you can define
306 how many signatures you accept to be treated as such. (Most
307 significant behaviour is when microsoft style quotes are
308 removed. Experts please see the code for the more subtle impli‐
309 cations of this option.)
310 Leave empty or specify zero to have an unlimited number of sigs.
311 Default is 1.
312 --spass
314 "SpamAssassin workaround": SpamAssassin (available at
315 http://spamassassin.org/) often is configured that it adds some
316 lines to the message body containing information about the spam
317 criteria which were found matching for the message. This option
318 enables an extra test to avoid false positives for Microsoft
319 style TOFU on such messages.
320 -t "TOFU deletion": Hides "traditional style" TOFU, where each line
322 begins with the indent string ">".
323 -w "whitespace deletion": Hides trailing whitespace (sequences of
325 space and tab). CAVEAT: This may lead to interesting effects
326 with crossposts between mailing lists or with undetected signa‐
327 ture attempts.
328
330 -P MESSAGE
331 "user defined bounce message for picky delivery": You may spec‐
332 ify your own bounce message to be returned when we try to
333 deliver an email and bounce it because there is TOFU inside. See
334 -p.
335 -p [ADDRESS]
337 "picky delivery": If we really find some TOFU, abort with exit
338 code EX_UNAVAILABLE. Otherwise redirect the message to ADDRESS
339 if given.
340 Intended for use from within mail delivery agents (MDAs) or mail
342 transport agents (MTAs), or even from within INN, so the message
343 bounces if TOFU is detected, and does not get on *your* nerves.
344 :)
345 As an example for usage with sendmail, put this line into your
347 alias file and invoke newaliases:
348 notofu: |"/usr/local/bin/t-prot -mt -p=user@mydomain"
350 This will bounce messages for <notofu@domainname> if any TOFU is
352 detected inside the message, and deliver it to <user@mydomain>
353 otherwise. Note that TOFU is only detected if you specify -t
354 respectively -m.
355 PLEASE be careful not to bounce messages to mailing lists!
357 --check[=FLAGS]
359 Run checks. If successful, print an error message and quit with
360 an appropriate exit code. Useful e.g. for rejecting messages
361 from within INN2.
362 Flags are separated by commas (no whitespaces), and can be the
364 following (right now just one flag):
365 ratio[=n]
367 If the quoting ratio is n or more, the message is rejected. Must
368 be between 0 and 1, or else it is entirely disabled. Default is
369 0.75 (i.e., 75% of the message lines are quotes).
370 -d, --debug
372 Print envelope info to syslog when bouncing TOFU contaminated
373 email. Default syslog facility is mail.debug. Requires -p.
374
376 -h, --help
377 Displays a short help text with a summary on all options, and
378 exits.
379 -v, --version
381 Prints the current version number and exits.
382
384 The environment variables LC_ALL, LC_MESSAGES, and LANG are read and
385 respected when interpreting output by mutt / mutt-kz or gnupg (unless
386 they are overruled by the --locale option). T-prot's own output is Eng‐
387 lish regardless of any locale setting.
388
390 On program exit, t-prot uses exit codes from /usr/include/sysexits.h
391 and thus behaves in a manner that sendmail and others understand when
392 calling t-prot.
393 Currently, the codes used are
395 EX_OK
396 EX_USAGE
397 EX_DATAERR
398 EX_UNAVAILABLE
399 EX_SOFTWARE
400 EX_IOERR
401 If, however, perl fails to compile and execute t-prot, perl's normal
403 exit codes will be returned.
404
406 TOFU is an abbreviation which mixes German and English words; it
407 expands to "text oben, full-quote unten" which means "text above - full
408 quote below" and describes the style of so many users who let their
409 mailer or newsreader quote everything of the previous message and just
410 add some text at the top; obviously they think that quoted text must
411 not be changed at all. This is quite annoying as it needlessly sends a
412 lot of data even when it is not required. Some editing of messages is
413 desired. Please point these people to the page
414 http://www.river.com/users/share/etiquette/edit.html - thank you!
415
417 There are several ways to fine-tune t-prot's performance:
418 Some command line options are quite grave a performance hit -- do not
420 use -k and especially --ms-smart if you are content without them.
421 Checking for special footers is very costly as well. Put as few footer
423 files as absolutely needed in any footer directory.
424 All PGP related options are eating up lots of CPU time. Try to avoid
426 them on unsigned and unencrypted messages.
427 When calling t-prot from within mutt (or mutt-kz), you might use mutt's
429 folder-hook and message-hook facilities to turn options on only when
430 needed, e.g. to set up a different footer directory for each mailing
431 list folder.
432
434 Q: I want to make my mailing list footer files match more different
435 mailing list footers. Can I use regular expressions, or how can
436 I accomplish that?
437 A: Nope, regexp's do not work here. The comparison is made by the
439 perl builtin index() function (see perldoc for more detailed
440 info), so you must exactly match the beginning of the line. The
441 longer the line you specify, the more precise the match; if your
442 line is empty you match unconditionally.
443 Q: I use the options -l and -L to suppress mailing list footers
445 when displaying messages in mutt(1). This does work sometimes,
446 but sometimes it does not: the footer is not detected, and
447 therefore full quotes are not deleted and signatures are
448 detected as too long (which they aren't).
449 A: This might occur if the message is badly encoded, so mutt cannot
451 resolve all encoded characters, e.g. if you have an encoded mes‐
452 sage on a mailing list, and majordomo appends a mailing list
453 footer in a different encoding (or even plain us-ascii). "-- "
454 simply does not match "--=20".
455 Another problem are non-us-ascii characters. Just avoid them,
456 and everything should work fine.
457 See the preceding Q+A for a solution.
458 Q: I want to write a message which contains parts that should *not*
460 be deleted even when filtered with t-prot. Is this possible?
461 A: Yes, but please do not spread word of it. Make unobstrusive use
463 of the verbatim instruction:
464 #v+
466 This line is protected from being filtered by t-prot !!!!!!!
467 #v-
468 Text coming now is not.
469
471 Written by Jochen Striepe <t-prot@tolot.escape.de>.
472
474 All of the documentation and software included in the t-prot releases
475 is copyrighted by Jochen Striepe (except when explicitly stated other‐
476 wise).
477 Copyright © 2001-2015 Jochen Striepe. All rights reserved.
479 Redistribution and use, with or without modification, are permitted
481 provided that the following conditions are met:
482 1. Redistributions of source code must retain the above copyright
484 notice, this list of conditions and the following disclaimer.
485 2. All advertising materials mentioning features or use of this soft‐
487 ware must display the following acknowledgement:
488 This product includes software developed by Jochen Striepe and oth‐
490 ers.
491 3. Neither the name of the author nor the names of any contributors may
493 be used to endorse or promote products derived from this software with‐
494 out specific prior written permission.
495 THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
497 ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
498 IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PUR‐
499 POSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE
500 LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
501 CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
502 SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSI‐
503 NESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
504 WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
505 OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
506 ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
507
509 Many good ideas, bug reports and support from (in alphabetical order)
510 Bjoern Buerger, Bjoern Laessig, Christian Borss, Gerfried Fuchs, Martin
511 Neitzel, Martin Dietze, Matthias Kilian, Ralf Doeblitz, Sven Guckes and
512 many more (see the ChangeLog for active contributors). Many thanks to
513 all of them!
514 Many thanks to Gerhard H. Wrodnigg who uses a TOFU protection script in
516 order to keep the responses to his cancel bot reasonably short. The
517 entire inspiration for this hack came from the "TOFU protection" line
518 of his script on many usenet postings.
519
521 You can get the latest version from
522 http://www.escape.de/users/tolot/mutt/.
523
525 There is a problem when mutt gives a PGP verified or even a multipart
526 message to t-prot: The information where the PGP encrypted/signed data
527 or even attachments begin and end is plainly embedded in the text, not
528 really cleanly recognizable for t-prot. The problem should be worked
529 around by now, please send a bug report if it does not work for you.
530
532 Please note that t-prot development happens on current stable perl ver‐
533 sions only. If you do run t-prot on earlier (or unstable) perl ver‐
534 sions, you might encounter perl compiler bugs (or funny t-prot behav‐
535 iour). One solution is to upgrade your perl, another is simply to write
536 a bug report. If you do not run a current perl version, please include
537 this information in your bug report.
538 Please do not report a bug if
540 * you found it in the TODO file coming with the distribution. We do
541 know those and try to fix them as soon as possible.
542 * you have an old t-prot version. If you encounter a problem, first
543 see if there is a new t-prot version which fixes the issue. If you
544 upgraded to the latest version and it still occurs, a bug report is
545 just great.
546 If you noticed a bug when processing a message and want to provide the
548 t-prot team with some useful info, please:
549 * if invoking t-prot by mutt's display_filter facility, just set dis‐
550 play_filter to something like
551 "tee ~/foobar | t-prot <your options>"
553 and include ~/foobar in the bug report -- this way we might reproduce
555 the bug much easier if you are using a different environment than we
556 do.
557 * provide information on what command line options you use t-prot
558 with, what perl version t-prot runs on your system, and what else might
559 be important to enable us reproducing the bug.
560 Send your bug report to <t-prot-bugs@tolot.escape.de>. Thank you.
562
564 Fix bugs (see the BUGS section). Beside that, all main features should
565 be implemented by now. See the TODO file for more information.
566
568 mutt(1), mutt-kz(1), muttrc(5) and the part about "display_filter",
569 perl(1), aliases(5),
570 RFCs 2045-2049, 3676 and 5322,
572 http://freshmeat.net/articles/t-prot/ (a nice, solid introduction),
574 http://got.to/quote/ (German language),
575 http://www.river.com/users/share/etiquette/edit.html (the Learn To Edit
576 Messages HowTo has found a new home).
577T-PROT March 2015 T-PROT(1)