1MIMEDEFANG-MULTIPLEXOR(8) System Manager's Manual MIMEDEFANG-MULTIPLEXOR(8)
2
6 mimedefang-multiplexor - Process pool controller for mail filters.
7
9 mimedefang-multiplexor [options]
10
13 mimedefang-multiplexor manages a pool of Perl processes for scanning e-
14 mail. It is designed to work in conjunction with mimedefang(8) and
15 mimedefang.pl(8).
16 mimedefang-multiplexor opens a UNIX-domain socket and listens for re‐
18 quests for work from mimedefang. As requests come in, mimedefang-mul‐
19 tiplexor creates Perl processes as needed to scan mail. The Perl pro‐
20 cesses are not killed when scanning is completed, but continue to run
21 in a loop. Perl processes are re-used for subsequent e-mail messages.
22 This eliminates the large overhead of starting a new Perl process for
23 each incoming message.
24 To avoid memory leaks, the Perl processes are killed after they have
26 handled some number of scans.
27
30 -U user
31 Runs the multiplexor as user rather than root. This option is
32 mandatory, and must match the -U option supplied to mimedefang.
33 -m minWorkers
36 The minimum number of Perl processes to keep running at all
37 times. The default is zero.
38 -x maxWorkers
41 The maximum number of Perl processes to run simultaneously. If
42 a request comes in and all processes are busy, a temporary fail‐
43 ure is signalled to the SMTP peer. The default is 2.
44 -r maxRequests
47 The maximum number of requests a given process handles before it
48 is killed and a replacement started. The default is 500.
49 -i idleTime
52 The idle time in seconds after which to kill of excess Perl pro‐
53 cesses. That is, if the process is idle for longer than this
54 time, and there are more than minWorkers running, the process is
55 killed. Note that this is implemented as a timer which ticks
56 every idleTime seconds; therefore, processes may be idle for up
57 to twice this time before they are killed. The default for
58 idleTime is 300 seconds.
59 -V maxLifetime
62 The maximum lifetime in seconds of a worker before it is killed
63 and a replacement started. The default is -1, which signifies
64 no maximum lifetime. Note that the lifetime check is done only
65 when a worker becomes idle after a request, or every time the
66 idle-timeout check is made. On a very quiet system, workers may
67 live for longer than maxLifetime by as much as idleTime. Note
68 also that the lifetime is measured not from when the worker
69 started running, but from when it was assigned its very first
70 request. A completely-idle worker that has never processed any
71 requests will not be terminated by the maxLifetime setting.
72 -b busyTime
75 The longest a Perl process is allowed to spend scanning an e-
76 mail before it is declared hung up and killed. The default is
77 120 seconds.
78 -Z This option specifies that the multiplexor should accept and
81 process "status updates" from busy workers. Note that this con‐
82 sumes one extra file descriptor per worker, plus a small amount
83 of CPU time per status update.
84 -c cmdTime
87 The timeout for communication between mimedefang-multiplexor and
88 mimedefang, or between mimedefang-multiplexor and a Perl scan‐
89 ning process. The default is 10 seconds. This timeout should
90 be kept quite short.
91 -w waitTime
94 When mimedefang-multiplexor starts the initial workers, or needs
95 to bring the number of running workers up to the number defined
96 by the -m option, it does not start all the workers at once, be‐
97 cause this could overload your server. Instead, it starts one
98 worker every waitTime seconds. The default value for waitTime
99 is 3.
100 -W waitTime
103 If you use this option, mimedefang-multiplexor will never acti‐
104 vate a worker until waitTime seconds have elapsed since the last
105 worker activation. This could result in mail being tempfailed
106 if worker activations do not keep pace with incoming mail. How‐
107 ever, it may be preferable to tempfail mail rather than allow
108 the load on your server to spike up too quickly. The default
109 value for this option is 0, meaning that mimedefang-multiplexor
110 will start workers as quickly as necessary to keep up with in‐
111 coming mail.
112 -z spooldir
115 Set the spool directory to spooldir. If this option is omitted,
116 the spool directory defaults to /var/spool/MIMEDefang.
117 -s pathName
120 The UNIX-domain socket on which mimedefang-multiplexor listens
121 for requests. This should be specified as an absolute pathname.
122 If this option is not supplied, it defaults to mimedefang-multi‐
123 plexor.sock under the spool directory.
124 -a socket
127 A socket for listening for requests. This is similar to the -s
128 socket, except that a restricted set of requests are processed.
129 On this socket, the multiplexor will only process requests ask‐
130 ing for status; it will not accept any commands to do scanning
131 or that would consume a worker. See the SOCKET SPECIFICATION
132 section for the format of socket.
133 -p fileName
136 Causes mimedefang-multiplexor to write its process-ID (after be‐
137 coming a daemon) to the specified file. The file will be owned
138 by root.
139 -o fileName
142 Causes mimedefang-multiplexor to use fileName as a lock file to
143 avoid multiple instances from running. If you supply -p but not
144 -o, then mimedefang-multiplexor constructs a lock file by ap‐
145 pending ".lock" to the pid file. However, this is less secure
146 than having a root-owned pid file in a root-owned directory and
147 a lock file writable by the user named by the -U option. (The
148 lock file must be writable by the -U user.)
149 -f filter_path
152 Normally, mimedefang-multiplexor executes a Perl filter script
153 called mimedefang.pl to scan the e-mail. However, you can have
154 it execute any program you like by specifying the full path to
155 the program with the -f option. This program must obey the pro‐
156 tocol documented in mimedefang-protocol(7); see that manual page
157 for details.
158 Note that the -f option does not specify the "filter" to use
160 with mimedefang.pl; instead, it specifies the program for
161 mimedefang-multiplexor to execute. You almost certainly should
162 not use this option unless you wish to replace mimedefang.pl
163 with your own program.
164 -F rules_path
167 Specifies the path to the filter rules. By default,
168 /etc/mail/mimedefang-filter is used. If you use the -F option,
169 its value is passed to the underlying Perl filter program using
170 -f.
171 -l Log certain events, including the output of the Perl workers'
174 standard-error, using syslog. Normally, the multiplexor does
175 not log much information.
176 -d Write debugging information about event-handling code in
179 /var/log/mimedefang-event-debug.log. This is only of use to
180 people debugging mimedefang-multiplexor.
181 -R kbytes
184 Limits the resident-set size of the worker filter processes to
185 kbytes kilobytes. This limit is not supported on all operating
186 systems; it is known to work on Linux.
187 -M kbytes
190 Limits the total memory space of worker filter processes to
191 kbytes kilobytes. This limit is supported on all operating sys‐
192 tems which support the setrlimit(2) system call. This should
193 include most modern UNIX systems.
194 We recommend that you monitor your worker filter processes and
196 get a feel for how much memory they use. You should then limit
197 the memory to two or three times the worst-case that you have
198 observed. This can help mitigate denial-of-service attacks
199 which use complicated MIME messages to force mimedefang.pl to
200 consume lots of memory.
201 -h Print usage information and exit.
204 -t filename
207 Log statistical information to filename. See the section STA‐
208 TISTICS for more information.
209 -T Log statistical information using syslog(2). You may use any -t
212 and -T together, in which case statistical information is logged
213 in a file and using syslog.
214 -u Flush the statistics file after every write. Normally, mimede‐
217 fang-multiplexor does not flush the file; this is the best
218 choice for minimizing disk I/O on a busy mail server. However,
219 if you wish to watch statistics entries in real-time, you should
220 enable flushing.
221 -D Do not fork into the background and become a daemon. Instead,
224 stay in the foreground. Useful mainly for debugging or if you
225 have a supervisory process managing mimedefang-multiplexor.
226 -q queue_size
229 Normally, if all workers are busy and mimedefang-multiplexor re‐
230 ceives another request, it fails it with the error "No free
231 workers." However, if you use the -q option, then up to
232 queue_size requests will be queued. As soon as a worker becomes
233 free, the queued requests will be handed off in FIFO order. If
234 the queue is full and another request comes in, then the request
235 is failed with "No free workers".
236 -Q queue_timeout
239 Queued requests should not stay on the queue indefinitely. If a
240 queued request cannot be processed within queue_timeout (default
241 30) seconds of being placed on the queue, it is failed with a
242 "Queued request timed out" message. See the section "QUEUEING
243 REQUESTS" for more discussion.
244 -O sock
247 Listen on a notification socket for connections from listeners.
248 mimedefang-multiplexor can inform external programs of state
249 changes by sending messages over a notification socket. The ex‐
250 ternal programs connect to this socket and then listen for noti‐
251 fications. See the section SOCKET SPECIFICATION for the format
252 of sock.
253 See the mimedefang-notify(7) man page for details of the notifi‐
255 cation protocol.
256 -N map_sock
259 Listen on a map socket for Sendmail SOCKETMAP connections. As
260 of Sendmail 8.13, you can define a Sendmail map type that talks
261 to a daemon over a socket. mimedefang-multiplexor implements
262 that protocol; consult the mimedefang-filter(5) man page for de‐
263 tils (see the SOCKET MAPS section).
264 See the section SOCKET SPECIFICATION for the format of map_sock.
266 -I backlog
271 When mimedefang-multiplexor creates a listening socket, it cal‐
272 culates the "backlog" argument to listen(2) based on the maximum
273 number of workers. However, you can explicitly set this backlog
274 with the -I option. Setting the backlog to a high value (around
275 30-50) may help on a very busy server. If you see mail log mes‐
276 sages saying "MXCommand: socket: Connection refused" during busy
277 periods, then that's an indication you need a higher listen
278 backlog.
279 -L interval
282 Log the worker status every interval seconds. This logs a line
283 using syslog; the line looks like this:
284 Worker status: Stopped=s Idle=i Busy=b Killed=k Queued=q Msgs=m Activations=a
287 Here, "Stopped" is the number of non-running workers, "Idle" is
289 the number of idle workers, "Busy" is the number of busy work‐
290 ers, "Killed" is the number of killed workers yet to be reaped,
291 "Queued" is the number of queued requests, "Msgs" is the total
292 number of messages processed since the multiplexor began run‐
293 ning, and "Activations" is the number of times a Perl process
294 has been started since the multiplexor began running.
295 If you supply an interval of 0 (which is the default), no peri‐
297 odic status logging is performed. If you supply an interval of
298 less than 5 seconds, it is silently reset to 5 seconds.
299 -S facility
302 Specifies the syslog facility for log messages. The default is
303 mail. See openlog(3) for a list of valid facilities. You can
304 use either the short name ("mail") or long name ("LOG_MAIL") for
305 the facility name.
306 -E Specifies that the multiplexor should create an embedded Perl
309 interpreter. This can improve performance dramatically. But
310 see the section "EMBEDDING PERL" for more information.
311 -X n Specifies that the multiplexor should initiate a "tick" request
314 every n seconds. This causes your filter_tick function (if de‐
315 fined) to be called. Note that you have no control over which
316 worker executes filter_tick. If all workers are busy when a
317 tick occurs, that tick request is skipped and a warning message
318 is logged.
319 -P n Specifies that the multiplexor should run n tick requests in
322 parallel. Each tick is run as often as specified with the -X
323 argument. (If you omit the -P option, then the multiplexor be‐
324 haves as if -P 1 had been specified.)
325 If you run parallel ticks, each tick is assigned an integer
327 identifying its "type". The type ranges from 0 to n-1. While
328 there may be as many as n tick requests running at a time, only
329 one tick of each type will be active at any time.
330 -Y label
333 Sets the tag used in the multiplexor's syslog messages to label
334 instead of mimedefang-multiplexor.
335 -G Normally, mimedefang-multiplexor uses a umask of 027 when creat‐
338 ing listening sockets. If you would like the sockets to be
339 readable and writeable by the group as well as the owner, supply
340 the -G option. This causes the umask to be 007 whenever UNIX-
341 domain sockets are created.
342 -y n Limits the maximum number of concurrent recipok checks to n on a
345 per-domain basis. The value of n can range from 0 (in which
346 case no limit is applied) to maxWorkers, where maxWorkers is the
347 argument to the -x option. If n is outside that range, it is
348 ignored (and no limit is applied.)
349 The recipok command ultimately invokes the filter_recipient
352 function in your filter. If you are doing recipient verifica‐
353 tion against servers that may be slow or unreliable, you can use
354 the -y option to limit the number of concurrent recipient veri‐
355 fications per domain. That way, if one domain's server becomes
356 very slow, it won't consume all available workers for recipient
357 verification. Instead, its RCPT commands will be tempfailed and
358 there will be workers available to handle RCPT commands for
359 other domains.
360
363 The -a, -N and -O options takes a socket as an argument. The format of
364 the socket parameter is similar to that of the Sendmail Milter library,
365 and is one of the following:
366 /path/to/socket
369 A UNIX-domain socket
370 inet:portnum
373 A TCP socket bound to port portnum, but which accepts connec‐
374 tions only from the IPv4 loopback address (127.0.0.1).
375 inet_any:portnum
378 A TCP socket bound to port portnum which will accept connections
379 from any address. Use inet_any with caution!
380 inet6:portnum
383 A TCP socket bound to port portnum listening on the IPv6 loop‐
384 back address.
385 inet6_any:portnum
388 A TCP socket bound to port portnum listening on the IPv6 wild‐
389 card address.
390
393 Normally, if all workers are busy, any additional requests are failed
394 immediately. However, the -q and -Q options allow you to queue re‐
395 quests for a short amount of time. This facility is intended to grace‐
396 fully handle a temporary overload; most of the time, your queue should
397 be empty.
398 Because mimedefang checks the number of free workers when a connection
400 is opened and fails the connection if there are no free workers, the
401 intent of the queue is to allow SMTP transactions that are already un‐
402 derway to continue if there is a slight overload. Any new connections
403 will be failed if all workers are busy, but existing connections are
404 allowed to continue. Queuing requests may improve throughput on ex‐
405 tremely busy servers.
406 Note that if you supply the -q option to mimedefang, then even new con‐
408 nections are allowed to queue. This may improve throughput by keeping
409 the worker utilization higher.
410 The -R option to mimedefang can be used to reserve a specified number
412 of workers for connections from the loopback address. Using the -R op‐
413 tion has the side-effect of permitting new connections from the loop‐
414 back address to queue.
415
418 Normally, when mimedefang-multiplexor activates a worker, it forks and
419 execs mimedefang.pl. However, if the multiplexor was compiled with em‐
420 bedded Perl support, and you supply the -E command-line option, the
421 multiplexor works like this:
422 1 It creates an embedded Perl interpreter, and sources mimede‐
425 fang.pl with a special command-line argument telling it to read
426 the filter, but not to enter the main loop.
427 2 Each time a worker is activated, the multiplexor calls fork()
430 and runs the mimedefang.pl main loop. This invokes filter_ini‐
431 tialize and then runs the main loop.
432 On some platforms (for example, Red Hat Linux 7.3 with Perl 5.6.1), it
435 is not safe to destroy and recreate a Perl interpreter without causing
436 a memory leak. On those platforms, if you attempt to reread the filter
437 file (by sending the multiplexor a HUP signal or reread command), the
438 filter will not be re-read, and a message will be logged to syslog. On
439 those platforms, you must kill and restart MIMEDefang if you change the
440 filter file.
441 On most platforms, however, a filter reread is accomplished by destroy‐
444 ing and re-creating the embedded interpreter, re-sourcing mimedefang.pl
445 and killing workers as soon as they are idle.
446
449 With the -t option, mimedefang-multiplexor logs certain events to a
450 file. This file can be post-processed to gather statistics about the
451 multiplexor. You can use it to tune the number of workers you run, ad‐
452 just timeouts, and so on.
453 Each line of the file looks like this:
456 YYYY/MM/DD:HH:MM:SS timestamp event key=val key=val...
458 Here, YYYY/MM/DD:HH:MM:SS is the local time of day. Timestamp is the
460 number of seconds since January 1, 1970. Event is the name of an
461 event. The valid events are:
462 StartWorker
465 A worker process has been started.
466 KillWorker
469 A worker process has been killed.
470 ReapWorker
473 A dead worker process has been reaped. It is possible to have a
474 ReapWorker event without a previous KillWorker event if the
475 worker process terminated abnormally.
476 StartFilter
479 A worker process has begun filtering an e-mail message.
480 EndFilter
483 A worker process has finished filtering an e-mail message.
484 The possible keys in the key=value pairs are:
487 worker=n
490 The worker involved in the event. Every worker is identified by
491 a small integer.
492 nworkers=n
495 The total number of running workers immediately after the event
496 happened.
497 nbusy=n
500 The number of busy workers (workers which are processing an e-
501 mail message) immediately after the event happened.
502 reason="string"
505 The reason for a StartWorker or KillWorker event. (Present only
506 for these events.)
507 numRequests=n
510 The number of e-mails processed by the worker. Present only for
511 an EndFilter event.
512 If you send the mimedefang-multiplexor process a SIGHUP signal (kill -1
515 pid), it closes and reopens the statistics file. This is useful during
516 log file rotation.
517 If you send the mimedefang-multiplexor process a SIGINT signal (kill
520 -INT pid), it terminates all active-but-idle workers. Also, any ac‐
521 tive-and-busy workers terminate as soon as they finish filtering the
522 current message. This is useful to force a reread of the filter rules
523 file without stopping and restarting Sendmail.
524 If you send the mimedefang-multiplexor process a SIGTERM signal (kill
527 pid), it terminates all workers and exits immediately.
528
531 mimedefang-mulitplexor was written by Dianne Skoll <dfs@roaringpen‐
532 guin.com>. The mimedefang home page is https://www.mimedefang.org/.
533
536 mimedefang.pl(8), mimedefang-filter(5), mimedefang(8), mimedefang-pro‐
537 tocol(7)
5384th Berkeley Distribution 8 February 2005 MIMEDEFANG-MULTIPLEXOR(8)