1innfeed.conf(5) File Formats Manual innfeed.conf(5)
2
6 innfeed.conf - configuration file for innfeed
7
9 This man page describes the configuration file for version 1.0 of inn‐
10 feed. This format has changed dramatically since version 0.9.3.
11 The file innfeed.conf is used to control the innfeed(1) program. It is
13 a fairly free-format file that consists of three types of entries:
14 key/value, peer and group. Comments are from the hash character ``#''
15 to the end of the line.
16 Key/value entries are a keyword and a value separated by a colon (which
18 can itself be surrounded by whitespace). For example:
19 max-connections: 10
21 A legal key starts with a letter and contains only letters, digits, and
23 ``_'', ``-''.
24 There are 5 different type of values: integers, floating-point numbers,
26 characters, booleans, and strings. Integer and floating point numbers
27 are as to be expected except that exponents in floating point numbers
28 are not supported. A boolean value is either ``true'' or ``false''
29 (case is not significant). A character value is a single-quoted charac‐
30 ter as defined by the C-language. A string value is any other sequence
31 of characters. If the string needs to contain whitespace, then it must
32 be quoted with double quotes, and uses the same format for embedding
33 non-printing characters as normal C-language string.
34 Peer entries look like:
36 peer <name> {
38 # body ...
39 }
40 The word ``peer'' is required. The ``<name>'' is the same as the site
42 name in INN's newsfeeds file. The body of a peer entry contains some
43 number (possibly zero) of key/value entries.
44 Group entries look like:
46 group <name> {
48 # body
49 }
50 The word ``group'' is required. The ``<name>'' is any string valid as a
52 key. The body of a group entry contains any number of the three types
53 of entries. So key/value pairs can be defined inside a group, and peers
54 can be nested inside a group, and other groups can be nested inside a
55 group.
56 Key/value entries that are defined outside of all peer and group
58 entries are said to be at ``global scope''. There are global key/value
59 entries that apply to the process as a whole (for example the location
60 of the backlog file directory), and there are global key/value entries
61 that act as defaults for peers. When innfeed looks for a specific value
62 in a peer entry (for example, the maximum number of connections to set
63 up), if the value is not defined in the peer entry, then the enclosing
64 groups are examined for the entry (starting at the closest enclosing
65 group). If there are no enclosing groups, or the enclosing groups don't
66 define the key/value, then the value at global scope is used.
67 A small example could be:
69 # Global value applied to all peers that have
71 # no value of their own.
72 max-connections: 5
73 # A peer definition. ``uunet'' is the name used by innd in
75 # the newsfeeds file.
76 peer uunet {
77 ip-name: usenet1.uu.net
78 }
79 peer vixie {
81 ip-name: gw.home.vix.com
82 max-connections: 10 # override global value.
83 }
84 # A group of two peers who can handle more connections
86 # than normal
87 group fast-sites {
88 max-connections: 15
89 # Another peer. The ``max-connections'' value from the
91 # ``fast-sites'' group scope is used. The ``ip-name'' value
92 # defaults to the peer's name.
93 peer data.ramona.vix.com {
94 }
95 peer bb.home.vix.com {
97 max-connections: 20 # he can really cook.
98 }
99 }
100 Given the above configuration file, the defined peers would have the
102 following values for the ``max-connections'' key.
103 uunet 5
105 vixie 10
106 data.ramona.vix.com 15
107 bb.home.vix.com 20
108 Innfeed ignores key/value pairs it is not interested in. Some config
110 file values can be set via a command line option, in which case that
111 setting overrides the settings in the file.
112 Config files can be included in other config files via the syntax:
114 $INCLUDE filename
116 There is a maximum nesting depth of 10.
118 For a fuller example config file, see the supplied innfeed.conf.
120
122 The following listing show all the keys that apply to the process as
123 whole. These are not required (compiled-in defaults are used where
124 needed).
125 news-spool
127 This key requires a pathname value. It specifies where the top
128 of the article spool is. This corresponds to the ``-a'' command-
129 line option.
130 input-file
132 This key requires a pathname value. It specifies the pathname
133 (relative to the backlog-directory) that should be read in fun‐
134 nel-file mode. This corresponds to giving a filename as an argu‐
135 ment on the command-line (i.e. its presence also implies that
136 funnel-file mode should be used).
137 pid-file
139 This key requires a pathname value. It specifies the pathname
140 (relative to the backlog-directory) where the pid of the innfeed
141 process should be stored. This corresponds to the ``-p'' com‐
142 mand-line option.
143 debug-level
145 This key defines the debug level for the process. A non-zero
146 number generates a lot of messages to stderr, or to the config-
147 defined ``log-file''. This corresponds to the ``-d'' command-
148 line option.
149 use-mmap
151 This key requires a boolean value. It specifies whether mmaping
152 should be used if innfeed has been built with mmap support. If
153 article data on disk is not in NNTP-ready format (CR/LF at the
154 end of each line), then after mmaping the article is read into
155 memory and fixed up, so mmaping has no positive effect (and pos‐
156 sibly some negative effect depending on your system), and so in
157 such a case this value should be false. This corresponds to the
158 ``-M'' command-line option.
159 log-file
161 This key requires a pathname value. It specifies where any log‐
162 ging messages that couldn't be sent via syslog(3) should go
163 (such as those generated when a positive value for ``debug-
164 value'', is used). This corresponds to the ``-l'' command-line
165 option. A relative pathname is relative to the ``backlog-direc‐
166 tory'' value.
167 backlog-directory
169 This key requires a pathname value. It specifies where the cur‐
170 rent innfeed process should store backlog files. This corre‐
171 sponds to the ``-b'' command-line option.
172 backlog-highwater
174 This key requires a positive integer value. It specifies how
175 many articles should be kept on the backlog file queue before
176 starting to write new entries to disk.
177 backlog-ckpt-period
179 This key requires a positive integer value. It specifies how
180 many seconds between checkpoints of the input backlog file. Too
181 small a number will mean frequent disk accesses, too large a
182 number will mean after a crash innfeed will re-offer more
183 already-processed articles than necessary.
184 backlog-newfile-period
186 This key requires a positive integer value. It specifies how
187 many seconds before each checks for externally generated backlog
188 files that are to be picked up and processed.
189 dns-retry
191 This key requires a positive integer value. It defines the num‐
192 ber of seconds between attempts to re-lookup host information
193 that previous failed to be resolved.
194 dns-expire
196 This key requires a positive integer value. It defines the num‐
197 ber of seconds between refreshes of name to address DNS transla‐
198 tion. This is so long-running processes don't get stuck with
199 stale data, should peer ip addresses change.
200 close-period
202 This key requires a positive integer value. It is the maximum
203 number of seconds a connection should be kept open. Some NNTP
204 servers don't deal well with connections being held open for
205 long periods.
206 gen-html
208 This key requires a boolean value. It specifies whether the sta‐
209 tus-file should be HTML-ified.
210 status-file
212 This key requires a pathname value. It specifies the pathname
213 (relative to the backlog-directory) where the periodic status of
214 the innfeed process should be stored. This corresponds to the
215 ``-S'' command-line option.
216 connection-stats
218 This key requires a boolean value. If the value is true, then
219 whenever the transmission statistics for a peer are logged, then
220 each active connection logs its own statistics. This corresponds
221 to the ``-z'' command-line option.
222 host-queue-highwater
224 This key requires a positive integer value. It defines how many
225 articles will be held internally for a peer before new arrivals
226 cause article information to be spooled to the backlog file.
227 stats-period
229 This key requires a positive integer value. It defines how many
230 seconds innfeed waits between generating statistics on transfer
231 rates.
232 stats-reset
234 This key requires a positive integer value. It defines how many
235 seconds innfeed waits before resetting all internal transfer
236 counters back to zero (after logging one final time). This is so
237 a innfeed-process running more than a day will generate
238 ``final'' stats that will be picked up by logfile processing
239 scripts.
240 initial-reconnect-time
242 This key requires a positive integer value. It defines how many
243 seconds to first wait before retrying to reconnect after a con‐
244 nection failure. If the next attempt fails too, then the recon‐
245 nect time is approximately doubled until the connection suc‐
246 ceeds, or max-reconnection-time is reached.
247 max-reconnect-time
249 This key requires an integer value. It defines the maximum num‐
250 ber of seconds to wait between attempt to reconnect to a peer.
251 The initial value for reconnection attempts is defined by ini‐
252 tial-reconnect-time, and it is doubled after each failure, up to
253 this value.
254 stdio-fdmax
256 This key requires a non-negative integer value. If the value is
257 greater than zero, then whenever a network socket file descrip‐
258 tor is created and it has a value less than this, the file
259 descriptor will be dup'ed to bring the value up greater than
260 this. This is to leave lower numbered file descriptors free for
261 stdio. Certain systems, Sun's in particular, require this. SunOS
262 4.1.x usually requires a value of 128 and Solaris requires a
263 value of 256. The default if this is not specified, is 0.
264 bindaddress
266 This key requires a string value. It specifies which outgoing
267 IPv4 address innfeed should bind the local end of its connec‐
268 tions to. Must be in dotted-quad format (nnn.nnn.nnn.nnn). If
269 not set, innfeed defaults to letting the kernel choose this
270 address. The default value is unset.
271 bindaddress6
273 This key requires a string value. It behave like bindaddress
274 except for outgoing IPv6 connections.
275
277 All the key/value pairs mentioned in this section must be specified at
278 global scope. They may also be specified inside a group or peer defini‐
279 tion. Note that when peers are added dynamically (i.e. when innfeed
280 receives an article for an unspecified peer), it will add the peer site
281 using the parameters specified at global scope.
282 article-timeout
284 This key requires a non-negative integer value. If no articles
285 need to be sent to the peer for this many seconds, then the peer
286 is considered idle and all its active connections are torn down.
287 response-timeout
289 This key requires a non-negative integer value. It defines the
290 maximum amount of time to wait for a response from the peer
291 after issuing a command.
292 initial-connections
294 This key requires a non-negative integer value. It defines the
295 number of connections to be opened immediately when setting up a
296 peer binding. A value of 0 means no connections will be created
297 until an article needs to be sent.
298 max-connections
300 This key requires positive integer value. It defines the maximum
301 number of connections to run in parallel to the peer. A value of
302 zero specifies an unlimited number of maximum connections. In
303 general use of an unlimited number of maximum connections is not
304 recommended. Do not ever set max-connections to zero with
305 dynamic-method 0 set, as this will saturate peer hosts with con‐
306 nections. [ Note that in previous versions of innfeed, a value
307 of 1 had a special meaning. This is no longer the case, 1 means
308 a maximum of 1 connection ].
309 dynamic-method
311 This key requires an integer value between 0 and 3. It controls
312 how connections (up to max-connections) are opened, up to the
313 maximum specified by max-connections. In general (and specifi‐
314 cally, with dynamic-method 0), a new connection is opened when
315 the current number of connections is below max-connections, and
316 an article is to be sent while no current connections are idle.
317 Without further restraint (i.e. using dynamic-method 0), in
318 practice this means that max-connections connections are estab‐
319 lished while articles are being sent. Use of other dynamic-
320 method settings imposes a further limit on the amount of connec‐
321 tions opened below that specified by max-connections. This limit
322 is calculated in different ways, depending of the value of
323 dynamic-method. Users should note that adding additional con‐
324 nections is not always productive - just because opening twice
325 as many connections results in a small percentage increase of
326 articles accepted by the remote peer, this may be at consider‐
327 able resource cost both locally and at the remote site, whereas
328 the remote site might well have received the extra articles sent
329 from another peer a fraction of a second later. Opening large
330 numbers of connections is considered antisocial. The meanings
331 of the various settings are:
332 0 no method
334 Increase of connections up to max-connections is unre‐
335 strained.
336 1 maximize articles per second
338 Connections are increased (up to max-connections) and
339 decreased so as to maximize the number of articles per
340 second sent, while using the fewest connections to do
341 this.
342 2 set target queue length
344 Connections are increased (up to max-connections) and
345 decreased so as to keep the queue of articles to be sent
346 within the bounds set by dynamic-backlog-low and dynamic-
347 backlog-high, while using the minimum resources possible.
348 As the queue will tend to fill if the site is not keeping
349 up, this method ensures that the maximum number of arti‐
350 cles are offered to the peer while using the minimum num‐
351 ber of connections to achieve this.
352 3 combination
354 This method uses a combination of methods 1 and 2 above.
355 For sites accepting a large percentage of articles,
356 method 2 will be used to ensure these sites are offered
357 as complete a feed as possible. For sites accepting a
358 small percentage of articles, method 1 is used, to mini‐
359 mize remote resource usage. For intermediate sites, an
360 appropriate combination is used.
361 dynamic-backlog-low
363 This key requires an integer value between 0 and 100. It repre‐
364 sents (as a percentage) the low water mark for the host queue.
365 If the host queue falls below this level while using dynamic-
366 method 2 or 3, and if 2 or more connections are open, innfeed
367 will attempt to drop connections to the host. An IIR filter is
368 applied to the value to prevent connection flap (see dynamic-
369 filter). A value of 25.0 is recommended. This value must be
370 smaller than dynamic-backlog-high.
371 dynamic-backlog-high
373 This key requries an integer value between 0 and 100. It repre‐
374 sents (as a percentage) the high water mark for the host queue.
375 If the host queue rises above this level while using dynamic-
376 method 2 or 3, and if less than max-connections are open to the
377 host, innfeed will attempt to open further connections to the
378 host. An IIR filter is applied to the value to prevent connec‐
379 tion flap (see dynamic-filter). A value of 50.0 is recommended.
380 This value must be larger than dynamic-backlog-low.
381 dynamic-backlog-filter
383 This key requires a floating-point value between 0 and 1. It
384 represents the filter coefficient used by the IIR filter used to
385 implement dynamic-method 2 and 3. The recommended value of this
386 filter is 0.7, giving a time constant of 1/(1-0.7) articles.
387 Higher values will result in slower response to queue fullness
388 changes, lower values in faster response.
389 max-queue-size
391 This key requires a positive integer value. It defines the maxi‐
392 mum number of articles to process at one time when using stream‐
393 ing to transmit to a peer. Larger numbers mean more memory con‐
394 sumed as articles usually get pulled into memory (see the
395 description of use-mmap).
396 streaming
398 This key requires a boolean value. It defines whether streaming
399 commands are used to transmit articles to the peers.
400 no-check-high
402 This key requires a floating-point number which must be in the
403 range [0.0, 100.0]. When running transmitting with the streaming
404 commands, innfeed attempts an optimization called ``no-CHECK''
405 mode. This involves not asking the peer if it wants the article,
406 but just sending it. This optimization occurs when the percent‐
407 age of the articles the peer has accepted gets larger than this
408 number. If this value is set to 100.0, then this effectively
409 turns off no-CHECK mode, as the percentage can never get above
410 100.0. If this value is too small, then the number of articles
411 the peer rejects will get bigger (and your bandwidth will be
412 wasted). A value of 95.0 usually works pretty well. NOTE: In
413 innfeed 0.9.3 and earlier this value was in the range [0.0,
414 9.0].
415 no-check-low:
417 This key requires a floating-point number which must be in the
418 range [0.0, 100.0), and it must be smaller that the value for
419 no-check-high. When running in no-CHECK mode, as described
420 above, if the percentage of articles the remote accepts drops
421 below this number, then the no-CHECK optimization is turned off
422 until the percentage gets above the no-check-high value again.
423 If there is small difference between this and the no-check-high
424 value (less than about 5.0), then innfeed may frequently go in
425 and out of no-CHECK mode. If the difference is too big, then it
426 will make it harder to get out of no-CHECK mode when necessary
427 (wasting bandwidth). Keeping this to between 5.0 and 10.0 less
428 than no-check-high usually works pretty well.
429 no-check-filter
431 This is a floating point value representing the time constant,
432 in articles, over which the CHECK / no-CHECK calculations are
433 done. The recommended value is 50.0 which will implement an IIR
434 filter of time constant 50. This roughly equates to making a
435 decision about the mode over the previous 50 articles. A higher
436 number will result in a slower response to changing percentages
437 of articles accepted; a lower number will result in a faster
438 response.
439 port-number
441 This key requires a positive integer value. It defines the
442 tcp/ip port number to use when connecting to the remote.
443 drop-deferred
445 This key requires a boolean value. By default it is set to
446 false. When set to true, and a peer replies with code 431 or 436
447 (try again later) just drop the article and don't try to re-send
448 it. This is useful for some peers that keep on deferring arti‐
449 cles for a long time to prevent innfeed from trying to offer the
450 same article over and over again.
451 min-queue-connection
453 This key requires a boolean value. By default it is set to
454 false. When set to true, innfeed will attempt to use a connec‐
455 tion with the least queue size (or the first empty connection).
456 If this key is set to true, it is recommended that dynamic-
457 method be set to 0. This allows for article propagation with the
458 least delay.
459 no-backlog
461 This key requires a boolean value. It specifies whether spooling
462 should be enabled (false, the default) or disabled (true). Note
463 that when no-backlog is set, articles reported as "spooled" are
464 actually silently discarded.
465 backlog-limit
467 This key requires a non-negative integer value. If the number is
468 0 then backlog files are allowed to grown without bound when the
469 peer is unable to keep up with the article flow. If this number
470 is greater than 0 then it specifies the size (in bytes) the
471 backlog file should get truncated to when the backlog file
472 reaches a certain limit. The limit depends on whether backlog-
473 factor or backlog-limit-high is used.
474 backlog-factor
476 This key requires a floating point value, which must be larger
477 than 1.0. It is used in conjunction with the peer key backlog-
478 limit. If backlog-limit has a value greater than zero, then when
479 the backlog file gets larger than the value backlog-limit *
480 backlog-factor, then the backlog file will be truncated to the
481 size backlog-limit. For example if backlog-limit has a value of
482 1000000, and backlog-factor has a value of 2.0, then when the
483 backlogfile gets to be larger than 2000000 bytes in size, it
484 will be truncated to 1000000 bytes. The front portion of the
485 file is removed, and the trimming happens on line boundaries, so
486 the final size may be a bit less than this number. If backlog-
487 limit-highwater is defined too, then backlog-factor takes prece‐
488 dence.
489 backlog-limit-highwater
491 This key requires a positive integer value that must be larger
492 than the value for backlog-limit. If the size of the backlog
493 file gets larger than this value (in bytes), then the backlog
494 file will be shrunk down to the size of backlog-limit. If both
495 backlog-factor and backlog-limit-highwater are defined, then the
496 value of backlog-factor is used.
497 backlog-feed-first
499 This key requires a boolean value. By default it is set to
500 false. When set to true, the backlog is fed before new files.
501 This is intended to enforce in-order delivery, so setting this
502 to true when initial-connections or max-connections is more than
503 1 is inconsistent.
504 username
506 This key requires a string value. If the value is defined, then
507 innfeed tries to authenticate by ``AUTHINFO USER'' and this
508 value used for user name. password must also be defined, if
509 this key is defined.
510 password
512 This key requires a string value. The value is the password
513 used for ``AUTHINFO PASS''. username must also be defined, if
514 this key is defined.
515 deliver
517 This key is used with imapfeed to authenticate to a remote host.
518 It is optional. There are several parameters that must be
519 included with deliver:
520 deliver-authname
522 The authname is who you want to authenticate as.
523 deliver-password
525 This is the appropriate password for authname.
526 deliver-username
528 The username is who you want to "act" as, that is, who is
529 actually going to be using the server.
530 deliver-realm
532 In this case, the "realm" is the realm in which the spec‐
533 ified authname is valid. Currently this is only needed
534 by the DIGEST-MD5 SASL mechanism.
535 deliver-rcpt-to
537 A printf-style format string for creating the envelope
538 recipient address. The pattern MUST include a single
539 string specifier which will be replaced with the newgroup
540 (e.g "bb+%s"). The default is "+%s".
541 deliver-to-header
543 An optional printf-style format string for creating a To:
544 header to be prepended to the article. The pattern MUST
545 include a single string specifier which will be replaced
546 with the newgroup (e.g "post+%s@domain"). If not speci‐
547 fied, the To: header will not be prepended.
548
550 As previously explained, the peer definitions can contain redefinitions
551 of any of the key/value pairs described in the GLOBAL PEER DEFAULTS
552 section above. There is one key/value pair that is specific to a peer
553 definition.
554 ip-name
556 This key requires a word value. The word is the host's FQDN, or
557 the dotted quad ip-address. If this value is not specified then
558 the name of the peer is taken to also be its ip-name. See the
559 entry for data.ramona.vix.com in the example below.
560
562 If innfeed gets a SIGHUP signal, then it will reread the config file.
563 All values at global scope except for ``backlog-directory'' can be
564 changed (although note that ``bindaddress'' and ``bindaddress6''
565 changes will only affect new connections). Any new peers are added and
566 any missing peers have their connections closed.
567
569 Below is the sample innfeed.conf file.
570 #
571 # innfeed.conf file. See the comment block at the
572 # end for a fuller description.
573 #
574 ##
576 ## Global values. Not specific to any peer. These
577 ## are optional, but if used will override the
578 ## compiled in values. Command-line options used
579 ## will override these values.
580 ##
581 pid-file: innfeed.pid
583 debug-level: 0
584 use-mmap: false
585 log-file: innfeed.log
586 stdio-fdmax: 0
587 backlog-directory: innfeed
589 backlog-rotate-period: 60
590 backlog-ckpt-period: 30
591 backlog-newfile-period: 600
592 dns-retry: 900
594 dns-expire: 86400
595 close-period: 3600
596 gen-html: false
597 status-file: innfeed.status
598 connection-stats: false
599 host-queue-highwater: 200
600 stats-period: 600
601 stats-reset: 43200
602 max-reconnect-time: 3600
604 initial-reconnect-time: 30
605 ##
608 ## Defaults for all peers. These must all exist at
609 ## global scope. Any of them can be redefined
610 ## inside a peer or group definition.
611 ##
612 article-timeout: 600
614 response-timeout: 300
615 initial-connections: 1
616 max-connections: 5
617 max-queue-size: 25
618 streaming: true
619 no-check-high: 95.0
620 no-check-low: 90.0
621 no-check-filter: 50.0
622 port-number: 119
623 backlog-limit: 0
624 backlog-factor: 1.10
625 backlog-limit-highwater:0
626 dynamic-method: 3
627 dynamic-backlog-filter: 0.7
628 dynamic-backlog-low: 25.0
629 dynamic-backlog-high: 50.0
630 no-backlog: false
631 ##
633 ## Peers.
634 ##
635 peer decwrl {
636 ip-name: news1.pa.dec.com
637 }
638 peer uunet {
640 ip-name: news.uunet.uu.net
641 max-connections: 10
642 }
643 peer data.ramona.vix.com {
645 # ip-name defaults to data.ramona.vix.com
646 streaming: false
647 }
648 peer bb.home.vix.com {
650 ip-name: 192.5.5.33
651 }
652 # Blank lines are ignored. Everything after a '#'
656 # is ignored too.
657 #
658 # Format is:
659 # key : value
660 #
661 # See innfeed.conf(5) for a description of
662 # necessary & useful keys. Unknown keys and their
663 # values are ignored.
664 #
665 # Values may be a integer, floating-point, c-style
666 # single-quoted characters, boolean, and strings.
667 #
668 # If a string value contains whitespace, or
669 # embedded quotes, or the comment character
670 # (``#''), then the whole string must be quoted
671 # with double quotes. Inside the quotes, you may
672 # use the standard c-escape sequence
673 # (\t,\n,\r,\f,\v,\",\').
674 #
675 # Examples:
676 # eg-string: "NewConfigfile0
677 # eg-long-string: "A long string that goes
678 # over multiple lines. The
679 # newline is kept in the
680 # string except when quoted
681 # with a backslash # as here."
682 # eg-simple-string: A-no-quote-string
683 # eg-integer: 10
684 # eg-boolean: true
685 # eg-char: 'a'
686 # eg-ctrl-g: ' 07'
687
690 Written by James Brister <brister@vix.com> for InterNetNews. This is
691 revision 7379, dated 2005-07-04.
692
694 innfeed(1), newsfeeds(5)
695 innfeed.conf(5)