1INNFEED.CONF(5) InterNetNews Documentation INNFEED.CONF(5)
2
6 innfeed.conf - Configuration file for innfeed
7
9 The configuration file innfeed.conf in pathetc is used to control the
10 innfeed(8) program. It is a fairly free-format file that consists of
11 three types of entries: key:value, peer and group. Comments are from
12 the hash character "#" to the end of the line.
13 key:value entries are a keyword and a value separated by a colon (which
15 can itself be surrounded by whitespace). For example:
16 max-connections: 10
18 A legal key starts with a letter and contains only letters, digits, and
20 the "_" and "-" characters. There are 5 different types of values:
21 integers, floating-point numbers, characters, booleans, and strings.
22 Integer and floating-point numbers are as to be expected, except that
24 exponents in floating-point numbers are not supported. A boolean value
25 is either "true" or "false" (case is not significant). A character
26 value is a single-quoted character as defined by the C-language. A
27 string value is any other sequence of characters. If the string needs
28 to contain whitespace, then it must be quoted with double quotes, and
29 uses the same format for embedding non-printing characters as normal
30 C-language string.
31 Peer entries look like:
33 peer <name> {
35 # body ...
36 }
37 The word "peer" is required. The <name> is the same as the site name
39 in INN's newsfeeds configuration file. The body of a peer entry
40 contains some number (possibly zero) of key:value entries.
41 Group entries look like:
43 group <name> {
45 # body ...
46 }
47 The word "group" is required. The <name> is any string valid as a key.
49 The body of a group entry contains any number of the three types of
50 entries. So key:value pairs can be defined inside a group, and peers
51 can be nested inside a group, and other groups can be nested inside a
52 group.
53 key:value entries that are defined outside of all peer and group
55 entries are said to be at "global scope". There are global key:value
56 entries that apply to the process as a whole (for example the location
57 of the backlog file directory), and there are global key:value entries
58 that act as defaults for peers. When innfeed looks for a specific
59 value in a peer entry (for example, the maximum number of connections
60 to set up), if the value is not defined in the peer entry, then the
61 enclosing groups are examined for the entry (starting at the closest
62 enclosing group). If there are no enclosing groups, or the enclosing
63 groups do not define the key:value, then the value at global scope is
64 used.
65 A small example could be:
67 # Global value applied to all peers that have
69 # no value of their own.
70 max-connections: 5
71 # A peer definition. "uunet" is the name used by innd
73 # in the newsfeeds configuration file.
74 peer uunet {
75 ip-name: usenet1.uu.net
76 }
77 peer vixie {
79 ip-name: gw.home.vix.com
80 max-connections: 10 # Override global value.
81 }
82 # A group of two peers which can handle more connections
84 # than normal.
85 group fast-sites {
86 max-connections: 15
87 # Another peer. The "max-connections" value from the
89 # "fast-sites" group scope is used. The "ip-name" value
90 # defaults to the peer's name.
91 peer data.ramona.vix.com {
92 }
93 peer bb.home.vix.com {
95 max-connections: 20 # He can really cook.
96 }
97 }
98 Given the above configuration file, the defined peers would have the
100 following values for the max-connections key:
101 uunet 5
103 vixie 10
104 data.ramona.vix.com 15
105 bb.home.vix.com 20
106 innfeed ignores key:value pairs it is not interested in. Some
108 configuration file values can be set via a command-line option, in
109 which case that setting overrides the settings in the file.
110 Configuration files can be included in other configuration files via
112 the syntax:
113 $INCLUDE filename
115 There is a maximum nesting depth of 10.
117 For a fuller example configuration file, see the supplied innfeed.conf.
119
121 The following listing show all the keys that apply to the process as
122 whole. These are not required (compiled-in defaults are used where
123 needed).
124 news-spool
126 This key requires a pathname value and defaults to patharticles in
127 inn.conf. It specifies where the top of the article spool is.
128 This corresponds to the -a command-line option.
129 input-file
131 This key requires a pathname value. It specifies the pathname
132 (relative to the backlog-directory value) that should be read in
133 funnel-file mode. This corresponds to giving a filename as an
134 argument on the command-line (i.e. its presence also implies that
135 funnel-file mode should be used).
136 The default is unset; innfeed then runs in channel or batch mode.
138 pid-file
140 This key requires a pathname value and defaults to innfeed.pid. It
141 specifies the pathname (relative to pathrun in inn.conf) where the
142 pid of the innfeed process should be stored. This corresponds to
143 the -p command-line option.
144 debug-level
146 This key defines the debug level for the process. Default is 0. A
147 non-zero number generates a lot of messages to stderr, or to the
148 config-defined log-file. This corresponds to the -d command-line
149 option.
150 If a file named innfeed.debug exists in the pathlog directory (as
152 set in inn.conf), then debug-level is automatically set to 1. This
153 is a cheap way of avoiding continual reloading of the newsfeeds
154 file when debugging. Note that debug messages still go to log-
155 file.
156 debug-shrinking
158 This key requires a boolean value and defaults to false (the debug
159 file is allowed to grow without bound). If set to true, this file
160 is truncated when its size reaches a certain limit. See backlog-
161 limit for more details.
162 initial-sleep
164 This key requires a positive integer. The default value is 2. It
165 defines the number of seconds to wait when innfeed (or a fork)
166 starts, before beginning to open connections to remote hosts.
167 fast-exit
169 This key requires a boolean value and defaults to false. If set to
170 true, when innfeed receives a SIGTERM or SIGQUIT signal, it will
171 close its listeners as soon as it can, even if it means dropping
172 articles.
173 use-mmap
175 This key requires a boolean value and defaults to true. When
176 innfeed is given file names to send (a fairly rare use case)
177 instead of storage API tokens, it specifies whether mmaping should
178 be used if innfeed has been built with mmap(2) support. If article
179 data on disk is not in NNTP-ready format (CR/LF at the end of each
180 line), then after mmaping, the article is read into memory and
181 fixed up, so mmaping has no positive effect (and possibly some
182 negative effect depending on your system), and so in such a case
183 this value should be "false", which corresponds to the -M command-
184 line option.
185 log-file
187 This key requires a pathname value and defaults to innfeed.log. It
188 specifies where any logging messages that could not be sent via
189 syslog(3) should go (such as those generated when a positive value
190 for debug-value is used). This corresponds to the -l command-line
191 option.
192 This pathname is relative to pathlog in inn.conf.
194 log-time-format
196 This key requires a format string suitable for strftime(3). It is
197 used for messages sent via syslog(3) and to the status-file.
198 Default value is "%a %b %d %H:%M:%S %Y".
199 backlog-directory
201 This key requires a pathname value and defaults to innfeed. It
202 specifies where the current innfeed process should store backlog
203 files. This corresponds to the -b command-line option.
204 This pathname is relative to pathspool in inn.conf.
206 backlog-highwater
208 This key requires a positive integer value and defaults to 5. It
209 specifies how many articles should be kept on the backlog file
210 queue before starting to write new entries to disk.
211 backlog-ckpt-period
213 This key requires a positive integer value and defaults to 30. It
214 specifies how many seconds elapse between checkpoints of the input
215 backlog file. Too small a number will mean frequent disk accesses;
216 too large a number will mean after a crash, innfeed will re-offer
217 more already-processed articles than necessary.
218 backlog-newfile-period
220 This key requires a positive integer value and defaults to 600. It
221 specifies how many seconds elapse before each check for externally
222 generated backlog files that are to be picked up and processed.
223 backlog-rotate-period
225 This key requires a positive integer value and defaults to 60. It
226 specifies how many seconds elapse before innfeed checks for a
227 manually created backlog file and moves the output backlog file to
228 the input backlog file.
229 dns-retry
231 This key requires a positive integer value and defaults to 900. It
232 defines the number of seconds between attempts to re-lookup host
233 information that previously failed to be resolved.
234 dns-expire
236 This key requires a positive integer value and defaults to 86400.
237 It defines the number of seconds between refreshes of name to
238 address DNS translation. This is so long-running processes do not
239 get stuck with stale data, should peer IP addresses change.
240 gen-html
242 This key requires a boolean value and defaults to false. It
243 specifies whether the status-file should be HTML-ified.
244 status-file
246 This key requires a pathname value and defaults to innfeed.status.
247 An absolute pathname can be used. It specifies the pathname
248 (relative to pathhttp when gen-html is true; otherwise, pathlog as
249 set in inn.conf) where the periodic status of the innfeed process
250 should be stored. This corresponds to the -S command-line option.
251 connection-stats
253 This key requires a boolean value and defaults to false. If the
254 value is true, then whenever the transmission statistics for a peer
255 are logged, each active connection logs its own statistics. This
256 corresponds to the -z command-line option.
257 host-queue-highwater
259 This key requires a positive integer value and defaults to 10. It
260 defines how many articles will be held internally for a peer before
261 new arrivals cause article information to be spooled to the backlog
262 file.
263 stats-period
265 This key requires a positive integer value and defaults to 600. It
266 defines how many seconds innfeed waits between generating
267 statistics on transfer rates.
268 stats-reset
270 This key requires a positive integer value and defaults to 43200.
271 It defines how many seconds innfeed waits before resetting all
272 internal transfer counters back to zero (after logging one final
273 time). This is so a innfeed process running more than a day will
274 generate "final" stats that will be picked up by logfile processing
275 scripts.
276 initial-reconnect-time
278 This key requires a positive integer value and defaults to 30. It
279 defines how many seconds to first wait before retrying to reconnect
280 after a connection failure. If the next attempt fails too, then
281 the reconnect time is approximately doubled until the connection
282 succeeds, or max-reconnect-time is reached.
283 max-reconnect-time
285 This key requires an integer value and defaults to 3600. It
286 defines the maximum number of seconds to wait between attempt to
287 reconnect to a peer. The initial value for reconnection attempts
288 is defined by initial-reconnect-time, and it is doubled after each
289 failure, up to this value.
290 stdio-fdmax
292 This key requires a non-negative integer value and defaults to 0.
293 If the value is greater than zero, then whenever a network socket
294 file descriptor is created and it has a value less than this, the
295 file descriptor will be dup'ed to bring the value up greater than
296 this. This is to leave lower numbered file descriptors free for
297 stdio. Certain systems, Sun's in particular, require this. SunOS
298 4.1.x usually requires a value of 128 and Solaris requires a value
299 of 256. The default if this is not specified, is 0.
300 Special keys for imapfeed
302 The following keys are used with imapfeed to authenticate to a remote
303 host. Several parameters may be included at global scope:
304 deliver-authname
306 The authname is who you want to authenticate as.
307 deliver-password
309 This is the appropriate password for authname.
310 deliver-username
312 The username is who you want to "act" as, that is, who is actually
313 going to be using the server.
314 deliver-realm
316 In this case, the "realm" is the realm in which the specified
317 authname is valid. Currently this is only needed by the DIGEST-MD5
318 SASL mechanism.
319 deliver-rcpt-to
321 A printf(3)-style format string for creating the envelope recipient
322 address. The pattern MUST include a single string specifier which
323 will be replaced with the newgroup (e.g. "bb+%s"). The default is
324 "+%s".
325 deliver-to-header
327 An optional printf(3)-style format string for creating a To: header
328 field to be prepended to the article. The pattern MUST include a
329 single string specifier which will be replaced with the newgroup
330 (e.g. "post+%s@domain"). If not specified, the To: header field
331 will not be prepended.
332
334 All the key:value pairs mentioned in this section can be specified at
335 global scope. They may also be specified inside a group or peer
336 definition. Note that when peers are added dynamically (i.e. when
337 innfeed receives an article for an unspecified peer), it will add the
338 peer site using the parameters specified at global scope.
339 Required keys
341 No keys are currently required. They all have a default value, if not
342 present in the configuration file.
343 Optional keys
345 The following keys are optional:
346 article-timeout
348 This key requires a non-negative integer value. The default value
349 is 600. If no articles need to be sent to the peer for this many
350 seconds, then the peer is considered idle and all its active
351 connections are torn down.
352 response-timeout
354 This key requires a non-negative integer value. The default value
355 is 300. It defines the maximum amount of time to wait for a
356 response from the peer after issuing a command.
357 initial-connections
359 This key requires a non-negative integer value. The default value
360 is 1. It defines the number of connections to be opened
361 immediately when setting up a peer binding. A value of 0 means no
362 connections will be created until an article needs to be sent.
363 max-connections
365 This key requires a positive integer value. The default value is 2
366 but may be increased if needed or for large feeds. It defines the
367 maximum number of connections to run in parallel to the peer. A
368 value of 0 specifies an unlimited number of maximum connections.
369 In general, use of an unlimited number of maximum connections is
370 not recommended. Do not ever set max-connections to zero with
371 dynamic-method 0 set, as this will saturate peer hosts with
372 connections.
373 close-period
375 This key requires a positive integer value and defaults to 86400.
376 It is the maximum number of seconds a connection should be kept
377 open. Some NNTP servers do not deal well with connections being
378 held open for long periods.
379 dynamic-method
381 This key requires an integer value between 0 and 3. The default
382 value is 3. It controls how connections are opened, up to the
383 maximum specified by max-connections. In general (and
384 specifically, with dynamic-method 0), a new connection is opened
385 when the current number of connections is below max-connections,
386 and an article is to be sent while no current connections are idle.
387 Without further restraint (i.e. using dynamic-method 0), in
388 practice this means that max-connections connections are
389 established while articles are being sent. Use of other dynamic-
390 method settings imposes a further limit on the amount of
391 connections opened below that specified by max-connections. This
392 limit is calculated in different ways, depending of the value of
393 dynamic-method.
394 Users should note that adding additional connections is not always
396 productive -- just because opening twice as many connections
397 results in a small percentage increase of articles accepted by the
398 remote peer, this may be at considerable resource cost both locally
399 and at the remote site, whereas the remote site might well have
400 received the extra articles sent from another peer a fraction of a
401 second later. Opening large numbers of connections is considered
402 antisocial.
403 The meanings of the various settings are:
405 0 (no method)
407 Increase of connections up to max-connections is unrestrained.
408 1 (maximize articles per second)
410 Connections are increased (up to max-connections) and decreased
411 so as to maximize the number of articles per second sent, while
412 using the fewest connections to do this.
413 2 (set target queue length)
415 Connections are increased (up to max-connections) and decreased
416 so as to keep the queue of articles to be sent within the bounds
417 set by dynamic-backlog-low and dynamic-backlog-high, while using
418 the minimum resources possible. As the queue will tend to fill
419 if the site is not keeping up, this method ensures that the
420 maximum number of articles are offered to the peer while using
421 the minimum number of connections to achieve this.
422 3 (combination)
424 This method uses a combination of methods 1 and 2 above. For
425 sites accepting a large percentage of articles, method 2 will be
426 used to ensure these sites are offered as complete a feed as
427 possible. For sites accepting a small percentage of articles,
428 method 1 is used, to minimize remote resource usage. For
429 intermediate sites, an appropriate combination is used.
430 dynamic-backlog-low
432 This key requires a floating-point value between 0 and 100. It
433 represents (as a percentage) the low water mark for the host queue.
434 If the host queue falls below this level while using dynamic-method
435 2 or 3, and if 2 or more connections are open, innfeed will attempt
436 to drop connections to the host. An Infinite Impulse Response
437 (IIR) filter is applied to the value to prevent connection flap
438 (see dynamic-filter). The default value is 20.0. This value must
439 be smaller than dynamic-backlog-high.
440 dynamic-backlog-high
442 This key requires a floating-point value between 0 and 100. It
443 represents (as a percentage) the high water mark for the host
444 queue. If the host queue rises above this level while using
445 dynamic-method 2 or 3, and if less than max-connections are open to
446 the host, innfeed will attempt to open further connections to the
447 host. An Infinite Impulse Response (IIR) filter is applied to the
448 value to prevent connection flap (see dynamic-filter). The default
449 value is 50.0. This value must be larger than dynamic-backlog-low.
450 dynamic-backlog-filter
452 This key requires a floating-point value between 0 and 1. It
453 represents the filter coefficient used by the Infinite Impulse
454 Response (IIR) filter used to implement dynamic-method 2 and 3.
455 The default value of this filter is 0.7, giving a time constant of
456 1/(1-0.7) articles. Higher values will result in slower response
457 to queue fullness changes; lower values in faster response.
458 max-queue-size
460 This key requires a positive integer value. The default value is
461 20. It defines the maximum number of articles to process at one
462 time when using streaming to transmit to a peer. Larger numbers
463 mean more memory consumed as articles usually get pulled into
464 memory (see the description of use-mmap).
465 streaming
467 This key requires a boolean value. Its default value is true. It
468 defines whether streaming commands are used to transmit articles to
469 the peers.
470 no-check-high
472 This key requires a floating-point number which must be in the
473 range [0.0, 100.0]. When running transmitting with the streaming
474 commands, innfeed attempts an optimization called "no-CHECK mode".
475 This involves not asking the peer if it wants the article, but just
476 sending it. This optimization occurs when the percentage of the
477 articles the peer has accepted gets larger than this number. If
478 this value is set to 100.0, then this effectively turns off no-
479 CHECK mode, as the percentage can never get above 100.0. If this
480 value is too small, then the number of articles the peer rejects
481 will get bigger (and your bandwidth will be wasted). The default
482 value of 95.0 usually works pretty well.
483 no-check-low
485 This key requires a floating-point number which must be in the
486 range [0.0, 100.0], and it must be smaller that the value for no-
487 check-high. When running in no-CHECK mode, as described above, if
488 the percentage of articles the remote server accepts drops below
489 this number, then the no-CHECK optimization is turned off until the
490 percentage gets above the no-check-high value again. If there is
491 small difference between this and the no-check-high value (less
492 than about 5.0), then innfeed may frequently go in and out of no-
493 CHECK mode. If the difference is too big, then it will make it
494 harder to get out of no-CHECK mode when necessary (wasting
495 bandwidth). Keeping this to between 5.0 and 10.0 less than no-
496 check-high usually works pretty well. The default value is 90.0.
497 no-check-filter
499 This is a floating-point value representing the time constant, in
500 articles, over which the CHECK/no-CHECK calculations are done. The
501 default value is 50.0, which will implement an Infinite Impulse
502 Response (IIR) filter of time constant 50. This roughly equates to
503 making a decision about the mode over the previous 50 articles. A
504 higher number will result in a slower response to changing
505 percentages of articles accepted; a lower number will result in a
506 faster response.
507 port-number
509 This key requires a positive integer value. It defines the TCP/IP
510 port number to use when connecting to the remote. Usually, port
511 number 119 is used, which is the default value.
512 force-ipv4
514 This key requires a boolean value. By default, it is set to false.
515 Setting it to true is the same as setting bindaddress6 to "none"
516 and removing bindaddress from "none" if it was set.
517 drop-deferred
519 This key requires a boolean value. By default, it is set to false.
520 When set to true, and a peer replies with code 431 or 436 (try
521 again later), innfeed just drops the article and does not try to
522 re-send it. This is useful for some peers that keep on deferring
523 articles for a long time to prevent innfeed from trying to offer
524 the same article over and over again.
525 min-queue-connection
527 This key requires a boolean value. By default, it is set to false.
528 When set to true, innfeed will attempt to use a connection with the
529 least queue size (or the first empty connection). If this key is
530 set to true, it is recommended that dynamic-method be set to 0.
531 This allows for article propagation with the least delay.
532 no-backlog
534 This key requires a boolean value. It specifies whether spooling
535 should be enabled (false, the default) or disabled (true). Note
536 that when no-backlog is set, articles reported as spooled are
537 actually silently discarded.
538 backlog-limit
540 This key requires a non-negative integer value. If the number is 0
541 (the default), then backlog files are allowed to grow without bound
542 when the peer is unable to keep up with the article flow. If this
543 number is greater than 0, then it specifies the size (in bytes) the
544 backlog file should get truncated to when the backlog file reaches
545 a certain limit. The limit depends on whether backlog-factor or
546 backlog-limit-highwater is used.
547 This parameter also applies to the debug file when debug-shrinking
549 is set to true, and has the same effect on this file as the one has
550 on backlog files.
551 backlog-factor
553 This key requires a floating-point value, which must be larger than
554 1.0. It is used in conjunction with the peer key backlog-limit.
555 If backlog-limit has a value greater than zero, then when the
556 backlog file gets larger than the value backlog-limit * backlog-
557 factor, then the backlog file will be truncated to the size
558 backlog-limit.
559 For example, if backlog-limit has a value of 1000000, and backlog-
561 factor has a value of 2.0, then when the backlog file gets to be
562 larger than 2000000 bytes in size, it will be truncated to 1000000
563 bytes. The front portion of the file is removed, and the trimming
564 happens on line boundaries, so the final size may be a bit less
565 than this number. If backlog-limit-highwater is defined too, then
566 backlog-factor takes precedence. The default value of backlog-
567 factor is 1.1.
568 This parameter also applies to the debug file when debug-shrinking
570 is set to true, and has the same effect on this file as the one has
571 on backlog files.
572 backlog-limit-highwater
574 This key requires a positive integer value that must be larger than
575 the value for backlog-limit. The default value is 0.
576 If the size of the backlog file gets larger than this value (in
578 bytes), then the backlog file will be shrunk down to the size of
579 backlog-limit. If both backlog-factor and backlog-limit-highwater
580 are defined, then the value of backlog-factor is used.
581 This parameter also applies to the debug file when debug-shrinking
583 is set to true, and has the same effect on this file as the one has
584 on backlog files.
585 backlog-feed-first
587 This key requires a boolean value. By default it is set to false.
588 When set to true, the backlog is fed before new files. This is
589 intended to enforce in-order delivery, so setting this to true when
590 initial-connections or max-connections is more than 1 is
591 inconsistent.
592 bindaddress
594 This key requires a string value. It specifies which outgoing IPv4
595 address innfeed should bind the local end of its connection to. It
596 must be an IPv4 address in dotted-quad format (nnn.nnn.nnn.nnn),
597 "any", or "none". If not set or set to "any", innfeed defaults to
598 letting the kernel choose this address. If set to "none", innfeed
599 will not use IPv4 for outgoing connections to peers in this scope
600 (i.e. it forces IPv6).
601 If not set in innfeed.conf, innfeed defaults to the value of
603 sourceaddress from inn.conf (which by default is unset).
604 bindaddress6
606 This key requires a string value. It behaves like bindaddress
607 except for outgoing IPv6 connections. It must be in numeric IPv6
608 format (note that a value containing colons must be enclosed in
609 double quotes), "any", or "none". If set to "none", innfeed will
610 not use IPv6 for outgoing connections to peers in this scope.
611 If not set in innfeed.conf, innfeed defaults to the value of
613 sourceaddress6 from inn.conf (which by default is unset).
614 username
616 This key requires a string value. If the value is defined, then
617 innfeed tries to authenticate by AUTHINFO USER and this value used
618 for user name. password must also be defined, if this key is
619 defined.
620 password
622 This key requires a string value. The value is the password used
623 for AUTHINFO PASS. username must also be defined, if this key is
624 defined.
625
627 As previously explained, the peer definitions can contain redefinitions
628 of any of the key:value pairs described in the section about global
629 peer defaults above. There is one key:value pair that is specific to a
630 peer definition.
631 ip-name
633 This key requires a word value. The word is either one of the
634 host's FQDNs, or the dotted-quad IP address of the peer for IPv4,
635 or the colon-separated IP address of the peer for IPv6. If this
636 value is not specified, then the name of the peer in the enclosing
637 peer block is taken to also be its ip-name.
638
640 If innfeed gets a SIGHUP signal, then it will reread the configuration
641 file. All values at global scope except for backlog-directory can be
642 changed (although note that bindaddress and bindaddress6 changes will
643 only affect new connections).
644 Any new peers are added and any missing peers have their connections
646 closed.
647 The log file is also reopened.
649
651 For a comprehensive example, see the sample innfeed.conf distributed
652 with INN and installed as a starting point.
653 Here are examples of how to format values:
655 eg-string: "New\tconfig\tfile\n"
657 eg-long-string: "A long string that goes
658 over multiple lines. The
659 newline is kept in the
660 string except when quoted
661 with a backslash \
662 as here."
663 eg-simple-string: A-no-quote-string
664 eg-integer: 10
665 eg-boolean: true
666 eg-char: 'a'
667 eg-ctrl-g: '\007'
668
670 Written by James Brister <brister@vix.com> for InterNetNews. Converted
671 to POD by Julien Elie.
672 Earlier versions of innfeed (up to 0.10.1) were shipped separately;
674 innfeed is now part of INN and shares the same version number. Please
675 note that the innfeed.conf format has changed dramatically since
676 version 0.9.3.
677 $Id: innfeed.conf.pod 10179 2017-09-18 20:13:48Z iulius $
679
681 inn.conf(5), innfeed(8), newsfeeds(5).
682INN 2.6.3 2018-01-28 INNFEED.CONF(5)