1INNFEED(1) General Commands Manual INNFEED(1)
2
6 innfeed - multi-host, multi-connection, streaming NNTP feeder.
7
9 innfeed [ -a spool-dir ] [ -b directory ] [ -C ] [ -c filename ] [ -d
10 num ] [ -e bytes ] [ -h ] [ -l filename ] [ -m ] [ -M ] [ -o bytes ] [
11 -p file ] [ -S file ] [ -x ] [ -y ] [ -z ] [ -v ] [ file ]
12
14 Innfeed implements the NNTP protocol for transferring news between com‐
15 puters. It handles the standard IHAVE protocol as well as the
16 CHECK/TAKETHIS streaming extension. Innfeed can feed any number of
17 remote hosts at once and will open multiple connections to each host if
18 configured to do so. The only limitations are the process limits for
19 open file descriptors and memory.
20 As an alternative to using NNTP, INN may also be fed to an IMAP server.
22 This is done by using an executable called imapfeed, which is identical
23 to innfeed except for the delivery process. The new version has two
24 types of connections: an LMTP connection to deliver regular messages
25 and an IMAP connection to handle control messages. The startinnfeed
26 process can then be told to start imapfeed instead of innfeed. (See
27 the INSTALL file for how to do this.)
28
30 Innfeed has three modes of operation: channel, funnel-file and batch.
31 Channel mode is used when no filename is given on the command line, the
33 ``input-file'' keyword is not given in the config file, and the ``-x''
34 option is not given. In channel mode innfeed runs with stdin connected
35 via a pipe to innd. Whenever innd closes this pipe (and it has several
36 reasons during normal processing to do so), innfeed will exit. It first
37 will try to finish sending all articles it was in the middle of trans‐
38 mitting, before issuing a QUIT command. This means innfeed may take a
39 while to exit depending on how slow your peers are. It never (well,
40 almost never) just drops the connection.
41 The recommended way to restart innfeed when run in channel mode is
43 therefore to tell innd to close the pipe and spawn a new innfeed
44 process. This can be done with ``ctlinnd flush <feed>'' where <feed>
45 is the name of the innfeed channel feed in ``newsfeeds''.
46 Funnel-file mode is used when a filename is given as an argument or the
48 ``input-file'' keyword is given in the config file. In funnel file
49 mode it reads the specified file for the same formatted information as
50 innd would give in channel mode. It is expected that innd is continu‐
51 ally writing to this file, so when innfeed reaches the end of the file
52 it will check periodically for new information. To prevent the funnel
53 file from growing without bounds, you will need to periodically move
54 the file to the side (or simply remove it) and have innd flush the
55 file. Then, after the file is flushed by innd, you can send innfeed a
56 SIGALRM, and it too will close the file and open the new file created
57 by innd. Something like:
58 innfeed -p /var/run/news/innfeed.pid my-funnel-file &
60 while true; do
61 sleep 43200
62 rm -f my-funnel-file
63 ctlinnd flush funnel-file-site
64 kill -ALRM `cat /var/run/news/innfeed.pid`
65 done
66 Batch mode is used when the ``-x'' flag is used. In batch mode innfeed
68 will ignore stdin, and will simply process any backlog created by a
69 previously running innfeed. This mode is not normally needed as innfeed
70 will take care of backlog processing.
71
73 Innfeed expects a couple of things to be able to run correctly: a
74 directory where it can store backlog files and a configuration file to
75 describe which peers it should handle.
76 The configuration file is described in innfeed.conf(5). The ``-c''
78 option can be used to specify a different file.
79 For each peer (say, ``foo''), innfeed manages up to 4 files in the
81 backlog directory: a ``foo.lock'' file, which prevents other instances
82 of innfeed from interfering with this one; a ``foo.input'' file which
83 has old article information innfeed is reading for re-processing; a
84 ``foo.output'' file where innfeed is writing information on articles
85 that couldn't be processed (normally due to a slow or blocked peer);
86 and a ``foo'' file.
87 This last file (``foo'') is never created by innfeed, but if innfeed
89 notices it, it will rename it to ``foo.input'' at the next opportunity
90 and will start reading from it. This lets you create a batch file and
91 put it in a place where innfeed will find it. You should never alter
92 the .input or .output files of a running innfeed.
93 The format of these last three files is one of the following:
95 /path/to/article <message-id>
97 @token@ <message-id>
98 This is the same as the first two fields of the lines innd feeds to
100 innfeed, and the same as the first two fields of the lines of the batch
101 file innd will write if innfeed is unavailable for some reason. When
102 innfeed processes its own batch files it ignores everything after the
103 first two whitespace separated fields, so moving the innd-created batch
104 file to the appropriate spot will work, even though the lines have
105 extra fields.
106 The first field can also be a storage API token. The two types of
108 lines can be intermingled; innfeed will use the storage manager if
109 appropriate and otherwise treat the first field as a filename to read
110 directly.
111 Innfeed writes its current status to the file ``innfeed.status'' (or
113 the file given by the ``-S'' option). This file contains details on the
114 process as a whole, and on each peer this instance of innfeed is manag‐
115 ing.
116 If innfeed is told to send an article to a host it is not managing,
118 then the article information will be put into a file matching the pat‐
119 tern ``innfeed-dropped.*'', with part of the file name matching the pid
120 of the innfeed process that is writing to it. Innfeed will not process
121 this file except to write to it. If nothing is written to the file then
122 it will be removed if innfeed exits normally.
123
125 Upon receipt of a SIGALRM innfeed will close the funnel-file specified
126 on the command line, and will reopen it (see funnel file description
127 above).
128 Innfeed with catch SIGINT and will write a large debugging snapshot of
130 the state of the running system.
131 Innfeed will catch SIGHUP and will reload the config file. See inn‐
133 feed.conf(5) for more details.
134 Innfeed will catch SIGCHLD and will close and reopen all backlog files.
136 Innfeed will catch SIGTERM and will do an orderly shutdown.
138 Upon receipt of a SIGUSR1 innfeed will increment the debugging level by
140 one; receipt of a SIGUSR2 will decrement it by one. The debugging level
141 starts at zero (unless the ``-d'' option it used), in which case no
142 debugging information is emitted. A larger value for the level means
143 more debugging information. Numbers up to 5 are currently useful.
144
146 There are 3 different categories of syslog entries for statistics:
147 Host, Connection and Global.
148 The Host statistics are generated for a given peer at regular intervals
150 after the first connection is made (or, if the remote is unreachable,
151 after spooling starts). The Host statistics give totals over all Con‐
152 nections that have been active during the given time frame. For example
153 (broken here to fit the page, with ``vixie'' being the peer):
154 May 23 12:49:08 data innfeed[16015]: vixie checkpoint
156 seconds 1381 offered 2744 accepted 1286
157 refused 1021 rejected 437 missing 0 spooled 990
158 on_close 0 unspooled 240 deferred 10 requeued 25
159 queue 42.1/100:14,35,13,4,24,10
160 These meanings of these fields are:
162 seconds The time since innfeed connected to the host or since the
164 statistics were reset by a ``final'' log entry.
165 offered The number of IHAVE commands sent to the host if it is not in
167 streaming mode. The sum of the number of TAKETHIS commands
168 sent when no-CHECK mode is in effect plus the number CHECK
169 commands sent in streaming mode (when no-CHECK mode is not in
170 effect).
171 accepted The number of articles which were sent to the remote host and
173 accepted by it.
174 refused The number of articles offered to the host that it it indi‐
176 cated it didn't want because it had already seen the Message-
177 ID. The remote host indicates this by sending a 435 response
178 to an IHAVE command or a 438 response to a CHECK command.
179 rejected The number of articles transferred to the host that it did
181 not accept because it determined either that it already had
182 the article or it did not want it because of the article's
183 Newsgroups: or Distribution: headers, etc. The remote host
184 indicates that it is rejecting the article by sending a 437
185 or 439 response after innfeed sent the entire article.
186 missing The number of articles which innfeed was told to offer to the
188 host but which were not present in the article spool. These
189 articles were probably cancelled or expired before innfeed
190 was able to offer them to the host.
191 spooled The number of article entries that were written to the .out‐
193 put backlog file because the articles could not either be
194 sent to the host or be refused by it. Articles are generally
195 spooled either because new articles are arriving more quickly
196 than they can be offered to the host, or because innfeed
197 closed all the connections to the host and pushed all the
198 articles currently in progress to the .output backlog file.
199 on_close The number of articles that were spooled when innfeed closed
201 all the connections to the host.
202 unspooled The number of article entries that were read from the .input
204 backlog file.
205 deferred The number of articles that the host told innfeed to retry
207 later by sending a 431 or 436 response. Innfeed immediately
208 puts these articles back on the tail of the queue.
209 requeued The number of articles that were in progress on connections
211 when innfeed dropped those connections and put the articles
212 back on the queue. These connections may have been broken by
213 a network problem or became unresponsive causing innfeed to
214 time them out.
215 queue The first number is the average (mean) queue size during the
217 previous logging interval. The second number is the maximum
218 allowable queue size. The third number is the percentage of
219 the time that the queue was empty. The fourth through sev‐
220 enth numbers are the percentages of the time that the queue
221 was >0% to 25% full, 25% to 50% full, 50% to 75% full, and
222 75% to <100% full. The last number is the percentage of the
223 time that the queue was totally full.
224 If the ``-z'' option is used (see below), then when the peer stats are
226 generated, each Connection will log its stats too. For example, for
227 connection number zero (from a set of five):
228 May 23 12:49:08 data innfeed[16015]: vixie:0 checkpoint
230 seconds 1381 offered 596 accepted 274
231 refused 225 rejected 97
232 If you only open a maximum of one Connection to a remote, then there
234 will be a close correlation between Connection numbers and Host num‐
235 bers, but in general you can't tie the two sets of number together in
236 any easy or very meaningful way. When a Connection closes it will
237 always log its stats.
238 If all Connections for a Host get closed together, then the Host logs
240 its stats as ``final'' and resets its counters. If the feed is so busy
241 that there's always at least one Connection open and running, then
242 after some amount of time (set via the config file), the Host stats are
243 logged as final and reset. This is to make generating higher level
244 stats from log files, by other programs, easier.
245 There is one log entry that is emitted for a Host just after its last
247 Connection closes and innfeed is preparing to exit. This entry contains
248 counts over the entire life of the process. The ``seconds'' field is
249 from the first time a Connection was successfully built, or the first
250 time spooling started. If a Host has been completely idle, it will have
251 no such log entry.
252 May 23 12:49:08 data innfeed[16015]: decwrl global
254 seconds 1381 offered 34 accepted 22
255 refused 3 rejected 7 missing 0
256 The final log entry is emitted immediately before exiting. It contains
258 a summary of the statistics over the entire life of the process.
259 Feb 13 14:43:41 data innfeed-0.9.4[22344]: ME global
261 seconds 15742 offered 273441 accepted 45750
262 refused 222008 rejected 3334 missing 217
263
265 -a The ``-a'' flag is used to specify the top of the article spool
266 tree. Innfeed does a chdir(2) to this directory, so it should
267 probably be an absolute path. The default is <patharti‐
268 cles in inn.conf>.
269 -b The ``-b'' flag may be used to specify a different directory for
271 backlog file storage and retrieval. If the path is relative then
272 it is relative to <pathspool in inn.conf>. The default is ``inn‐
273 feed''.
274 -c The ``-c'' flag may be used to specify a different config file
276 from the default value. If the path is relative then it is rela‐
277 tive to <pathetc in inn.conf>. The default is ``innfeed.conf''.
278 -C The ``-C'' flag is used to have innfeed simply check the config
280 file, report on any errors and then exit.
281 -d The ``-d'' flag may be used to specify the initial logging
283 level. All debugging messages go to stderr (which may not be
284 what you want, see the ``-l'' flag below).
285 -e The ``-e'' flag may be used to specify the size limit (in bytes)
287 for the .output backlog files innfeed creates. If the output
288 file gets bigger than 10% more than the given number, innfeed
289 will replace the output file with the tail of the original ver‐
290 sion. The default value is 0, which means there is no limit.
291 -h Use the ``-h'' flag to print the usage message.
293 -l The ``-l'' flag may be used to specify a different log file
295 from stderr. As innd starts innfeed with stderr attached to
296 /dev/null, using this option can be useful in catching any
297 abnormal error messages, or any debugging messages (all ``nor‐
298 mal'' errors messages go to syslog).
299 -M If innfeed has been built with mmap support, then the ``-M''
301 flag turns OFF the use of mmap(); otherwise it has no effect.
302 -m The ``-m'' flag is used to turn on logging of all missing arti‐
304 cles. Normally if an article is missing, innfeed keeps a count,
305 but logs no further information. When this flag is used, details
306 about message-id and expected pathname are logged.
307 -o The ``-o'' flag sets a value of the maximum number of bytes of
309 article data innfeed is supposed to keep in memory. This doesn't
310 work properly yet.
311 -p The ``-p'' flag is used to specify the filename to write the pid
313 of the process into. A relative path is relative to
314 <pathrun in inn.conf>. The default is ``innfeed.pid''.
315 -S The ``-S'' flag specifies the name of the file to write the
317 periodic staus to. If the path is relative it is considered rel‐
318 ative to <pathlog in inn.conf>. The default is ``innfeed.sta‐
319 tus''.
320 -v When the ``-v'' flag is given, version information is printed to
322 stderr and then innfeed exits.
323 -x The ``-x'' flag is used to tell innfeed not to expect any arti‐
325 cle information from innd but just to process any backlog files
326 that exist and then exit.
327 -y The ``-y'' flag is used to allow dynamic peer binding. If this
329 flag is used and article information is received from innd that
330 specifies an unknown peer, then the peer name is taken to be the
331 IP name too, and an association with it is created. Using this
332 it is possible to only have the global defaults in the inn‐
333 feed.conf file, provided the peername as used by innd is the
334 same as the ip name. Note that innfeed with ``-y'' and no peer
335 in innfeed.conf would cause a problem that innfeed drops the
336 first article.
337 -z The ``-z'' flag is used to cause each connection, in a parallel
339 feed configuration, to report statistics when the controller for
340 the connections prints its statistics.
341 BUGS
344 When using the ``-x'' option, the config file entry's ``initial-connec‐
346 tions'' field will be the total number of connections created and used,
347 no matter how many big the batch file, and no matter how big the ``max-
348 connectiond'' field specifies. Thus a value of 0 for ``initial-connec‐
349 tions'' means nothing will happen in ``-x'' mode.
350 Innfeed does not automatically grab the file out of out.going--this
352 needs to be prepared for it by external means.
353 Probably too many other bugs to count.
355
357 infeed.conf config file.
358 innfeed directory for backlog files.
359
361 Written by James Brister <brister@vix.com> for InterNetNews. This is
362 revision 7381, dated 2005-07-04.
363
365 innfeed.conf(5)
366 INNFEED(1)