1INNFEED(1) General Commands Manual INNFEED(1)
2
6 innfeed - multi-host, multi-connection, streaming NNTP feeder.
7
9 innfeed [ -a spool-dir ] [ -b directory ] [ -C ] [ -c filename ] [ -d
10 num ] [ -e bytes ] [ -h ] [ -l filename ] [ -m ] [ -M ] [ -o bytes ] [
11 -p file ] [ -S file ] [ -x ] [ -y ] [ -z ] [ -v ] [ file ]
12
14 Innfeed implements the NNTP protocol for transferring news between com‐
15 puters. It handles the standard IHAVE protocol as well as the
16 CHECK/TAKETHIS streaming extension. Innfeed can feed any number of
17 remote hosts at once and will open multiple connections to each host if
18 configured to do so. The only limitations are the process limits for
19 open file descriptors and memory.
20 As an alternative to using NNTP, INN may also be fed to an IMAP server.
22 This is done by using an executable called imapfeed, which is identical
23 to innfeed except for the delivery process. The new version has two
24 types of connections: an LMTP connection to deliver regular messages
25 and an IMAP connection to handle control messages.
26
28 Innfeed has three modes of operation: channel, funnel-file and batch.
29 Channel mode is used when no filename is given on the command line, the
31 ``input-file'' keyword is not given in the config file, and the ``-x''
32 option is not given. In channel mode innfeed runs with stdin connected
33 via a pipe to innd. Whenever innd closes this pipe (and it has several
34 reasons during normal processing to do so), innfeed will exit. It first
35 will try to finish sending all articles it was in the middle of trans‐
36 mitting, before issuing a QUIT command. This means innfeed may take a
37 while to exit depending on how slow your peers are. It never (well,
38 almost never) just drops the connection.
39 The recommended way to restart innfeed when run in channel mode is
41 therefore to tell innd to close the pipe and spawn a new innfeed
42 process. This can be done with ``ctlinnd flush <feed>'' where <feed>
43 is the name of the innfeed channel feed in ``newsfeeds''.
44 Funnel-file mode is used when a filename is given as an argument or the
46 ``input-file'' keyword is given in the config file. In funnel file
47 mode it reads the specified file for the same formatted information as
48 innd would give in channel mode. It is expected that innd is continu‐
49 ally writing to this file, so when innfeed reaches the end of the file
50 it will check periodically for new information. To prevent the funnel
51 file from growing without bounds, you will need to periodically move
52 the file to the side (or simply remove it) and have innd flush the
53 file. Then, after the file is flushed by innd, you can send innfeed a
54 SIGALRM, and it too will close the file and open the new file created
55 by innd. Something like:
56 innfeed -p /var/run/news/innfeed.pid my-funnel-file &
58 while true; do
59 sleep 43200
60 rm -f my-funnel-file
61 ctlinnd flush funnel-file-site
62 kill -ALRM `cat /var/run/news/innfeed.pid`
63 done
64 Batch mode is used when the ``-x'' flag is used. In batch mode innfeed
66 will ignore stdin, and will simply process any backlog created by a
67 previously running innfeed. This mode is not normally needed as innfeed
68 will take care of backlog processing.
69
71 Innfeed expects a couple of things to be able to run correctly: a
72 directory where it can store backlog files and a configuration file to
73 describe which peers it should handle.
74 The configuration file is described in innfeed.conf(5). The ``-c''
76 option can be used to specify a different file.
77 For each peer (say, ``foo''), innfeed manages up to 4 files in the
79 backlog directory: a ``foo.lock'' file, which prevents other instances
80 of innfeed from interfering with this one; a ``foo.input'' file which
81 has old article information innfeed is reading for re-processing; a
82 ``foo.output'' file where innfeed is writing information on articles
83 that couldn't be processed (normally due to a slow or blocked peer);
84 and a ``foo'' file.
85 This last file (``foo'') is never created by innfeed, but if innfeed
87 notices it, it will rename it to ``foo.input'' at the next opportunity
88 and will start reading from it. This lets you create a batch file and
89 put it in a place where innfeed will find it. You should never alter
90 the .input or .output files of a running innfeed.
91 The format of these last three files is one of the following:
93 /path/to/article <message-id>
95 @token@ <message-id>
96 This is the same as the first two fields of the lines innd feeds to
98 innfeed, and the same as the first two fields of the lines of the batch
99 file innd will write if innfeed is unavailable for some reason. When
100 innfeed processes its own batch files it ignores everything after the
101 first two whitespace separated fields, so moving the innd-created batch
102 file to the appropriate spot will work, even though the lines have
103 extra fields.
104 The first field can also be a storage API token. The two types of
106 lines can be intermingled; innfeed will use the storage manager if
107 appropriate and otherwise treat the first field as a filename to read
108 directly.
109 Innfeed writes its current status to the file ``innfeed.status'' (or
111 the file given by the ``-S'' option). This file contains details on the
112 process as a whole, and on each peer this instance of innfeed is manag‐
113 ing.
114 If innfeed is told to send an article to a host it is not managing,
116 then the article information will be put into a file matching the pat‐
117 tern ``innfeed-dropped.*'', with part of the file name matching the pid
118 of the innfeed process that is writing to it. Innfeed will not process
119 this file except to write to it. If nothing is written to the file then
120 it will be removed if innfeed exits normally.
121
123 Upon receipt of a SIGALRM innfeed will close the funnel-file specified
124 on the command line, and will reopen it (see funnel file description
125 above).
126 Innfeed with catch SIGINT and will write a large debugging snapshot of
128 the state of the running system.
129 Innfeed will catch SIGHUP and will reload the config file. See inn‐
131 feed.conf(5) for more details.
132 Innfeed will catch SIGCHLD and will close and reopen all backlog files.
134 Innfeed will catch SIGTERM and will do an orderly shutdown.
136 Upon receipt of a SIGUSR1 innfeed will increment the debugging level by
138 one; receipt of a SIGUSR2 will decrement it by one. The debugging level
139 starts at zero (unless the ``-d'' option it used), in which case no
140 debugging information is emitted. A larger value for the level means
141 more debugging information. Numbers up to 5 are currently useful.
142
144 There are 3 different categories of syslog entries for statistics:
145 Host, Connection and Global.
146 The Host statistics are generated for a given peer at regular intervals
148 after the first connection is made (or, if the remote is unreachable,
149 after spooling starts). The Host statistics give totals over all Con‐
150 nections that have been active during the given time frame. For example
151 (broken here to fit the page, with ``vixie'' being the peer):
152 May 23 12:49:08 data innfeed[16015]: vixie checkpoint
154 seconds 1381 offered 2744 accepted 1286
155 refused 1021 rejected 437 missing 0 spooled 990
156 on_close 0 unspooled 240 deferred 10 requeued 25
157 queue 42.1/100:14,35,13,4,24,10
158 These meanings of these fields are:
160 seconds The time since innfeed connected to the host or since the
162 statistics were reset by a ``final'' log entry.
163 offered The number of IHAVE commands sent to the host if it is not in
165 streaming mode. The sum of the number of TAKETHIS commands
166 sent when no-CHECK mode is in effect plus the number CHECK
167 commands sent in streaming mode (when no-CHECK mode is not in
168 effect).
169 accepted The number of articles which were sent to the remote host and
171 accepted by it.
172 refused The number of articles offered to the host that it it indi‐
174 cated it didn't want because it had already seen the Message-
175 ID. The remote host indicates this by sending a 435 response
176 to an IHAVE command or a 438 response to a CHECK command.
177 rejected The number of articles transferred to the host that it did
179 not accept because it determined either that it already had
180 the article or it did not want it because of the article's
181 Newsgroups: or Distribution: headers, etc. The remote host
182 indicates that it is rejecting the article by sending a 437
183 or 439 response after innfeed sent the entire article.
184 missing The number of articles which innfeed was told to offer to the
186 host but which were not present in the article spool. These
187 articles were probably cancelled or expired before innfeed
188 was able to offer them to the host.
189 spooled The number of article entries that were written to the .out‐
191 put backlog file because the articles could not either be
192 sent to the host or be refused by it. Articles are generally
193 spooled either because new articles are arriving more quickly
194 than they can be offered to the host, or because innfeed
195 closed all the connections to the host and pushed all the
196 articles currently in progress to the .output backlog file.
197 on_close The number of articles that were spooled when innfeed closed
199 all the connections to the host.
200 unspooled The number of article entries that were read from the .input
202 backlog file.
203 deferred The number of articles that the host told innfeed to retry
205 later by sending a 431 or 436 response. Innfeed immediately
206 puts these articles back on the tail of the queue.
207 requeued The number of articles that were in progress on connections
209 when innfeed dropped those connections and put the articles
210 back on the queue. These connections may have been broken by
211 a network problem or became unresponsive causing innfeed to
212 time them out.
213 queue The first number is the average (mean) queue size during the
215 previous logging interval. The second number is the maximum
216 allowable queue size. The third number is the percentage of
217 the time that the queue was empty. The fourth through sev‐
218 enth numbers are the percentages of the time that the queue
219 was >0% to 25% full, 25% to 50% full, 50% to 75% full, and
220 75% to <100% full. The last number is the percentage of the
221 time that the queue was totally full.
222 If the ``-z'' option is used (see below), then when the peer stats are
224 generated, each Connection will log its stats too. For example, for
225 connection number zero (from a set of five):
226 May 23 12:49:08 data innfeed[16015]: vixie:0 checkpoint
228 seconds 1381 offered 596 accepted 274
229 refused 225 rejected 97
230 If you only open a maximum of one Connection to a remote, then there
232 will be a close correlation between Connection numbers and Host num‐
233 bers, but in general you can't tie the two sets of number together in
234 any easy or very meaningful way. When a Connection closes it will
235 always log its stats.
236 If all Connections for a Host get closed together, then the Host logs
238 its stats as ``final'' and resets its counters. If the feed is so busy
239 that there's always at least one Connection open and running, then
240 after some amount of time (set via the config file), the Host stats are
241 logged as final and reset. This is to make generating higher level
242 stats from log files, by other programs, easier.
243 There is one log entry that is emitted for a Host just after its last
245 Connection closes and innfeed is preparing to exit. This entry contains
246 counts over the entire life of the process. The ``seconds'' field is
247 from the first time a Connection was successfully built, or the first
248 time spooling started. If a Host has been completely idle, it will have
249 no such log entry.
250 May 23 12:49:08 data innfeed[16015]: decwrl global
252 seconds 1381 offered 34 accepted 22
253 refused 3 rejected 7 missing 0
254 The final log entry is emitted immediately before exiting. It contains
256 a summary of the statistics over the entire life of the process.
257 Feb 13 14:43:41 data innfeed-0.9.4[22344]: ME global
259 seconds 15742 offered 273441 accepted 45750
260 refused 222008 rejected 3334 missing 217
261
263 -a The ``-a'' flag is used to specify the top of the article spool
264 tree. Innfeed does a chdir(2) to this directory, so it should
265 probably be an absolute path. The default is <patharti‐
266 cles in inn.conf>.
267 -b The ``-b'' flag may be used to specify a different directory for
269 backlog file storage and retrieval. If the path is relative then
270 it is relative to <pathspool in inn.conf>. The default is ``inn‐
271 feed''.
272 -c The ``-c'' flag may be used to specify a different config file
274 from the default value. If the path is relative then it is rela‐
275 tive to <pathetc in inn.conf>. The default is ``innfeed.conf''.
276 -C The ``-C'' flag is used to have innfeed simply check the config
278 file, report on any errors and then exit.
279 -d The ``-d'' flag may be used to specify the initial logging
281 level. All debugging messages go to stderr (which may not be
282 what you want, see the ``-l'' flag below).
283 -e The ``-e'' flag may be used to specify the size limit (in bytes)
285 for the .output backlog files innfeed creates. If the output
286 file gets bigger than 10% more than the given number, innfeed
287 will replace the output file with the tail of the original ver‐
288 sion. The default value is 0, which means there is no limit.
289 -h Use the ``-h'' flag to print the usage message.
291 -l The ``-l'' flag may be used to specify a different log file
293 from stderr. As innd starts innfeed with stderr attached to
294 /dev/null, using this option can be useful in catching any
295 abnormal error messages, or any debugging messages (all ``nor‐
296 mal'' errors messages go to syslog).
297 -M If innfeed has been built with mmap support, then the ``-M''
299 flag turns OFF the use of mmap(); otherwise it has no effect.
300 -m The ``-m'' flag is used to turn on logging of all missing arti‐
302 cles. Normally if an article is missing, innfeed keeps a count,
303 but logs no further information. When this flag is used, details
304 about message-id and expected pathname are logged.
305 -o The ``-o'' flag sets a value of the maximum number of bytes of
307 article data innfeed is supposed to keep in memory. This doesn't
308 work properly yet.
309 -p The ``-p'' flag is used to specify the filename to write the pid
311 of the process into. A relative path is relative to
312 <pathrun in inn.conf>. The default is ``innfeed.pid''.
313 -S The ``-S'' flag specifies the name of the file to write the
315 periodic staus to. If the path is relative it is considered rel‐
316 ative to <pathlog in inn.conf>. The default is ``innfeed.sta‐
317 tus''.
318 -v When the ``-v'' flag is given, version information is printed to
320 stderr and then innfeed exits.
321 -x The ``-x'' flag is used to tell innfeed not to expect any arti‐
323 cle information from innd but just to process any backlog files
324 that exist and then exit.
325 -y The ``-y'' flag is used to allow dynamic peer binding. If this
327 flag is used and article information is received from innd that
328 specifies an unknown peer, then the peer name is taken to be the
329 IP name too, and an association with it is created. Using this
330 it is possible to only have the global defaults in the inn‐
331 feed.conf file, provided the peername as used by innd is the
332 same as the ip name. Note that innfeed with ``-y'' and no peer
333 in innfeed.conf would cause a problem that innfeed drops the
334 first article.
335 -z The ``-z'' flag is used to cause each connection, in a parallel
337 feed configuration, to report statistics when the controller for
338 the connections prints its statistics.
339 BUGS
342 When using the ``-x'' option, the config file entry's ``initial-connec‐
344 tions'' field will be the total number of connections created and used,
345 no matter how many big the batch file, and no matter how big the ``max-
346 connectiond'' field specifies. Thus a value of 0 for ``initial-connec‐
347 tions'' means nothing will happen in ``-x'' mode.
348 Innfeed does not automatically grab the file out of out.going--this
350 needs to be prepared for it by external means.
351 Probably too many other bugs to count.
353
355 innfeed.conf config file.
356 innfeed directory for backlog files.
357
359 Written by James Brister <brister@vix.com> for InterNetNews. This is
360 revision 7799, dated 2008-04-26.
361
363 innfeed.conf(5)
364 INNFEED(1)