1XFERSTATS(8) System Manager's Manual XFERSTATS(8)
2
6 xferstats - compiles information about file transfers from logfiles
7
9 xferstats [- | + ] [-c <config file>] [-T <number>] [[-+]C] [[-+]H]
10 [[-+]n] [[-+]s] [[-+]r] [[-+]a] [[-+]g] [[-+]i] [[-+]u] [[-+]h] [[-+]m]
11 [[-+]w] [[-+]M] [[-+]f [number]] [[-+]d [number]] [[-+]t [number]]
12 [[-+]O [number]] [[-+]o [number]] [+L <number>] [+A] [-D <number>] [+D
13 <number>] [+P <prefix>] [+B <timestamp>] [+E <timestamp>] [-v] [--ver‐
14 sion] [--help] [--] [filename(s)]
15
18 xferstats is a utility that parses wu-ftpd, ncftpd, and (special)
19 apache xferlogs (although it's not difficult to add support for other
20 log types) and gives various information and totals about the data
21 transferred.
22
25 + The single plus tells xferstats to get the xferlog from stdin.
26 For example:
27 cat xferlog | xferstats +
29 - The single hyphen tells xferstats to get the xferlog from a
31 file. While this is the default behaviour, the configuration
32 file may change the default, and this option allows you to over‐
33 ride it. For example:
34 xferstats - /var/log/xferlog
36 -c configuration file
38 Specify an alternate path and filename for the configuration
39 file. The packaged default is /etc/xferstats.cfg but your
40 administrator may have changed this. A new default can be set
41 by changing the appropriate variable in xferstats.h and recom‐
42 piling.
43 -T log type
45 Specify the type of log file to be parsed. Currently, only
46 three types are defined, wu-ftpd for wu-ftpd-style logs, ncftpd
47 for ncftpd-style logs, and apache for special Apache logs.
48 Note: you can also use "wu-ftp", "wuftp", "wuftpd", or "ncftp".
49 In order for xferstats to parse Apache logfiles, you must use a
50 special LogFile line: LogFormat "%{%a %b %d %X %Y}t %T %h %b %f
51 %u". xferstats may or may not support regular Apache logfiles
52 in the future. If this is something you want, please let me
53 know.
54 -v, --version
56 Display what version of xferstats you're running.
57 --help Display a quick summary of the command line options.
59 -- Use a -- (double hyphen) to indicate that you are finished giv‐
61 ing parameters and will follow with a filename. This is partic‐
62 ularly useful when you want to specify a filename that begins
63 with a + or -
64 TOGGLE OPTIONS
67 The options in this section can be either forced on with +[option] or
68 forced off with -[option]. Options in this section, like all other
69 options, will always override any settings in the xferstats configura‐
70 tion file. Any arguments to toggle options should only be used when
71 enabling an option. Supplying an argument when disabling an option
72 will generate an error.
73 C When enabled, xferstats will try to weed out corrupt log lines
76 by strictly checking to make sure that the data is valid. If
77 you --enable-debug_flow at compile time, xferstats will notify
78 you when corrupt lines are detected.
79 H When enabled, xferstats will generate HTML tables and graphs
81 instead of text tables. In order to actually generate tables
82 instead of broken graphic icons, you must put the graph JPGs in
83 the directory pointed to by the GRAPH_PATH configuration com‐
84 mand. For more information, see the CONFIGURATION FILE section.
85 n When enabled, xferstats will refrain from adding <HTML>, <BODY>,
87 and <TITLE> tags, so that the output can be inserted into pages
88 containing custom headers.
89 s When enabled, xferstats will output HTML as a single page (file‐
91 name "xferstats.html") instead of a page for each section. This
92 option also disables <HTML> and <BODY> tags, although it does
93 produce a <TITLE> tag unless the NO_HTML_HEADERS option is also
94 specified.
95 r When enabled, xferstats will generate statistics about real user
97 connections (but not necessarily only real user connections)
98 a When enabled, xferstats will generate statistics about anonymous
100 connections (but not necessarily only anonymous connections)
101 g When enabled, xferstats will generate statistics about guest
103 connections (but not necessarily only guest connections)
104 i When enabled, xferstats will generate statistics about inbound
106 (the product of a PUT) files (but not necessarily only inbound
107 files)
108 u When enabled, xferstats will generate statistics about outbound
110 (the product of a GET) files (but not necessarily only outbound
111 files)
112 h When enabled, xferstats will generate a report listing traffic
114 by hour
115 m When enabled, xferstats will generate a report listing traffic
117 by month (January, February, etc.)
118 S When enabled, xferstats will generate a report listing traffic
120 by file size
121 w When enabled, xferstats will generate a report listing traffic
123 by day of the week (Monday, Tuesday, etc.)
124 M When enabled, xferstats will generate a report listing traffic
126 by day of the month (1st, 2nd, 3rd, etc.)
127 f [number]
129 When enabled, xferstats will generate a report listing traffic
130 by file. You can supply an optional argument that will deter‐
131 mine how many file records xferstats displays (0 (zero) disables
132 the limit)
133 d [number]
135 When enabled, xferstats will generate a report listing traffic
136 by directory. You can supply an optional argument that will
137 determine how many directory records xferstats displays (0
138 (zero) disables the limit). See the -D and +D options, as well
139 as the DIR_DEPTH and DIR_FILTER configuration commands for more
140 information.
141 t [number]
143 When enabled, xferstats will generate a report listing traffic
144 by top-level domain. You can supply an optional argument that
145 will determine how many TLD records xferstats displays (0 (zero)
146 disables the limit)
147 O [number]
149 When enabled, xferstats will generate a report listing traffic
150 by domain. You can supply an optional argument that will deter‐
151 mine how many domain records xferstats displays (0 (zero) dis‐
152 ables the limit)
153 o [number]
155 When enabled, xferstats will generate a report listing traffic
156 by host. You can supply an optional argument that will deter‐
157 mine how many host records xferstats displays (0 (zero) disables
158 the limit)
159 OTHER OPTIONS
162 +A Include all users, generate all reports
163 +L <number>
165 Limit the daily report to the last <number> days
166 -D <number>
168 Depth of path detail for directories (default 3)
169 +D <directory>
171 Generate reports on files located only in <directory>. For
172 example:
173 +D /pub will report only on paths under /pub. Specifying multi‐
175 ple +D options will instruct xferstats to include data from all
176 specified directories.
177 +P <pathprefix>
179 Strip the <prefix> from any pathnames that begin with it. This
180 is the same as specifying a STRIP_PREFIX configuration file com‐
181 mand. Keep in mind that the DIR_FILTER (or +D) is checked
182 before the STRIP_PREFIX (or +P) when using these.
183 +B <timestamp>
185 Providing a +B argument instructs xferstats to only generate
186 data on transfers occuring after that time. The time stamp for‐
187 mat is "Jan DD HH:MM:ss YYYY". One may combine +B and +E to
188 generate a report about a specific window in time.
189 +E <timestamp>
191 Providing a +E argument instructs xferstats to only generate
192 data on transfers occuring before that time. The time stamp
193 format is "Jan DD HH:MM:ss YYYY". One may combine +B and +E to
194 generate a report about a specific window in time.
195 The default compile-time setting for the xferlog location is
197 /var/log/xferlog but your administrator may have changed this. If
198 xferstats reports an error, try specifying the filename as the last
199 command line parameter.
200
203 The default configuration file path is /etc/xferstats.cfg although your
204 administrator may have changed this.
205 Any text after a '#' is ignored, as long as the '#' begins a line or is
207 prepended by whitespace.
208 Commands (LOGFILE, ANON_TRAFFIC, etc) are not case sensitive, although
210 some arguments may be.
211 LOGFILE <filename>
213 sets the default log file to process. It can be overridden by
214 adding "<filename>" at the end of the command line options. You
215 may specify as many LOGFILE lines as you like; they will be pro‐
216 cessed in the order specified (although the order processed
217 should not change the output). Example:
218 LOGFILE /var/log/xferlog LOGFILE /var/log/xferlog.1 LOGFILE
220 /var/log/xferlog.2
221 STRICT_CHECK
223 takes no arguments. When enabled, xferstats will try to weed
224 out corrupt log lines by strictly checking to make sure that the
225 data is valid. If you --enable-debug_flow at compile time,
226 xferstats will notify you when corrupt lines are detected. It
227 can be enabled with "+C" or disabled with "-C" on the command
228 line.
229 ANON_TRAFFIC
231 takes no arguments. When enabled, xferstats will generate sta‐
232 tistics about anonymous connections (but not necessarily only
233 anonymous connections). It can be enabled with "+a" or disabled
234 with "-a" on the command line.
235 GUEST_TRAFFIC
237 takes no arguments. When enabled, xferstats will generate sta‐
238 tistics about guest connections (but not necessarily only guest
239 connections). This option has no effect when parsing non-wuftpd
240 logfiles. It can be enabled with "+g" or disabled with "-g" on
241 the command line.
242 REAL_TRAFFIC
244 takes no arguments. When enabled, xferstats will generate sta‐
245 tistics about real user connections (but not necessarily only
246 real user connections). It can be enabled with "+r" or disabled
247 with "-r" on the command line.
248 INBOUND
250 takes no arguments. When enabled, xferstats will generate sta‐
251 tistics about inbound (the product of a PUT) files (but not nec‐
252 essarily only inbound files). It can be enabled with "+i" or
253 disabled with "-i" on the command line.
254 OUTBOUND
256 takes no arguments. When enabled, xferstats will generate sta‐
257 tistics about outbound (the product of a GET) files (but not
258 necessarily only outbound files). It can be enabled with "+u" or
259 disabled with "-u" on the command line.
260 HOURLY_REPORT
262 takes no arguments. When enabled, xferstats will generate a
263 report listing traffic per hour. It can be enabled with "+h" or
264 disabled with "-h" on the command line.
265 DOW_REPORT
267 takes no arguments. When enabled, xferstats will generate a
268 report listing traffic by day of the week (Monday, Tuesday,
269 etc). It can be enabled with "+w" or disabled with "-w" on the
270 command line.
271 DOM_REPORT
273 takes no arguments. When enabled, xferstats will generate a
274 report listing traffic by day of the month (1st, 2nd, etc). It
275 can be enabled with "+M" or disabled with "-M" on the command
276 line.
277 TLD_REPORT
279 takes no arguments. When enabled, xferstats will generate a
280 report listing traffic by top-level domain (com, edu, net, us,
281 uk, etc). It can be enabled with "+t <number>" or disabled with
282 "-t" on the command line.
283 DOMAIN_REPORT
285 takes no arguments. When enabled, xferstats will generate a
286 report listing traffic by domain (foo.com, blah.edu, etc). It
287 can be enabled with "+O <number>" or disabled with "-O" on the
288 command line.
289 HOST_REPORT
291 takes no arguments. When enabled, xferstats will generate a
292 report listing traffic by host. It can be enabled with "+o" or
293 disabled with "-o" on the command line.
294 DIR_REPORT
296 takes no arguments. When enabled, xferstats will generate a
297 report listing traffic by directory. The directory depth that
298 xferstats will keep track of can be controlled by the DIR_DEPTH
299 configuration command or with the "-l <depth>" command line
300 option. It can be enabled with "+d" or disabled with "-d" on
301 the command line.
302 FILE_REPORT
304 takes no arguments. When enabled, xferstats will generate a
305 report listing traffic by file. It can be enabled with "+f" or
306 disabled with "-f" on the command line.
307 MONTHLY_REPORT
309 takes no arguments. When enabled, xferstats will generate a
310 report listing traffic per month (January, February, etc). It
311 can be enabled with "+m" or disabled with "-m" on the command
312 line.
313 HTML_OUTPUT
315 takes no arguments. When enabled, xferstats will output all
316 reports into individual HTML files. Future versions will con‐
317 tain a configuration option to set the directory in which to
318 place the HTML files. For now they are placed in the present
319 working# directory. It can be enabled with "+H" or disabled with
320 "-H" on the command line.
321 USE_STDIN
323 takes no arguments. When enabled, xferstats will, by default,
324 read the logfile from the standard input instead of from a file
325 on disk. It can be enabled with "+" or disabled with "-" on the
326 command line.
327 LOG_TYPE
329 takes one argument, the type of log that will be processed by
330 default. The argument can be either a number or a case insensi‐
331 tive string and are as follows:
332 wu-ftpd log: "1" or "wu-ftp" or "wu-ftpd" or "wuftp" or "wuftpd"
334 ncftpd log : "2" or "ncftp" or "ncftpd"
336 Apache log : "3" or "apache"
338 Therefore:
340 LOG_TYPE 1
342 LOG_TYPE wu-ftp
344 LOG_TYPE wu-ftpd
346 LOG_TYPE wuftp
348 LOG_TYPE wuftpd
350 are all equivalent.
352 Future versions will contain support for additional logfile
354 types. It can be set on the command line with "-T <number>"
355 DIR_DEPTH
357 takes one argument, the number of subdirectories deep that xfer‐
358 stats should go when generating statistics by directory. This
359 configuration command has no effect when DIR_REPORT has not been
360 enabled. DIR_DEPTH has no upper bound. A DIR_DEPTH of '0'
361 (zero) tells xferstats to process all subdirectories. The
362 default value is 3. It can be set on the command line with "-D
363 <number>". Example:
364 DIR_DEPTH 3
366 DIR_FILTER
368 takes one argument, the directory that xferstats should report
369 on. xferstats will discard any data that does not have a path
370 that begins with the value set by DIR_FILTER. This value is
371 case-sensitive, as it is part of a path name. Specifying multi‐
372 ple statements will instruct xferstats to include data from all
373 specified directories. It can be set on the command line with
374 "+D <path>". Example:
375 DIR_FILTER /pub/foobar
377 NUMBER_FILE_STATS
379 takes one argument, the number of files to report on in the file
380 statistics section. It must be a number greater than or equal
381 to zero, with zero denoting no limit. It can be set on the com‐
382 mand line with "+f <number>". The default is 50.
383 NUMBER_FILE_STATS 50
385 NUMBER_DIR_STATS
387 takes one argument, the number of directories to report on in
388 the directory statistics section. It must be a number greater
389 than or equal to zero, with zero denoting no limit. It can be
390 set on the command line with "+d <number>". The default is 50.
391 NUMBER_DAILY_STATS
393 takes one argument, the number of days to report on in the daily
394 statistics section. It must be a number greater than or equal
395 to zero, with zero denoting no limit. It can be set on the com‐
396 mand line with "+L <number>". The default is 50.
397 Note: the daily statistics section will always be displayed.
399 NUMBER_TLD_STATS
401 takes one argument, the number of top-level domains to report on
402 in the TLD statistics section. It must be a number greater than
403 or equal to zero, with zero denoting no limit. It can be set on
404 the command line with "+t <number>". The default is 50.
405 NUMBER_DOMAIN_STATS
407 takes one argument, the number of top-level domains to report on
408 in the domain statistics section. It must be a number greater
409 than or equal to zero, with zero denoting no limit. It can be
410 set on the command line with "+O <number>". The default is 50.
411 NUMBER_HOST_STATS
413 takes one argument, the number of hosts to report on in the host
414 statistics section. It must be a number greater than or equal
415 to zero, with zero denoting no limit. It can be set on the com‐
416 mand line with "+o <number>". The default is 50.
417 MAX_REPORT_SIZE
419 takes one argument, the maximum size of any one output table.
420 This is particularly useful to keep xferstats from generating
421 gigantic tables that make graphical browsers choke. It will
422 split tables to be no larger than# this value. It must be
423 greater than or equal to ten, or zero, zero denoting no limit.
424 It can be set on the command line with "+R <number>". The
425 default is 30.
426 GRAPH_PATH
428 takes one argument, the location of the JPGs used to draw graphs
429 in the HTML output. It can be an absolute path (ie,
430 http://foo.bar.com/graphs/) or a relative path (ie, ../graphs/).
431 You must include a trailing '/' or it will not work. This
432 option cannot be set on the command line. The default is noth‐
433 ing (which will make the browser look for the graphs in the
434 directory with the HTML). Example:
435 GRAPH_PATH ../graphs/
437 NO_HTML_HEADERS
439 takes no arguments. When enabled, xferstats will refrain from
440 adding <HTML>, <BODY>, and <TITLE> tags, so that the output can
441 be inserted into pages containing custom headers. It can be
442 enabled with "+n" or disabled with "-n" on the command line.
443 REFRESH
445 takes one argument, the number of seconds after being displayed
446 that the browser should refresh it. Unspecified or zero indi‐
447 cates no refresh.
448 SINGLE_PAGE
450 takes no arguments. When enabled, xferstats will output HTML as
451 a single page (entitled "xferstats.html") instead of a page for
452 each section. This option also disables <HTML> and <BODY> tags,
453 although it does produce a <TITLE> tag unless the NO_HTML_HEAD‐
454 ERS option is also specified. It can be enabled with "+s" or
455 disabled with "-s" on the command line.
456 STRIP_PREFIX
458 takes one argument. When specified, xferstats will strip the
459 given string from the beginning of any paths. Keep in mind when
460 using this option with DIR_FILTER that the DIR_FILTER is applied
461 -before- stripping, so that you must take this into account when
462 specifying the DIR_FILTER string. This option cannot be set on
463 the command line. Example:
464 STRIP_PREFIX /home/ftp
466 The next 11 entries are all basically the same--they define how the
468 reports are to be sorted.
469 0 -- Sort by the number of bytes downloaded
471 1 -- Sort by the number of files downloaded
473 2 -- Sort by the "name". This varies by report; for the file,
475 directory, TLD, etc reports, it sorts them alphabetically; for
476 the file size report it lists the file sizes in alphabetical
477 order; for the daily report, it lists the days in order. You
478 get the idea. 3 -- Sort by average throughput, in K/s
479 FILE_SORT_PREF, DIR_SORT_PREF, DOMAIN_SORT_PREF, TLD_SORT_PREF,
481 HOST_SORT_PREF, DOM_SORT_PREF, DOW_SORT_PREF, HOURLY_SORT_PREF,
482 DAILY_SORT_PREF, SIZE_SORT_PREF, MONTHLY_SORT_PREF
483 CHUNK_INPUT
485 takes one argument. When specified, xferstats will parse the
486 logfiles in chunks of the given size (in number of lines),
487 process them, and continue. This will keep it from trying to
488 parse the entire log at once, using up a lot of memory for large
489 logs. Specifying too small a value will cause xferstats to
490 waste a lot of time switching tasks, while too large a value may
491 slow it down as your system swaps. Toy with this value a bit
492 and see what works best for you. If left unspecified, xferstats
493 will continue to parse the logs in one pass.
494
497 xferlog (generated by wu-ftpd or ncftpd)
498 /etc/xferstats.cfg contains all of the default settings and many set‐
500 tings which cannot be set from the command line.
501 If you installed the RPM version of xferstats, you'll find the JPGs for
503 the graphs in /usr/share/xferstats/graphs.
504
507 The JPG graph pieces included are not ideal; sometimes artifacts show
508 up. I will eventually get around to making new GIFs or something.
509 Please send bug reports to phil@off.net If the xferlog that you're
511 using is smaller than 1 megabyte, please send a gzipped copy, along
512 with the exact parameters you're using.
513
516 I greatly appreciate the suggestions and assistance of Nat Friedman
517 <ndf@mit.edu>.
518 Thanks to Mike Shaver <shaver@mozilla.org> and Jamie Zawinski <jwz@net‐
520 scape.com> for coughing up gigantic mozilla.org logs to aid my testing.
521 Thanks to Phil Copeland <bryce@zen.ics.uwe.ac.uk> for getting on my
523 back constantly to add new stuff, fix nagging bugs, and speed it up.
524 Oh, and for using it, at least someone does :).
525 Extreme gratitude to Mark Hagler <hagler@hagler.org>, someone whom I've
527 left off of this list for too long. Mark was kind enough to Amazon me
528 a copy of O'Reillys _Pthreads Programming_ which was the catalyst for
529 the threaded version and really helped get me interested in the project
530 again. While the threaded version doesn't exist anymore, the book is
531 still terribly useful for other projects.
532
534 ftpd(8), ncftpd(8), xferlog(5)
535
538 xferstats is available from the following locations. The old addresses
539 are valid, but are deprecated! Please use the new addresses whenever
540 possible.
541 http://xferstats.off.net/
543 ftp://xferstats.off.net/pub/xferstats/
545
548 xferstats is Copyright (c) 1997-2000 Phil Schwan <phil@off.net> and is
549 released under the terms and provisions of the GNU Public License.
550 31 Oct 2000 XFERSTATS(8)