1XFERSTATS(8) System Manager's Manual XFERSTATS(8)
2
6 xferstats - compiles information about file transfers from logfiles
7
9 xferstats [- | + ] [-c <config file>] [-T <number>] [[-+]C] [[-+]H]
10 [[-+]n] [[-+]s] [[-+]r] [[-+]a] [[-+]g] [[-+]i] [[-+]u] [[-+]h] [[-+]m]
11 [[-+]w] [[-+]M] [[-+]f [number]] [[-+]d [number]] [[-+]t [number]]
12 [[-+]O [number]] [[-+]o [number]] [+L <number>] [+A] [-D <number>] [+D
13 <number>] [+P <prefix>] [+B <timestamp>] [+E <timestamp>] [-v] [--ver‐
14 sion] [--help] [--] [filename(s)]
15
18 xferstats is a utility that parses wu-ftpd, ncftpd, and (special)
19 apache xferlogs (although it's not difficult to add support for other
20 log types) and gives various information and totals about the data
21 transferred.
22
25 + The single plus tells xferstats to get the xferlog from stdin.
26 For example:
27 cat xferlog | xferstats +
29 - The single hyphen tells xferstats to get the xferlog from a
31 file. While this is the default behaviour, the configuration
32 file may change the default, and this option allows you to over‐
33 ride it. For example:
34 xferstats - /var/log/xferlog
36 -c configuration file
38 Specify an alternate path and filename for the configuration
39 file. The packaged default is /etc/xferstats.cfg but your
40 administrator may have changed this. A new default can be set
41 by changing the appropriate variable in xferstats.h and recom‐
42 piling.
43 -T log type
45 Specify the type of log file to be parsed. Currently, only
46 three types are defined, wu-ftpd for wu-ftpd-style logs, ncftpd
47 for ncftpd-style logs, and apache for special Apache logs.
48 Note: you can also use "wu-ftp", "wuftp", "wuftpd", or "ncftp".
49 In order for xferstats to parse Apache logfiles, you must use a
50 special LogFile line: LogFormat "%{%a %b %d %X %Y}t %T %h %b %f
51 %u". xferstats may or may not support regular Apache logfiles
52 in the future. If this is something you want, please let me
53 know.
54 -v, --version
56 Display what version of xferstats you're running.
57 --help Display a quick summary of the command line options.
59 -- Use a -- (double hyphen) to indicate that you are finished giv‐
61 ing parameters and will follow with a filename. This is partic‐
62 ularly useful when you want to specify a filename that begins
63 with a + or -
64 TOGGLE OPTIONS
67 The options in this section can be either forced on with +[option] or
68 forced off with -[option]. Options in this section, like all other
69 options, will always override any settings in the xferstats configura‐
70 tion file. Any arguments to toggle options should only be used when
71 enabling an option. Supplying an argument when disabling an option
72 will generate an error.
73 C When enabled, xferstats will try to weed out corrupt log lines
76 by strictly checking to make sure that the data is valid. If
77 you --enable-debug_flow at compile time, xferstats will notify
78 you when corrupt lines are detected.
79 H When enabled, xferstats will generate HTML tables and graphs
81 instead of text tables. In order to actually generate tables
82 instead of broken graphic icons, you must put the graph JPGs in
83 the directory pointed to by the GRAPH_PATH configuration com‐
84 mand. For more information, see the CONFIGURATION FILE section.
85 n When enabled, xferstats will refrain from adding <HTML>, <BODY>,
87 and <TITLE> tags, so that the output can be inserted into pages
88 containing custom headers.
89 s When enabled, xferstats will output HTML as a single page (file‐
91 name "xferstats.html") instead of a page for each section. This
92 option also disables <HTML> and <BODY> tags, although it does
93 produce a <TITLE> tag unless the NO_HTML_HEADERS option is also
94 specified.
95 r When enabled, xferstats will generate statistics about real user
97 connections (but not necessarily only real user connections)
98 a When enabled, xferstats will generate statistics about anonymous
100 connections (but not necessarily only anonymous connections)
101 g When enabled, xferstats will generate statistics about guest
103 connections (but not necessarily only guest connections)
104 i When enabled, xferstats will generate statistics about inbound
106 (the product of a PUT) files (but not necessarily only inbound
107 files)
108 u When enabled, xferstats will generate statistics about outbound
110 (the product of a GET) files (but not necessarily only outbound
111 files)
112 h When enabled, xferstats will generate a report listing traffic
114 by hour
115 m When enabled, xferstats will generate a report listing traffic
117 by month (January, February, etc.)
118 w When enabled, xferstats will generate a report listing traffic
120 by day of the week (Monday, Tuesday, etc.)
121 M When enabled, xferstats will generate a report listing traffic
123 by day of the month (1st, 2nd, 3rd, etc.)
124 f [number]
126 When enabled, xferstats will generate a report listing traffic
127 by file. You can supply an optional argument that will deter‐
128 mine how many file records xferstats displays (0 (zero) disables
129 the limit)
130 d [number]
132 When enabled, xferstats will generate a report listing traffic
133 by directory. You can supply an optional argument that will
134 determine how many directory records xferstats displays (0
135 (zero) disables the limit). See the -D and +D options, as well
136 as the DIR_DEPTH and DIR_FILTER configuration commands for more
137 information.
138 t [number]
140 When enabled, xferstats will generate a report listing traffic
141 by top-level domain. You can supply an optional argument that
142 will determine how many TLD records xferstats displays (0 (zero)
143 disables the limit)
144 O [number]
146 When enabled, xferstats will generate a report listing traffic
147 by domain. You can supply an optional argument that will deter‐
148 mine how many domain records xferstats displays (0 (zero) dis‐
149 ables the limit)
150 o [number]
152 When enabled, xferstats will generate a report listing traffic
153 by host. You can supply an optional argument that will deter‐
154 mine how many host records xferstats displays (0 (zero) disables
155 the limit)
156 OTHER OPTIONS
159 +A Include all users, generate all reports
160 +L <number>
162 Limit the daily report to the last <number> days
163 -D <number>
165 Depth of path detail for directories (default 3)
166 +D <directory>
168 Generate reports on files located only in <directory>. For
169 example:
170 +D /pub will report only on paths under /pub. Specifying multi‐
172 ple +D options will instruct xferstats to include data from all
173 specified directories.
174 +P <pathprefix>
176 Strip the <prefix> from any pathnames that begin with it. This
177 is the same as specifying a STRIP_PREFIX configuration file com‐
178 mand. Keep in mind that the DIR_FILTER (or +D) is checked
179 before the STRIP_PREFIX (or +P) when using these.
180 +B <timestamp>
182 Providing a +B argument instructs xferstats to only generate
183 data on transfers occuring after that time. The time stamp for‐
184 mat is "Jan DD HH:MM:ss YYYY". One may combine +B and +E to
185 generate a report about a specific window in time.
186 +E <timestamp>
188 Providing a +E argument instructs xferstats to only generate
189 data on transfers occuring before that time. The time stamp
190 format is "Jan DD HH:MM:ss YYYY". One may combine +B and +E to
191 generate a report about a specific window in time.
192 The default compile-time setting for the xferlog location is
194 /var/log/xferlog but your administrator may have changed this. If
195 xferstats reports an error, try specifying the filename as the last
196 command line parameter.
197
200 The default configuration file path is /etc/xferstats.cfg although your
201 administrator may have changed this.
202 Any text after a '#' is ignored, as long as the '#' begins a line or is
204 prepended by whitespace.
205 Commands (LOGFILE, ANON_TRAFFIC, etc) are not case sensitive, although
207 some arguments may be.
208 LOGFILE <filename>
210 sets the default log file to process. It can be overridden by
211 adding "<filename>" at the end of the command line options. You
212 may specify as many LOGFILE lines as you like; they will be pro‐
213 cessed in the order specified (although the order processed
214 should not change the output). Example:
215 LOGFILE /var/log/xferlog LOGFILE /var/log/xferlog.1 LOGFILE
217 /var/log/xferlog.2
218 STRICT_CHECK
220 takes no arguments. When enabled, xferstats will try to weed
221 out corrupt log lines by strictly checking to make sure that the
222 data is valid. If you --enable-debug_flow at compile time,
223 xferstats will notify you when corrupt lines are detected. It
224 can be enabled with "+C" or disabled with "-C" on the command
225 line.
226 ANON_TRAFFIC
228 takes no arguments. When enabled, xferstats will generate sta‐
229 tistics about anonymous connections (but not necessarily only
230 anonymous connections). It can be enabled with "+a" or disabled
231 with "-a" on the command line.
232 GUEST_TRAFFIC
234 takes no arguments. When enabled, xferstats will generate sta‐
235 tistics about guest connections (but not necessarily only guest
236 connections). This option has no effect when parsing non-wuftpd
237 logfiles. It can be enabled with "+g" or disabled with "-g" on
238 the command line.
239 REAL_TRAFFIC
241 takes no arguments. When enabled, xferstats will generate sta‐
242 tistics about real user connections (but not necessarily only
243 real user connections). It can be enabled with "+r" or disabled
244 with "-r" on the command line.
245 INBOUND
247 takes no arguments. When enabled, xferstats will generate sta‐
248 tistics about inbound (the product of a PUT) files (but not nec‐
249 essarily only inbound files). It can be enabled with "+i" or
250 disabled with "-i" on the command line.
251 OUTBOUND
253 takes no arguments. When enabled, xferstats will generate sta‐
254 tistics about outbound (the product of a GET) files (but not
255 necessarily only outbound files). It can be enabled with "+u" or
256 disabled with "-u" on the command line.
257 HOURLY_REPORT
259 takes no arguments. When enabled, xferstats will generate a
260 report listing traffic per hour. It can be enabled with "+h" or
261 disabled with "-h" on the command line.
262 DOW_REPORT
264 takes no arguments. When enabled, xferstats will generate a
265 report listing traffic by day of the week (Monday, Tuesday,
266 etc). It can be enabled with "+w" or disabled with "-w" on the
267 command line.
268 DOM_REPORT
270 takes no arguments. When enabled, xferstats will generate a
271 report listing traffic by day of the month (1st, 2nd, etc). It
272 can be enabled with "+M" or disabled with "-M" on the command
273 line.
274 TLD_REPORT
276 takes no arguments. When enabled, xferstats will generate a
277 report listing traffic by top-level domain (com, edu, net, us,
278 uk, etc). It can be enabled with "+t <number>" or disabled with
279 "-t" on the command line.
280 DOMAIN_REPORT
282 takes no arguments. When enabled, xferstats will generate a
283 report listing traffic by domain (foo.com, blah.edu, etc). It
284 can be enabled with "+O <number>" or disabled with "-O" on the
285 command line.
286 HOST_REPORT
288 takes no arguments. When enabled, xferstats will generate a
289 report listing traffic by host. It can be enabled with "+o" or
290 disabled with "-o" on the command line.
291 DIR_REPORT
293 takes no arguments. When enabled, xferstats will generate a
294 report listing traffic by directory. The directory depth that
295 xferstats will keep track of can be controlled by the DIR_DEPTH
296 configuration command or with the "-l <depth>" command line
297 option. It can be enabled with "+d" or disabled with "-d" on
298 the command line.
299 FILE_REPORT
301 takes no arguments. When enabled, xferstats will generate a
302 report listing traffic by file. It can be enabled with "+f" or
303 disabled with "-f" on the command line.
304 MONTHLY_REPORT
306 takes no arguments. When enabled, xferstats will generate a
307 report listing traffic per month (January, February, etc). It
308 can be enabled with "+m" or disabled with "-m" on the command
309 line.
310 HTML_OUTPUT
312 takes no arguments. When enabled, xferstats will output all
313 reports into individual HTML files. Future versions will con‐
314 tain a configuration option to set the directory in which to
315 place the HTML files. For now they are placed in the present
316 working# directory. It can be enabled with "+H" or disabled with
317 "-H" on the command line.
318 USE_STDIN
320 takes no arguments. When enabled, xferstats will, by default,
321 read the logfile from the standard input instead of from a file
322 on disk. It can be enabled with "+" or disabled with "-" on the
323 command line.
324 LOG_TYPE
326 takes one argument, the type of log that will be processed by
327 default. The argument can be either a number or a case insensi‐
328 tive string and are as follows:
329 wu-ftpd log: "1" or "wu-ftp" or "wu-ftpd" or "wuftp" or "wuftpd"
331 ncftpd log : "2" or "ncftp" or "ncftpd"
333 Apache log : "3" or "apache"
335 Therefore:
337 LOG_TYPE 1
339 LOG_TYPE wu-ftp
341 LOG_TYPE wu-ftpd
343 LOG_TYPE wuftp
345 LOG_TYPE wuftpd
347 are all equivalent.
349 Future versions will contain support for additional logfile
351 types. It can be set on the command line with "-T <number>"
352 DIR_DEPTH
354 takes one argument, the number of subdirectories deep that xfer‐
355 stats should go when generating statistics by directory. This
356 configuration command has no effect when DIR_REPORT has not been
357 enabled. DIR_DEPTH has no upper bound. A DIR_DEPTH of '0'
358 (zero) tells xferstats to process all subdirectories. The
359 default value is 3. It can be set on the command line with "-D
360 <number>". Example:
361 DIR_DEPTH 3
363 DIR_FILTER
365 takes one argument, the directory that xferstats should report
366 on. xferstats will discard any data that does not have a path
367 that begins with the value set by DIR_FILTER. This value is
368 case-sensitive, as it is part of a path name. Specifying multi‐
369 ple statements will instruct xferstats to include data from all
370 specified directories. It can be set on the command line with
371 "+D <path>". Example:
372 DIR_FILTER /pub/foobar
374 NUMBER_FILE_STATS
376 takes one argument, the number of files to report on in the file
377 statistics section. It must be a number greater than or equal
378 to zero, with zero denoting no limit. It can be set on the com‐
379 mand line with "+f <number>". The default is 50.
380 NUMBER_FILE_STATS 50
382 NUMBER_DIR_STATS
384 takes one argument, the number of directories to report on in
385 the directory statistics section. It must be a number greater
386 than or equal to zero, with zero denoting no limit. It can be
387 set on the command line with "+d <number>". The default is 50.
388 NUMBER_DAILY_STATS
390 takes one argument, the number of days to report on in the daily
391 statistics section. It must be a number greater than or equal
392 to zero, with zero denoting no limit. It can be set on the com‐
393 mand line with "+L <number>". The default is 50.
394 Note: the daily statistics section will always be displayed.
396 NUMBER_TLD_STATS
398 takes one argument, the number of top-level domains to report on
399 in the TLD statistics section. It must be a number greater than
400 or equal to zero, with zero denoting no limit. It can be set on
401 the command line with "+t <number>". The default is 50.
402 NUMBER_DOMAIN_STATS
404 takes one argument, the number of top-level domains to report on
405 in the domain statistics section. It must be a number greater
406 than or equal to zero, with zero denoting no limit. It can be
407 set on the command line with "+O <number>". The default is 50.
408 NUMBER_HOST_STATS
410 takes one argument, the number of hosts to report on in the host
411 statistics section. It must be a number greater than or equal
412 to zero, with zero denoting no limit. It can be set on the com‐
413 mand line with "+o <number>". The default is 50.
414 MAX_REPORT_SIZE
416 takes one argument, the maximum size of any one output table.
417 This is particularly useful to keep xferstats from generating
418 gigantic tables that make graphical browsers choke. It will
419 split tables to be no larger than# this value. It must be
420 greater than or equal to ten, or zero, zero denoting no limit.
421 It can be set on the command line with "+R <number>". The
422 default is 30.
423 GRAPH_PATH
425 takes one argument, the location of the JPGs used to draw graphs
426 in the HTML output. It can be an absolute path (ie,
427 http://foo.bar.com/graphs/) or a relative path (ie, ../graphs/).
428 You must include a trailing '/' or it will not work. This
429 option cannot be set on the command line. The default is noth‐
430 ing (which will make the browser look for the graphs in the
431 directory with the HTML). Example:
432 GRAPH_PATH ../graphs/
434 NO_HTML_HEADERS
436 takes no arguments. When enabled, xferstats will refrain from
437 adding <HTML>, <BODY>, and <TITLE> tags, so that the output can
438 be inserted into pages containing custom headers. It can be
439 enabled with "+n" or disabled with "-n" on the command line.
440 REFRESH
442 takes one argument, the number of seconds after being displayed
443 that the browser should refresh it. Unspecified or zero indi‐
444 cates no refresh.
445 SINGLE_PAGE
447 takes no arguments. When enabled, xferstats will output HTML as
448 a single page (entitled "xferstats.html") instead of a page for
449 each section. This option also disables <HTML> and <BODY> tags,
450 although it does produce a <TITLE> tag unless the NO_HTML_HEAD‐
451 ERS option is also specified. It can be enabled with "+s" or
452 disabled with "-s" on the command line.
453 STRIP_PREFIX
455 takes one argument. When specified, xferstats will strip the
456 given string from the beginning of any paths. Keep in mind when
457 using this option with DIR_FILTER that the DIR_FILTER is applied
458 -before- stripping, so that you must take this into account when
459 specifying the DIR_FILTER string. This option cannot be set on
460 the command line. Example:
461 STRIP_PREFIX /home/ftp
463 The next 11 entries are all basically the same--they define how the
465 reports are to be sorted.
466 0 -- Sort by the number of bytes downloaded
468 1 -- Sort by the number of files downloaded
470 2 -- Sort by the "name". This varies by report; for the file,
472 directory, TLD, etc reports, it sorts them alphabetically; for
473 the file size report it lists the file sizes in alphabetical
474 order; for the daily report, it lists the days in order. You
475 get the idea. 3 -- Sort by average throughput, in K/s
476 FILE_SORT_PREF, DIR_SORT_PREF, DOMAIN_SORT_PREF, TLD_SORT_PREF,
478 HOST_SORT_PREF, DOM_SORT_PREF, DOW_SORT_PREF, HOURLY_SORT_PREF,
479 DAILY_SORT_PREF, SIZE_SORT_PREF, MONTHLY_SORT_PREF
480 CHUNK_INPUT
482 takes one argument. When specified, xferstats will parse the
483 logfiles in chunks of the given size (in number of lines),
484 process them, and continue. This will keep it from trying to
485 parse the entire log at once, using up a lot of memory for large
486 logs. Specifying too small a value will cause xferstats to
487 waste a lot of time switching tasks, while too large a value may
488 slow it down as your system swaps. Toy with this value a bit
489 and see what works best for you. If left unspecified, xferstats
490 will continue to parse the logs in one pass.
491
494 xferlog (generated by wu-ftpd or ncftpd)
495 /etc/xferstats.cfg contains all of the default settings and many set‐
497 tings which cannot be set from the command line.
498 If you installed the RPM version of xferstats, you'll find the JPGs for
500 the graphs in /usr/share/xferstats/graphs.
501
504 The JPG graph pieces included are not ideal; sometimes artifacts show
505 up. I will eventually get around to making new GIFs or something.
506 Please send bug reports to phil@off.net If the xferlog that you're
508 using is smaller than 1 megabyte, please send a gzipped copy, along
509 with the exact parameters you're using.
510
513 I greatly appreciate the suggestions and assistance of Nat Friedman
514 <ndf@mit.edu>.
515 Thanks to Mike Shaver <shaver@mozilla.org> and Jamie Zawinski <jwz@net‐
517 scape.com> for coughing up gigantic mozilla.org logs to aid my testing.
518 Thanks to Phil Copeland <bryce@zen.ics.uwe.ac.uk> for getting on my
520 back constantly to add new stuff, fix nagging bugs, and speed it up.
521 Oh, and for using it, at least someone does :).
522 Extreme gratitude to Mark Hagler <hagler@hagler.org>, someone whom I've
524 left off of this list for too long. Mark was kind enough to Amazon me
525 a copy of O'Reillys _Pthreads Programming_ which was the catalyst for
526 the threaded version and really helped get me interested in the project
527 again. While the threaded version doesn't exist anymore, the book is
528 still terribly useful for other projects.
529
531 ftpd(8), ncftpd(8), xferlog(5)
532
535 xferstats is available from the following locations. The old addresses
536 are valid, but are deprecated! Please use the new addresses whenever
537 possible.
538 http://xferstats.off.net/
540 ftp://xferstats.off.net/pub/xferstats/
542
545 xferstats is Copyright (c) 1997-2000 Phil Schwan <phil@off.net> and is
546 released under the terms and provisions of the GNU Public License.
547 31 Oct 2000 XFERSTATS(8)