1CALAMARIS(1) Calamaris Manual CALAMARIS(1)
2
6 calamaris - generate text and graphical statistics out of log files
7 from Proxy-Cache-Servers
8
11 cat log | calamaris [ --config-file .../calamaris.conf ] [ switches ]
12
15 Calamaris is used to produce statistical output from Squid, NetCache,
16 Inktomi Traffic Server, Oops! proxy server, Compaq Tasksmart, Cisco
17 Content Engines or related Proxy log files. The resulting output can
18 be ascii or html with or without graphic and with or without frames. It
19 is possible to cache calculated data in a file to use them in later
20 runs.
21 This manual page describes the options of Calamaris and gives a few ex‐
23 amples.
24
27 Configuration File
28 --config-file file
29 Not all reports and modification can be made through command-
31 line-switches. To use all options of Calamaris you´ll have to
32 use the configuration file. You'll find the configuration-di‐
33 rectives below, always inside of braces. Examples are in the
34 calamaris.conf which should come with this package.
35 Reports
38 --all-useful-reports|-a
39 extracts all useful reports available, --all-useful-reports
40 equals --size-distribution-report 10 --domain-report 20 --per‐
41 formance-report 60 --requester-report 20 --status-report --type-
42 report 20 --response-time-report --errorcode-distribution-report
43 --domain-report|-d n ($domain_report)
46 switches the top level and the second level report on. The data
47 is derived from the URL. The output is limited by n. (-1 is un‐
48 limited)
49 --domain-report-limit n ($domain_report_limit)
52 limit the domain-report to lines which have n or more requests.
53 --domain-report-n-level|-N n ($domain_report_n_level)
56 All URL-Host reports will be switched from 2nd-level to n-level-
57 reports. (-1 shows a full urlhost-report)
58 Note: This option is only useful with activated domain-report.
60 --errorcode-distribution-report ($errorcode_distribution_report)
63 shows the Response code distribution over all objects
64 ($object_freshness_report)
67 shows the freshness of objects in your cache. Calamaris looks
68 for freshness tags like 'TCP_HIT', 'TCP_REFRESH_MISS', ... and
69 make statistics on it. With this information you can optimize
70 the caching behaviour of your cache depending on the objects
71 content type. E.g. squid admins could use this information to
72 configure the refresh_pattern. This option needs more configura‐
73 tion in the configuration-file.
74 --peak-report|-p type ($peak_report)
77 Measures the peaks of the Proxy usage in requests per second,
78 per minute and per hour. It reports peaks for TCP, UDP and ALL
79 requests. If set to 'old' these values were calculated with the
80 old slow method, if set to 'new' the new faster (but still slow)
81 method is used.
82 --performance-report|-P n ($performance_report)
85 Shows the throughput of TCP requests for every n minutes.
86 --performance-report-adjust|-T n ($performance_report_adjust)
89 Time: Adjust the Performancereport in minutes for non GMT-Time‐
90 zoners.
91 --requester-report|-r n ($requester_report)
94 Switches the UDP and TCP requester reports on. The output is
95 limited by n. (-1 is unlimited)
96 --requester-report-no-dns-lookup|-n ($requester_report_no_dns_lookup)
99 Switches the IP lookup for the requesters off.
100 --requester-report-use-user-info|-u ($requester_report_use_user_info)
103 Switches the usage of eventually available ident information for
104 requester reports on.
105 Warning: This breaks the privacy of your users! (see PRIVACY-
107 Section below)
108 --requester-report-with-targets|-R n ($requester_report_with_targets)
111 adds to each line of the requester report the requested URLs.
112 The output is limited by n. (-1 is unlimited, and can result in
113 very very long reports.)
114 Warning: Using this option breaks the privacy of your users!
116 (see PRIVACY-Section below)
117 --response-time-report ($response_time_report)
120 sums up the time distribution over all objects
121 (@response_time_report_interval)
124 This array defines the time steps, which should be reported in
125 the response-time-report.
126 --size-distribution-report|-D n ($size_distribution_report)
129 shows size-based distribution of requested objects, smaller num‐
130 bers result in more verbose reports. (choose 2, 10 or 100 for
131 useful output.)
132 --status-report|-s ($status_report)
135 alters the default behaviour of Calamaris and makes the status
136 reports more verbose.
137 --type-report|-t n ($type_report)
140 switches the content type and the file extension report on. The
141 output is limited by n. (-1 is unlimited)
142 --type-report-ignore-case|-c ($type_report_ignore_case)
145 Switch to case-insensitive. This is useful for the 'Requested
146 extensions' report.
147 Input
150 --input-format|-f type ($input_format)
151 sets the type of input logfiles. If set to
152 'auto' Calamaris tries to guess the input file format. This is
154 the Default.
155 Note: If the first line of your input file is corrupted, Cala‐
157 maris will stop with an error.
158 'squid' Calamaris expects native logfile derived from Squid
160 V1.1.beta26-V2.x or OOPS.
161 'squid-extended' Calamaris expects native logfile derived from
163 Squid V1.1.alpha1-V2.x with log_mime_hdrs enabled or Squid with
164 Smartfilter-Patch or squid-style logfiles out of Cisco Content
165 Engines. (This only enables parsing of these kind of logfile,
166 the additional data will be ignored.)
167 (Logging of MIME-Headers breaks the privacy of your users! (see
169 PRIVACY-Section below)
170 'squid-old' Calamaris expects native logfile derived from Squid
172 V1.1.alpha1-V1.1.beta25.
173 'nc' Calamaris expects Logfiles from NetCache up to V4.x.
175 (Please see the README on this.)
176 'its' Calamaris expects Logfiles from Inktomi Traffic Server.
178 'elff' Calamaris expects Logfiles in Extended Logfile Format
180 (i.e. from Compaq Tasksmart, Novell Internet Caching System or
181 NetCache V5.x)
182 'nse' Calamaris expects Logfiles in Netscape Extended-1 or Net‐
184 scape Extended-2 Logfile Format (from Netscape/iPlanet/SunOne
185 Proxy-Server )
186 --ipfilter-exclude IP/range ($ipfilter_exclude)
189 all IPs are analyzed, except IP/range. Format:
190 1.1.1.1/32:1.1.2.0/24 or
191 1.1.1.1/255.255.255.255:1.1.2.0/255.255.255.0
192 IP list separated by ':' This switch needs the perl Module Ne‐
194 tAddr::IP.
195 Warning: This breaks the privacy of your users! (see PRIVACY-
197 Section below)
198 --ipfilter-include IP/range ($ipfilter_include)
201 no IPs are analyzed, except IP/range. Format: see --ipfilter-ex‐
202 clude
203 Warning: This breaks the privacy of your users! (see PRIVACY-
205 Section below)
206 --no-input|-z ($no_input)
209 Switches reading from standard input off. You can use this to
210 merge many cache files to one (see --cache-input-file and
211 --cache-output-file) or to generate a report out of cache files.
212 --time-interval|-I t-t ($time_interval)
215 defines which time-interval should be parsed. t has to be the
216 format yyyymmddhhmmss (localtime)
217 Note: omitting the beginning or ending date is allowed.
219 Output
222 Standard output format is plain ascii with 80 chars width.
223 ($column1_color)
226 ($column2_color) defines the colors for the columns in graphics.
227 (only useful with --output-format graph)
228 ($formats[n])
231 Through the config-file you are able to modify the width of the
232 report and alter the culomns that are displayed in the reports.
233 n is the number of the report, as displayed by --help in the
234 --show-reports-option.
235 --hostname|-H name ($hostname)
238 The name for the title or subject of the output. If set to
239 'lookup' Calamaris looks up the host name of the system its been
240 run on.
241 --image-type ($image_type)
244 Sets the image type to gif, png, jpeg, gd or gd2. Only useful
245 when --output-format graph is set. The available images types
246 are depending on your GD::Graph installation. Default is 'png'.
247 --logo|-l string ($logo)
250 add a custom string to a HTML-Report. It'll be added to a table
251 on the top of the output. -l '<A HREF="https://cord.de/"><IMG
252 BORDER=0 SRC="https://cord.de/sites/default/files/cordlog2n.gif"
253 ALT="Cord"></A>' will add my logo with a link to the Report.
254 Note: --logo works only in combination with --output-format html
256 or html-frame
257 --meta|-M string ($meta)
260 Meta: adds a custom string or the content of a file into the
261 <HEAD> of a HTML-Report. Useful if you want to add Stylesheets
262 or something to the Report.
263 Note: --meta works only in combination with --output-format html
265 or html-frame
266 --output-format|-F type[,type[,type[,...]]] ($output_format)
269 Format: sets the format of the output-report. If set to
270 'mail' adds a subject header to the beginning of the report.
272 'html' all output is given in html with tables. Can be combined
274 with 'mail' to send html mails.
275 'html-frame' all output is given in html frames with tables.
277 'html-embed' all output is given in html with tables without
279 HTML-Headers. Useful for Server-Side-Includes.
280 'graph' enables graphics for html, html-embed or html-frame.
282 'unformatted' gives out the raw numbers separated by spaces.
284 Useful for re-using the output in other scripts. If you use this
285 along with -U, the byte values are calculated in the given Unit,
286 and displayed without indication along with the numbers. the in‐
287 dication moves up to the header of the report.
288 --output-path ($output_path)
291 output calamaris statistics to /path. In case of graph output,
292 the graphics destination is /path and the filename is in‐
293 dex.html, else it is calamaris.txt. If --output-path is not
294 given, all graphics are written to the working directory.
295 --output-file ($output_file)
298 alter the filename of --output-path.
299 --output-file-prefix ($output_file_prefix)
302 adds a prefix to --output-file %t is replaced by the timerange
303 of the report, %h by the hostname (see --hostname )
304 --show-reports|-S n[,n[,n[,...]]] ($show_reports)
307 Show: Shows only the defined reports in the specified order. De‐
308 fault is to display the reports as they are defined through the
309 report-switches above. The following numbers are defined:
310 0 Summary
312 1 Incoming request peak per protocol
313 2 Incoming transfer volume peak per protocol
314 3 Incoming requests by method
315 4 Incoming UDP-requests by status
316 5 Incoming TCP-requests by status
317 6 Outgoing requests by status
318 7 Outgoing requests by destination
319 8 Request-destinations by 2ndlevel-domain
320 9 Request-destinations by toplevel-domain
321 10 TCP-Request-protocol
322 11 Requested content-type
323 12 Requested extensions
324 13 Incoming UDP-requests by host
325 14 Incoming TCP-requests by host
326 15 Size Distribution Diagram
327 16 Performance in n minute steps
328 17 UDP-Request duration distribution in msec
329 18 TCP-Request duration distribution in msec
330 19 UDP Response code distribution
331 20 TCP Response code distribution
332 Note: Using this doesn't make Calamaris any faster, the internal calcu‐
334 lations will be done as the report-switches were set (see above).
335 --sort-order|-O ($sort_order)
338 Changes the sort order in the reports to request size, default
339 is sorting by number of requests.
340 ($text_color)
343 defines the colors for text/axis/legend/labels in graphics.
344 (only useful with --output-format graph )
345 --unit|-U string ($unit)
348 You can define this to K(ilo), M(ega), G(iga) or T(era) for the
349 Byte-units.
350 ($width)
353 defines the width of the graphics. height is calculated from
354 this with a 3:2-ratio. (only useful with --output-format graph )
355 ($x_scale)
358 defines how many datasets should be drawn on the graph. 30 is a
359 good value, but you can play with this. if $x_scale gets to big,
360 you're on your own ;-)
361 --generate-index ($generate_index)
364 generates an index for all reports that match --output-file-pre‐
365 fix.
366 Caching
369 --cache-input-file|-i file ($cache_input_file)
370 You can reuse a cache file generated with --cache-output-file
371 file to add old data to a new report. Several files can be sep‐
372 arated with a ':'.
373 Note: if you use more than one cache file, make sure they are
375 chronologicaly ordered (oldest first).
376 Note: if you reuse cache-files, which were not created with -d
378 -1 -r -1 -t -1 -R -1 the number of 'others' would be wrong ev‐
379 erywhere. In this case the number of 'others' are omitted.
380 --cache-output-file|-o file ($cache_output_file)
383 Calamaris stores a summary of the computed information in file
384 and you can reuse it at a later time with --cache-input-file
385 Note: The output file can be the same as the input file: it is
387 simply overwritten after reading the data. It is not recom‐
388 mended to change the options between different runs if you in‐
389 clude older data as this can result in strange measurements.
390 Misc
393 --benchmark|-b n ($benchmark)
394 benchmark: A switch for the impatient as it prints a '#' for ev‐
395 ery n parsed lines.
396 --copyright|-C
399 Prints the copyright information of Calamaris
400 --help|-h
403 Prints a brief description of the command line options.
404 --version|-V
407 Prints out the Version-Number.
408 Debug
411 --dump-loop|-L
412 prints the internal loop to STDERR. (for Debugging)
413 ($test)
416 activates some small tests for the programmer.
417 --verbose|-v ($verbose)
420 print more information about what is Calamaris is doing and be‐
421 lieving.
422
425 This example mails the daily statistics to root:
426 cat /var/log/squid/access.log | nice -39 calamaris --all-useful-
427 reports --hostname "daily worf" --output-format mail | mail root
428 This one only caches a summary for later use:
431 cat /var/log/squid/access.log | calamaris --all-useful-reports
432 --cache-output-file daily.`date +"%w"` > /dev/null
433 You can then use the caches to have weekly statistics:
436 if [ $DAYOFWEEK = "0" ]; then
438 calamaris --all-useful-reports --cache-input-file
439 daily.1:daily.2:daily.3:daily.4:daily.5:daily.6:daily.0 --no-in‐
440 put --output-format mail --hostname "weekly worf" | mail root ;
441 fi
443
446 If you have a problem with Calamaris , please make sure that you use
447 the recent version of Calamaris (see VERSION below). Also check that
448 your proxy works correctly and doesn't produce invalid Logfiles. (see
449 the README for buglist and pointers.)
450 If you're sure that you've encountered a bug in Calamaris please report
452 it to Calamaris-bug@cord.de. This also applies if Calamaris itself says
453 'please report this'.
454
457 Calamaris can be (mis-)used to track what users are requesting.
458 So please read the following and think about it, before using Calamaris
460 to be the Big Brother.
461 - If you don't trust your users than there is something more wrong
464 than the loss of productivity.
465 - Squid has some nice acl-mechanisms. If you think that your users
468 don't use the net properly, don't let them use it. (You can also
469 open the net at specific times or to specific sites, if you
470 want.)
471 - If you still want to use Calamaris that way, let your
474 vict^Wusers know, that they'll be monitored. (in Germany you
475 have to let them know!)
476
479 squid(8)
480
483 Cord Beermann <Cord@Wunder-Nett.org>, Michael Pophal
484 <michael.pophal@nefkom.net>. There are also a lot of people who con‐
485 tributed code, gave ideas or requested features. Look them up in the
486 executable.
487 This man page was written by Philipp Frauenfelder <pfrauenf@de‐
489 bian.org>, maintainer of the Debian package. Maintenance is now taken
490 over by Cord Beermann.
491
494 Version of this manpage: $Id: calamaris.1,v 3.1 2006-03-19 17:52:48
495 cord Exp $
496 It describes the usage of Calamaris V3.0 and later.
498 Information about new releases, mailing lists, and other related issues
500 can be found from the Calamaris home page at URL https://Cala‐
501 maris.Cord.de/
502
505 Calamaris comes with "absolutely no warranty".
506
509 Copyright (C) 1997-2006, 2013, 2015, 2017, 2021 Cord Beermann
510 This program is free software; you can redistribute it and/or modify it
512 under the terms of the GNU General Public License as published by the
513 Free Software Foundation; either version 2 of the License, or (at your
514 option) any later version.
515 (If you modify and want to publish it under the name Calamaris , please
517 ask me. I don't want to confuse the 'audience' with many different
518 versions of the same name and/or Version number. (This is not part of
519 the license, it is only a favour i asked of you.))
520 This program is distributed in the hope that it will be useful, but
522 WITHOUT ANY WARRANTY; without even the implied warranty of MER‐
523 CHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
524 Public License for more details.
525 You should have received a copy of the GNU General Public License along
527 with this program; if not, write to the Free Software Foundation, Inc.,
528 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
529Handmade $Date: 2006-03-19 17:52:48 $ CALAMARIS(1)
