1VARNISHNCSA(1) VARNISHNCSA(1)
2
6 varnishncsa - Display Varnish logs in Apache / NCSA combined log format
7
9 varnishncsa [-a] [-b] [-c] [-C] [-d] [-D] [-E] [-F <format>] [-f <for‐
10 matfile>] [-g <request|vxid>] [-h] [-j] [-k <num>] [-L <limit>] [-n
11 <dir>] [-P <file>] [-Q <file>] [-q <query>] [-r <filename>] [-R
12 <limit[/duration]>] [-t <seconds|off>] [-V] [-w <filename>]
13
15 The varnishncsa utility reads varnishd(1) shared memory logs and
16 presents them in the Apache / NCSA "combined" log format.
17 Each log line produced is based on a single Request type transaction
19 gathered from the shared memory log. The Request transaction is then
20 scanned for the relevant parts in order to output one log line. To fil‐
21 ter the log lines produced, use the query language to select the appli‐
22 cable transactions. Non-request transactions are ignored.
23 The following options are available:
25 -a When writing output to a file, append to it rather than over‐
27 write it. This option has no effect without the -w option.
28 -b Log backend requests. If -c is not specified, then only backend
30 requests will trigger log lines.
31 -c Log client requests. This is the default. If -b is specified,
33 then -c is needed to also log client requests
34 -C Do all regular expression and string matching caseless.
36 -d Process log records at the head of the log and exit.
38 -D Daemonize.
40 -E Show ESI requests, implies client mode.
42 -F <format>
44 Set the output log format string.
45 -f <formatfile>
47 Read output format from a file. Will read a single line from the
48 specified file, and use that line as the format.
49 -g <request|vxid>
51 The grouping of the log records. The default is to group by
52 vxid.
53 -h Print program usage and exit
55 -j Make format-specifier replacements JSON-compatible. When escap‐
57 ing characters, use JSON-style \uXXXX escape sequences instead
58 of C-style \xXX sequences. Empty strings will be replaced with
59 "" instead of "-", and empty integers will be replaced with
60 null. Use -F or -f in combination with -j to write JSON logs.
61 -k <num>
63 Process this number of matching log transactions before exiting.
64 -L <limit>
66 Sets the upper limit of incomplete transactions kept before the
67 oldest transaction is force completed. A warning record is syn‐
68 thesized when this happens. This setting keeps an upper bound on
69 the memory usage of running queries. Defaults to 1000 transac‐
70 tions.
71 -n <dir>
73 Specify the varnishd working directory (also known as instance
74 name) to get logs from. If -n is not specified, the host name is
75 used.
76 -P <file>
78 Write the process' PID to the specified file.
79 -Q <file>
81 Specifies the file containing the VSL query to use. When multi‐
82 ple -Q or -q options are specified, all queries are considered
83 as if the 'or' operator was used to combine them.
84 -q <query>
86 Specifies the VSL query to use. When multiple -q or -Q options
87 are specified, all queries are considered as if the 'or' opera‐
88 tor was used to combine them.
89 -r <filename>
91 Read log in binary file format from this file. The file can be
92 created with varnishlog -w filename. If the filename is -, logs
93 are read from the standard input. and cannot work as a daemon.
94 -R <limit[/duration]>
96 Restrict the output to the specified limit. Transactions exceed‐
97 ing the limit will be suppressed. The limit is specified as the
98 maximum number of transactions (with respect to the chosen
99 grouping method) and an optional time period. If no duration is
100 specified, a default of s is used. The duration field can be
101 formatted as in VCL (e.g. -R 10/2m) or as a simple time period
102 without the prefix (e.g. -R 5/m). When in -g raw grouping mode,
103 this setting can not be used alongside -i, -I, -x or -X, and we
104 advise using -q instead.
105 -t <seconds|off>
107 Timeout before returning error on initial VSM connection. If set
108 the VSM connection is retried every 0.5 seconds for this many
109 seconds. If zero the connection is attempted only once and will
110 fail immediately if unsuccessful. If set to "off", the connec‐
111 tion will not fail, allowing the utility to start and wait inde‐
112 finetely for the Varnish instance to appear. Defaults to 5 sec‐
113 onds.
114 -V Print version information and exit.
116 -w <filename>
118 Redirect output to file. The file will be overwritten unless the
119 -a option was specified. If the application receives a SIGHUP in
120 daemon mode the file will be reopened allowing the old one to be
121 rotated away. This option is required when running in daemon
122 mode. If the filename is -, varnishncsa writes to the standard
123 output and cannot work as a daemon.
124 --optstring
126 Print the optstring parameter to getopt(3) to help writing wrap‐
127 per scripts.
128
130 The default mode of varnishncsa is "client mode". In this mode, the
131 log will be similar to what a web server would produce in the absence
132 of varnish. Client mode can be explicitly selected by using -c.
133 If the -b switch is specified, varnishncsa will operate in "backend
135 mode". In this mode, requests generated by varnish to the backends
136 will be logged. Unless -c is also specified, client requests received
137 by varnish will be ignored.
138 When running varnishncsa in both backend and client mode, it is
140 strongly advised to include the format specifier %{Varnish:side}x to
141 distinguish between backend and client requests.
142 Client requests that results in a pipe (ie. return(pipe) in vcl), will
144 not generate logging in backend mode. This is because varnish is not
145 generating requests, but blindly passes on bytes in both directions.
146 However, a varnishncsa instance running in normal mode can see this
147 case by using the formatter %{Varnish:handling}x, which will be 'pipe'.
148 In backend mode, some of the fields in the format string get different
150 meanings. Most notably, the byte counting formatters (%b, %I, %O) con‐
151 siders varnish to be the client.
152 It is possible to keep two varnishncsa instances running, one in back‐
154 end mode, and one in client mode, logging to different files.
155
157 Specify the log format to use. If no format is specified the default
158 log format is used:
159 %h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i"
161 Escape sequences \n and \t are supported.
163 Supported formatters are:
165 %b In client mode, size of response in bytes, excluding HTTP head‐
167 ers. In backend mode, the number of bytes received from the
168 backend, excluding HTTP headers. In CLF format, i.e. a '-'
169 rather than a 0 when no bytes are sent.
170 %D In client mode, time taken to serve the request, in microsec‐
172 onds. In backend mode, time from the request was sent to the
173 entire body had been received. This is equivalent to %{us}T.
174 %H The request protocol. Defaults to HTTP/1.0 if not known.
176 %h Remote host. Defaults to '-' if not known. In backend mode this
178 is the IP of the backend server.
179 %I In client mode, total bytes received from client. In backend
181 mode, total bytes sent to the backend.
182 %{X}i The contents of request header X. If the header appears multiple
184 times in a single transaction, the last occurrence is used.
185 %l Remote logname. Always '-'.
187 %m Request method. Defaults to '-' if not known.
189 %{X}o The contents of response header X. If the header appears multi‐
191 ple times in a single transaction, the last occurrence is used.
192 %O In client mode, total bytes sent to client. In backend mode,
194 total bytes received from the backend.
195 %q The query string. Defaults to an empty string if not present.
197 %r The first line of the request. Synthesized from other fields, so
199 it may not be the request verbatim. See the NOTES section.
200 %s Status sent to the client. In backend mode, status received
202 from the backend.
203 %t In client mode, time when the request was received, in HTTP
205 date/time format. In backend mode, time when the request was
206 sent.
207 %{X}t In client mode, time when the request was received, in the for‐
209 mat specified by X. In backend mode, time when the request was
210 sent. The time specification format is the same as for strf‐
211 time(3) with these extensions:
212 • %{sec}: number of seconds since the Epoch
214 • %{msec}: number of milliseconds since the Epoch
216 • %{usec}: number of milliseconds since the Epoch
218 • %{msec_frac}: millisecond fraction
220 • %{usec_frac}: microsecond fraction
222 The extensions can not be combined with each other or strf‐
224 time(3) in the same specification. Use multiple %{X}t specifica‐
225 tions instead.
226 %T In client mode, time taken to serve the request, in seconds. In
228 backend mode, time from the request was sent to the entire body
229 had been received. This is equivalent to %{s}T.
230 %{X}T In client mode, time taken to serve the request, in the format
232 specified by X. In backend mode, time from the request was sent
233 to the entire body had been received. The time specification
234 format can be one of the following: s (same as %T), ms or us
235 (same as %D).
236 %U The request URL without the query string. Defaults to '-' if not
238 known.
239 %u Remote user from auth.
241 %{X}x Extended variables. Supported variables are:
243 Varnish:time_firstbyte
245 Time from when the request processing starts until the
246 first byte is sent to the client, in seconds. For back‐
247 end mode: Time from the request was sent to the backend
248 to the entire header had been received.
249 Varnish:hitmiss
251 One of the 'hit' or 'miss' strings, depending on whether
252 the request was a cache hit or miss. Pipe, pass and synth
253 are considered misses.
254 Varnish:handling
256 One of the 'hit', 'miss', 'pass', 'pipe' or 'synth'
257 strings indicating how the request was handled.
258 Varnish:side
260 Backend or client side. One of two values, 'b' or 'c',
261 depending on where the request was made. In pure backend
262 or client mode, this field will be constant.
263 Varnish:vxid
265 The VXID of the varnish transaction.
266 VCL_Log:key
268 The value set by std.log("key:value") in VCL.
269 VSL:tag:record-prefix[field]
271 The value of the VSL entry for the given tag-record pre‐
272 fix-field combination. Tag is mandatory, the other compo‐
273 nents are optional.
274 The record prefix will limit the matches to those records
276 that have this prefix as the first part of the record
277 content followed by a colon.
278 The field will, if present, treat the log record as a
280 white space separated list of fields, and only the nth
281 part of the record will be matched against. Fields start
282 counting at 1 and run up to 255.
283 Defaults to '-' when the tag is not seen, the record pre‐
285 fix does not match or the field is out of bounds. If a
286 tag appears multiple times in a single transaction, the
287 first occurrence is used.
288
290 • SIGHUP
291 Rotate the log file (see -w option) in daemon mode, abort the loop
293 and die gracefully when running in the foreground.
294 • SIGUSR1
296 Flush any outstanding transactions.
298
300 The %r formatter is equivalent to "%m http://%{Host}i%U%q %H". This
301 differs from apache's %r behavior, equivalent to "%m %U%q %H". Fur‐
302 thermore, when using the %r formatter, if the Host header appears mul‐
303 tiple times in a single transaction, the first occurrence is used.
304
306 Log the second field of the Begin record, corresponding to the VXID of
307 the parent transaction:
308 varnishncsa -F "%{VSL:Begin[2]}x"
310 Log the entire Timestamp record associated with the processing length:
312 varnishncsa -F "%{VSL:Timestamp:Process}x"
314 Log in JSON, using the -j flag to ensure that the output is valid JSON
316 for all inputs:
317 varnishncsa -j -F '{"size": %b, "time": "%t", "ua": "%{User-Agent}i"}'
319
321 varnishd(1) varnishlog(1) varnishstat(1) vsl(7)
322
324 The varnishncsa utility was developed by Poul-Henning Kamp in coopera‐
325 tion with Verdens Gang AS and Varnish Software AS. This manual page was
326 initially written by Dag-Erling Smørgrav <des@des.no>, and later up‐
327 dated by Martin Blix Grydeland and Pål Hermunn Johansen.
328
330 This document is licensed under the same licence as Varnish itself. See
331 LICENCE for details.
332 • Copyright (c) 2006 Verdens Gang AS
334 • Copyright (c) 2006-2016 Varnish Software AS
336 VARNISHNCSA(1)