1VSL-QUERY(7)                                                      VSL-QUERY(7)
2
3
4

NAME

6       vsl-query - Varnish VSL Query Expressions
7

OVERVIEW

9       The  Varnish  VSL Query Expressions extracts transactions from the Var‐
10       nish shared memory log, and perform queries on the transactions  before
11       reporting matches.
12
13       A  transaction  is  a  set  of  log lines that belongs together, e.g. a
14       client request or a backend request. The API monitors the log, and col‐
15       lects  all  log  records that make up a transaction before reporting on
16       that transaction. Transactions can also  be  grouped,  meaning  backend
17       transactions  are  reported  together  with the client transaction that
18       initiated it.
19
20       A query is run on a group of transactions. A query expression  is  true
21       if there is a log record within the group that satisfies the condition.
22       It is false only if none of the log records  satisfies  the  condition.
23       Query expressions can be combined using boolean functions.  In addition
24       to log records, it is possible  to  query  transaction  ids  (vxid)  in
25       query.
26

GROUPING

28       When  grouping  transactions,  there  is  a hierarchy structure showing
29       which transaction initiated what. The level  increases  by  one  on  an
30       'initiated by' relation, so for example a backend transaction will have
31       one higher level than the client transaction that  initiated  it  on  a
32       cache   miss.  Request  restart  transactions  don't  get  their  level
33       increased to make it predictable.
34
35       Levels start counting at 1, except when using raw where it will  always
36       be 0.
37
38       The grouping modes are:
39
40       · session
41
42         All  transactions  initiated  by  a  client  connection  are reported
43         together.  Client  connections  are  open  ended  when   using   HTTP
44         keep-alives, so it is undefined when the session will be reported. If
45         the transaction timeout period is exceeded an incomplete session will
46         be reported. Non-transactional data (vxid == 0) is not reported.
47
48       · request
49
50         Transactions  are  grouped by request, where the set will include the
51         request itself as well as any backend  requests  or  ESI-subrequests.
52         Session data and non-transactional data (vxid == 0) is not reported.
53
54       · vxid
55
56         Transactions  are  not  grouped,  so  each  vxid  is  reported in its
57         entirety. Sessions, requests, ESI-requests and backend  requests  are
58         all  reported  individually.  Non-transactional  data is not reported
59         (vxid == 0). This is the default.
60
61       · raw
62
63         Every log record will make up a transaction of  its  own.  All  data,
64         including non-transactional data will be reported.
65
66   Transaction Hierarchy
67       Example transaction hierarchy using request grouping mode
68
69          Lvl 1: Client request (cache miss)
70            Lvl 2: Backend request
71            Lvl 2: ESI subrequest (cache miss)
72              Lvl 3: Backend request
73              Lvl 3: Backend request (VCL restart)
74              Lvl 3: ESI subrequest (cache miss)
75                Lvl 4: Backend request
76            Lvl 2: ESI subrequest (cache hit)
77

MEMORY USAGE

79       The API will use pointers to shared memory log data as long as possible
80       to keep memory usage at a minimum. But as the shared memory  log  is  a
81       ring  buffer,  data will get overwritten eventually, so the API creates
82       local copies of referenced log data when varnishd comes close to  over‐
83       writing still unreported content.
84
85       This  process  avoids loss of log data in many scenarios, but it is not
86       failsafe: Overruns where varnishd "overtakes" the log reader process in
87       the  ring buffer can still happen when API clients cannot keep up read‐
88       ing and/or copying, for instance due to output blocking.
89
90       Though being unrelated to grouping in principle, copying of log data is
91       particularly  relevant  for session grouping together with long lasting
92       client connections - for this grouping, the logging API client  process
93       is  likely  to consume relevant amounts of memory. As the vxid grouping
94       also logs (potentially long lasting) sessions, it  is  also  likely  to
95       require  memory  for  copies  of log entries, but far less than session
96       grouping.
97

QUERY LANGUAGE

99       A query expression consists of record selection criteria,  and  option‐
100       ally an operator and a value to match against the selected records.
101
102          <record selection criteria> <operator> <operand>
103
104       Additionally,  a  query  expression can occur on the transaction itself
105       rather than log records belonging to the transaction.
106
107          vxid <numerical operator> <integer>
108
109       A vxid query allows you to directly target a specific transacion, whose
110       id  can  be  obtained  from an X-Varnish HTTP header, the default "guru
111       meditation" error page, or Begin and Link log records.
112
113       A query must fit on a single line, but it is possible to pass  multiple
114       queries  at  once, one query per line. Empty lines are ignored, and the
115       list of queries is treated as if the 'or' operator was used to  combine
116       them.
117
118       For example this list of queries:
119
120          # catch varnish errors
121          *Error
122
123          # catch backend errors
124          BerespStatus >= 500
125
126       is identical to this query:
127
128          (*Error) or (BerespStatus >= 500)
129
130       Comments can be used and will be ignored, they start with the '#' char‐
131       acter, which may be more useful when the query is read from a file.
132
133       For very long queries that  couldn't  easily  be  split  into  multiple
134       queries  it  is possible to break them into multiple lines with a back‐
135       slash preceding an end of line.
136
137       For example this query:
138
139          BerespStatus >= 500
140
141       is identical to this query:
142
143          BerespStatus \
144          >= \
145          500
146
147       A backslash-newline sequence doesn't continue a  comment  on  the  next
148       line and isn't allowed in a quoted string.
149
150   Record selection criteria
151       The  record  selection  criteria  determines what kind records from the
152       transaction group the expression applies to. Syntax:
153
154          {level}taglist:record-prefix[field]
155
156       Taglist is mandatory, the other components are optional.
157
158       The level limits the expression to a transaction at that level. If left
159       unspecified,  the  expression is applied to transactions at all levels.
160       Level is a positive integer or zero. If level  is  followed  by  a  '+'
161       character,  it expresses greater than or equal. If level is followed by
162       a '-', it expresses less than or equal.
163
164       The taglist is a comma-separated list of  VSL  record  tags  that  this
165       expression  should  be  checked against. Each list element can be a tag
166       name or a tag glob. Globs allow a '*' either in the  beginning  of  the
167       name or at the end, and will select all tags that match either the pre‐
168       fix or subscript. A single '*' will select all tags.
169
170       The record prefix will further limit the matches to those records  that
171       has this prefix as their first part of the record content followed by a
172       colon. The part of the log record matched against will then be  limited
173       to  what  follows  the  prefix  and colon. This is useful when matching
174       against specific HTTP headers. The record prefix matching is done  case
175       insensitive.
176
177       The field will, if present, treat the log record as a white space sepa‐
178       rated list of fields, and only the nth  part  of  the  record  will  be
179       matched against. Fields start counting at 1.
180
181       An  expression  using  only a record selection criteria will be true if
182       there is any record in the transaction group that is  selected  by  the
183       criteria.
184
185   Operators
186       The following matching operators are available:
187
188       · == != < <= > >=
189
190         Numerical comparison. The record contents will be converted to either
191         an integer or a float before comparison, depending on the type of the
192         operand.
193
194       · eq ne
195
196         String  comparison.  'eq'  tests  string equality, 'ne' tests for not
197         equality.
198
199       · ~ !~
200
201         Regular expression matching. '~' is  a  positive  match,  '!~'  is  a
202         non-match.
203
204   Operand
205       The operand is the value the selected records will be matched against.
206
207       An  operand  can  be quoted or unquoted. Quotes can be either single or
208       double quotes, and for quoted operands  a  backslash  can  be  used  to
209       escape the quotes.
210
211       Unquoted operands can only consist of the following characters:
212
213          a-z A-Z 0-9 + - _ . *
214
215       The following types of operands are available:
216
217       · Integer
218
219         A number without any fractional part, valid for the numerical compar‐
220         ison operators. The integer type is used when the  operand  does  not
221         contain any period (.) characters.
222
223       · Float
224
225         A  number  with a fractional part, valid for the numerical comparison
226         operators. The float type is used when the  operand  does  contain  a
227         period (.) character.
228
229       · String
230
231         A sequence of characters, valid for the string equality operators.
232
233       · Regular expression
234
235         A  PCRE  regular  expression. Valid for the regular expression opera‐
236         tors.
237
238   Boolean functions
239       Query expressions can be linked together using boolean  functions.  The
240       following are available, in decreasing precedence:
241
242       · not <expr>
243
244         Inverts the result of <expr>
245
246       · <expr1> and <expr2>
247
248         True only if both expr1 and expr2 are true
249
250       · <expr1> or <expr2>
251
252         True if either of expr1 or expr2 is true
253
254       Expressions can be grouped using parenthesis.
255

QUERY EXPRESSION EXAMPLES

257       · Transaction group contains a request URL that equals to "/foo"
258
259            ReqURL eq "/foo"
260
261       · Transaction group contains a request cookie header
262
263            ReqHeader:cookie
264
265       · Transaction group doesn't contain a request cookie header
266
267            not ReqHeader:cookie
268
269       · Client request where internal handling took more than 800ms.:
270
271            Timestamp:Process[2] > 0.8
272
273       · Transaction  group contains a request user-agent header that contains
274         "iPod" and the request delivery time exceeds 1 second
275
276            ReqHeader:user-agent ~ "iPod" and Timestamp:Resp[2] > 1.
277
278       · Transaction group contains a backend response status larger  than  or
279         equal to 500
280
281            BerespStatus >= 500
282
283       · Transaction  group  contains  a  request  response status of 304, but
284         where the request did not contain an if-modified-since header
285
286            RespStatus == 304 and not ReqHeader:if-modified-since
287
288       · Transactions that have had backend failures or long delivery time  on
289         their ESI subrequests. (Assumes request grouping mode).
290
291            BerespStatus >= 500 or {2+}Timestamp:Process[2] > 1.
292
293       · Log non-transactional errors. (Assumes raw grouping mode).
294
295            vxid == 0 and Error
296

HISTORY

298       This document was written by Martin Blix Grydeland.
299
301       This document is licensed under the same licence as Varnish itself. See
302       LICENCE for details.
303
304       · Copyright (c) 2006 Verdens Gang AS
305
306       · Copyright (c) 2006-2015 Varnish Software AS
307
308
309
310
311                                                                  VSL-QUERY(7)
Impressum