1VSL-QUERY(7)                                                      VSL-QUERY(7)
2
3
4

NAME

6       vsl-query - Varnish VSL Query Expressions
7

OVERVIEW

9       The  Varnish  VSL Query Expressions extracts transactions from the Var‐
10       nish shared memory log, and perform queries on the transactions  before
11       reporting matches.
12
13       A  transaction  is  a  set  of  log lines that belongs together, e.g. a
14       client request or a backend request. The API monitors the log, and col‐
15       lects  all  log  records that make up a transaction before reporting on
16       that transaction. Transactions can also  be  grouped,  meaning  backend
17       transactions  are  reported  together  with the client transaction that
18       initiated it.
19
20       A query is run on a group of transactions. A query expression  is  true
21       if there is a log record within the group that satisfies the condition.
22       It is false only if none of the log records  satisfies  the  condition.
23       Query expressions can be combined using boolean functions.  In addition
24       to log records, it is possible  to  query  transaction  ids  (vxid)  in
25       query.
26

GROUPING

28       When  grouping  transactions,  there  is  a hierarchy structure showing
29       which transaction initiated what. The level  increases  by  one  on  an
30       'initiated by' relation, so for example a backend transaction will have
31       one higher level than the client transaction that  initiated  it  on  a
32       cache  miss.  Request  restart  transactions  don't get their level in‐
33       creased to make it predictable.
34
35       Levels start counting at 1, except when using raw where it will  always
36       be 0.
37
38       The grouping modes are:
39
40session
41
42         All  transactions  initiated  by a client connection are reported to‐
43         gether.  Client  connections  are  open   ended   when   using   HTTP
44         keep-alives, so it is undefined when the session will be reported. If
45         the transaction timeout period is exceeded an incomplete session will
46         be reported. Non-transactional data (vxid == 0) is not reported.
47
48request
49
50         Transactions  are  grouped by request, where the set will include the
51         request itself as well as any backend  requests  or  ESI-subrequests.
52         Session data and non-transactional data (vxid == 0) is not reported.
53
54vxid
55
56         Transactions  are  not  grouped,  so each vxid is reported in its en‐
57         tirety. Sessions, requests, ESI-requests and backend requests are all
58         reported  individually.  Non-transactional data is not reported (vxid
59         == 0). This is the default.
60
61raw
62
63         Every log record will make up a transaction of its own. All data, in‐
64         cluding non-transactional data will be reported.
65
66   Transaction Hierarchy
67       Example transaction hierarchy using request grouping mode
68
69          Lvl 1: Client request (cache miss)
70            Lvl 2: Backend request
71            Lvl 2: ESI subrequest (cache miss)
72              Lvl 3: Backend request
73              Lvl 3: Backend request (VCL restart)
74              Lvl 3: ESI subrequest (cache miss)
75                Lvl 4: Backend request
76            Lvl 2: ESI subrequest (cache hit)
77

MEMORY USAGE

79       The API will use pointers to shared memory log data as long as possible
80       to keep memory usage at a minimum. But as the shared memory  log  is  a
81       ring  buffer,  data will get overwritten eventually, so the API creates
82       local copies of referenced log data when varnishd comes close to  over‐
83       writing still unreported content.
84
85       This  process  avoids loss of log data in many scenarios, but it is not
86       failsafe: Overruns where varnishd "overtakes" the log reader process in
87       the  ring buffer can still happen when API clients cannot keep up read‐
88       ing and/or copying, for instance due to output blocking.
89
90       Though being unrelated to grouping in principle, copying of log data is
91       particularly  relevant  for session grouping together with long lasting
92       client connections - for this grouping, the logging API client  process
93       is  likely  to consume relevant amounts of memory. As the vxid grouping
94       also logs (potentially long lasting) sessions, it is also likely to re‐
95       quire  memory  for  copies  of  log  entries, but far less than session
96       grouping.
97

QUERY LANGUAGE

99       A query expression consists of record selection criteria,  and  option‐
100       ally an operator and a value to match against the selected records.
101
102          <record selection criteria> <operator> <operand>
103
104       Additionally,  a  query  expression can occur on the transaction itself
105       rather than log records belonging to the transaction.
106
107          vxid <numerical operator> <integer>
108
109       A vxid query allows you to directly target a specific transacion, whose
110       id  can  be  obtained  from an X-Varnish HTTP header, the default "guru
111       meditation" error page, or Begin and Link log records.
112
113       A query must fit on a single line, but it is possible to pass  multiple
114       queries  at  once, one query per line. Empty lines are ignored, and the
115       list of queries is treated as if the 'or' operator was used to  combine
116       them.
117
118       For example this list of queries:
119
120          # catch varnish errors
121          *Error
122
123          # catch backend errors
124          BerespStatus >= 500
125
126       is identical to this query:
127
128          (*Error) or (BerespStatus >= 500)
129
130       Comments can be used and will be ignored, they start with the '#' char‐
131       acter, which may be more useful when the query is read from a file.
132
133       For very long queries that  couldn't  easily  be  split  into  multiple
134       queries  it  is possible to break them into multiple lines with a back‐
135       slash preceding an end of line.
136
137       For example this query:
138
139          BerespStatus >= 500
140
141       is identical to this query:
142
143          BerespStatus \
144          >= \
145          500
146
147       A backslash-newline sequence doesn't continue a  comment  on  the  next
148       line and isn't allowed in a quoted string.
149
150   Record selection criteria
151       The  record  selection  criteria  determines what kind records from the
152       transaction group the expression applies to. Syntax:
153
154          {level}taglist:record-prefix[field]
155
156       Taglist is mandatory, the other components are optional.
157
158       The level limits the expression to a transaction at that level. If left
159       unspecified,  the  expression is applied to transactions at all levels.
160       Level is a positive integer or zero. If level  is  followed  by  a  '+'
161       character,  it expresses greater than or equal. If level is followed by
162       a '-', it expresses less than or equal.
163
164       The taglist is a comma-separated list of VSL record tags that this  ex‐
165       pression should be checked against. Each list element can be a tag name
166       or a tag glob. Globs allow a '*' either in the beginning of the name or
167       at  the  end,  and will select all tags that match either the prefix or
168       subscript. A single '*' will select all tags.
169
170       The record prefix will further limit the matches to those records  that
171       has this prefix as their first part of the record content followed by a
172       colon. The part of the log record matched against will then be  limited
173       to  what  follows  the  prefix  and colon. This is useful when matching
174       against specific HTTP headers. The record prefix matching is done  case
175       insensitive.
176
177       The field will, if present, treat the log record as a white space sepa‐
178       rated list of fields, and only the nth  part  of  the  record  will  be
179       matched against. Fields start counting at 1.
180
181       An  expression  using  only a record selection criteria will be true if
182       there is any record in the transaction group that is  selected  by  the
183       criteria.
184
185   Operators
186       The following matching operators are available:
187
188       • == != < <= > >=
189
190         Numerical comparison. The record contents will be converted to either
191         an integer or a float before comparison, depending on the type of the
192         operand.
193
194       • eq ne
195
196         String  comparison.  'eq'  tests  string equality, 'ne' tests for not
197         equality.
198
199       • ~ !~
200
201         Regular expression matching. '~' is  a  positive  match,  '!~'  is  a
202         non-match.
203
204   Operand
205       The operand is the value the selected records will be matched against.
206
207       An  operand  can  be quoted or unquoted. Quotes can be either single or
208       double quotes, and for quoted operands a backslash can be used  to  es‐
209       cape the quotes.
210
211       Unquoted operands can only consist of the following characters:
212
213          a-z A-Z 0-9 + - _ . *
214
215       The following types of operands are available:
216
217       • Integer
218
219         A number without any fractional part, valid for the numerical compar‐
220         ison operators. The integer type is used when the  operand  does  not
221         contain  any  period  (.) nor exponent (e) characters. However if the
222         record evaluates as a float, only its integral part is used  for  the
223         comparison.
224
225       • Float
226
227         A  number  with a fractional part, valid for the numerical comparison
228         operators. The float type is used when the operand does contain a pe‐
229         riod (.) or exponent (e) character.
230
231       • String
232
233         A sequence of characters, valid for the string equality operators.
234
235       • Regular expression
236
237         A  PCRE  regular  expression. Valid for the regular expression opera‐
238         tors.
239
240   Boolean functions
241       Query expressions can be linked together using boolean  functions.  The
242       following are available, in decreasing precedence:
243
244       • not <expr>
245
246         Inverts the result of <expr>
247
248       • <expr1> and <expr2>
249
250         True only if both expr1 and expr2 are true
251
252       • <expr1> or <expr2>
253
254         True if either of expr1 or expr2 is true
255
256       Expressions can be grouped using parenthesis.
257

QUERY EXPRESSION EXAMPLES

259       • Transaction group contains a request URL that equals to "/foo"
260
261            ReqURL eq "/foo"
262
263       • Transaction group contains a request cookie header
264
265            ReqHeader:cookie
266
267       • Transaction group doesn't contain a request cookie header
268
269            not ReqHeader:cookie
270
271       • Client request where internal handling took more than 800ms.:
272
273            Timestamp:Process[2] > 0.8
274
275       • Transaction  group contains a request user-agent header that contains
276         "iPod" and the request delivery time exceeds 1 second
277
278            ReqHeader:user-agent ~ "iPod" and Timestamp:Resp[2] > 1.
279
280       • Transaction group contains a backend response status larger  than  or
281         equal to 500
282
283            BerespStatus >= 500
284
285       • Transaction  group  contains  a  request  response status of 304, but
286         where the request did not contain an if-modified-since header
287
288            RespStatus == 304 and not ReqHeader:if-modified-since
289
290       • Transactions that have had backend failures or long delivery time  on
291         their ESI subrequests. (Assumes request grouping mode).
292
293            BerespStatus >= 500 or {2+}Timestamp:Process[2] > 1.
294
295       • Log non-transactional errors. (Assumes raw grouping mode).
296
297            vxid == 0 and Error
298

HISTORY

300       This  document  was  initially  written  by  Martin  Blix Grydeland and
301       amended by others.
302
304       This document is licensed under the same licence as Varnish itself. See
305       LICENCE for details.
306
307       • Copyright (c) 2006 Verdens Gang AS
308
309       • Copyright (c) 2006-2015 Varnish Software AS
310
311
312
313
314                                                                  VSL-QUERY(7)
Impressum