1VSL-QUERY(7)                                                      VSL-QUERY(7)
2
3
4

NAME

6       vsl-query - Varnish VSL Query Expressions
7

OVERVIEW

9       The  Varnish  VSL Query Expressions extracts transactions from the Var‐
10       nish shared memory log, and perform queries on the transactions  before
11       reporting matches.
12
13       A  transaction  is  a  set  of  log lines that belongs together, e.g. a
14       client request or a backend request. The API monitors the log, and col‐
15       lects  all  log  records that make up a transaction before reporting on
16       that transaction. Transactions can also  be  grouped,  meaning  backend
17       transactions  are  reported  together  with the client transaction that
18       initiated it.
19
20       A query is run on a group of transactions. A query expression  is  true
21       if there is a log record within the group that satisfies the condition.
22       It is false only if none of the log records  satisfies  the  condition.
23       Query expressions can be combined using boolean functions.  In addition
24       to log records, it is possible  to  query  transaction  ids  (vxid)  in
25       query.
26

GROUPING

28       When  grouping  transactions,  there  is  a hierarchy structure showing
29       which transaction initiated what. The level  increases  by  one  on  an
30       'initiated by' relation, so for example a backend transaction will have
31       one higher level than the client transaction that  initiated  it  on  a
32       cache   miss.  Request  restart  transactions  don't  get  their  level
33       increased to make it predictable.
34
35       Levels start counting at 1, except when using raw where it will  always
36       be 0.
37
38       The grouping modes are:
39
40       · session
41
42         All  transactions  initiated  by  a  client  connection  are reported
43         together.  Client  connections  are  open  ended  when   using   HTTP
44         keep-alives, so it is undefined when the session will be reported. If
45         the transaction timeout period is exceeded an incomplete session will
46         be reported. Non-transactional data (vxid == 0) is not reported.
47
48       · request
49
50         Transactions  are  grouped by request, where the set will include the
51         request itself as well as any backend  requests  or  ESI-subrequests.
52         Session data and non-transactional data (vxid == 0) is not reported.
53
54       · vxid
55
56         Transactions  are  not  grouped,  so  each  vxid  is  reported in its
57         entirety. Sessions, requests, ESI-requests and backend  requests  are
58         all  reported  individually.  Non-transactional  data is not reported
59         (vxid == 0). This is the default.
60
61       · raw
62
63         Every log record will make up a transaction of  its  own.  All  data,
64         including non-transactional data will be reported.
65
66   Transaction Hierarchy
67       Example transaction hierarchy using request grouping mode
68
69          Lvl 1: Client request (cache miss)
70            Lvl 2: Backend request
71            Lvl 2: ESI subrequest (cache miss)
72              Lvl 3: Backend request
73              Lvl 3: Backend request (VCL restart)
74              Lvl 3: ESI subrequest (cache miss)
75                Lvl 4: Backend request
76            Lvl 2: ESI subrequest (cache hit)
77

MEMORY USAGE

79       The API will use pointers to shared memory log data as long as possible
80       to keep memory usage at a minimum. But as the shared memory  log  is  a
81       ring  buffer,  data will get overwritten eventually, so the API creates
82       local copies of referenced log data when varnishd comes close to  over‐
83       writing still unreported content.
84
85       This  process  avoids loss of log data in many scenarios, but it is not
86       failsafe: Overruns where varnishd "overtakes" the log reader process in
87       the  ring buffer can still happen when API clients cannot keep up read‐
88       ing and/or copying, for instance due to output blocking.
89
90       Though being unrelated to grouping in principle, copying of log data is
91       particularly  relevant  for session grouping together with long lasting
92       client connections - for this grouping, the logging API client  process
93       is  likely  to consume relevant amounts of memory. As the vxid grouping
94       also logs (potentially long lasting) sessions, it  is  also  likely  to
95       require  memory  for  copies  of log entries, but far less than session
96       grouping.
97

QUERY LANGUAGE

99       A query expression consists of record selection criteria,  and  option‐
100       ally an operator and a value to match against the selected records.
101
102          <record selection criteria> <operator> <operand>
103
104       Additionally,  a  query  expression can occur on the transaction itself
105       rather than log records belonging to the transaction.
106
107          vxid <numerical operator> <integer>
108
109       A vxid query allows you to directly target a specific transacion, whose
110       id  can  be  obtained  from an X-Varnish HTTP header, the default "guru
111       meditation" error page, or Begin and Link log records.
112
113   Record selection criteria
114       The record selection criteria determines what  kind  records  from  the
115       transaction group the expression applies to. Syntax:
116
117          {level}taglist:record-prefix[field]
118
119       Taglist is mandatory, the other components are optional.
120
121       The level limits the expression to a transaction at that level. If left
122       unspecified, the expression is applied to transactions at  all  levels.
123       Level  is  a  positive  integer  or zero. If level is followed by a '+'
124       character, it expresses greater than or equal. If level is followed  by
125       a '-', it expresses less than or equal.
126
127       The  taglist  is  a  comma-separated  list of VSL record tags that this
128       expression should be checked against. Each list element can  be  a  tag
129       name  or  a  tag glob. Globs allow a '*' either in the beginning of the
130       name or at the end, and will select all tags that match either the pre‐
131       fix or subscript. A single '*' will select all tags.
132
133       The  record prefix will further limit the matches to those records that
134       has this prefix as their first part of the record content followed by a
135       colon.  The part of the log record matched against will then be limited
136       to what follows the prefix and colon.  This  is  useful  when  matching
137       against  specific HTTP headers. The record prefix matching is done case
138       insensitive.
139
140       The field will, if present, treat the log record as a white space sepa‐
141       rated  list  of  fields,  and  only  the nth part of the record will be
142       matched against. Fields start counting at 1.
143
144       An expression using only a record selection criteria will  be  true  if
145       there  is  any  record in the transaction group that is selected by the
146       criteria.
147
148   Operators
149       The following matching operators are available:
150
151       · == != < <= > >=
152
153         Numerical comparison. The record contents will be converted to either
154         an integer or a float before comparison, depending on the type of the
155         operand.
156
157       · eq ne
158
159         String comparison. 'eq' tests string equality,  'ne'  tests  for  not
160         equality.
161
162       · ~ !~
163
164         Regular  expression  matching.  '~'  is  a  positive match, '!~' is a
165         non-match.
166
167   Operand
168       The operand is the value the selected records will be matched against.
169
170       An operand can be quoted or unquoted. Quotes can be  either  single  or
171       double  quotes,  and  for  quoted  operands  a backslash can be used to
172       escape the quotes.
173
174       Unquoted operands can only consist of the following characters:
175
176          a-z A-Z 0-9 + - _ . *
177
178       The following types of operands are available:
179
180       · Integer
181
182         A number without any fractional part, valid for the numerical compar‐
183         ison  operators.  The  integer type is used when the operand does not
184         contain any period (.) characters.
185
186       · Float
187
188         A number with a fractional part, valid for the  numerical  comparison
189         operators.  The  float  type  is used when the operand does contain a
190         period (.) character.
191
192       · String
193
194         A sequence of characters, valid for the string equality operators.
195
196       · Regular expression
197
198         A PCRE regular expression. Valid for the  regular  expression  opera‐
199         tors.
200
201   Boolean functions
202       Query  expressions  can be linked together using boolean functions. The
203       following are available, in decreasing precedence:
204
205       · not <expr>
206
207         Inverts the result of <expr>
208
209       · <expr1> and <expr2>
210
211         True only if both expr1 and expr2 are true
212
213       · <expr1> or <expr2>
214
215         True if either of expr1 or expr2 is true
216
217       Expressions can be grouped using parenthesis.
218

QUERY EXPRESSION EXAMPLES

220       · Transaction group contains a request URL that equals to "/foo"
221
222            ReqURL eq "/foo"
223
224       · Transaction group contains a request cookie header
225
226            ReqHeader:cookie
227
228       · Transaction group doesn't contain a request cookie header
229
230            not ReqHeader:cookie
231
232       · Client request where internal handling took more than 800ms.:
233
234            Timestamp:Process[2] > 0.8
235
236       · Transaction group contains a request user-agent header that  contains
237         "iPod" and the request delivery time exceeds 1 second
238
239            ReqHeader:user-agent ~ "iPod" and Timestamp:Resp[2] > 1.
240
241       · Transaction  group  contains a backend response status larger than or
242         equal to 500
243
244            BerespStatus >= 500
245
246       · Transaction group contains a request  response  status  of  304,  but
247         where the request did not contain an if-modified-since header
248
249            RespStatus == 304 and not ReqHeader:if-modified-since
250
251       · Transactions  that have had backend failures or long delivery time on
252         their ESI subrequests. (Assumes request grouping mode).
253
254            BerespStatus >= 500 or {2+}Timestamp:Process[2] > 1.
255
256       · Log non-transactional errors. (Assumes raw grouping mode).
257
258            vxid == 0 and Error
259

HISTORY

261       This document was written by Martin Blix Grydeland.
262
264       This document is licensed under the same licence as Varnish itself. See
265       LICENCE for details.
266
267       · Copyright (c) 2006 Verdens Gang AS
268
269       · Copyright (c) 2006-2015 Varnish Software AS
270
271
272
273
274                                                                  VSL-QUERY(7)
Impressum