1VSL-QUERY(7) VSL-QUERY(7)
2
3
4
6 vsl-query - Varnish VSL Query Expressions
7
9 The Varnish VSL Query Expressions extracts transactions from the Var‐
10 nish shared memory log, and perform queries on the transactions before
11 reporting matches.
12
13 A transaction is a set of log lines that belongs together, e.g. a
14 client request or a backend request. The API monitors the log, and col‐
15 lects all log records that make up a transaction before reporting on
16 that transaction. Transactions can also be grouped, meaning backend
17 transactions are reported together with the client transaction that
18 initiated it.
19
20 A query is run on a group of transactions. A query expression is true
21 if there is a log record within the group that satisfies the condition.
22 It is false only if none of the log records satisfies the condition.
23 Query expressions can be combined using boolean functions. In addition
24 to log records, it is possible to query transaction ids (vxid) in
25 query.
26
28 When grouping transactions, there is a hierarchy structure showing
29 which transaction initiated what. The level increases by one on an
30 'initiated by' relation, so for example a backend transaction will have
31 one higher level than the client transaction that initiated it on a
32 cache miss. Request restart transactions don't get their level in‐
33 creased to make it predictable.
34
35 Levels start counting at 1, except when using raw where it will always
36 be 0.
37
38 The grouping modes are:
39
40 • session
41
42 All transactions initiated by a client connection are reported to‐
43 gether. Client connections are open ended when using HTTP
44 keep-alives, so it is undefined when the session will be reported. If
45 the transaction timeout period is exceeded an incomplete session will
46 be reported. Non-transactional data (vxid == 0) is not reported.
47
48 • request
49
50 Transactions are grouped by request, where the set will include the
51 request itself as well as any backend requests or ESI-subrequests.
52 Session data and non-transactional data (vxid == 0) is not reported.
53
54 • vxid
55
56 Transactions are not grouped, so each vxid is reported in its en‐
57 tirety. Sessions, requests, ESI-requests and backend requests are all
58 reported individually. Non-transactional data is not reported (vxid
59 == 0). This is the default.
60
61 • raw
62
63 Every log record will make up a transaction of its own. All data, in‐
64 cluding non-transactional data will be reported.
65
66 Transaction Hierarchy
67 Example transaction hierarchy using request grouping mode
68
69 Lvl 1: Client request (cache miss)
70 Lvl 2: Backend request
71 Lvl 2: ESI subrequest (cache miss)
72 Lvl 3: Backend request
73 Lvl 3: Backend request (VCL restart)
74 Lvl 3: ESI subrequest (cache miss)
75 Lvl 4: Backend request
76 Lvl 2: ESI subrequest (cache hit)
77
79 The API will use pointers to shared memory log data as long as possible
80 to keep memory usage at a minimum. But as the shared memory log is a
81 ring buffer, data will get overwritten eventually, so the API creates
82 local copies of referenced log data when varnishd comes close to over‐
83 writing still unreported content.
84
85 This process avoids loss of log data in many scenarios, but it is not
86 failsafe: Overruns where varnishd "overtakes" the log reader process in
87 the ring buffer can still happen when API clients cannot keep up read‐
88 ing and/or copying, for instance due to output blocking.
89
90 Though being unrelated to grouping in principle, copying of log data is
91 particularly relevant for session grouping together with long lasting
92 client connections - for this grouping, the logging API client process
93 is likely to consume relevant amounts of memory. As the vxid grouping
94 also logs (potentially long lasting) sessions, it is also likely to re‐
95 quire memory for copies of log entries, but far less than session
96 grouping.
97
99 A query expression consists of record selection criteria, and option‐
100 ally an operator and a value to match against the selected records.
101
102 <record selection criteria> <operator> <operand>
103
104 Additionally, a query expression can occur on the transaction itself
105 rather than log records belonging to the transaction.
106
107 vxid <numerical operator> <integer>
108
109 A vxid query allows you to directly target a specific transacion, whose
110 id can be obtained from an X-Varnish HTTP header, the default "guru
111 meditation" error page, or Begin and Link log records.
112
113 A query must fit on a single line, but it is possible to pass multiple
114 queries at once, one query per line. Empty lines are ignored, and the
115 list of queries is treated as if the 'or' operator was used to combine
116 them.
117
118 For example this list of queries:
119
120 # catch varnish errors
121 *Error
122
123 # catch backend errors
124 BerespStatus >= 500
125
126 is identical to this query:
127
128 (*Error) or (BerespStatus >= 500)
129
130 Comments can be used and will be ignored, they start with the '#' char‐
131 acter, which may be more useful when the query is read from a file.
132
133 For very long queries that couldn't easily be split into multiple
134 queries it is possible to break them into multiple lines with a back‐
135 slash preceding an end of line.
136
137 For example this query:
138
139 BerespStatus >= 500
140
141 is identical to this query:
142
143 BerespStatus \
144 >= \
145 500
146
147 A backslash-newline sequence doesn't continue a comment on the next
148 line and isn't allowed in a quoted string.
149
150 Record selection criteria
151 The record selection criteria determines what kind records from the
152 transaction group the expression applies to. Syntax:
153
154 {level}taglist:record-prefix[field]
155
156 Taglist is mandatory, the other components are optional.
157
158 The level limits the expression to a transaction at that level. If left
159 unspecified, the expression is applied to transactions at all levels.
160 Level is a positive integer or zero. If level is followed by a '+'
161 character, it expresses greater than or equal. If level is followed by
162 a '-', it expresses less than or equal.
163
164 The taglist is a comma-separated list of VSL record tags that this ex‐
165 pression should be checked against. Each list element can be a tag name
166 or a tag glob. Globs allow a '*' either in the beginning of the name or
167 at the end, and will select all tags that match either the prefix or
168 subscript. A single '*' will select all tags.
169
170 The record prefix will further limit the matches to those records that
171 has this prefix as their first part of the record content followed by a
172 colon. The part of the log record matched against will then be limited
173 to what follows the prefix and colon. This is useful when matching
174 against specific HTTP headers. The record prefix matching is done case
175 insensitive.
176
177 The field will, if present, treat the log record as a white space sepa‐
178 rated list of fields, and only the nth part of the record will be
179 matched against. Fields start counting at 1.
180
181 An expression using only a record selection criteria will be true if
182 there is any record in the transaction group that is selected by the
183 criteria.
184
185 Operators
186 The following matching operators are available:
187
188 • == != < <= > >=
189
190 Numerical comparison. The record contents will be converted to either
191 an integer or a float before comparison, depending on the type of the
192 operand.
193
194 • eq ne
195
196 String comparison. 'eq' tests string equality, 'ne' tests for not
197 equality.
198
199 • ~ !~
200
201 Regular expression matching. '~' is a positive match, '!~' is a
202 non-match.
203
204 Operand
205 The operand is the value the selected records will be matched against.
206
207 An operand can be quoted or unquoted. Quotes can be either single or
208 double quotes, and for quoted operands a backslash can be used to es‐
209 cape the quotes.
210
211 Unquoted operands can only consist of the following characters:
212
213 a-z A-Z 0-9 + - _ . *
214
215 The following types of operands are available:
216
217 • Integer
218
219 A number without any fractional part, valid for the numerical compar‐
220 ison operators. The integer type is used when the operand does not
221 contain any period (.) nor exponent (e) characters. However if the
222 record evaluates as a float, only its integral part is used for the
223 comparison.
224
225 • Float
226
227 A number with a fractional part, valid for the numerical comparison
228 operators. The float type is used when the operand does contain a pe‐
229 riod (.) or exponent (e) character.
230
231 • String
232
233 A sequence of characters, valid for the string equality operators.
234
235 • Regular expression
236
237 A PCRE regular expression. Valid for the regular expression opera‐
238 tors.
239
240 Boolean functions
241 Query expressions can be linked together using boolean functions. The
242 following are available, in decreasing precedence:
243
244 • not <expr>
245
246 Inverts the result of <expr>
247
248 • <expr1> and <expr2>
249
250 True only if both expr1 and expr2 are true
251
252 • <expr1> or <expr2>
253
254 True if either of expr1 or expr2 is true
255
256 Expressions can be grouped using parenthesis.
257
259 • Transaction group contains a request URL that equals to "/foo"
260
261 ReqURL eq "/foo"
262
263 • Transaction group contains a request cookie header
264
265 ReqHeader:cookie
266
267 • Transaction group doesn't contain a request cookie header
268
269 not ReqHeader:cookie
270
271 • Client request where internal handling took more than 800ms.:
272
273 Timestamp:Process[2] > 0.8
274
275 • Transaction group contains a request user-agent header that contains
276 "iPod" and the request delivery time exceeds 1 second
277
278 ReqHeader:user-agent ~ "iPod" and Timestamp:Resp[2] > 1.
279
280 • Transaction group contains a backend response status larger than or
281 equal to 500
282
283 BerespStatus >= 500
284
285 • Transaction group contains a request response status of 304, but
286 where the request did not contain an if-modified-since header
287
288 RespStatus == 304 and not ReqHeader:if-modified-since
289
290 • Transactions that have had backend failures or long delivery time on
291 their ESI subrequests. (Assumes request grouping mode).
292
293 BerespStatus >= 500 or {2+}Timestamp:Process[2] > 1.
294
295 • Log non-transactional errors. (Assumes raw grouping mode).
296
297 vxid == 0 and Error
298
300 This document was initially written by Martin Blix Grydeland and
301 amended by others.
302
304 This document is licensed under the same licence as Varnish itself. See
305 LICENCE for details.
306
307 • Copyright (c) 2006 Verdens Gang AS
308
309 • Copyright (c) 2006-2015 Varnish Software AS
310
311
312
313
314 VSL-QUERY(7)