1VMOD_QUERYSTRING(3) VMOD_QUERYSTRING(3)
2
6 vmod_querystring - Varnish Query-String Module
7
9 This module is a tool for query-string filtering in Varnish Cache. It
10 works with application/x-www-form-urlencoded strings that are commonly
11 used on the web. Such a query-string is interpreted as a key/values
12 store encoded with the following syntax:
13 key=value&other=value1&other=value2
15 The query-string parsing is very lenient and will tolerate malformed
17 strings. Assuming that a legitimate client might build a malformed
18 query-string that would be empty or include a trailing & or spurious &s
19 in the middle, it may be a good idea not to fail such requests and in‐
20 stead get rid of the noise that might misguide Varnish and hurt your
21 hit rate. Empty parameters are ignored, and parameters are considered
22 empty when their names are empty:
23 next=empty&&previous=empty&=no-name-means-empty&name-only-is-fine
25 Once cleaned:
27 next=empty&previous=empty&name-only-is-fine
28 And since this module works with query-strings, a filter's input is as‐
30 sumed to be a URL. The query-string is then the part of the URL after a
31 question mark when there's one. The rest of the URL is always left un‐
32 touched by the filters. Proper encoding of the URL isn't checked and
33 only separators (?, & and =) are considered.
34 For example:
36 /path?query-string
38 If it doesn't make any difference to your back-end application, you may
40 also sort the parameters inside the query-string and remove duplicates.
41 It will make the hashing more deterministic for Varnish, possibly im‐
42 proving your hit rate even more.
43 new xfilter = querystring.filter(BOOL sort, BOOL uniq, ENUM match)
45 new xfilter = querystring.filter(
46 BOOL sort=0,
47 BOOL uniq=0,
48 ENUM {name, param} match=name
49 )
50 A filter is first set up in vcl_init and then used during transactions.
52 The setup includes the creation of the filter to which you add crite‐
53 ria. During transactions, you may apply the filter to URLs (like
54 req.url or a Location header) or extract the filtered query-string.
55 A filter may sort its parameters, but by default it will maintain or‐
57 der. The filter constructor takes an optional sort argument, you may
58 use its name to improve readability.
59 Example
61 import querystring;
63 sub vcl_init {
65 new qf = querystring.filter(sort = true);
66 qf.add_string("_"); # a timestamp used to bypass caches
67 qf.add_glob("utm_*"); # google analytics parameters
68 qf.add_regex("sess[0-9]+"); # anti-CSRF token
69 }
70 sub vcl_hash {
72 hash_data(qf.apply()); # implicitly on req.url
73 hash_data(req.http.host);
74 return (lookup);
75 }
76 It is also possible for a filter to either match parameters by name, or
78 by themselves. By default the criteria are tested against the name
79 only, leaving alone the contents starting at the equal sign when
80 there's one. For that the constructor takes another optional match ar‐
81 gument that can take either name (default) or param.
82 Example
84 sub vcl_init {
86 new fr = querystring.filter(match = param);
87 fr.add_regex("^lang=fr(-..)?$");
88 }
89 sub vcl_recv {
91 if (fr.extract()) {
92 set req.backend_hint = www_fr;
93 }
94 }
95 Finally, a filter can drop consecutive duplicate parameters if the op‐
97 tional uniq argument. Combined with a sort, this effectively removes
98 all the duplicates from the URL. The match argument determines how pa‐
99 rameters are considered duplicates.
100 Since most of the time it is either req.url or bereq.url that is fil‐
102 tered, omitting the url argument for functions that take one will de‐
103 fault to one of them depending on whether the function was called dur‐
104 ing a client or backend transaction.
105 VOID xfilter.add_string(STRING)
107 Description
108 Use the string argument as an exact-match criterion.
109 VOID xfilter.add_glob(STRING)
111 Description
112 Use the string argument as a GLOB pattern matching criterion.
113 The underlying fnmatch function may fail, in which case the
114 query-param is kept to avoid spurious filtering and the error is
115 logged.
116 VOID xfilter.add_regex(REGEX)
118 Description
119 The string argument is compiled to a regular expression that
120 will be used as the matching criterion. If the regular expres‐
121 sion is invalid, it will fail the vcl.load command and report
122 the error in the varnish-cli output.
123 STRING xfilter.apply([STRING url], ENUM mode)
125 STRING xfilter.apply([STRING url], ENUM {keep, drop} mode=drop)
126 Description
128 The url argument is filtered against the filter's criteria, pre‐
129 viously added using the add_*() methods in vcl_init. The result
130 is a new URL with a clean (possibly sorted and de-duplicated)
131 query-string.
132 Depending on the optional mode argument the matching criteria
134 act as a black list for drop (default) or a white list for keep.
135 In the resulting URL the query-string will either contain only
136 the matching parameters or everything but them.
137 Example
139 set req.url = myfilter.apply(mode = drop);
141 STRING xfilter.extract([STRING url], ENUM mode)
143 STRING xfilter.extract(
144 [STRING url],
145 ENUM {keep, drop} mode=drop
146 )
147 Description
149 This method works exactly like .apply and discards all URL parts
150 but the query-string.
151 STRING clean([STRING url])
153 Description
154 This is a shorthand function that works like applying a filter
155 with no criteria. It will keep all parameters and shave off the
156 empty ones.
157 Example
159 set req.url = querystring.clean();
161 STRING sort([STRING url], BOOL uniq=0)
163 Description
164 This is a shorthand function that works like applying a sort‐
165 ing-enabled filter with no criteria matching full parameters,
166 not just their names. It will keep all parameters and shave off
167 the empty ones. If the uniq argument is true duplicate parame‐
168 ters are also removed.
169 Example
171 set req.url = querystring.sort();
173 STRING remove([STRING url])
175 Description
176 This is a shorthand function that works like applying a filter
177 with a catch-all criteria. It will return the given URL with its
178 query-string removed. For better efficiency, it is not backed by
179 an actual filter.
180 Example
182 set req.url = querystring.remove();
184
186 Copyright (C) 2012-2021 Dridi Boukelmoune. License GPLv3+: GNU GPL
187 version 3 or later <http://gnu.org/licenses/gpl.html>.
188 This is free software: you are free to change and redistribute it.
190 There is NO WARRANTY, to the extent permitted by law.
191
193 vcl(7), varnishd(1), varnish-cli(7), glob(7)
194 RFC 1866 Section 8.2.1, The form-urlencoded Media Type
196 RFC 3986 Section 3, Syntax Components
197 RFC 7234 Section 2, Overview of Cache Operation
198 VMOD_QUERYSTRING(3)