1VMOD_QUERYSTRING(3)                                        VMOD_QUERYSTRING(3)
2
3
4

NAME

6       vmod_querystring - Varnish Query-String Module
7

DESCRIPTION

9       This  module  is a tool for query-string filtering in Varnish Cache. It
10       works with application/x-www-form-urlencoded strings that are  commonly
11       used  on  the  web.  Such a query-string is interpreted as a key/values
12       store encoded with the following syntax:
13
14          key=value&other=value1&other=value2
15
16       The query-string parsing is very lenient and  will  tolerate  malformed
17       strings.   Assuming  that  a  legitimate client might build a malformed
18       query-string that would be empty or include a trailing & or spurious &s
19       in  the middle, it may be a good idea not to fail such requests and in‐
20       stead get rid of the noise that might misguide Varnish  and  hurt  your
21       hit  rate.  Empty parameters are ignored, and parameters are considered
22       empty when their names are empty:
23
24          next=empty&&previous=empty&=no-name-means-empty&no-value-is-fine
25
26          Once cleaned:
27          next=empty&previous=empty&no-value-is-fine
28
29       And since this module works with query-strings, a filter's input is as‐
30       sumed to be a URL. The query-string is then the part of the URL after a
31       question mark when there's one. The rest of the URL is always left  un‐
32       touched  by  the filters.  Proper encoding of the URL isn't checked and
33       only separators (?, & and =) are considered.
34
35       For example:
36
37          /path?query-string
38
39       If it doesn't make any difference to your back-end application, you may
40       also sort the parameters inside the query-string and remove duplicates.
41       It will make the hashing more deterministic for Varnish,  possibly  im‐
42       proving your hit rate even more.
43
44   new xfilter = querystring.filter(BOOL sort, BOOL uniq, ENUM match)
45          new xfilter = querystring.filter(
46             BOOL sort=0,
47             BOOL uniq=0,
48             ENUM {name, param} match=name
49          )
50
51       A filter is first set up in vcl_init and then used during transactions.
52       The setup includes the creation of the filter to which you  add  crite‐
53       ria.  During  transactions,  you  may  apply  the  filter to URLs (like
54       req.url or a Location header) or extract the filtered query-string.
55
56       A filter may sort its parameters, but by default it will  maintain  or‐
57       der.  The  filter  constructor takes an optional sort argument, you may
58       use its name to improve readability.
59
60       Example
61
62                 import querystring;
63
64                 sub vcl_init {
65                     new qf = querystring.filter(sort = true);
66                     qf.add_string("_"); # a timestamp used to bypass caches
67                     qf.add_glob("utm_*"); # google analytics parameters
68                     qf.add_regex("sess[0-9]+"); # anti-CSRF token
69                 }
70
71                 sub vcl_hash {
72                     hash_data(qf.apply()); # implicitly on req.url
73                     hash_data(req.http.host);
74                     return (lookup);
75                 }
76
77       It is also possible for a filter to either match parameters by name, or
78       by  themselves.  By  default  the  criteria are tested against the name
79       only, leaving alone the  contents  starting  at  the  equal  sign  when
80       there's  one. For that the constructor takes another optional match ar‐
81       gument that can take either name (default) or param.
82
83       Example
84
85                 sub vcl_init {
86                     new fr = querystring.filter(match = param);
87                     fr.add_regex("^lang=fr(-..)?$");
88                 }
89
90                 sub vcl_recv {
91                     if (fr.extract()) {
92                         set req.backend_hint = www_fr;
93                     }
94                 }
95
96       Finally, a filter can drop consecutive duplicate parameters if the  op‐
97       tional  uniq  argument.  Combined with a sort, this effectively removes
98       all the duplicates from the URL. The match argument determines how  pa‐
99       rameters are considered duplicates.
100
101       Since  most  of the time it is either req.url or bereq.url that is fil‐
102       tered, omitting the url argument for functions that take one  will  de‐
103       fault  to one of them depending on whether the function was called dur‐
104       ing a client or backend transaction.
105
106   VOID xfilter.add_string(STRING)
107       Description
108              Use the string argument as an exact-match criterion.
109
110   VOID xfilter.add_glob(STRING)
111       Description
112              Use the string argument as a GLOB  pattern  matching  criterion.
113              The  underlying  fnmatch  function  may  fail, in which case the
114              query-param is kept to avoid spurious filtering and the error is
115              logged.
116
117   VOID xfilter.add_regex(STRING)
118       Description
119              The  string  argument  is  compiled to a regular expression that
120              will be used as the matching criterion. If the  regular  expres‐
121              sion  is  invalid,  it will fail the vcl.load command and report
122              the error in the varnish-cli output.
123
124   STRING xfilter.apply([STRING url], ENUM mode)
125          STRING xfilter.apply([STRING url], ENUM {keep, drop} mode=drop)
126
127       Description
128              The url argument is filtered against the filter's criteria, pre‐
129              viously  added using the add_*() methods in vcl_init. The result
130              is a new URL with a clean (possibly  sorted  and  de-duplicated)
131              query-string.
132
133              Depending  on  the  optional mode argument the matching criteria
134              act as a black list for drop (default) or a white list for keep.
135              In  the  resulting URL the query-string will either contain only
136              the matching parameters or everything but them.
137
138       Example
139
140                 set req.url = myfilter.apply(mode = drop);
141
142   STRING xfilter.extract([STRING url], ENUM mode)
143          STRING xfilter.extract(
144                [STRING url],
145                ENUM {keep, drop} mode=drop
146          )
147
148       Description
149              This method works exactly like .apply and discards all URL parts
150              but the query-string.
151
152   STRING clean([STRING url])
153       Description
154              This  is  a shorthand function that works like applying a filter
155              with no criteria. It will keep all parameters and shave off  the
156              empty ones.
157
158       Example
159
160                 set req.url = querystring.clean();
161
162   STRING sort([STRING url], BOOL uniq=0)
163       Description
164              This  is  a  shorthand function that works like applying a sort‐
165              ing-enabled filter with no criteria  matching  full  parameters,
166              not  just their names. It will keep all parameters and shave off
167              the empty ones. If the uniq argument is true  duplicate  parame‐
168              ters are also removed.
169
170       Example
171
172                 set req.url = querystring.sort();
173
174   STRING remove([STRING url])
175       Description
176              This  is  a shorthand function that works like applying a filter
177              with a catch-all criteria. It will return the given URL with its
178              query-string removed. For better efficiency, it is not backed by
179              an actual filter.
180
181       Example
182
183                 set req.url = querystring.remove();
184
186       Copyright (C) 2012-2018  Dridi Boukelmoune.  License  GPLv3+:  GNU  GPL
187       version 3 or later <http://gnu.org/licenses/gpl.html>.
188
189       This  is  free  software:  you  are free to change and redistribute it.
190       There is NO WARRANTY, to the extent permitted by law.
191

SEE ALSO

193       vcl(7), varnishd(1), varnish-cli(7), glob(7)
194
195       RFC 1866 Section 8.2.1, The form-urlencoded Media Type
196       RFC 3986 Section 3, Syntax Components
197       RFC 7234 Section 2, Overview of Cache Operation
198
199
200
201
202
203                                                           VMOD_QUERYSTRING(3)
Impressum