1VMOD_BLOB(3) VMOD_BLOB(3)
2
6 vmod_blob - utilities for the VCL blob type
7
9 import blob [from "path"] ;
10 BLOB decode(ENUM decoding, INT length, STRING encoded)
12 STRING encode(ENUM encoding, ENUM case, BLOB blob)
14 STRING transcode(ENUM decoding, ENUM encoding, ENUM case, INT length, STRING encoded)
16 BOOL same(BLOB, BLOB)
18 BOOL equal(BLOB, BLOB)
20 INT length(BLOB)
22 BLOB sub(BLOB, BYTES length, BYTES offset)
24 new xblob = blob.blob(ENUM decoding, STRING encoded)
26 BLOB xblob.get()
28 STRING xblob.encode(ENUM encoding, ENUM case)
30
32 This VMOD provides utility functions and an object for the VCL data
33 type BLOB, which may contain arbitrary data of any length.
34 Examples:
36 sub vcl_init {
38 # Create blob objects from encodings such as base64 or hex.
39 new myblob = blob.blob(BASE64, "Zm9vYmFy");
40 new yourblob = blob.blob(encoded="666F6F", decoding=HEX);
41 }
42 sub vcl_deliver {
44 # The .get() method retrieves the BLOB from an object.
45 set resp.http.MyBlob-As-Hex
46 = blob.encode(blob=myblob.get(), encoding=HEX);
47 # The .encode() method efficiently retrieves an encoding.
49 set resp.http.YourBlob-As-Base64 = yourblob.encode(BASE64);
50 # decode() and encode() functions convert blobs to text and
52 # vice versa at runtime.
53 set resp.http.Base64-Encoded
54 = blob.encode(BASE64,
55 blob=blob.decode(HEX,
56 encoded=req.http.Hex-Encoded));
57 }
58 sub vcl_recv {
60 # transcode() converts from one encoding to another.
61 # case=UPPER specifies upper-case hex digits A-F.
62 set req.http.Hex-Encoded
63 = blob.transcode(decoding=BASE64, encoding=HEX,
64 case=UPPER, encoded="YmF6");
65 # transcode() from URL to IDENTITY effects a URL decode.
67 set req.url = blob.transcode(encoded=req.url, decoding=URL);
68 # transcode() from IDENTITY to URL effects a URL encode.
70 set req.http.url_urlcoded
71 = blob.transcode(encoded=req.url, encoding=URL);
72 }
73
75 Binary-to-text encoding schemes are specified by ENUMs in the VMOD's
76 constructor, methods and functions. Decodings convert a (possibly con‐
77 catenated) string into a blob, while encodings convert a blob into a
78 string.
79 ENUM values for an encoding scheme can be one of:
81 · IDENTITY
83 · BASE64
85 · BASE64URL
87 · BASE64URLNOPAD
89 · HEX
91 · URL
93 Empty strings are decoded into a "null blob" (of length 0), and con‐
95 versely a null blob is encoded as the empty string.
96 For encodings with HEX or URL, you may also specify a case ENUM with
98 one of the values LOWER, UPPER or DEFAULT to produce a string with
99 lower- or uppercase hex digits (in [a-f] or [A-F]). The default value
100 for case is DEFAULT, which for HEX and URL means the same as LOWER.
101 The case ENUM is not relevant for decodings; HEX or URL strings to be
103 decoded as BLOBs may have hex digits in either case, or in mixed case.
104 The case ENUM MUST be set to DEFAULT for the other encodings (BASE64*
106 and IDENTITY). You cannot, for example, produce an uppercase string by
107 using the IDENTITY scheme with case=UPPER. To change the case of a
108 string, use the toupper or tolower functions from vmod_std(3).
109 IDENTITY
111 The simplest encoding converts between the BLOB and STRING data types,
112 leaving the contents byte-identical.
113 Note that a BLOB may contain a null byte at any position before its
115 end; if such a BLOB is decoded with IDENTITY, the resulting STRING will
116 have a null byte at that position. Since VCL strings, like C strings,
117 are represented with a terminating null byte, the string will be trun‐
118 cated, appearing to contain less data than the original blob. For exam‐
119 ple:
120 # Decode from the hex encoding for "foo\0bar".
122 # The header will be seen as "foo".
123 set resp.http.Trunced-Foo1
124 = blob.encode(IDENTITY, blob=blob.decode(HEX,
125 encoded="666f6f00626172"));
126 IDENTITY is the default encoding and decoding. So the above can also be
128 written as:
129 # Decode from the hex encoding for "foo\0bar".
131 # The header will be seen as "foo".
132 set resp.http.Trunced-Foo2
133 = blob.encode(blob=blob.decode(HEX, encoded="666f6f00626172"));
134 The case ENUM MUST be set to DEFAULT for IDENTITY encodings.
136 BASE64*
138 The base64 encoding schemes use 4 characters to encode 3 bytes. There
139 are no newlines or maximal line lengths -- whitespace is not permitted.
140 The BASE64 encoding uses the alphanumeric characters, + and /; and
142 encoded strings are padded with the = character so that their length is
143 always a multiple of four.
144 The BASE64URL encoding also uses the alphanumeric characters, but - and
146 _ instead of + and /, so that an encoded string can be used safely in a
147 URL. This scheme also uses the padding character =.
148 The BASE64URLNOPAD encoding uses the same alphabet as BASE6URL, but
150 leaves out the padding. Thus the length of an encoding with this scheme
151 is not necessarily a multiple of four.
152 The case ENUM MUST be set to DEFAULT for for all of the BASE64* encod‐
154 ings.
155 HEX
157 The HEX encoding scheme converts hex strings into blobs and vice versa.
158 For encodings, you may use the case ENUM to specify upper- or lowercase
159 hex digits A through f (default DEFAULT, which means the same as
160 LOWER). A prefix such as 0x is not used for an encoding and is illegal
161 for a decoding.
162 If a hex string to be decoded has an odd number of digits, it is
164 decoded as if a 0 is prepended to it; that is, the first digit is
165 interpreted as representing the least significant nibble of the first
166 byte. For example:
167 # The concatenated string is "abcdef0", and is decoded as "0abcdef0".
169 set resp.http.First = "abc";
170 set resp.http.Second = "def0";
171 set resp.http.Hex-Decoded
172 = blob.encode(HEX, blob=blob.decode(HEX,
173 encoded=resp.http.First + resp.http.Second));
174 URL
176 The URL decoding replaces any %<2-hex-digits> substrings with the
177 binary value of the hexadecimal number after the % sign.
178 The URL encoding implements "percent encoding" as per RFC3986. The case
180 ENUM determines the case of the hex digits, but does not affect alpha‐
181 betic characters that are not percent-encoded.
182 BLOB decode(ENUM decoding, INT length, STRING encoded)
184 BLOB decode(
185 ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} decoding=IDENTITY,
186 INT length=0,
187 STRING encoded
188 )
189 Returns the BLOB derived from the string encoded according to the
191 scheme specified by decoding.
192 If length > 0, only decode the first length characters of the encoded
194 string. If length <= 0 or greater than the length of the string, then
195 decode the entire string. The default value of length is 0.
196 decoding defaults to IDENTITY.
198 Example:
200 blob.decode(BASE64, encoded="Zm9vYmFyYmF6");
202 # same with named parameters
204 blob.decode(encoded="Zm9vYmFyYmF6", decoding=BASE64);
205 # convert string to blob
207 blob.decode(encoded="foo");
208 STRING encode(ENUM encoding, ENUM case, BLOB blob)
210 STRING encode(
211 ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} encoding=IDENTITY,
212 ENUM {LOWER, UPPER, DEFAULT} case=DEFAULT,
213 BLOB blob
214 )
215 Returns a string representation of the BLOB blob as specified by encod‐
217 ing. case determines the case of hex digits for the HEX and URL encod‐
218 ings, and is ignored for the other encodings.
219 encoding defaults to IDENTITY, and case defaults to DEFAULT. DEFAULT
221 is interpreted as LOWER for the HEX and URL encodings, and is the
222 required value for the other encodings.
223 Example:
225 set resp.http.encode1
227 = blob.encode(HEX,
228 blob=blob.decode(BASE64, encoded="Zm9vYmFyYmF6"));
229 # same with named parameters
231 set resp.http.encode2
232 = blob.encode(blob=blob.decode(encoded="Zm9vYmFyYmF6",
233 decoding=BASE64),
234 encoding=HEX);
235 # convert blob to string
237 set resp.http.encode3
238 = blob.encode(blob=blob.decode(encoded="foo"));
239 transcode(...)
241 STRING transcode(
242 ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} decoding=IDENTITY,
243 ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} encoding=IDENTITY,
244 ENUM {LOWER, UPPER, DEFAULT} case=DEFAULT,
245 INT length=0,
246 STRING encoded
247 )
248 Translates from one encoding to another, by first decoding the string
250 encoded according to the scheme decoding, and then returning the encod‐
251 ing of the resulting blob according to the scheme encoding. case deter‐
252 mines the case of hex digits for the HEX and URL encodings, and is
253 ignored for other encodings.
254 As with decode(): If length > 0, only decode the first length charac‐
256 ters of the encoded string, otherwise decode the entire string. The
257 default value of length is 0.
258 decoding and encoding default to IDENTITY, and case defaults to
260 DEFAULT. DEFAULT is interpreted as LOWER for the HEX and URL encodings,
261 and is the required value for the other encodings.
262 Example:
264 set resp.http.Hex2Base64-1
266 = blob.transcode(HEX, BASE64, encoded="666f6f");
267 # same with named parameters
269 set resp.http.Hex2Base64-2
270 = blob.transcode(encoded="666f6f",
271 encoding=BASE64, decoding=HEX);
272 # URL decode -- recall that IDENTITY is the default encoding.
274 set resp.http.urldecoded
275 = blob.transcode(encoded="foo%20bar", decoding=URL);
276 # URL encode
278 set resp.http.urlencoded
279 = blob.transcode(encoded="foo bar", encoding=URL);
280 BOOL same(BLOB, BLOB)
282 Returns true if and only if the two BLOB arguments are the same object,
283 i.e. they specify exactly the same region of memory, or both are empty.
284 If the BLOBs are both empty (length is 0 and/or the internal pointer is
286 NULL), then same() returns true. If any non-empty BLOB is compared to
287 an empty BLOB, then same() returns false.
288 BOOL equal(BLOB, BLOB)
290 Returns true if and only if the two BLOB arguments have equal contents
291 (possibly in different memory regions).
292 As with same(): If the BLOBs are both empty, then equal() returns true.
294 If any non-empty BLOB is compared to an empty BLOB, then equal()
295 returns false.
296 INT length(BLOB)
298 Returns the length of the BLOB.
299 BLOB sub(BLOB, BYTES length, BYTES offset=0)
301 Returns a new BLOB formed from length bytes of the BLOB argument start‐
302 ing at offset bytes from the start of its memory region. The default
303 value of offset is 0B.
304 sub() fails and returns NULL if the BLOB argument is empty, or if off‐
306 set + length requires more bytes than are available in the BLOB.
307 new xblob = blob.blob(ENUM decoding, STRING encoded)
309 new xblob = blob.blob(
310 ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} decoding=IDENTITY,
311 STRING encoded
312 )
313 Creates an object that contains the BLOB derived from the string
315 encoded according to the scheme decoding.
316 Example:
318 new theblob1 = blob.blob(BASE64, encoded="YmxvYg==");
320 # same with named arguments
322 new theblob2 = blob.blob(encoded="YmxvYg==", decoding=BASE64);
323 # string as a blob
325 new stringblob = blob.blob(encoded="bazz");
326 BLOB xblob.get()
328 Returns the BLOB created by the constructor.
329 Example:
331 set resp.http.The-Blob1 =
333 blob.encode(blob=theblob1.get());
334 set resp.http.The-Blob2 =
336 blob.encode(blob=theblob2.get());
337 set resp.http.The-Stringblob =
339 blob.encode(blob=stringblob.get());
340 STRING xblob.encode(ENUM encoding, ENUM case)
342 STRING xblob.encode(
343 ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} encoding=IDENTITY,
344 ENUM {LOWER, UPPER, DEFAULT} case=DEFAULT
345 )
346 Returns an encoding of BLOB created by the constructor, according to
348 the scheme encoding. case determines the case of hex digits for the HEX
349 and URL encodings, and MUST be set to DEFAULT for the other encodings.
350 Example:
352 # blob as text
354 set resp.http.The-Blob = theblob1.encode();
355 # blob as base64
357 set resp.http.The-Blob-b64 = theblob1.encode(BASE64);
358 For any blob object, encoding ENC and case CASE, encodings via the
360 .encode() method and the encode() function are equal:
361 # Always true:
363 blob.encode(ENC, CASE, blob.get()) == blob.encode(ENC, CASE)
364 But the object method is more efficient -- the encoding is computed
366 once and cached (with allocation in heap memory), and the cached encod‐
367 ing is retrieved on every subsequent call. The encode() function com‐
368 putes the encoding on every call, allocating space for the string in
369 Varnish workspaces.
370 So if the data in a BLOB are fixed at VCL initialization time, so that
372 its encodings will always be the same, it is better to create a blob
373 object. The VMOD's functions should be used for data that are not known
374 until runtime.
375
377 The encoders, decoders and sub() may fail if there is insufficient
378 space to create the new blob or string. Decoders may also fail if the
379 encoded string is an illegal format for the decoding scheme. Encoders
380 will fail for the IDENTITY and BASE64* encoding schemes if the case
381 ENUM is not set to DEFAULT.
382 If any of the VMOD's methods, functions or constructor fail, then VCL
384 failure is invoked, just as if return(fail) had been called in the VCL
385 source. This means that:
386 · If the blob object constructor fails, or if any methods or functions
388 fail during vcl_init, then the VCL program will fail to load, and the
389 VCC compiler will emit an error message.
390 · If a method or function fails in any other VCL subroutine besides
392 vcl_synth, then control is directed to vcl_synth. The response status
393 is set to 503 with the reason string "VCL failed", and an error mes‐
394 sage will be written to the Varnish log using the tag VCL_Error.
395 · If the failure occurs during vcl_synth, then vcl_synth is aborted.
397 The response line "503 VCL failed" is returned, and the VCL_Error
398 message is written to the log.
399
401 The VMOD allocates memory in various ways for new blobs and strings.
402 The blob object and its methods allocate memory from the heap, and
403 hence they are only limited by available virtual memory.
404 The encode(), decode() and transcode() functions allocate Varnish
406 workspace, as does sub() for the newly created BLOB. If these func‐
407 tions are failing, as indicated by "out of space" messages in the Var‐
408 nish log (with the VCL_Error tag), then you will need to increase the
409 varnishd parameters workspace_client and/or workspace_backend.
410 The transcode() function also allocates space on the stack for a tempo‐
412 rary BLOB. If this function causes stack overflow, you may need to
413 increase the varnishd parameter thread_pool_stack.
414
416 · varnishd(1)
417 · vcl(7)
419 · vmod_std(3)
421
423 This document is licensed under the same conditions as Varnish itself.
424 See LICENSE for details.
425 Authors: Nils Goroll <nils.goroll@uplex.de>
427 Geoffrey Simmons <geoffrey.simmons@uplex.de>
428 VMOD_BLOB(3)