1VMOD_BLOB(3) VMOD_BLOB(3)
2
6 vmod_blob - Utilities for the VCL blob type, encoding and decoding
7
9 import blob [as name] [from "path"]
10 BLOB decode(ENUM decoding, INT length, STRING encoded)
12 STRING encode(ENUM encoding, ENUM case, BLOB blob)
14 STRING transcode(ENUM decoding, ENUM encoding, ENUM case, INT length, STRING encoded)
16 BOOL same(BLOB, BLOB)
18 BOOL equal(BLOB, BLOB)
20 INT length(BLOB)
22 BLOB sub(BLOB, BYTES length, BYTES offset)
24 new xblob = blob.blob(ENUM decoding, STRING encoded)
26 BLOB xblob.get()
28 STRING xblob.encode(ENUM encoding, ENUM case)
30
32 This VMOD provides utility functions and an object for the VCL data
33 type BLOB, which may contain arbitrary data of any length.
34 Examples:
36 sub vcl_init {
38 # Create blob objects from encodings such as base64 or hex.
39 new myblob = blob.blob(BASE64, "Zm9vYmFy");
40 new yourblob = blob.blob(encoded="666F6F", decoding=HEX);
41 }
42 sub vcl_deliver {
44 # The .get() method retrieves the BLOB from an object.
45 set resp.http.MyBlob-As-Hex
46 = blob.encode(blob=myblob.get(), encoding=HEX);
47 # The .encode() method efficiently retrieves an encoding.
49 set resp.http.YourBlob-As-Base64 = yourblob.encode(BASE64);
50 # decode() and encode() functions convert blobs to text and
52 # vice versa at runtime.
53 set resp.http.Base64-Encoded
54 = blob.encode(BASE64,
55 blob=blob.decode(HEX,
56 encoded=req.http.Hex-Encoded));
57 }
58 sub vcl_recv {
60 # transcode() converts from one encoding to another.
61 # case=UPPER specifies upper-case hex digits A-F.
62 set req.http.Hex-Encoded
63 = blob.transcode(decoding=BASE64, encoding=HEX,
64 case=UPPER, encoded="YmF6");
65 # transcode() from URL to IDENTITY effects a URL decode.
67 set req.url = blob.transcode(encoded=req.url, decoding=URL);
68 # transcode() from IDENTITY to URL effects a URL encode.
70 set req.http.url_urlcoded
71 = blob.transcode(encoded=req.url, encoding=URL);
72 }
73 ENCODING SCHEMES
75 Binary-to-text encoding schemes are specified by ENUMs in the VMOD's
76 constructor, methods and functions. Decodings convert a (possibly con‐
77 catenated) string into a blob, while encodings convert a blob into a
78 string.
79 ENUM values for an encoding scheme can be one of:
81 • IDENTITY
83 • BASE64
85 • BASE64URL
87 • BASE64URLNOPAD
89 • HEX
91 • URL
93 Empty strings are decoded into a "null blob" (of length 0), and con‐
95 versely a null blob is encoded as the empty string.
96 For encodings with HEX or URL, you may also specify a case ENUM with
98 one of the values LOWER, UPPER or DEFAULT to produce a string with
99 lower- or uppercase hex digits (in [a-f] or [A-F]). The default value
100 for case is DEFAULT, which for HEX and URL means the same as LOWER.
101 The case ENUM is not relevant for decodings; HEX or URL strings to be
103 decoded as BLOBs may have hex digits in either case, or in mixed case.
104 The case ENUM MUST be set to DEFAULT for the other encodings (BASE64*
106 and IDENTITY). You cannot, for example, produce an uppercase string by
107 using the IDENTITY scheme with case=UPPER. To change the case of a
108 string, use the std.toupper() or std.tolower() functions from
109 vmod_std(3).
110 IDENTITY
112 The simplest encoding converts between the BLOB and STRING data types,
113 leaving the contents byte-identical.
114 Note that a BLOB may contain a null byte at any position before its
116 end; if such a BLOB is decoded with IDENTITY, the resulting STRING will
117 have a null byte at that position. Since VCL strings, like C strings,
118 are represented with a terminating null byte, the string will be trun‐
119 cated, appearing to contain less data than the original blob. For exam‐
120 ple:
121 # Decode from the hex encoding for "foo\0bar".
123 # The header will be seen as "foo".
124 set resp.http.Trunced-Foo1
125 = blob.encode(IDENTITY, blob=blob.decode(HEX,
126 encoded="666f6f00626172"));
127 IDENTITY is the default encoding and decoding. So the above can also be
129 written as:
130 # Decode from the hex encoding for "foo\0bar".
132 # The header will be seen as "foo".
133 set resp.http.Trunced-Foo2
134 = blob.encode(blob=blob.decode(HEX, encoded="666f6f00626172"));
135 The case ENUM MUST be set to DEFAULT for IDENTITY encodings.
137 BASE64*
139 The base64 encoding schemes use 4 characters to encode 3 bytes. There
140 are no newlines or maximal line lengths -- whitespace is not permitted.
141 The BASE64 encoding uses the alphanumeric characters, + and /; and en‐
143 coded strings are padded with the = character so that their length is
144 always a multiple of four.
145 The BASE64URL encoding also uses the alphanumeric characters, but - and
147 _ instead of + and /, so that an encoded string can be used safely in a
148 URL. This scheme also uses the padding character =.
149 The BASE64URLNOPAD encoding uses the same alphabet as BASE6URL, but
151 leaves out the padding. Thus the length of an encoding with this scheme
152 is not necessarily a multiple of four.
153 The case ENUM MUST be set to DEFAULT for for all of the BASE64* encod‐
155 ings.
156 HEX
158 The HEX encoding scheme converts hex strings into blobs and vice versa.
159 For encodings, you may use the case ENUM to specify upper- or lowercase
160 hex digits A through f (default DEFAULT, which means the same as
161 LOWER). A prefix such as 0x is not used for an encoding and is illegal
162 for a decoding.
163 If a hex string to be decoded has an odd number of digits, it is de‐
165 coded as if a 0 is prepended to it; that is, the first digit is inter‐
166 preted as representing the least significant nibble of the first byte.
167 For example:
168 # The concatenated string is "abcdef0", and is decoded as "0abcdef0".
170 set resp.http.First = "abc";
171 set resp.http.Second = "def0";
172 set resp.http.Hex-Decoded
173 = blob.encode(HEX, blob=blob.decode(HEX,
174 encoded=resp.http.First + resp.http.Second));
175 URL
177 The URL decoding replaces any %<2-hex-digits> substrings with the bi‐
178 nary value of the hexadecimal number after the % sign.
179 The URL encoding implements "percent encoding" as per RFC3986. The case
181 ENUM determines the case of the hex digits, but does not affect alpha‐
182 betic characters that are not percent-encoded.
183 BLOB decode(ENUM decoding, INT length, STRING encoded)
185 BLOB decode(
186 ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} decoding=IDENTITY,
187 INT length=0,
188 STRING encoded
189 )
190 Returns the BLOB derived from the string encoded according to the
192 scheme specified by decoding.
193 If length > 0, only decode the first length characters of the encoded
195 string. If length <= 0 or greater than the length of the string, then
196 decode the entire string. The default value of length is 0.
197 decoding defaults to IDENTITY.
199 Example:
201 blob.decode(BASE64, encoded="Zm9vYmFyYmF6");
203 # same with named parameters
205 blob.decode(encoded="Zm9vYmFyYmF6", decoding=BASE64);
206 # convert string to blob
208 blob.decode(encoded="foo");
209 STRING encode(ENUM encoding, ENUM case, BLOB blob)
211 STRING encode(
212 ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} encoding=IDENTITY,
213 ENUM {LOWER, UPPER, DEFAULT} case=DEFAULT,
214 BLOB blob
215 )
216 Returns a string representation of the BLOB blob as specified by encod‐
218 ing. case determines the case of hex digits for the HEX and URL encod‐
219 ings, and is ignored for the other encodings.
220 encoding defaults to IDENTITY, and case defaults to DEFAULT. DEFAULT
222 is interpreted as LOWER for the HEX and URL encodings, and is the re‐
223 quired value for the other encodings.
224 Example:
226 set resp.http.encode1
228 = blob.encode(HEX,
229 blob=blob.decode(BASE64, encoded="Zm9vYmFyYmF6"));
230 # same with named parameters
232 set resp.http.encode2
233 = blob.encode(blob=blob.decode(encoded="Zm9vYmFyYmF6",
234 decoding=BASE64),
235 encoding=HEX);
236 # convert blob to string
238 set resp.http.encode3
239 = blob.encode(blob=blob.decode(encoded="foo"));
240 STRING transcode(ENUM decoding, ENUM encoding, ENUM case, INT length,
242 STRING encoded)
243 STRING transcode(
244 ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} decoding=IDENTITY,
245 ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} encoding=IDENTITY,
246 ENUM {LOWER, UPPER, DEFAULT} case=DEFAULT,
247 INT length=0,
248 STRING encoded
249 )
250 Translates from one encoding to another, by first decoding the string
252 encoded according to the scheme decoding, and then returning the encod‐
253 ing of the resulting blob according to the scheme encoding. case deter‐
254 mines the case of hex digits for the HEX and URL encodings, and is ig‐
255 nored for other encodings.
256 As with blob.decode(): If length > 0, only decode the first length
258 characters of the encoded string, otherwise decode the entire string.
259 The default value of length is 0.
260 decoding and encoding default to IDENTITY, and case defaults to DE‐
262 FAULT. DEFAULT is interpreted as LOWER for the HEX and URL encodings,
263 and is the required value for the other encodings.
264 Example:
266 set resp.http.Hex2Base64-1
268 = blob.transcode(HEX, BASE64, encoded="666f6f");
269 # same with named parameters
271 set resp.http.Hex2Base64-2
272 = blob.transcode(encoded="666f6f",
273 encoding=BASE64, decoding=HEX);
274 # URL decode -- recall that IDENTITY is the default encoding.
276 set resp.http.urldecoded
277 = blob.transcode(encoded="foo%20bar", decoding=URL);
278 # URL encode
280 set resp.http.urlencoded
281 = blob.transcode(encoded="foo bar", encoding=URL);
282 BOOL same(BLOB, BLOB)
284 Returns true if and only if the two BLOB arguments are the same object,
285 i.e. they specify exactly the same region of memory, or both are empty.
286 If the BLOBs are both empty (length is 0 and/or the internal pointer is
288 NULL), then blob.same() returns true. If any non-empty BLOB is compared
289 to an empty BLOB, then blob.same() returns false.
290 BOOL equal(BLOB, BLOB)
292 Returns true if and only if the two BLOB arguments have equal contents
293 (possibly in different memory regions).
294 As with blob.same(): If the BLOBs are both empty, then blob.equal() re‐
296 turns true. If any non-empty BLOB is compared to an empty BLOB, then
297 blob.equal() returns false.
298 INT length(BLOB)
300 Returns the length of the BLOB.
301 BLOB sub(BLOB, BYTES length, BYTES offset=0)
303 Returns a new BLOB formed from length bytes of the BLOB argument start‐
304 ing at offset bytes from the start of its memory region. The default
305 value of offset is 0B.
306 blob.sub() fails and returns NULL if the BLOB argument is empty, or if
308 offset + length requires more bytes than are available in the BLOB.
309 new xblob = blob.blob(ENUM decoding, STRING encoded)
311 new xblob = blob.blob(
312 ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} decoding=IDENTITY,
313 STRING encoded
314 )
315 Creates an object that contains the BLOB derived from the string en‐
317 coded according to the scheme decoding.
318 Example:
320 new theblob1 = blob.blob(BASE64, encoded="YmxvYg==");
322 # same with named arguments
324 new theblob2 = blob.blob(encoded="YmxvYg==", decoding=BASE64);
325 # string as a blob
327 new stringblob = blob.blob(encoded="bazz");
328 BLOB xblob.get()
330 Returns the BLOB created by the constructor.
331 Example:
333 set resp.http.The-Blob1 =
335 blob.encode(blob=theblob1.get());
336 set resp.http.The-Blob2 =
338 blob.encode(blob=theblob2.get());
339 set resp.http.The-Stringblob =
341 blob.encode(blob=stringblob.get());
342 STRING xblob.encode(ENUM encoding, ENUM case)
344 STRING xblob.encode(
345 ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} encoding=IDENTITY,
346 ENUM {LOWER, UPPER, DEFAULT} case=DEFAULT
347 )
348 Returns an encoding of BLOB created by the constructor, according to
350 the scheme encoding. case determines the case of hex digits for the HEX
351 and URL encodings, and MUST be set to DEFAULT for the other encodings.
352 Example:
354 # blob as text
356 set resp.http.The-Blob = theblob1.encode();
357 # blob as base64
359 set resp.http.The-Blob-b64 = theblob1.encode(BASE64);
360 For any blob.blob() object, encoding and case, encodings via the
362 xblob.encode() method and the blob.encode() function are equal:
363 # Always true:
365 blob.encode(ENC, CASE, blob.get()) == blob.encode(ENC, CASE)
366 But the xblob.encode() object method is more efficient -- the encoding
368 is computed once and cached (with allocation in heap memory), and the
369 cached encoding is retrieved on every subsequent call. The
370 blob.encode() function computes the encoding on every call, allocating
371 space for the string in Varnish workspaces.
372 So if the data in a BLOB are fixed at VCL initialization time, so that
374 its encodings will always be the same, it is better to create a
375 blob.blob() object. The VMOD's functions should be used for data that
376 are not known until runtime.
377
379 The encoders, decoders and blob.sub() may fail if there is insufficient
380 space to create the new blob or string. Decoders may also fail if the
381 encoded string is an illegal format for the decoding scheme. Encoders
382 will fail for the IDENTITY and BASE64* encoding schemes if the case
383 ENUM is not set to DEFAULT.
384 If any of the VMOD's methods, functions or constructor fail, then VCL
386 failure is invoked, just as if return(fail) had been called in the VCL
387 source. This means that:
388 • If the blob.blob() object constructor fails, or if any methods or
390 functions fail during vcl_init{}, then the VCL program will fail to
391 load, and the VCC compiler will emit an error message.
392 • If a method or function fails in any other VCL subroutine besides
394 vcl_synth{}, then control is directed to vcl_synth{}. The response
395 status is set to 503 with the reason string "VCL failed", and an er‐
396 ror message will be written to the vsl(7) using the tag VCL_Error.
397 • If the failure occurs during vcl_synth{}, then vcl_synth{} is
399 aborted. The response line "503 VCL failed" is returned, and the
400 VCL_Error message is written to the log.
401
403 The VMOD allocates memory in various ways for new blobs and strings.
404 The blob.blob() object and its methods allocate memory from the heap,
405 and hence they are only limited by available virtual memory.
406 The blob.encode(), blob.decode() and blob.transcode() functions allo‐
408 cate Varnish workspace, as does blob.sub() for the newly created BLOB.
409 If these functions are failing, as indicated by "out of space" messages
410 in the Varnish log (with the VCL_Error tag), then you will need to in‐
411 crease the varnishd parameters workspace_client and/or workspace_back‐
412 end.
413 The blob.transcode() function also allocates space on the stack for a
415 temporary BLOB. If this function causes stack overflow, you may need to
416 increase the varnishd parameter thread_pool_stack.
417
419 • varnishd(1)
420 • vcl(7)
422 • vsl(7)
424 • vmod_std(3)
426
428 This document is licensed under the same conditions as Varnish itself.
429 See LICENSE for details.
430 SPDX-License-Identifier: BSD-2-Clause
432 Authors: Nils Goroll <nils.goroll@uplex.de>
434 Geoffrey Simmons <geoffrey.simmons@uplex.de>
435 VMOD_BLOB(3)