1VMOD_BLOB(3) VMOD_BLOB(3)
2
6 vmod_blob - Utilities for the VCL blob type, encoding and decoding
7
9 import blob [as name] [from "path"]
10 BLOB decode(ENUM decoding, INT length, STRING encoded)
12 STRING encode(ENUM encoding, ENUM case, BLOB blob)
14 STRING transcode(ENUM decoding, ENUM encoding, ENUM case, INT length, STRING encoded)
16 BOOL same(BLOB, BLOB)
18 BOOL equal(BLOB, BLOB)
20 INT length(BLOB)
22 BLOB sub(BLOB, BYTES length, BYTES offset)
24 new xblob = blob.blob(ENUM decoding, STRING encoded)
26 BLOB xblob.get()
28 STRING xblob.encode(ENUM encoding, ENUM case)
30
32 This VMOD provides utility functions and an object for the VCL data
33 type BLOB, which may contain arbitrary data of any length.
34 Examples:
36 sub vcl_init {
38 # Create blob objects from encodings such as base64 or hex.
39 new myblob = blob.blob(BASE64, "Zm9vYmFy");
40 new yourblob = blob.blob(encoded="666F6F", decoding=HEX);
41 }
42 sub vcl_deliver {
44 # The .get() method retrieves the BLOB from an object.
45 set resp.http.MyBlob-As-Hex
46 = blob.encode(blob=myblob.get(), encoding=HEX);
47 # The .encode() method efficiently retrieves an encoding.
49 set resp.http.YourBlob-As-Base64 = yourblob.encode(BASE64);
50 # decode() and encode() functions convert blobs to text and
52 # vice versa at runtime.
53 set resp.http.Base64-Encoded
54 = blob.encode(BASE64,
55 blob=blob.decode(HEX,
56 encoded=req.http.Hex-Encoded));
57 }
58 sub vcl_recv {
60 # transcode() converts from one encoding to another.
61 # case=UPPER specifies upper-case hex digits A-F.
62 set req.http.Hex-Encoded
63 = blob.transcode(decoding=BASE64, encoding=HEX,
64 case=UPPER, encoded="YmF6");
65 # transcode() from URL to IDENTITY effects a URL decode.
67 set req.url = blob.transcode(encoded=req.url, decoding=URL);
68 # transcode() from IDENTITY to URL effects a URL encode.
70 set req.http.url_urlcoded
71 = blob.transcode(encoded=req.url, encoding=URL);
72 }
73 ENCODING SCHEMES
75 Binary-to-text encoding schemes are specified by ENUMs in the VMOD's
76 constructor, methods and functions. Decodings convert a (possibly con‐
77 catenated) string into a blob, while encodings convert a blob into a
78 string.
79 ENUM values for an encoding scheme can be one of:
81 • IDENTITY
83 • BASE64
85 • BASE64URL
87 • BASE64URLNOPAD
89 • BASE64CF
91 • HEX
93 • URL
95 Empty strings are decoded into a "null blob" (of length 0), and con‐
97 versely a null blob is encoded as the empty string.
98 For encodings with HEX or URL, you may also specify a case ENUM with
100 one of the values LOWER, UPPER or DEFAULT to produce a string with
101 lower- or uppercase hex digits (in [a-f] or [A-F]). The default value
102 for case is DEFAULT, which for HEX and URL means the same as LOWER.
103 The case ENUM is not relevant for decodings; HEX or URL strings to be
105 decoded as BLOBs may have hex digits in either case, or in mixed case.
106 The case ENUM MUST be set to DEFAULT for the other encodings (BASE64*
108 and IDENTITY). You cannot, for example, produce an uppercase string by
109 using the IDENTITY scheme with case=UPPER. To change the case of a
110 string, use the std.toupper() or std.tolower() functions from
111 vmod_std(3).
112 IDENTITY
114 The simplest encoding converts between the BLOB and STRING data types,
115 leaving the contents byte-identical.
116 Note that a BLOB may contain a null byte at any position before its
118 end; if such a BLOB is decoded with IDENTITY, the resulting STRING will
119 have a null byte at that position. Since VCL strings, like C strings,
120 are represented with a terminating null byte, the string will be trun‐
121 cated, appearing to contain less data than the original blob. For exam‐
122 ple:
123 # Decode from the hex encoding for "foo\0bar".
125 # The header will be seen as "foo".
126 set resp.http.Trunced-Foo1
127 = blob.encode(IDENTITY, blob=blob.decode(HEX,
128 encoded="666f6f00626172"));
129 IDENTITY is the default encoding and decoding. So the above can also be
131 written as:
132 # Decode from the hex encoding for "foo\0bar".
134 # The header will be seen as "foo".
135 set resp.http.Trunced-Foo2
136 = blob.encode(blob=blob.decode(HEX, encoded="666f6f00626172"));
137 The case ENUM MUST be set to DEFAULT for IDENTITY encodings.
139 BASE64*
141 The base64 encoding schemes use 4 characters to encode 3 bytes. There
142 are no newlines or maximal line lengths -- whitespace is not permitted.
143 The BASE64 encoding uses the alphanumeric characters, + and /; and en‐
145 coded strings are padded with the = character so that their length is
146 always a multiple of four.
147 The BASE64URL encoding also uses the alphanumeric characters, but - and
149 _ instead of + and /, so that an encoded string can be used safely in a
150 URL. This scheme also uses the padding character =.
151 The BASE64URLNOPAD encoding uses the same alphabet as BASE6URL, but
153 leaves out the padding. Thus the length of an encoding with this scheme
154 is not necessarily a multiple of four.
155 The BASE64CF` is similar to ``BASE64URL, with the following changes to
157 BASE64: + replaced with -, / replaced with ~ and _ as the padding char‐
158 acter. It is used by a certain CDN provider who also inspired the name.
159 The case ENUM MUST be set to DEFAULT for for all of the BASE64* encod‐
161 ings.
162 HEX
164 The HEX encoding scheme converts hex strings into blobs and vice versa.
165 For encodings, you may use the case ENUM to specify upper- or lowercase
166 hex digits A through f (default DEFAULT, which means the same as
167 LOWER). A prefix such as 0x is not used for an encoding and is illegal
168 for a decoding.
169 If a hex string to be decoded has an odd number of digits, it is de‐
171 coded as if a 0 is prepended to it; that is, the first digit is inter‐
172 preted as representing the least significant nibble of the first byte.
173 For example:
174 # The concatenated string is "abcdef0", and is decoded as "0abcdef0".
176 set resp.http.First = "abc";
177 set resp.http.Second = "def0";
178 set resp.http.Hex-Decoded
179 = blob.encode(HEX, blob=blob.decode(HEX,
180 encoded=resp.http.First + resp.http.Second));
181 URL
183 The URL decoding replaces any %<2-hex-digits> substrings with the bi‐
184 nary value of the hexadecimal number after the % sign.
185 The URL encoding implements "percent encoding" as per RFC3986. The case
187 ENUM determines the case of the hex digits, but does not affect alpha‐
188 betic characters that are not percent-encoded.
189 BLOB decode(ENUM decoding, INT length, STRING encoded)
191 BLOB decode(
192 ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, BASE64CF, HEX, URL} decoding=IDENTITY,
193 INT length=0,
194 STRING encoded
195 )
196 Returns the BLOB derived from the string encoded according to the
198 scheme specified by decoding.
199 If length > 0, only decode the first length characters of the encoded
201 string. If length <= 0 or greater than the length of the string, then
202 decode the entire string. The default value of length is 0.
203 decoding defaults to IDENTITY.
205 Example:
207 blob.decode(BASE64, encoded="Zm9vYmFyYmF6");
209 # same with named parameters
211 blob.decode(encoded="Zm9vYmFyYmF6", decoding=BASE64);
212 # convert string to blob
214 blob.decode(encoded="foo");
215 STRING encode(ENUM encoding, ENUM case, BLOB blob)
217 STRING encode(
218 ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, BASE64CF, HEX, URL} encoding=IDENTITY,
219 ENUM {LOWER, UPPER, DEFAULT} case=DEFAULT,
220 BLOB blob
221 )
222 Returns a string representation of the BLOB blob as specified by encod‐
224 ing. case determines the case of hex digits for the HEX and URL encod‐
225 ings, and is ignored for the other encodings.
226 encoding defaults to IDENTITY, and case defaults to DEFAULT. DEFAULT
228 is interpreted as LOWER for the HEX and URL encodings, and is the re‐
229 quired value for the other encodings.
230 Example:
232 set resp.http.encode1
234 = blob.encode(HEX,
235 blob=blob.decode(BASE64, encoded="Zm9vYmFyYmF6"));
236 # same with named parameters
238 set resp.http.encode2
239 = blob.encode(blob=blob.decode(encoded="Zm9vYmFyYmF6",
240 decoding=BASE64),
241 encoding=HEX);
242 # convert blob to string
244 set resp.http.encode3
245 = blob.encode(blob=blob.decode(encoded="foo"));
246 STRING transcode(ENUM decoding, ENUM encoding, ENUM case, INT length,
248 STRING encoded)
249 STRING transcode(
250 ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, BASE64CF, HEX, URL} decoding=IDENTITY,
251 ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, BASE64CF, HEX, URL} encoding=IDENTITY,
252 ENUM {LOWER, UPPER, DEFAULT} case=DEFAULT,
253 INT length=0,
254 STRING encoded
255 )
256 Translates from one encoding to another, by first decoding the string
258 encoded according to the scheme decoding, and then returning the encod‐
259 ing of the resulting blob according to the scheme encoding. case deter‐
260 mines the case of hex digits for the HEX and URL encodings, and is ig‐
261 nored for other encodings.
262 As with blob.decode(): If length > 0, only decode the first length
264 characters of the encoded string, otherwise decode the entire string.
265 The default value of length is 0.
266 decoding and encoding default to IDENTITY, and case defaults to DE‐
268 FAULT. DEFAULT is interpreted as LOWER for the HEX and URL encodings,
269 and is the required value for the other encodings.
270 Example:
272 set resp.http.Hex2Base64-1
274 = blob.transcode(HEX, BASE64, encoded="666f6f");
275 # same with named parameters
277 set resp.http.Hex2Base64-2
278 = blob.transcode(encoded="666f6f",
279 encoding=BASE64, decoding=HEX);
280 # URL decode -- recall that IDENTITY is the default encoding.
282 set resp.http.urldecoded
283 = blob.transcode(encoded="foo%20bar", decoding=URL);
284 # URL encode
286 set resp.http.urlencoded
287 = blob.transcode(encoded="foo bar", encoding=URL);
288 BOOL same(BLOB, BLOB)
290 Returns true if and only if the two BLOB arguments are the same object,
291 i.e. they specify exactly the same region of memory, or both are empty.
292 If the BLOBs are both empty (length is 0 and/or the internal pointer is
294 NULL), then blob.same() returns true. If any non-empty BLOB is compared
295 to an empty BLOB, then blob.same() returns false.
296 BOOL equal(BLOB, BLOB)
298 Returns true if and only if the two BLOB arguments have equal contents
299 (possibly in different memory regions).
300 As with blob.same(): If the BLOBs are both empty, then blob.equal() re‐
302 turns true. If any non-empty BLOB is compared to an empty BLOB, then
303 blob.equal() returns false.
304 INT length(BLOB)
306 Returns the length of the BLOB.
307 BLOB sub(BLOB, BYTES length, BYTES offset=0)
309 Returns a new BLOB formed from length bytes of the BLOB argument start‐
310 ing at offset bytes from the start of its memory region. The default
311 value of offset is 0B.
312 blob.sub() fails and returns NULL if the BLOB argument is empty, or if
314 offset + length requires more bytes than are available in the BLOB.
315 new xblob = blob.blob(ENUM decoding, STRING encoded)
317 new xblob = blob.blob(
318 ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, BASE64CF, HEX, URL} decoding=IDENTITY,
319 STRING encoded
320 )
321 Creates an object that contains the BLOB derived from the string en‐
323 coded according to the scheme decoding.
324 Example:
326 new theblob1 = blob.blob(BASE64, encoded="YmxvYg==");
328 # same with named arguments
330 new theblob2 = blob.blob(encoded="YmxvYg==", decoding=BASE64);
331 # string as a blob
333 new stringblob = blob.blob(encoded="bazz");
334 BLOB xblob.get()
336 Returns the BLOB created by the constructor.
337 Example:
339 set resp.http.The-Blob1 =
341 blob.encode(blob=theblob1.get());
342 set resp.http.The-Blob2 =
344 blob.encode(blob=theblob2.get());
345 set resp.http.The-Stringblob =
347 blob.encode(blob=stringblob.get());
348 STRING xblob.encode(ENUM encoding, ENUM case)
350 STRING xblob.encode(
351 ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, BASE64CF, HEX, URL} encoding=IDENTITY,
352 ENUM {LOWER, UPPER, DEFAULT} case=DEFAULT
353 )
354 Returns an encoding of BLOB created by the constructor, according to
356 the scheme encoding. case determines the case of hex digits for the HEX
357 and URL encodings, and MUST be set to DEFAULT for the other encodings.
358 Example:
360 # blob as text
362 set resp.http.The-Blob = theblob1.encode();
363 # blob as base64
365 set resp.http.The-Blob-b64 = theblob1.encode(BASE64);
366 For any blob.blob() object, encoding and case, encodings via the
368 xblob.encode() method and the blob.encode() function are equal:
369 # Always true:
371 blob.encode(ENC, CASE, blob.get()) == blob.encode(ENC, CASE)
372 But the xblob.encode() object method is more efficient -- the encoding
374 is computed once and cached (with allocation in heap memory), and the
375 cached encoding is retrieved on every subsequent call. The
376 blob.encode() function computes the encoding on every call, allocating
377 space for the string in Varnish workspaces.
378 So if the data in a BLOB are fixed at VCL initialization time, so that
380 its encodings will always be the same, it is better to create a
381 blob.blob() object. The VMOD's functions should be used for data that
382 are not known until runtime.
383
385 The encoders, decoders and blob.sub() may fail if there is insufficient
386 space to create the new blob or string. Decoders may also fail if the
387 encoded string is an illegal format for the decoding scheme. Encoders
388 will fail for the IDENTITY and BASE64* encoding schemes if the case
389 ENUM is not set to DEFAULT.
390 If any of the VMOD's methods, functions or constructor fail, then VCL
392 failure is invoked, just as if return(fail) had been called in the VCL
393 source. This means that:
394 • If the blob.blob() object constructor fails, or if any methods or
396 functions fail during vcl_init{}, then the VCL program will fail to
397 load, and the VCC compiler will emit an error message.
398 • If a method or function fails in any other VCL subroutine besides
400 vcl_synth{}, then control is directed to vcl_synth{}. The response
401 status is set to 503 with the reason string "VCL failed", and an er‐
402 ror message will be written to the vsl(7) using the tag VCL_Error.
403 • If the failure occurs during vcl_synth{}, then vcl_synth{} is
405 aborted. The response line "503 VCL failed" is returned, and the
406 VCL_Error message is written to the log.
407
409 The VMOD allocates memory in various ways for new blobs and strings.
410 The blob.blob() object and its methods allocate memory from the heap,
411 and hence they are only limited by available virtual memory.
412 The blob.encode(), blob.decode() and blob.transcode() functions allo‐
414 cate Varnish workspace, as does blob.sub() for the newly created BLOB.
415 If these functions are failing, as indicated by "out of space" messages
416 in the Varnish log (with the VCL_Error tag), then you will need to in‐
417 crease the varnishd parameters workspace_client and/or workspace_back‐
418 end.
419 The blob.transcode() function also allocates space on the stack for a
421 temporary BLOB. If this function causes stack overflow, you may need to
422 increase the varnishd parameter thread_pool_stack.
423
425 • varnishd(1)
426 • vcl(7)
428 • vsl(7)
430 • vmod_std(3)
432
434 This document is licensed under the same conditions as Varnish itself.
435 See LICENSE for details.
436 SPDX-License-Identifier: BSD-2-Clause
438 Authors: Nils Goroll <nils.goroll@uplex.de>
440 Geoffrey Simmons <geoffrey.simmons@uplex.de>
441 VMOD_BLOB(3)