1Sereal::Decoder(3) User Contributed Perl Documentation Sereal::Decoder(3)
2
6 Sereal::Decoder - Fast, compact, powerful binary deserialization
7
9 use Sereal::Decoder
10 qw(decode_sereal sereal_decode_with_object scalar_looks_like_sereal);
11 my $decoder = Sereal::Decoder->new({...options...});
13 my $structure;
15 $decoder->decode($blob, $structure); # deserializes into $structure
16 # or if you don't have references to the top level structure, this works, too:
18 $structure = $decoder->decode($blob);
19 # alternatively functional interface: (See Sereal::Performance)
21 sereal_decode_with_object($decoder, $blob, $structure);
22 $structure = sereal_decode_with_object($decoder, $blob);
23 # much slower functional interface with no persistent objects:
25 decode_sereal($blob, {... options ...}, $structure);
26 $structure = decode_sereal($blob, {... options ...});
27 # Not a full validation, but just a quick check for a reasonable header:
29 my $is_likely_sereal = scalar_looks_like_sereal($some_string);
30 # or:
31 $is_likely_sereal = $decoder->looks_like_sereal($some_string);
32
34 This library implements a deserializer for an efficient, compact-
35 output, and feature-rich binary protocol called Sereal. Its sister
36 module Sereal::Encoder implements an encoder for this format. The two
37 are released separately to allow for independent and safer upgrading.
38 The Sereal protocol versions that are compatible with this decoder
40 implementation are currently protocol versions 1, 2, 3 and 4. As it
41 stands, it will refuse to attempt to decode future versions of the
42 protocol, but if necessary there is likely going to be an option to
43 decode the parts of the input that are compatible with version 4 of the
44 protocol. The protocol was designed to allow for this.
45 The protocol specification and many other bits of documentation can be
47 found in the github repository. Right now, the specification is at
48 <https://github.com/Sereal/Sereal/blob/master/sereal_spec.pod>, there
49 is a discussion of the design objectives in
50 <https://github.com/Sereal/Sereal/blob/master/README.pod>, and the
51 output of our benchmarks can be seen at
52 <https://github.com/Sereal/Sereal/wiki/Sereal-Comparison-Graphs>.
53
55 new
56 Constructor. Optionally takes a hash reference as first parameter. This
57 hash reference may contain any number of options that influence the
58 behaviour of the encoder.
59 Currently, the following options are recognized, none of them are on by
61 default.
62 refuse_snappy
64 If set, the decoder will refuse Snappy-compressed input data. This can
66 be desirable for robustness. See the section "ROBUSTNESS" below.
67 refuse_objects
69 If set, the decoder will refuse deserializing any objects in the input
71 stream and instead throw an exception. Defaults to off. See the section
72 "ROBUSTNESS" below.
73 no_bless_objects
75 If set, the decoder will deserialize any objects in the input stream
77 but without blessing them. Defaults to off. See the section
78 "ROBUSTNESS" below.
79 validate_utf8
81 If set, the decoder will refuse invalid UTF-8 byte sequences. This is
83 off by default, but it's strongly encouraged to be turned on if you're
84 dealing with any data that has been encoded by an external source (e.g.
85 http cookies).
86 max_recursion_depth
88 "Sereal::Decoder" is recursive. If you pass it a Sereal document that
90 is deeply nested, it will eventually exhaust the C stack. Therefore,
91 there is a limit on the depth of recursion that is accepted. It
92 defaults to 10000 nested calls. You may choose to override this value
93 with the "max_recursion_depth" option. Beware that setting it too high
94 can cause hard crashes.
95 Do note that the setting is somewhat approximate. Setting it to 10000
97 may break at somewhere between 9997 and 10003 nested structures
98 depending on their types.
99 max_num_hash_entries
101 If set to a non-zero value (default: 0), then "Sereal::Decoder" will
103 refuse to deserialize any hash/dictionary (or hash-based object) with
104 more than that number of entries. This is to be able to respond quickly
105 to any future hash-collision attacks on Perl's hash function, and also
106 the memory exhaustion attacks on Sereal itself. For a gentle
107 introduction to the topic from the cryptographic point of view, see
108 <http://en.wikipedia.org/wiki/Collision_attack>.
109 max_num_array_entries
111 If set to a non-zero value (default: 0), then "Sereal::Decoder" will
113 refuse to deserialize any array with more than that number of entries.
114 This is to be able to respond quickly to any future memory exhaustion
115 attacks on Sereal.
116 max_string_length
118 If set to a non-zero value (default: 0), then "Sereal::Decoder" will
120 refuse to deserialize any string with more than that number of
121 characters. This is to be able to respond quickly to any future memory
122 exhaustion attacks on Sereal.
123 max_uncompressed_size
125 If set to a non-zero value (default: 0), then "Sereal::Decoder" will
127 refuse to deserialize any blob with a size that exceds the value when
128 uncompressed. This is to be able to respond quickly to any future
129 memory exhaustion attacks on Sereal.
130 incremental
132 If set to a non-zero value (default: 0), then "Sereal::Decoder" will
134 destructively parse Sereal documents out of a variable. Every time a
135 Sereal document is successfully parsed it is removed from the front of
136 the string it is parsed from.
137 This means you can do this:
139 while (length $buffer) {
141 my $data= decode_sereal($buffer,{incremental=>1});
142 }
143 alias_smallint
145 If set to a true value then "Sereal::Decoder" will share integers from
147 -16 to 15 (encoded as either SRL_HDR_NEG and SRL_HDR_POS) as read-only
148 aliases to a common SV.
149 The result of this may be significant space savings in data structures
151 with many integers in the specified range. The cost is more memory used
152 by the decoder and a very modest speed penalty when deserializing.
153 Note this option changes the structure of the dumped data. Use with
155 caution.
156 See also the "alias_varint_under" option.
158 alias_varint_under
160 If set to a true positive integer smaller than 16 then this option is
162 similar to setting "alias_smallint" and causes all integers from -16 to
163 15 to be shared as read-only aliases to the same SV, except that this
164 treatment ALSO applies to SRL_HDR_VARINT. If set to a value larger than
165 16 then this applies to all varints varints under the value set. (In
166 general SRL_HDR_VARINT is used only for integers larger than 15, and
167 SRL_HDR_NEG and SRL_HDR_POS are used for -16 to -1 and 0 to 15
168 respectively.)
169 In simple terms if you want to share values larger than 16 then you
171 should use this option, if you want to share only values in the -16 to
172 15 range then you should use the "alias_smallint" option instead.
173 The result of this may be significant space savings in data structures
175 with many integers in the desire range. The cost is more memory used by
176 the decoder and a very modest speed penalty when deserializing.
177 Note this option changes the structure of the dumped data. Use with
179 caution.
180 use_undef
182 If set to a true value then this any undef value to be deserialized as
184 PL_sv_undef. This may change the structure of the data structure being
185 dumped, do not enable this unless you know what you are doing.
186 set_readonly
188 If set to a true value then the output will be completely readonly
190 (deeply).
191 set_readonly_scalars
193 If set to a true value then scalars in the output will be readonly
195 (deeply). References won't be readonly.
196
198 decode
199 Given a byte string of Sereal data, the "decode" call deserializes that
200 data structure. The result can be obtained in one of two ways: "decode"
201 accepts a second parameter, which is a scalar to write the result to,
202 AND "decode" will return the resulting data structure.
203 The two are subtly different in case of data structures that contain
205 references to the root element. In that case, the return value will be
206 a (non-recursive) copy of the reference. The pass-in style is more
207 correct. In other words,
208 $decoder->decode($sereal_string, my $out);
210 # is almost the same but safer than:
211 my $out = $decoder->decode($sereal_string);
212 This is an unfortunate side-effect of perls standard copy semantics of
214 assignment. Possibly one day we will have an alternative to this.
215 decode_with_header
217 Given a byte string of Sereal data, the "decode_with_header" call
218 deserializes that data structure as "decode" would do, however it also
219 decodes the optional user data structure that can be embedded into a
220 Sereal document, inside the header (see Sereal::Encoder::encode).
221 It accepts an optional second parameter, which is a scalar to write the
223 body to, and an optional third parameter, which is a scalar to write
224 the header to.
225 Regardless of the number of parameters received, "decode_with_header"
227 returns an ArrayRef containing the deserialized header, and the
228 deserialized body, in this order.
229 See "decode" for the subtle difference between the one, two and three
231 parameters versions.
232 If there is no header in a Sereal document, corresponding variable or
234 return value will be set to undef.
235 decode_only_header
237 Given a byte string of Sereal data, the "decode_only_header"
238 deserializes only the optional user data structure that can be embedded
239 into a Sereal document, inside the header (see
240 Sereal::Encoder::encode).
241 It accepts an optional second parameter, which is a scalar to write the
243 header to.
244 Regardless of the number of parameters received, "decode_only_header"
246 returns the resulting data structure.
247 See "decode" for the subtle difference between the one and two
249 parameters versions.
250 If there is no header in a Sereal document, corresponding variable or
252 return value will be set to undef.
253 decode_with_offset
255 Same as the "decode" method, except as second parameter, you must pass
256 an integer offset into the input string, at which the decoding is to
257 start. The optional "pass-in" style scalar (see "decode" above) is
258 relegated to being the third parameter.
259 decode_only_header_with_offset
261 Same as the "decode_only_header" method, except as second parameter,
262 you must pass an integer offset into the input string, at which the
263 decoding is to start. The optional "pass-in" style scalar (see
264 "decode_only_header" above) is relegated to being the third parameter.
265 decode_with_header_and_offset
267 Same as the "decode_with_header" method, except as second parameter,
268 you must pass an integer offset into the input string, at which the
269 decoding is to start. The optional "pass-in" style scalars (see
270 "decode_with_header" above) are relegated to being the third and fourth
271 parameters.
272 bytes_consumed
274 After using the various "decode" methods documented previously,
275 "bytes_consumed" can return the number of bytes from the body of the
276 input string that were actually consumed by the decoder. That is, if
277 you append random garbage to a valid Sereal document, "decode" will
278 happily decode the data and ignore the garbage. If that is an error in
279 your use case, you can use "bytes_consumed" to catch it.
280 my $out = $decoder->decode($sereal_string);
282 if (length($sereal_string) != $decoder->bytes_consumed) {
283 die "Not all input data was consumed!";
284 }
285 Chances are that if you do this, you're violating UNIX philosophy in
287 "be strict in what you emit but lenient in what you accept".
288 You can also use this to deserialize a list of Sereal documents that is
290 concatenated into the same string (code not very robust...):
291 my @out;
293 my $pos = 0;
294 eval {
295 while (1) {
296 push @out, $decoder->decode_with_offset($sereal_string, $pos);
297 $pos += $decoder->bytes_consumed;
298 last if $pos >= length($sereal_string)
299 or not $decoder->bytes_consumed;
300 }
301 };
302 As mentioned, only the bytes consumed from the body are considered. So
304 the following example is correct, as only the header is deserialized:
305 my $header = $decoder->decode_only_header($sereal_string);
307 my $count = $decoder->bytes_consumed;
308 # $count is 0
309 decode_from_file
311 Sereal::Decoder->decode_from_file($file);
312 $decoder->decode_from_file($file);
313 Read and decode the file specified. If called in list context and
315 incremental mode is enabled then decodes all packets contained in the
316 file and returns a list, otherwise decodes the first (or only) packet
317 in the file. Accepts an optinal "target" variable as a second argument.
318 looks_like_sereal
320 Performs some rudimentary check to determine if the argument appears to
321 be a valid Sereal packet or not. These tests are not comprehensive and
322 a true result does not mean that the document is valid, merely that it
323 appears to be valid. On the other hand a false result is always
324 reliable.
325 The return of this method may be treated as a simple boolean but is in
327 fact a more complex return. When the argument does not look anything
328 like a Sereal document then the return is perl's FALSE, which has the
329 property of being string equivalent to "" and numerically equivalent to
330 0. However when the argument appears to be a UTF-8 encoded protocol 3
331 Sereal document (by noticing that the \xF3 in the magic string has been
332 replaced by \xC3\xB3) then it returns 0 (the number, which is string
333 equivalent to "0"), and otherwise returns the protocol version of the
334 document. This means you can write something like this:
335 $type= Sereal::Decoder->looks_like_sereal($thing);
337 if ($type eq '') {
338 say "Not a Sereal document";
339 } elsif ($type eq '0') {
340 say "Possibly utf8 encoded Sereal document";
341 } else {
342 say "Sereal document version $type";
343 }
344 For reference, Sereal's magic value is a four byte string which is
346 either "=srl" for protocol version 1 and 2 or "=\xF3rl" for protocol
347 version 3 and later. This function checks that the magic string
348 corresponds with the reported version number, as well as other checks,
349 which may be enhanced in the future.
350 Note that looks_like_sereal() may be called as a class or object
352 method, and may also be called as a single argument function. See the
353 related scalar_looks_like_sereal() for a version which may ONLY be
354 called as a function, not as a method (and which is typically much
355 faster).
356
358 sereal_decode_with_object
359 The functional interface that is equivalent to using "decode". Takes a
360 decoder object reference as first parameter, followed by a byte string
361 to deserialize. Optionally takes a third parameter, which is the
362 output scalar to write to. See the documentation for "decode" above for
363 details.
364 This functional interface is marginally faster than the OO interface
366 since it avoids method resolution overhead and, on sufficiently modern
367 Perl versions, can usually avoid subroutine call overhead. See
368 Sereal::Performance for a discussion on how to tune Sereal for maximum
369 performance if you need to.
370 sereal_decode_with_header_with_object
372 The functional interface that is equivalent to using
373 "decode_with_header". Takes a decoder object reference as first
374 parameter, followed by a byte string to deserialize. Optionally takes
375 third and fourth parameters, which are the output scalars to write to.
376 See the documentation for "decode_with_header" above for details.
377 This functional interface is marginally faster than the OO interface
379 since it avoids method resolution overhead and, on sufficiently modern
380 Perl versions, can usually avoid subroutine call overhead. See
381 Sereal::Performance for a discussion on how to tune Sereal for maximum
382 performance if you need to.
383 sereal_decode_only_header_with_object
385 The functional interface that is equivalent to using
386 "decode_only_header". Takes a decoder object reference as first
387 parameter, followed by a byte string to deserialize. Optionally takes a
388 third parameters, which outputs scalars to write to. See the
389 documentation for "decode_with_header" above for details.
390 This functional interface is marginally faster than the OO interface
392 since it avoids method resolution overhead and, on sufficiently modern
393 Perl versions, can usually avoid subroutine call overhead. See
394 Sereal::Performance for a discussion on how to tune Sereal for maximum
395 performance if you need to.
396 sereal_decode_only_header_with_offset_with_object
398 The functional interface that is equivalent to using
399 "decode_only_header_with_offset". Same as the
400 "sereal_decode_only_header_with_object" function, except as the third
401 parameter, you must pass an integer offset into the input string, at
402 which the decoding is to start. The optional "pass-in" style scalar
403 (see "sereal_decode_only_header_with_object" above) is relegated to
404 being the fourth parameter.
405 This functional interface is marginally faster than the OO interface
407 since it avoids method resolution overhead and, on sufficiently modern
408 Perl versions, can usually avoid subroutine call overhead. See
409 Sereal::Performance for a discussion on how to tune Sereal for maximum
410 performance if you need to.
411 sereal_decode_with_header_and_offset_with_object
413 The functional interface that is equivalent to using
414 "decode_with_header_and_offset". Same as the
415 "sereal_decode_with_header_with_object" function, except as the third
416 parameter, you must pass an integer offset into the input string, at
417 which the decoding is to start. The optional "pass-in" style scalars
418 (see "sereal_decode_with_header_with_object" above) are relegated to
419 being the fourth and fifth parameters.
420 This functional interface is marginally faster than the OO interface
422 since it avoids method resolution overhead and, on sufficiently modern
423 Perl versions, can usually avoid subroutine call overhead. See
424 Sereal::Performance for a discussion on how to tune Sereal for maximum
425 performance if you need to.
426 sereal_decode_with_offset_with_object
428 The functional interface that is equivalent to using
429 "decode_with_offset". Same as the "sereal_decode_with_object"
430 function, except as the third parameter, you must pass an integer
431 offset into the input string, at which the decoding is to start. The
432 optional "pass-in" style scalar (see "sereal_decode_with_object" above)
433 is relegated to being the third parameter.
434 This functional interface is marginally faster than the OO interface
436 since it avoids method resolution overhead and, on sufficiently modern
437 Perl versions, can usually avoid subroutine call overhead. See
438 Sereal::Performance for a discussion on how to tune Sereal for maximum
439 performance if you need to.
440 decode_sereal
442 The functional interface that is equivalent to using "new" and
443 "decode". Expects a byte string to deserialize as first argument,
444 optionally followed by a hash reference of options (see documentation
445 for "new()"). Finally, "decode_sereal" supports a third parameter,
446 which is the output scalar to write to. See the documentation for
447 "decode" above for details.
448 This functional interface is significantly slower than the OO interface
450 since it cannot reuse the decoder object.
451 decode_sereal_with_header_data
453 The functional interface that is equivalent to using "new" and
454 "decode_with_header". Expects a byte string to deserialize as first
455 argument, optionally followed by a hash reference of options (see
456 documentation for "new()"). Finally, "decode_sereal" supports third and
457 fourth parameters, which are the output scalars to write to. See the
458 documentation for "decode_with_header" above for details.
459 This functional interface is significantly slower than the OO interface
461 since it cannot reuse the decoder object.
462 scalar_looks_like_sereal
464 The functional interface that is equivalent to using
465 "looks_like_sereal".
466 Note that this version cannot be called as a method. It is normally
468 executed as a custom opcode, as such errors about its usage may be
469 caught at compile time, and it should be much faster than
470 looks_like_sereal.
471
473 This implementation of a Sereal decoder tries to be as robust to
474 invalid input data as reasonably possible. This means that it should
475 never (though read on) segfault. It may, however, cause a large malloc
476 to fail. Generally speaking, invalid data should cause a Perl-trappable
477 exception. The one exception is that for Snappy-compressed Sereal
478 documents, the Snappy library may cause segmentation faults (invalid
479 reads or writes). This should only be a problem if you do not checksum
480 your data (internal checksum support is a To-Do) or if you accept data
481 from potentially malicious sources.
482 It requires a lot of run-time boundary checks to prevent decoder
484 segmentation faults on invalid data. We implemented them in the
485 lightest way possible. Adding robustness against running out of memory
486 would cause an very significant run-time overhead. In most cases of
487 random garbage (with valid header no less) when a malloc() fails due to
488 invalid data, the problem was caused by a very large array or string
489 length. This kind of very large malloc can then fail, being trappable
490 from Perl. Only when packet causes many repeated allocations do you
491 risk causing a hard OOM error from the kernel that cannot be trapped
492 because Perl may require some small allocations to succeed before the
493 now-invalid memory is released. It is at least not entirely trivial to
494 craft a Sereal document that causes this behaviour.
495 Finally, deserializing proper objects is potentially a problem because
497 classes can define a destructor. Thus, the data fed to the decoder can
498 cause the (deferred) execution of any destructor in your application.
499 That's why the "refuse_objects" option exists and what the
500 "no_bless_objects" can be used for as well. Later on, we may or may not
501 provide a facility to whitelist classes. Furthermore, if the encoder
502 emitted any objects using "FREEZE" callbacks, the "THAW" class method
503 may be invoked on the respective classes. If you can't trust the source
504 of your Sereal documents, you may want to use the "refuse_objects"
505 option. For more details on the "FREEZE/THAW" mechanism, please refer
506 to Sereal::Encoder.
507
509 Please refer to the Sereal::Performance document that has more detailed
510 information about Sereal performance and tuning thereof.
511
513 "Sereal::Decoder" is thread-safe on Perl's 5.8.7 and higher. This means
514 "thread-safe" in the sense that if you create a new thread, all
515 "Sereal::Decoder" objects will become a reference to undef in the new
516 thread. This might change in a future release to become a full clone of
517 the decoder object.
518
520 For reporting bugs, please use the github bug tracker at
521 <http://github.com/Sereal/Sereal/issues>.
522 For support and discussion of Sereal, there are two Google Groups:
524 Announcements around Sereal (extremely low volume):
526 <https://groups.google.com/forum/?fromgroups#!forum/sereal-announce>
527 Sereal development list:
529 <https://groups.google.com/forum/?fromgroups#!forum/sereal-dev>
530
532 Yves Orton <demerphq@gmail.com>
533 Damian Gryski
535 Steffen Mueller <smueller@cpan.org>
537 Rafaël Garcia-Suarez
539 Ævar Arnfjörð Bjarmason <avar@cpan.org>
541 Tim Bunce
543 Daniel Dragan <bulkdd@cpan.org> (Windows support and bugfixes)
545 Zefram
547 Borislav Nikolov
549 Ivan Kruglov <ivan.kruglov@yahoo.com>
551 Eric Herman <eric@freesa.org>
553 Some inspiration and code was taken from Marc Lehmann's excellent
555 JSON::XS module due to obvious overlap in problem domain.
556
558 This module was originally developed for Booking.com. With approval
559 from Booking.com, this module was generalized and published on CPAN,
560 for which the authors would like to express their gratitude.
561
563 Copyright (C) 2012, 2013, 2014 by Steffen Mueller Copyright (C) 2012,
564 2013, 2014 by Yves Orton
565 The license for the code in this distribution is the following, with
567 the exceptions listed below:
568 This library is free software; you can redistribute it and/or modify it
570 under the same terms as Perl itself.
571 Except portions taken from Marc Lehmann's code for the JSON::XS module,
573 which is licensed under the same terms as this module. (Many thanks to
574 Marc for inspiration, and code.)
575 Also except the code for Snappy compression library, whose license is
577 reproduced below and which, to the best of our knowledge, is compatible
578 with this module's license. The license for the enclosed Snappy code
579 is:
580 Copyright 2011, Google Inc.
582 All rights reserved.
583 Redistribution and use in source and binary forms, with or without
585 modification, are permitted provided that the following conditions are
586 met:
587 * Redistributions of source code must retain the above copyright
589 notice, this list of conditions and the following disclaimer.
590 * Redistributions in binary form must reproduce the above
591 copyright notice, this list of conditions and the following disclaimer
592 in the documentation and/or other materials provided with the
593 distribution.
594 * Neither the name of Google Inc. nor the names of its
595 contributors may be used to endorse or promote products derived from
596 this software without specific prior written permission.
597 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
599 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
600 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
601 A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
602 OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
603 SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
604 LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
605 DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
606 THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
607 (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
608 OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
609perl v5.32.0 2020-08-04 Sereal::Decoder(3)