1FLATBUFFERS(7) User Manuals FLATBUFFERS(7)
2
6 flatbuffers - memory efficient serialization library
7
9 Before you get started
10 Before diving into the FlatBuffers usage in C++, it should be
11 noted that the Tutorial ⟨http://google.github.io/flat‐
12 buffers/flatbuffers_guide_tutorial.html⟩ page has a complete
13 guide to general FlatBuffers usage in all of the supported lan‐
14 guages (including C++). This page is designed to cover the
15 nuances of FlatBuffers usage, specific to C++.
16 This page assumes you have written a FlatBuffers schema and com‐
18 piled it with the Schema Compiler. If you have not, please see
19 Using the schema compiler ⟨http://google.github.io/flat‐
20 buffers/flatbuffers_guide_using_schema_compiler.html⟩ and Writ‐
21 ing a schema ⟨http://google.github.io/flatbuffers/flat‐
22 buffers_guide_writing_schema.html⟩.
23 Assuming you wrote a schema, say mygame.fbs (though the exten‐
25 sion doesn't matter), you've generated a C++ header called
26 mygame_generated.h using the compiler (e.g. flatc -c
27 mygame.fbs), you can now start using this in your program by
28 including the header. As noted, this header relies on flat‐
29 buffers/flatbuffers.h, which should be in your include path.
30 FlatBuffers C++ library code location
32 The code for the FlatBuffers C++ library can be found at flat‐
33 buffers/include/flatbuffers. You can browse the library code on
34 the FlatBuffers GitHub page ⟨https://github.com/google/flat‐
35 buffers/tree/master/include/flatbuffers⟩.
36 Testing the FlatBuffers C++ library
38 The code to test the C++ library can be found at flat‐
39 buffers/tests. The test code itself is located in test.cpp
40 ⟨https://github.com/google/flatbuffers/blob/mas‐
41 ter/tests/test.cpp⟩.
42 This test file is built alongside flatc. To review how to build
44 the project, please read the Building
45 ⟨http://google.github.io/flatbuffers/flatbuffers_guide_build‐
46 ing.html⟩ documenation.
47 To run the tests, execute flattests from the root flatbuffers/
49 directory. For example, on Linux
50 ⟨https://en.wikipedia.org/wiki/Linux⟩, you would simply run:
51 ./flattests.
52 Using the FlatBuffers C++ library
54 Note: See Tutorial ⟨http://google.github.io/flatbuffers/flat‐
55 buffers_guide_tutorial.html⟩ for a more in-depth example of how
56 to use FlatBuffers in C++.
57 FlatBuffers supports both reading and writing FlatBuffers in
59 C++.
60 To use FlatBuffers in your code, first generate the C++ classes
62 from your schema with the --cpp option to flatc. Then you can
63 include both FlatBuffers and the generated code to read or write
64 FlatBuffers.
65 For example, here is how you would read a FlatBuffer binary file
67 in C++: First, include the library and generated code. Then read
68 the file into a char * array, which you pass to GetMonster().
69 #include "flatbuffers/flatbuffers.h"
71 #include "monster_test_generate.h"
72 #include <iostream> // C++ header file for printing
73 #include <fstream> // C++ header file for file access
74 std::ifstream infile;
77 infile.open("monsterdata_test.mon", std::ios::binary | std::ios::in);
78 infile.seekg(0,std::ios::end);
79 int length = infile.tellg();
80 infile.seekg(0,std::ios::beg);
81 char *data = new char[length];
82 infile.read(data, length);
83 infile.close();
84 auto monster = GetMonster(data);
86 monster is of type Monster *, and points to somewhere inside
88 your buffer (root object pointers are not the same as buf‐
89 fer_pointer !). If you look in your generated header, you'll
90 see it has convenient accessors for all fields, e.g. hp(),
91 mana(), etc:
92 std::cout << "hp : " << monster->hp() << std::endl; // `80`
94 std::cout << "mana : " << monster->mana() << std::endl; // default value of `150`
95 std::cout << "name : " << monster->name()->c_str() << std::endl; // "MyMonster"
96 Note: That we never stored a mana value, so it will return the
98 default.
99 Object based API
101 FlatBuffers is all about memory efficiency, which is why its
102 base API is written around using as little as possible of it.
103 This does make the API clumsier (requiring pre-order construc‐
104 tion of all data, and making mutation harder).
105 For times when efficiency is less important a more convenient
107 object based API can be used (through --gen-object-api) that is
108 able to unpack & pack a FlatBuffer into objects and standard STL
109 containers, allowing for convenient construction, access and
110 mutation.
111 To use:
113 // Autogenerated class from table Monster.
115 MonsterT monsterobj;
116 // Deserialize from buffer into object.
118 UnPackTo(&monsterobj, flatbuffer);
119 // Update object directly like a C++ class instance.
121 cout << monsterobj->name; // This is now a std::string!
122 monsterobj->name = "Bob"; // Change the name.
123 // Serialize into new flatbuffer.
125 FlatBufferBuilder fbb;
126 Pack(fbb, &monsterobj);
127 The following attributes are specific to the object-based API
129 code generation:
130 · native_inline: (on a field): Because FlatBuffer tables and
132 structs are optionally present in a given buffer, they are
133 best represented as pointers (specifically std::uniqueptrs) in
134 the native class since they can be null. This attribute
135 changes the member declaration to use the type directly rather
136 than wrapped in a uniqueptr.
137 · native_default: "value" (on a field): For members that are
139 declared "native_inline", the value specified with this
140 attribute will be included verbatim in the class constructor
141 initializer list for this member.
142 · native_custom_alloc:"customallocator" (on a table or struct):
144 When using the object-based API all generated NativeTables
145 that are allocated when unpacking your flatbuffer will use
146 "custom allocator". The allocator is also used by any
147 std::vector that appears in a table defined with `nativecus‐
148 tom_alloc`. This can be used to provide allocation from a
149 pool for example, for faster unpacking when using the
150 object-based API.
151 Minimal Example:
153 schema:
155 table mytable(native_custom_alloc:"custom_allocator") {
157 ...
158 }
159 with custom_allocator defined before flatbuffers.h is included,
161 as:
162 template <typename T> struct custom_allocator : public std::allocator<T> {
164 typedef T *pointer;
166 template <class U>
168 struct rebind {
169 typedef custom_allocator<U> other;
170 };
171 pointer allocate(const std::size_t n) {
173 return std::allocator<T>::allocate(n);
174 }
175 void deallocate(T* ptr, std::size_t n) {
177 return std::allocator<T>::deallocate(ptr,n);
178 }
179 custom_allocator() throw() {}
181 template <class U>
182 custom_allocator(const custom_allocator<U>&) throw() {}
183 };
184 · native_type: "type" (on a struct): In some cases, a more opti‐
186 mal C++ data type exists for a given struct. For example, the
187 following schema:
188 struct Vec2 {
190 x: float;
191 y: float;
192 }
193 generates the following Object-Based API class:
195 struct Vec2T : flatbuffers::NativeTable {
197 float x;
198 float y;
199 };
200 However, it can be useful to instead use a user-defined
202 C++ type since it can provide more functionality, eg.
203 struct vector2 {
205 float x = 0, y = 0;
206 vector2 operator+(vector2 rhs) const { ... }
207 vector2 operator-(vector2 rhs) const { ... }
208 float length() const { ... }
209 // etc.
210 };
211 The native_type attribute will replace the usage of the
213 generated class with the given type. So, continuing with
214 the example, the generated code would use |vector2| in
215 place of |Vec2T| for all generated code.
216 However, because the native_type is unknown to flat‐
218 buffers, the user must provide the following functions to
219 aide in the serialization process:
220 namespace flatbuffers {
222 FlatbufferStruct Pack(const native_type& obj);
223 native_type UnPack(const FlatbufferStruct& obj);
224 }
225 · native_include: "path" (at file level): Because the
227 native_type attribute can be used to introduce types that are
228 unknown to flatbuffers, it may be necessary to include "exter‐
229 nal" header files in the generated code. This attribute can be
230 used to directly add an #include directive to the top of the
231 generated code that includes the specified path directly.
232 External references
234 An additional feature of the object API is the ability to allow
235 you to load multiple independent FlatBuffers, and have them
236 refer to eachothers objects using hashes which are then repre‐
237 sented as typed pointers in the object API.
238 To make this work have a field in the objects you want to
240 referred to which is using the string hashing feature (see hash
241 attribute in the schema ⟨http://google.github.io/flat‐
242 buffers/flatbuffers_guide_writing_schema.html⟩ documentation).
243 Then you have a similar hash in the field referring to it, along
244 with a cpp_type attribute specifying the C++ type this will
245 refer to (this can be any C++ type, and will get a * added).
246 Then, in JSON or however you create these buffers, make sure
248 they use the same string (or hash).
249 When you call UnPack (or Create), you'll need a function that
251 maps from hash to the object (see resolver_function_t for
252 details).
253 Using different pointer types
255 By default the object tree is built out of std::unique_ptr, but
256 you can influence this either globally (using the --cpp-ptr-type
257 argument to flatc) or per field (using the cpp_ptr_type
258 attribute) to by any smart pointer type (my_ptr<T>), or by spec‐
259 ifying naked as the type to get T * pointers. Unlike the smart
260 pointers, naked pointers do not manage memory for you, so you'll
261 have to manage their lifecycles manually.
262 Using different string type
264 By default the object tree is built out of std::string, but you
265 can influence this either globally (using the --cpp-str-type
266 argument to flatc) or per field using the cpp_str_type
267 attribute.
268 The type must support T::c_str() and T::length() as member func‐
270 tions.
271 Reflection (& Resizing)
273 There is experimental support for reflection in FlatBuffers,
274 allowing you to read and write data even if you don't know the
275 exact format of a buffer, and even allows you to change sizes of
276 strings and vectors in-place.
277 The way this works is very elegant; there is actually a Flat‐
279 Buffer schema that describes schemas (!) which you can find in
280 reflection/reflection.fbs. The compiler, flatc, can write out
281 any schemas it has just parsed as a binary FlatBuffer, corre‐
282 sponding to this meta-schema.
283 Loading in one of these binary schemas at runtime allows you
285 traverse any FlatBuffer data that corresponds to it without
286 knowing the exact format. You can query what fields are present,
287 and then read/write them after.
288 For convenient field manipulation, you can include the header
290 flatbuffers/reflection.h which includes both the generated code
291 from the meta schema, as well as a lot of helper functions.
292 And example of usage, for the time being, can be found in
294 test.cpp/ReflectionTest().
295 Mini Reflection
297 A more limited form of reflection is available for direct inclu‐
299 sion in generated code, which doesn't any (binary) schema access
300 at all. It was designed to keep the overhead of reflection as
301 low as possible (on the order of 2-6 bytes per field added to
302 your executable), but doesn't contain all the information the
303 (binary) schema contains.
304 You add this information to your generated code by specifying
306 --reflect-types (or instead --reflect-names if you also want
307 field / enum names).
308 You can now use this information, for example to print a Flat‐
310 Buffer to text:
311 auto s = flatbuffers::FlatBufferToString(flatbuf, MonsterTypeTable());
313 MonsterTypeTable() is declared in the generated code for each
315 type. The string produced is very similar to the JSON produced
316 by the Parser based text generator.
317 You'll need flatbuffers/minireflect.h for this functionality. In
319 there is also a convenient visitor/iterator so you can write
320 your own output / functionality based on the mini reflection
321 tables without having to know the FlatBuffers or reflection
322 encoding.
323 Storing maps / dictionaries in a FlatBuffer
325 FlatBuffers doesn't support maps natively, but there is support
326 to emulate their behavior with vectors and binary search, which
327 means you can have fast lookups directly from a FlatBuffer with‐
328 out having to unpack your data into a std::map or similar.
329 To use it:
331 · Designate one of the fields in a table as they "key" field.
333 You do this by setting the key attribute on this field, e.g.
334 name:string (key).
335 You may only have one key field, and it must be of string or
337 scalar type.
338 · Write out tables of this type as usual, collect their offsets
340 in an array or vector.
341 · Instead of CreateVector, call CreateVectorOfSortedTables,
343 which will first sort all offsets such that the tables they
344 refer to are sorted by the key field, then serialize it.
345 · Now when you're accessing the FlatBuffer, you can use Vec‐
347 tor::LookupByKey instead of just Vector::Get to access ele‐
348 ments of the vector, e.g.: myvector->LookupByKey("Fred"),
349 which returns a pointer to the corresponding table type, or
350 nullptr if not found. LookupByKey performs a binary search,
351 so should have a similar speed to std::map, though may be
352 faster because of better caching. LookupByKey only works if
353 the vector has been sorted, it will likely not find elements
354 if it hasn't been sorted.
355 Direct memory access
357 As you can see from the above examples, all elements in a buffer
358 are accessed through generated accessors. This is because every‐
359 thing is stored in little endian format on all platforms (the
360 accessor performs a swap operation on big endian machines), and
361 also because the layout of things is generally not known to the
362 user.
363 For structs, layout is deterministic and guaranteed to be the
365 same across platforms (scalars are aligned to their own size,
366 and structs themselves to their largest member), and you are
367 allowed to access this memory directly by using sizeof() and
368 memcpy on the pointer to a struct, or even an array of structs.
369 To compute offsets to sub-elements of a struct, make sure they
371 are a structs themselves, as then you can use the pointers to
372 figure out the offset without having to hardcode it. This is
373 handy for use of arrays of structs with calls like glVertexAt‐
374 tribPointer in OpenGL or similar APIs.
375 It is important to note is that structs are still little endian
377 on all machines, so only use tricks like this if you can guaran‐
378 tee you're not shipping on a big endian machine (an assert(FLAT‐
379 BUFFERS_LITTLEENDIAN) would be wise).
380 Access of untrusted buffers
382 The generated accessor functions access fields over offsets,
383 which is very quick. These offsets are not verified at run-time,
384 so a malformed buffer could cause a program to crash by access‐
385 ing random memory.
386 When you're processing large amounts of data from a source you
388 know (e.g. your own generated data on disk), this is accept‐
389 able, but when reading data from the network that can poten‐
390 tially have been modified by an attacker, this is undesirable.
391 For this reason, you can optionally use a buffer verifier before
393 you access the data. This verifier will check all offsets, all
394 sizes of fields, and null termination of strings to ensure that
395 when a buffer is accessed, all reads will end up inside the buf‐
396 fer.
397 Each root type will have a verification function generated for
399 it, e.g. for Monster, you can call:
400 bool ok = VerifyMonsterBuffer(Verifier(buf, len));
402 if ok is true, the buffer is safe to read.
404 Besides untrusted data, this function may be useful to call in
406 debug mode, as extra insurance against data being corrupted
407 somewhere along the way.
408 While verifying a buffer isn't "free", it is typically faster
410 than a full traversal (since any scalar data is not actually
411 touched), and since it may cause the buffer to be brought into
412 cache before reading, the actual overhead may be even lower than
413 expected.
414 In specialized cases where a denial of service attack is possi‐
416 ble, the verifier has two additional constructor arguments that
417 allow you to limit the nesting depth and total amount of tables
418 the verifier may encounter before declaring the buffer mal‐
419 formed. The default is Verifier(buf, len, 64 /* max depth */,
420 1000000, /* max tables */) which should be sufficient for most
421 uses.
422 Text & schema parsing
424 Using binary buffers with the generated header provides a super
425 low overhead use of FlatBuffer data. There are, however, times
426 when you want to use text formats, for example because it inter‐
427 acts better with source control, or you want to give your users
428 easy access to data.
429 Another reason might be that you already have a lot of data in
431 JSON format, or a tool that generates JSON, and if you can write
432 a schema for it, this will provide you an easy way to use that
433 data directly.
434 (see the schema documentation for some specifics on the JSON
436 format accepted).
437 There are two ways to use text formats:
439 · Using the compiler as a conversion tool.
441 This is the preferred path, as it doesn't require you to add
443 any new code to your program, and is maximally
444 efficient since you can ship with binary data. The disadvantage
445 is that it is an extra step for your
446 users/developers to perform, though you might be able to auto‐
447 mate it.
448 flatc -b myschema.fbs mydata.json
450 This will generate the binary file mydata_wire.bin which can be
452 loaded as before.
453 · Making your program capable of loading text directly.
455 This gives you maximum flexibility. You could even opt to sup‐
457 port both, i.e. check for both files, and regenerate
458 the binary from text when required, otherwise just load the
459 binary. This option is currently only available for
460 C++, or Java through JNI.
461 As mentioned in the section "Building" above, this technique
463 requires you to link a few more files into your
464 program, and you'll want to include flatbuffers/idl.h.
465 Load text (either a schema or json) into an in-memory buffer
467 (there is a convenient LoadFile() utility function
468 in flatbuffers/util.h if you wish). Construct a parser:
469 flatbuffers::Parser parser;
471 Now you can parse any number of text files in sequence:
473 parser.Parse(text_file.c_str());
475 This works similarly to how the command-line compiler works: a
477 sequence of files parsed by the same Parser object
478 allow later files to reference definitions in earlier files.
479 Typically this means you first load a schema file
480 (which populates Parser with definitions), followed by one or
481 more JSON files.
482 As optional argument to Parse, you may specify a null-termi‐
484 nated list of include paths. If not specified, any
485 include statements try to resolve from the current directory.
486 If there were any parsing errors, Parse will return false, and
488 Parser::err contains a human readable error
489 string with a line number etc, which you should present to the
490 creator of that file.
491 After each JSON file, the Parser::fbb member variable is the
493 FlatBufferBuilder that contains the binary buffer
494 version of that file, that you can access as described above.
495 samples/sample_text.cpp is a code sample showing
496 the above operations.
497 Threading
499 Reading a FlatBuffer does not touch any memory outside the orig‐
500 inal buffer, and is entirely read-only (all const), so is safe
501 to access from multiple threads even without synchronisation
502 primitives.
503 Creating a FlatBuffer is not thread safe. All state related to
505 building a FlatBuffer is contained in a FlatBufferBuilder
506 instance, and no memory outside of it is touched. To make this
507 thread safe, either do not share instances of FlatBufferBuilder
508 between threads (recommended), or manually wrap it in synchroni‐
509 sation primites. There's no automatic way to accomplish this, by
510 design, as we feel multithreaded construction of a single buffer
511 will be rare, and synchronisation overhead would be costly.
512 Advanced union features
514 The C++ implementation currently supports vectors of unions
515 (i.e. you can declare a field as [T] where T is a union type
516 instead of a table type). It also supports structs and strings
517 in unions, besides tables.
518 For an example of these features, see tests/union_vector, and
520 UnionVectorTest in test.cpp.
521 Since these features haven't been ported to other languages yet,
523 if you choose to use them, you won't be able to use these buf‐
524 fers in other languages (flatc will refuse to compile a schema
525 that uses these features).
526 These features reduce the amount of "table wrapping" that was
528 previously needed to use unions.
529 To use scalars, simply wrap them in a struct.
531
533 flatc(1), Official documentation ⟨http://google.github.io/flatbuffers⟩
534Linux APRIL 2018 FLATBUFFERS(7)