1FLATBUFFERS(7)                   User Manuals                   FLATBUFFERS(7)
2
3
4

NAME

6       flatbuffers - memory efficient serialization library
7

DESCRIPTION

9       Before you get started
10              Before  diving  into  the FlatBuffers usage in C++, it should be
11              noted   that   the    Tutorial    ⟨http://google.github.io/flat
12              buffers/flatbuffers_guide_tutorial.html⟩  page  has  a  complete
13              guide to general FlatBuffers usage in all of the supported  lan‐
14              guages  (including  C++).  This  page  is  designed to cover the
15              nuances of FlatBuffers usage, specific to C++.
16
17              This page assumes you have written a FlatBuffers schema and com‐
18              piled  it  with the Schema Compiler. If you have not, please see
19              Using   the   schema   compiler   ⟨http://google.github.io/flat
20              buffers/flatbuffers_guide_using_schema_compiler.html⟩  and Writ‐
21              ing    a    schema    ⟨http://google.github.io/flatbuffers/flat
22              buffers_guide_writing_schema.html⟩.
23
24              Assuming  you  wrote a schema, say mygame.fbs (though the exten‐
25              sion doesn't matter),  you've  generated  a  C++  header  called
26              mygame_generated.h   using   the   compiler   (e.g.   flatc   -c
27              mygame.fbs), you can now start using this  in  your  program  by
28              including  the  header.  As  noted,  this header relies on flat‐
29              buffers/flatbuffers.h, which should be in your include path.
30
31       FlatBuffers C++ library code location
32              The code for the FlatBuffers C++ library can be found  at  flat‐
33              buffers/include/flatbuffers.  You can browse the library code on
34              the  FlatBuffers  GitHub  page  ⟨https://github.com/google/flat
35              buffers/tree/master/include/flatbuffers⟩.
36
37       Testing the FlatBuffers C++ library
38              The  code  to  test  the  C++  library  can  be  found  at flat‐
39              buffers/tests. The test  code  itself  is  located  in  test.cpp
40https://github.com/google/flatbuffers/blob/mas
41              ter/tests/test.cpp⟩.
42
43              This test file is built alongside flatc. To review how to  build
44              the       project,       please      read      the      Building
45http://google.github.io/flatbuffers/flatbuffers_guide_build
46              ing.html⟩ documenation.
47
48              To  run  the tests, execute flattests from the root flatbuffers/
49              directory.         For         example,         on         Linux
50https://en.wikipedia.org/wiki/Linux⟩,  you  would  simply  run:
51              ./flattests.
52
53       Using the FlatBuffers C++ library
54              Note:  See  Tutorial  ⟨http://google.github.io/flatbuffers/flat
55              buffers_guide_tutorial.html⟩  for a more in-depth example of how
56              to use FlatBuffers in C++.
57
58              FlatBuffers supports both reading  and  writing  FlatBuffers  in
59              C++.
60
61              To  use FlatBuffers in your code, first generate the C++ classes
62              from your schema with the --cpp option to flatc.  Then  you  can
63              include both FlatBuffers and the generated code to read or write
64              FlatBuffers.
65
66              For example, here is how you would read a FlatBuffer binary file
67              in C++: First, include the library and generated code. Then read
68              the file into a char * array, which you pass to GetMonster().
69
70                #include "flatbuffers/flatbuffers.h"
71                #include "monster_test_generate.h"
72                #include <iostream> // C++ header file for printing
73                #include <fstream> // C++ header file for file access
74
75
76                std::ifstream infile;
77                infile.open("monsterdata_test.mon", std::ios::binary | std::ios::in);
78                infile.seekg(0,std::ios::end);
79                int length = infile.tellg();
80                infile.seekg(0,std::ios::beg);
81                char *data = new char[length];
82                infile.read(data, length);
83                infile.close();
84
85                auto monster = GetMonster(data);
86
87              monster is of type Monster *, and  points  to  somewhere  inside
88              your  buffer  (root  object  pointers  are  not the same as buf‐
89              fer_pointer !).  If you look in your  generated  header,  you'll
90              see  it  has  convenient  accessors  for  all fields, e.g. hp(),
91              mana(), etc:
92
93                std::cout << "hp : " << monster->hp() << std::endl;              // `80`
94                std::cout << "mana : " << monster->mana() << std::endl;          // default value of `150`
95                std::cout << "name : " << monster->name()->c_str() << std::endl; // "MyMonster"
96
97              Note: That we never stored a mana value, so it will  return  the
98              default.
99
100       Object based API
101              FlatBuffers  is  all  about  memory efficiency, which is why its
102              base API is written around using as little as  possible  of  it.
103              This  does  make the API clumsier (requiring pre-order construc‐
104              tion of all data, and making mutation harder).
105
106              For times when efficiency is less important  a  more  convenient
107              object  based API can be used (through --gen-object-api) that is
108              able to unpack & pack a FlatBuffer into objects and standard STL
109              containers,  allowing  for  convenient  construction, access and
110              mutation.
111
112              To use:
113
114                // Autogenerated class from table Monster.
115                MonsterT monsterobj;
116
117                // Deserialize from buffer into object.
118                UnPackTo(&monsterobj, flatbuffer);
119
120                // Update object directly like a C++ class instance.
121                cout << monsterobj->name;  // This is now a std::string!
122                monsterobj->name = "Bob";  // Change the name.
123
124                // Serialize into new flatbuffer.
125                FlatBufferBuilder fbb;
126                Pack(fbb, &monsterobj);
127
128              The following attributes are specific to  the  object-based  API
129              code generation:
130
131              · native_inline:  (on  a  field):  Because FlatBuffer tables and
132                structs are optionally present in a  given  buffer,  they  are
133                best represented as pointers (specifically std::uniqueptrs) in
134                the native class since  they  can  be  null.   This  attribute
135                changes the member declaration to use the type directly rather
136                than wrapped in a uniqueptr.
137
138              · native_default: "value" (on a field):  For  members  that  are
139                declared   "native_inline",  the  value  specified  with  this
140                attribute will be included verbatim in the  class  constructor
141                initializer list for this member.
142
143              · native_custom_alloc:"customallocator"  (on a table or struct):
144                When using the object-based  API  all  generated  NativeTables
145                that   are  allocated when unpacking your  flatbuffer will use
146                "custom  allocator".  The  allocator  is  also  used  by   any
147                std::vector  that  appears in a table defined with `nativecus‐
148                tom_alloc`.  This can be  used to provide  allocation  from  a
149                pool   for  example,  for  faster  unpacking  when  using  the
150                object-based API.
151
152              Minimal Example:
153
154              schema:
155
156                     table mytable(native_custom_alloc:"custom_allocator") {
157                       ...
158                     }
159
160              with custom_allocator defined before flatbuffers.h is  included,
161              as:
162
163                     template <typename T> struct custom_allocator : public std::allocator<T> {
164
165                       typedef T *pointer;
166
167                       template <class U>
168                       struct rebind {
169                         typedef custom_allocator<U> other;
170                       };
171
172                       pointer allocate(const std::size_t n) {
173                         return std::allocator<T>::allocate(n);
174                       }
175
176                       void deallocate(T* ptr, std::size_t n) {
177                         return std::allocator<T>::deallocate(ptr,n);
178                       }
179
180                       custom_allocator() throw() {}
181                       template <class U>
182                       custom_allocator(const custom_allocator<U>&) throw() {}
183                     };
184
185              · native_type: "type" (on a struct): In some cases, a more opti‐
186                mal C++ data type exists for a given struct. For example,  the
187                following schema:
188
189                     struct Vec2 {
190                       x: float;
191                       y: float;
192                     }
193
194                     generates the following Object-Based API class:
195
196                     struct Vec2T : flatbuffers::NativeTable {
197                       float x;
198                       float y;
199                     };
200
201                     However,  it  can be useful to instead use a user-defined
202                     C++ type since it can provide more functionality, eg.
203
204                     struct vector2 {
205                       float x = 0, y = 0;
206                       vector2 operator+(vector2 rhs) const { ... }
207                       vector2 operator-(vector2 rhs) const { ... }
208                       float length() const { ... }
209                       // etc.
210                     };
211
212                     The native_type attribute will replace the usage  of  the
213                     generated  class with the given type. So, continuing with
214                     the example, the generated code would  use  |vector2|  in
215                     place of |Vec2T| for all generated code.
216
217                     However,  because  the  native_type  is  unknown to flat‐
218                     buffers, the user must provide the following functions to
219                     aide in the serialization process:
220
221                     namespace flatbuffers {
222                       FlatbufferStruct Pack(const native_type& obj);
223                       native_type UnPack(const FlatbufferStruct& obj);
224                     }
225
226              · native_include:   "path"   (at   file   level):   Because  the
227                native_type attribute can be used to introduce types that  are
228                unknown to flatbuffers, it may be necessary to include "exter‐
229                nal" header files in the generated code. This attribute can be
230                used  to  directly add an #include directive to the top of the
231                generated code that includes the specified path directly.
232
233       External references
234              An additional feature of the object API is the ability to  allow
235              you  to  load  multiple  independent  FlatBuffers, and have them
236              refer to eachothers objects using hashes which are  then  repre‐
237              sented as typed pointers in the object API.
238
239              To  make  this  work  have  a  field  in the objects you want to
240              referred to which is using the string hashing feature (see  hash
241              attribute    in    the   schema   ⟨http://google.github.io/flat
242              buffers/flatbuffers_guide_writing_schema.html⟩   documentation).
243              Then you have a similar hash in the field referring to it, along
244              with a cpp_type attribute specifying  the  C++  type  this  will
245              refer to (this can be any C++ type, and will get a * added).
246
247              Then,  in  JSON  or  however you create these buffers, make sure
248              they use the same string (or hash).
249
250              When you call UnPack (or Create), you'll need  a  function  that
251              maps  from  hash  to  the  object  (see  resolver_function_t for
252              details).
253
254       Using different pointer types
255              By default the object tree is built out of std::unique_ptr,  but
256              you can influence this either globally (using the --cpp-ptr-type
257              argument  to  flatc)  or  per  field  (using  the   cpp_ptr_type
258              attribute) to by any smart pointer type (my_ptr<T>), or by spec‐
259              ifying naked as the type to get T * pointers. Unlike  the  smart
260              pointers, naked pointers do not manage memory for you, so you'll
261              have to manage their lifecycles manually.
262
263       Using different string type
264              By default the object tree is built out of std::string, but  you
265              can  influence  this  either  globally (using the --cpp-str-type
266              argument  to  flatc)  or  per  field  using   the   cpp_str_type
267              attribute.
268
269              The type must support T::c_str() and T::length() as member func‐
270              tions.
271
272       Reflection (& Resizing)
273              There is experimental support  for  reflection  in  FlatBuffers,
274              allowing  you  to read and write data even if you don't know the
275              exact format of a buffer, and even allows you to change sizes of
276              strings and vectors in-place.
277
278              The  way  this  works is very elegant; there is actually a Flat‐
279              Buffer schema that describes schemas (!) which you can  find  in
280              reflection/reflection.fbs.   The  compiler, flatc, can write out
281              any schemas it has just parsed as a  binary  FlatBuffer,  corre‐
282              sponding to this meta-schema.
283
284              Loading  in  one  of  these binary schemas at runtime allows you
285              traverse any FlatBuffer data  that  corresponds  to  it  without
286              knowing the exact format. You can query what fields are present,
287              and then read/write them after.
288
289              For convenient field manipulation, you can  include  the  header
290              flatbuffers/reflection.h  which includes both the generated code
291              from the meta schema, as well as a lot of helper functions.
292
293              And example of usage, for  the  time  being,  can  be  found  in
294              test.cpp/ReflectionTest().
295
296       Mini Reflection
297
298              A more limited form of reflection is available for direct inclu‐
299              sion in generated code, which doesn't any (binary) schema access
300              at  all.  It  was designed to keep the overhead of reflection as
301              low as possible (on the order of 2-6 bytes per  field  added  to
302              your  executable),  but  doesn't contain all the information the
303              (binary) schema contains.
304
305              You add this information to your generated  code  by  specifying
306              --reflect-types  (or  instead  --reflect-names  if you also want
307              field / enum names).
308
309              You can now use this information, for example to print  a  Flat‐
310              Buffer to text:
311
312                auto s = flatbuffers::FlatBufferToString(flatbuf, MonsterTypeTable());
313
314              MonsterTypeTable()  is  declared  in the generated code for each
315              type. The string produced is very similar to the  JSON  produced
316              by the Parser based text generator.
317
318              You'll need flatbuffers/minireflect.h for this functionality. In
319              there is also a convenient visitor/iterator  so  you  can  write
320              your  own  output  /  functionality based on the mini reflection
321              tables without having to  know  the  FlatBuffers  or  reflection
322              encoding.
323
324       Storing maps / dictionaries in a FlatBuffer
325              FlatBuffers  doesn't support maps natively, but there is support
326              to emulate their behavior with vectors and binary search,  which
327              means you can have fast lookups directly from a FlatBuffer with‐
328              out having to unpack your data into a std::map or similar.
329
330              To use it:
331
332              · Designate one of the fields in a table as  they  "key"  field.
333                You  do  this by setting the key attribute on this field, e.g.
334                name:string (key).
335
336                You may only have one key field, and it must be of  string  or
337                scalar type.
338
339              · Write  out tables of this type as usual, collect their offsets
340                in an array or vector.
341
342              · Instead  of  CreateVector,  call   CreateVectorOfSortedTables,
343                which  will  first  sort all offsets such that the tables they
344                refer to are sorted by the key field, then serialize it.
345
346              · Now when you're accessing the FlatBuffer,  you  can  use  Vec‐
347                tor::LookupByKey  instead  of  just Vector::Get to access ele‐
348                ments  of  the  vector,  e.g.:  myvector->LookupByKey("Fred"),
349                which  returns  a  pointer to the corresponding table type, or
350                nullptr if not found.  LookupByKey performs a  binary  search,
351                so  should  have  a  similar  speed to std::map, though may be
352                faster because of better caching. LookupByKey  only  works  if
353                the  vector  has been sorted, it will likely not find elements
354                if it hasn't been sorted.
355
356       Direct memory access
357              As you can see from the above examples, all elements in a buffer
358              are accessed through generated accessors. This is because every‐
359              thing is stored in little endian format on  all  platforms  (the
360              accessor  performs a swap operation on big endian machines), and
361              also because the layout of things is generally not known to  the
362              user.
363
364              For  structs,  layout  is deterministic and guaranteed to be the
365              same across platforms (scalars are aligned to  their  own  size,
366              and  structs  themselves  to  their largest member), and you are
367              allowed to access this memory directly  by  using  sizeof()  and
368              memcpy on the pointer to a struct, or even an array of structs.
369
370              To  compute  offsets to sub-elements of a struct, make sure they
371              are a structs themselves, as then you can use  the  pointers  to
372              figure  out  the  offset  without having to hardcode it. This is
373              handy for use of arrays of structs with calls  like  glVertexAt‐
374              tribPointer in OpenGL or similar APIs.
375
376              It  is important to note is that structs are still little endian
377              on all machines, so only use tricks like this if you can guaran‐
378              tee you're not shipping on a big endian machine (an assert(FLAT‐
379              BUFFERS_LITTLEENDIAN) would be wise).
380
381       Access of untrusted buffers
382              The generated accessor functions  access  fields  over  offsets,
383              which is very quick. These offsets are not verified at run-time,
384              so a malformed buffer could cause a program to crash by  access‐
385              ing random memory.
386
387              When  you're  processing large amounts of data from a source you
388              know (e.g.  your own generated data on disk),  this  is  accept‐
389              able,  but  when  reading  data from the network that can poten‐
390              tially have been modified by an attacker, this is undesirable.
391
392              For this reason, you can optionally use a buffer verifier before
393              you  access  the data. This verifier will check all offsets, all
394              sizes of fields, and null termination of strings to ensure  that
395              when a buffer is accessed, all reads will end up inside the buf‐
396              fer.
397
398              Each root type will have a verification function  generated  for
399              it, e.g. for Monster, you can call:
400
401                bool ok = VerifyMonsterBuffer(Verifier(buf, len));
402
403              if ok is true, the buffer is safe to read.
404
405              Besides  untrusted  data, this function may be useful to call in
406              debug mode, as extra  insurance  against  data  being  corrupted
407              somewhere along the way.
408
409              While  verifying  a  buffer isn't "free", it is typically faster
410              than a full traversal (since any scalar  data  is  not  actually
411              touched),  and  since it may cause the buffer to be brought into
412              cache before reading, the actual overhead may be even lower than
413              expected.
414
415              In  specialized cases where a denial of service attack is possi‐
416              ble, the verifier has two additional constructor arguments  that
417              allow  you to limit the nesting depth and total amount of tables
418              the verifier may encounter  before  declaring  the  buffer  mal‐
419              formed.  The  default  is Verifier(buf, len, 64 /* max depth */,
420              1000000, /* max tables */) which should be sufficient  for  most
421              uses.
422
423       Text & schema parsing
424              Using  binary buffers with the generated header provides a super
425              low overhead use of FlatBuffer data. There are,  however,  times
426              when you want to use text formats, for example because it inter‐
427              acts better with source control, or you want to give your  users
428              easy access to data.
429
430              Another  reason  might be that you already have a lot of data in
431              JSON format, or a tool that generates JSON, and if you can write
432              a  schema  for it, this will provide you an easy way to use that
433              data directly.
434
435              (see the schema documentation for some  specifics  on  the  JSON
436              format accepted).
437
438              There are two ways to use text formats:
439
440              · Using the compiler as a conversion tool.
441
442               This  is  the  preferred path, as it doesn't require you to add
443              any new code to your program, and is maximally
444               efficient since you can ship with binary data. The disadvantage
445              is that it is an extra step for your
446               users/developers  to perform, though you might be able to auto‐
447              mate it.
448
449                     flatc -b myschema.fbs mydata.json
450
451               This will generate the binary file mydata_wire.bin which can be
452              loaded as before.
453
454              · Making your program capable of loading text directly.
455
456               This  gives you maximum flexibility. You could even opt to sup‐
457              port both, i.e. check for both files, and regenerate
458               the binary from text when required,  otherwise  just  load  the
459              binary. This option is currently only available for
460               C++, or Java through JNI.
461
462               As  mentioned  in  the section "Building" above, this technique
463              requires you to link a few more files into your
464               program, and you'll want to include flatbuffers/idl.h.
465
466               Load text (either a schema or json) into  an  in-memory  buffer
467              (there is a convenient LoadFile() utility function
468               in flatbuffers/util.h if you wish).  Construct a parser:
469
470                     flatbuffers::Parser parser;
471
472               Now you can parse any number of text files in sequence:
473
474                     parser.Parse(text_file.c_str());
475
476               This  works similarly to how the command-line compiler works: a
477              sequence of files parsed by the same Parser object
478               allow later files to reference definitions  in  earlier  files.
479              Typically this means you first load a schema file
480               (which  populates  Parser with definitions), followed by one or
481              more JSON files.
482
483               As optional argument to Parse, you may  specify  a  null-termi‐
484              nated list of include paths. If not specified, any
485               include statements try to resolve from the current directory.
486
487               If  there were any parsing errors, Parse will return false, and
488              Parser::err contains a human readable error
489               string with a line number etc, which you should present to  the
490              creator of that file.
491
492               After  each  JSON  file, the Parser::fbb member variable is the
493              FlatBufferBuilder that contains the binary buffer
494               version of that file, that you can access as  described  above.
495              samples/sample_text.cpp is a code sample showing
496               the above operations.
497
498       Threading
499              Reading a FlatBuffer does not touch any memory outside the orig‐
500              inal buffer, and is entirely read-only (all const), so  is  safe
501              to  access  from  multiple  threads even without synchronisation
502              primitives.
503
504              Creating a FlatBuffer is not thread safe. All state  related  to
505              building  a  FlatBuffer  is  contained  in  a  FlatBufferBuilder
506              instance, and no memory outside of it is touched. To  make  this
507              thread  safe, either do not share instances of FlatBufferBuilder
508              between threads (recommended), or manually wrap it in synchroni‐
509              sation primites. There's no automatic way to accomplish this, by
510              design, as we feel multithreaded construction of a single buffer
511              will be rare, and synchronisation overhead would be costly.
512
513       Advanced union features
514              The  C++  implementation  currently  supports  vectors of unions
515              (i.e. you can declare a field as [T] where T  is  a  union  type
516              instead  of  a table type). It also supports structs and strings
517              in unions, besides tables.
518
519              For an example of these features,  see  tests/union_vector,  and
520              UnionVectorTest in test.cpp.
521
522              Since these features haven't been ported to other languages yet,
523              if you choose to use them, you won't be able to use  these  buf‐
524              fers  in  other languages (flatc will refuse to compile a schema
525              that uses these features).
526
527              These features reduce the amount of "table  wrapping"  that  was
528              previously needed to use unions.
529
530              To use scalars, simply wrap them in a struct.
531

SEE ALSO

533       flatc(1), Official documentation ⟨http://google.github.io/flatbuffers
534
535
536
537Linux                             APRIL 2018                    FLATBUFFERS(7)
Impressum