1SYNCTHING-BEP(7)                   Syncthing                  SYNCTHING-BEP(7)
2
3
4

NAME

6       syncthing-bep - Block Exchange Protocol v1
7

INTRODUCTION AND DEFINITIONS

9       The  Block  Exchange Protocol (BEP) is used between two or more devices
10       thus forming a cluster. Each device has one or more  folders  of  files
11       described by the local model, containing metadata and block hashes. The
12       local model is sent to the other devices in the cluster. The  union  of
13       all  files  in the local models, with files selected for highest change
14       version, forms the global model. Each device strives to get its folders
15       in  sync with the global model by requesting missing or outdated blocks
16       from the other devices in the cluster.
17
18       File data is described and transferred in units of blocks,  each  being
19       from  128  KiB  (131072 bytes) to 16 MiB in size, in steps of powers of
20       two. The block size may vary between files but is constant in any given
21       file, except for the last block which may be smaller.
22
23       The  key  words  “MUST”,  “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”,
24       “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and  “OPTIONAL”  in  this
25       document are to be interpreted as described in RFC 2119.
26

TRANSPORT AND AUTHENTICATION

28       BEP  is  deployed  as  the  highest level in a protocol stack, with the
29       lower level protocols providing encryption and authentication.
30
31          +-----------------------------+
32          |   Block Exchange Protocol   |
33          |-----------------------------|
34          | Encryption & Auth (TLS 1.2) |
35          |-----------------------------|
36          |      Reliable Transport     |
37          |-----------------------------|
38          v             ...             v
39
40       The encryption and authentication layer SHALL use TLS 1.2 or  a  higher
41       revision.  A  strong  cipher  suite  SHALL be used, with “strong cipher
42       suite” being defined as being without known  weaknesses  and  providing
43       Perfect  Forward  Secrecy  (PFS).  Examples of strong cipher suites are
44       given at the end of this document. This  is  not  to  be  taken  as  an
45       exhaustive  list of allowed cipher suites but represents best practices
46       at the time of writing.
47
48       The exact nature of the authentication is up to the  application,  how‐
49       ever it SHALL be based on the TLS certificate presented at the start of
50       the connection. Possibilities include certificates signed by  a  common
51       trusted  CA, preshared certificates, preshared certificate fingerprints
52       or certificate pinning combined with some out of band  first  verifica‐
53       tion.  The  reference implementation uses preshared certificate finger‐
54       prints (SHA-256) referred to as “Device IDs”.
55
56       There is no required order or synchronization among BEP messages except
57       as  noted  per  message type - any message type may be sent at any time
58       and the sender need not await a response to one message before  sending
59       another.
60
61       The underlying transport protocol MUST guarantee reliable packet deliv‐
62       ery.
63
64       In this document, in diagrams and text, “bit 0” refers to the most sig‐
65       nificant bit of a word; “bit 15” is thus the least significant bit of a
66       16 bit word (int16) and “bit 31” is the least significant bit of  a  32
67       bit  word  (int32). Non protocol buffer integers are always represented
68       in network byte order (i.e., big endian) and are signed  unless  stated
69       otherwise,  but  when describing message lengths negative values do not
70       make sense and the most significant bit MUST be zero.
71
72       The protocol buffer schemas in this document are in proto3 syntax. This
73       means, among other things, that all fields are optional and will assume
74       their default value when missing. This does not necessarily mean that a
75       message  is  valid  with all fields empty - for example, an index entry
76       for a file that does not have a name is not useful and MAY be  rejected
77       by  the  implementation. However the folder label is for human consump‐
78       tion only so an empty label should be  accepted  -  the  implementation
79       will  have to choose some way to represent the folder, perhaps by using
80       the ID in it’s place or automatically generating a label.
81

PRE-AUTHENTICATION MESSAGES

83       AFTER establishing a connection, but BEFORE performing any  authentica‐
84       tion, devices MUST exchange Hello messages.
85
86       Hello messages are used to carry additional information about the peer,
87       which might be of interest to the user even if the peer is not  permit‐
88       ted  to  communicate  due to failing authentication. Note that the cer‐
89       tificate based authentication may be considered part of the  TLS  hand‐
90       shake  that  precedes  the Hello message exchange, but even in the case
91       that a connection is rejected a Hello message must be sent  before  the
92       connection is terminated.
93
94       Hello messages MUST be prefixed with an int32 containing the magic num‐
95       ber 0x2EA7D90B, followed by an int16 representing the size of the  mes‐
96       sage, followed by the contents of the Hello message itself.
97
98           0                   1
99           0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
100          +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
101          |             Magic             |
102          |           (32 bits)           |
103          +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
104          |             Length            |
105          +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
106          /                               /
107          \             Hello             \
108          /                               /
109          +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
110
111       The  Hello message itself is in protocol buffer format with the follow‐
112       ing schema:
113
114          message Hello {
115              string device_name    = 1;
116              string client_name    = 2;
117              string client_version = 3;
118          }
119
120   Fields (Hello message)
121       The device_name is a  human  readable  (configured  or  auto  detected)
122       device name or host name, for the remote device.
123
124       The  client_name  and client_version identifies the implementation. The
125       values SHOULD  be simple strings identifying the  implementation  name,
126       as  a  user  would expect to see it, and the version string in the same
127       manner. An example client name is “syncthing”  and  an  example  client
128       version  is  “v0.7.2”.  The client version field SHOULD follow the pat‐
129       terns laid out in the Semantic  Versioning  <http://semver.org/>  stan‐
130       dard.
131
132       Immediately  after  exchanging  Hello  messages, the connection MUST be
133       dropped if the remote device does not pass authentication.
134

POST-AUTHENTICATION MESSAGES

136       Every message post authentication is made up of several parts:
137
138       · A header length word
139
140       · A Header
141
142       · A message length word
143
144       · A Message
145
146           0                   1
147           0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
148          +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
149          |         Header Length         |
150          +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
151          /                               /
152          \            Header             \
153          /                               /
154          +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
155          |         Message Length        |
156          |           (32 bits)           |
157          +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
158          /                               /
159          \            Message            \
160          /                               /
161          +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
162
163       The header length word is 16 bits. It indicates the length of the  fol‐
164       lowing  Header  message.  The  Header is in protocol buffer format. The
165       Header describes the type and compression status of the following  mes‐
166       sage.
167
168       The message is preceded by the 32 bit message length word and is one of
169       the concrete BEP messages described below, identified by the type field
170       of the Header.
171
172       As always, the length words are in network byte order (big endian).
173
174          message Header {
175              MessageType        type        = 1;
176              MessageCompression compression = 2;
177          }
178
179          enum MessageType {
180              CLUSTER_CONFIG    = 0;
181              INDEX             = 1;
182              INDEX_UPDATE      = 2;
183              REQUEST           = 3;
184              RESPONSE          = 4;
185              DOWNLOAD_PROGRESS = 5;
186              PING              = 6;
187              CLOSE             = 7;
188          }
189
190          enum MessageCompression {
191              NONE = 0;
192              LZ4  = 1;
193          }
194
195       When the compression field is NONE, the message is directly in protocol
196       buffer format.
197
198       When the compression field is LZ4, the message consists  of  a  32  bit
199       integer describing the uncompressed message length followed by a single
200       LZ4 block. After decompressing the LZ4 block it should  be  interpreted
201       as a protocol buffer message just as in the uncompressed case.
202

MESSAGE SUBTYPES

204   Cluster Config
205       This  informational message provides information about the cluster con‐
206       figuration as it pertains to the current connection. A  Cluster  Config
207       message  MUST  be  the  first post authentication message sent on a BEP
208       connection. Additional Cluster Config messages MUST NOT be  sent  after
209       the initial exchange.
210
211   Protocol Buffer Schema
212          message ClusterConfig {
213              repeated Folder folders = 1;
214          }
215
216          message Folder {
217              string id                   = 1;
218              string label                = 2;
219              bool   read_only            = 3;
220              bool   ignore_permissions   = 4;
221              bool   ignore_delete        = 5;
222              bool   disable_temp_indexes = 6;
223              bool   paused               = 7;
224
225              repeated Device devices = 16;
226          }
227
228          message Device {
229              bytes           id                         = 1;
230              string          name                       = 2;
231              repeated string addresses                  = 3;
232              Compression     compression                = 4;
233              string          cert_name                  = 5;
234              int64           max_sequence               = 6;
235              bool            introducer                 = 7;
236              uint64          index_id                   = 8;
237              bool            skip_introduction_removals = 9;
238          }
239
240          enum Compression {
241              METADATA = 0;
242              NEVER    = 1;
243              ALWAYS   = 2;
244          }
245
246   Fields (Cluster Config Message)
247       The  folders  field  contains the list of folders that will be synchro‐
248       nized over the current connection.
249
250   Fields (Folder Message)
251       The id field contains the folder ID, which is the unique identifier  of
252       the folder.
253
254       The  label  field contains the folder label, the human readable name of
255       the folder.
256
257       The read only field is set for folders that the device will  accept  no
258       updates from the network for.
259
260       The  ignore  permissions  field is set for folders that the device will
261       not accept or announce file permissions for.
262
263       The ignore delete field is set for folders that the device will  ignore
264       deletes for.
265
266       The  disable  temp  indexes field is set for folders that will not dis‐
267       patch and do not wish to receive progress updates about partially down‐
268       loaded files via Download Progress messages.
269
270       The paused field is set for folders that are currently paused.
271
272       The  devices  field  is a list of devices participating in sharing this
273       folder.
274
275   Fields (Device Message)
276       The device id field is a 32 byte number that  uniquely  identifies  the
277       device.  For instance, the reference implementation uses the SHA-256 of
278       the device X.509 certificate.
279
280       The name field is a human  readable  name  assigned  to  the  described
281       device  by  the  sending  device.  It  MAY  be empty and it need not be
282       unique.
283
284       The list of addresses is that used by the sending device to connect  to
285       the described device.
286
287       The  compression  field  indicates the compression mode in use for this
288       device and folder. The following values are valid:
289
290       0      Compress metadata. This enables compression of metadata messages
291              such as Index.
292
293       1      Compression disabled. No compression is used on any message.
294
295       2      Compress  always. Metadata messages as well as Response messages
296              are compressed.
297
298       The cert name field indicates the expected certificate  name  for  this
299       device.  It  is  commonly  blank,  indicating to use the implementation
300       default.
301
302       The max sequence field contains the  highest  sequence  number  of  the
303       files  in  the  index.  See  Delta Index Exchange for the usage of this
304       field.
305
306       The introducer field is set for devices that  are  trusted  as  cluster
307       introducers.
308
309       The  index  id field contains the unique identifier for the current set
310       of index data. See Delta Index Exchange for the usage of this field.
311
312       The skip introduction removals field signifies if the remote device has
313       opted  to  ignore introduction removals for the given device. This set‐
314       ting is copied across as we are being introduced to a new device.
315
316   Index and Index Update
317       The Index and Index Update messages define the contents of the  senders
318       folder. An Index message represents the full contents of the folder and
319       thus supersedes any previous index. An Index Update amends an  existing
320       index  with  new information, not affecting any entries not included in
321       the message. An Index Update MAY NOT be  sent  unless  preceded  by  an
322       Index,  unless a non-zero Max Sequence has been announced for the given
323       folder by the peer device.
324
325       The Index and Index Update messages are currently identical in  format,
326       although this is not guaranteed to be the case in the future.
327
328   Protocol Buffer Schema
329          message Index {
330              string            folder = 1;
331              repeated FileInfo files  = 2;
332          }
333
334          message IndexUpdate {
335              string            folder = 1;
336              repeated FileInfo files  = 2;
337          }
338
339          message FileInfo {
340              string       name           = 1;
341              FileInfoType type           = 2;
342              int64        size           = 3;
343              uint32       permissions    = 4;
344              int64        modified_s     = 5;
345              int32        modified_ns    = 11;
346              uint64       modified_by    = 12;
347              bool         deleted        = 6;
348              bool         invalid        = 7;
349              bool         no_permissions = 8;
350              Vector       version        = 9;
351              int64        sequence       = 10;
352              int32        block_size     = 13;
353
354              repeated BlockInfo Blocks         = 16;
355              string             symlink_target = 17;
356          }
357
358          enum FileInfoType {
359              FILE              = 0;
360              DIRECTORY         = 1;
361              SYMLINK_FILE      = 2 [deprecated = true];
362              SYMLINK_DIRECTORY = 3 [deprecated = true];
363              SYMLINK           = 4;
364          }
365
366          message BlockInfo {
367              int64 offset     = 1;
368              int32 size       = 2;
369              bytes hash       = 3;
370              uint32 weak_hash = 4;
371          }
372
373          message Vector {
374              repeated Counter counters = 1;
375          }
376
377          message Counter {
378              uint64 id    = 1;
379              uint64 value = 2;
380          }
381
382   Fields (Index Message)
383       The  folder field identifies the folder that the index message pertains
384       to.
385
386       The files field is a list of files making up the index information.
387
388   Fields (FileInfo Message)
389       The name is the file name path relative to the folder  root.  Like  all
390       strings in BEP, the Name is always in UTF-8 NFC regardless of operating
391       system or file system specific conventions. The  name  field  uses  the
392       slash  character (“/”) as path separator, regardless of the implementa‐
393       tion’s operating system conventions. The combination of folder and name
394       uniquely identifies each file in a cluster.
395
396       The type field contains the type of the described item. The type is one
397       of file (0), directory (1), or symlink (4).
398
399       The size field contains the size of the file, in bytes. For directories
400       and symlinks the size is zero.
401
402       The  permissions field holds the common Unix permission bits. An imple‐
403       mentation MAY ignore or interpret these as  is  suitable  on  the  host
404       operating system.
405
406       The  modified_s  time  is  expressed as the number of seconds since the
407       Unix Epoch (1970-01-01 00:00:00 UTC). The modified_ns field  holds  the
408       nanosecond part of the modification time.
409
410       The  modified_by  field holds the short id of the client that last made
411       any modification to the file whether add, change or delete.  This  will
412       be  overwritten  every  time  a  change is made to the file by the last
413       client to do so and so does not hold history.
414
415       The deleted field is set when the file has been deleted. The block list
416       SHALL be of length zero and the modification time indicates the time of
417       deletion or, if the time of deletion is not reliably determinable,  the
418       last known modification time.
419
420       The  invalid  field is set when the file is invalid and unavailable for
421       synchronization. A peer MAY set this bit to indicate that it can tempo‐
422       rarily not serve data for the file.
423
424       The no permissions field is set when there is no permission information
425       for the file. This is the case when it  originates  on  a  file  system
426       which  does  not  support  permissions. Changes to only permission bits
427       SHOULD be disregarded on files with this bit set. The permissions  bits
428       MUST be set to the octal value 0666.
429
430       The  version field is a version vector describing the updates performed
431       to a file by all members in the cluster. Each counter  in  the  version
432       vector  is an ID-Value tuple. The ID is the first 64 bits of the device
433       ID. The Value is a simple incrementing counter, starting at  zero.  The
434       combination  of  Folder,  Name and Version uniquely identifies the con‐
435       tents of a file at a given point in time.
436
437       The sequence field is the value of a device local  monotonic  clock  at
438       the  time  of  last local database update to a file. The clock ticks on
439       every local database update, thus forming a sequence number over  data‐
440       base updates.
441
442       The block_size field is the size, in bytes, of each individual block in
443       the block list (except, possibly, the last block).  If  this  field  is
444       missing  or  zero,  the  block  size  is  assumed to be 128 KiB (131072
445       bytes). Valid values of this field are the powers of two from  128  KiB
446       through 16 MiB. See also Selection of Block Size.
447
448       The  blocks list contains the size and hash for each block in the file.
449       Each block represents a block_size-sized slice of the file, except  for
450       the  last block which may represent a smaller amount of data. The block
451       list is empty for directories and symlinks.
452
453       The symlink_target field contains the symlink target,  for  entries  of
454       symlink type. It is empty for all other entry types.
455
456   Request
457       The Request message expresses the desire to receive a data block corre‐
458       sponding to a part of a certain file in the peer’s folder.
459
460   Protocol Buffer Schema
461          message Request {
462              int32  id             = 1;
463              string folder         = 2;
464              string name           = 3;
465              int64  offset         = 4;
466              int32  size           = 5;
467              bytes  hash           = 6;
468              bool   from_temporary = 7;
469          }
470
471   Fields
472       The id is the request identifier. It will be matched in the correspond‐
473       ing Response message. Each outstanding request must have a unique ID.
474
475       The  folder  and  name  fields are as documented for the Index message.
476       The offset and size fields specify the region of the file to be  trans‐
477       ferred.  This  SHOULD  equate  to exactly one block as seen in an Index
478       message.
479
480       The hash field MAY be set to the expected hash value of the  block.  If
481       set,  the other device SHOULD ensure that the transmitted block matches
482       the requested hash. The other device MAY reuse a block from a different
483       file and offset having the same size and hash, if one exists.
484
485       The  from  temporary  field  is set to indicate that the read should be
486       performed from the temporary file (converting name  to  it’s  temporary
487       form)  and  falling back to the non temporary file if any error occurs.
488       Knowledge of contents of temporary files  comes  from  DownloadProgress
489       messages.
490
491   Response
492       The Response message is sent in response to a Request message.
493
494   Protocol Buffer Schema
495          message Response {
496              int32     id   = 1;
497              bytes     data = 2;
498              ErrorCode code = 3;
499          }
500
501          enum ErrorCode {
502              NO_ERROR     = 0;
503              GENERIC      = 1;
504              NO_SUCH_FILE = 2;
505              INVALID_FILE = 3;
506          }
507
508   Fields
509       The  id  field  is  the request identifier. It must match the ID of the
510       Request that is being responded to.
511
512       The data field contains either the requested data block or is empty  if
513       the requested block is not available.
514
515       The  code  field contains an error code describing the reason a Request
516       could not be  fulfilled,  in  the  case  where  zero  length  data  was
517       returned. The following values are defined:
518
519       0      No Error (data should be present)
520
521       1      Generic Error
522
523       2      No  Such  File (the requested file does not exist, or the offset
524              is outside the acceptable range for the file)
525
526       3      Invalid (file exists but has invalid bit  set  or  is  otherwise
527              unavailable)
528
529   DownloadProgress
530       The  DownloadProgress  message  is  used to notify remote devices about
531       partial availability of files. By  default,  these  messages  are  sent
532       every  5 seconds, and only in the cases where progress or state changes
533       have been detected.  Each DownloadProgress message is  addressed  to  a
534       specific  folder  and MUST contain zero or more FileDownloadProgressUp‐
535       date messages.
536
537   Protocol Buffer Schema
538          message DownloadProgress {
539              string                              folder  = 1;
540              repeated FileDownloadProgressUpdate updates = 2;
541          }
542
543          message FileDownloadProgressUpdate {
544              FileDownloadProgressUpdateType update_type   = 1;
545              string                         name          = 2;
546              Vector                         version       = 3;
547              repeated int32                 block_indexes = 4;
548          }
549
550          enum FileDownloadProgressUpdateType {
551              APPEND = 0;
552              FORGET = 1;
553          }
554
555   Fields (DownloadProgress Message)
556       The folder field represents the ID of the folder for which  the  update
557       is being provided.
558
559       The updates field is a list of progress update messages.
560
561   Fields (FileDownloadProgressUpdate Message)
562       The update type indicates whether the update is of type append (0) (new
563       blocks are available) or forget (1) (the file transfer has completed or
564       failed).
565
566       The  name  field  defines the file name from the global index for which
567       this update is being sent.
568
569       The version message defines the version of  the  file  for  which  this
570       update is being sent.
571
572       The  block  indexes  field  is  a list of positive integers, where each
573       integer represents the index of  the  block  in  the  FileInfo  message
574       Blocks array that has become available for download.
575
576       For example an integer with value 3 represents that the data defined in
577       the fourth BlockInfo message of the FileInfo message of  that  file  is
578       now  available.  Please  note  that matching should be done on name AND
579       version. Furthermore, each update received is incremental, for  example
580       the  initial  update message might contain indexes 0, 1, 2, an update 5
581       seconds later might contain indexes 3, 4, 5 which should be appended to
582       the  original  list, which implies that blocks 0-5 are currently avail‐
583       able.
584
585       Block indexes MAY be added in any order.  An  implementation  MUST  NOT
586       assume that block indexes are added in any specific order.
587
588       The  forget  field being set implies that previously advertised file is
589       no longer available, therefore the list  of  block  indexes  should  be
590       truncated.
591
592       Messages with the forget field set MUST NOT have any block indexes.
593
594       Any  update  message which is being sent for a different version of the
595       same file name must be preceded with an update message for the old ver‐
596       sion of that file with the forget field set.
597
598       As  a  safeguard  on  the receiving side, the value of version changing
599       between update messages implies that the file has changed and that  any
600       indexes  previously  advertised  are  no  longer available. The list of
601       available block indexes MUST be replaced (rather  than  appended)  with
602       the indexes specified in this message.
603
604   Ping
605       The  Ping  message is used to determine that a connection is alive, and
606       to keep connections alive through state tracking network elements  such
607       as firewalls and NAT gateways. A Ping message is sent every 90 seconds,
608       if no other message has been sent in the preceding 90 seconds.
609
610   Protocol Buffer Schema
611          message Ping {
612          }
613
614   Close
615       The Close message MAY be sent to indicate that the connection  will  be
616       torn  down  due to an error condition. A Close message MUST NOT be fol‐
617       lowed by further messages.
618
619   Protocol Buffer Schema
620          message Close {
621              string reason = 1;
622          }
623
624   Fields
625       The reason field contains a human readable  description  of  the  error
626       condition.
627

SHARING MODES

629   Trusted
630       Trusted mode is the default sharing mode. Updates are exchanged in both
631       directions.
632
633          +------------+     Updates      /---------\
634          |            |  ----------->   /           \
635          |   Device   |                 |  Cluster  |
636          |            |  <-----------   \           /
637          +------------+     Updates      \---------/
638
639   Send Only
640       In send only mode, a device does not apply any updates from  the  clus‐
641       ter, but publishes changes of its local folder to the cluster as usual.
642
643          +------------+     Updates      /---------\
644          |            |  ----------->   /           \
645          |   Device   |                 |  Cluster  |
646          |            |                 \           /
647          +------------+                  \---------/
648
649   Receive Only
650       In  receive  only mode, a device does not send any updates to the clus‐
651       ter, but accepts changes to its local folder from the cluster as usual.
652
653          +------------+     Updates      /---------\
654          |            |  <-----------   /           \
655          |   Device   |                 |  Cluster  |
656          |            |                 \           /
657          +------------+                  \---------/
658

DELTA INDEX EXCHANGE

660       Index data must be exchanged whenever two devices connect so  that  one
661       knows  the  files  available  on the other. In the most basic case this
662       happens by way of sending an Index message  followed  by  one  or  more
663       Index  Update  messages.  Any  previous  index  data known for a remote
664       device is removed and replaced with the new index data received  in  an
665       Index  message, while the contents of an Index Update message is simply
666       added to the existing index data.
667
668       For situations with large indexes or frequent reconnects  this  can  be
669       quite  inefficient.  A  mechanism can then be used to retain index data
670       between connections and only transmit any changes since  that  data  on
671       connection start. This is called “delta indexes”. To enable this mecha‐
672       nism the sequence and index ID fields are used.
673
674       Sequence:
675              Each index  item  (i.e.,  file,  directory  or  symlink)  has  a
676              sequence number field. It contains the value of a counter at the
677              time the index item was updated. The counter increments  by  one
678              for  each change. That is, as files are scanned and added to the
679              index they get assigned sequence numbers 1, 2, 3 and so on.  The
680              next  file to be changed or detected gets sequence number 4, and
681              future updates continue in the same fashion.
682
683       Index ID:
684              Each folder has an Index ID. This is a 64 bit random  identifier
685              set at index creation time.
686
687       Given  the  above,  we  know that the tuple {index ID, maximum sequence
688       number} uniquely identifies a point in time of a given index. Any  fur‐
689       ther  changes  will increase the sequence number of some item, and thus
690       the maximum sequence number for the index itself. Should the  index  be
691       reset or removed (i.e., the sequence number reset to zero), a new index
692       ID must be generated.
693
694       By letting a device know the {index ID,  maximum  sequence  number}  we
695       have  for  their  index  data, that device can arrange to only transmit
696       Index Update messages for items with a higher sequence number. This  is
697       the delta index mechanism.
698
699       The  index  ID  and  maximum  sequence  number known for each device is
700       transmitted in the Cluster Config message at connection start.
701
702       For this mechanism to be reliable it is essential that  outgoing  index
703       information  is ordered by increasing sequence number. Devices announc‐
704       ing a non-zero index ID in the Cluster Config  message  MUST  send  all
705       index data ordered by increasing sequence number. Devices not intending
706       to participate in delta index exchange MUST send a zero  index  ID  or,
707       equivalently, not send the index_id attribute at all.
708

MESSAGE LIMITS

710       An  implementation  MAY  impose reasonable limits on the length of mes‐
711       sages and message fields to aid robustness in the face of corruption or
712       broken  implementations.  An  implementation should strive to keep mes‐
713       sages short and to the point, favouring more and smaller messages  over
714       fewer and larger.  For example, favour a smaller Index message followed
715       by one or more Index Update messages rather than sending a  very  large
716       Index message.
717
718       The  Syncthing implementation imposes a hard limit of 500,000,000 bytes
719       on all messages. Attempting to send or receive a  larger  message  will
720       result in a connection close. This size was chosen to accommodate Index
721       messages containing a large block list. It’s intended  that  the  limit
722       may  be further reduced in a future protocol update supporting variable
723       block sizes (and thus shorter block lists for large files).
724

SELECTION OF BLOCK SIZE

726       The desired block size for any given file is the  smallest  block  size
727       that  results  in fewer than 2000 blocks, or the maximum block size for
728       larger files. This rule results in the following table of  block  sizes
729       per file size:
730
731                          ┌──────────────────┬────────────┐
732                          │File Size         │ Block Size │
733                          ├──────────────────┼────────────┤
734                          │0 - 250 MiB       │ 128 KiB    │
735                          ├──────────────────┼────────────┤
736                          │250 MiB - 500 MiB │ 256 KiB    │
737                          ├──────────────────┼────────────┤
738                          │500 MiB - 1 GiB   │ 512 KiB    │
739                          ├──────────────────┼────────────┤
740                          │1 GiB - 2 GiB     │ 1 MiB      │
741                          ├──────────────────┼────────────┤
742                          │2 GiB - 4 GiB     │ 2 MiB      │
743                          ├──────────────────┼────────────┤
744                          │4 GiB - 8 GiB     │ 4 MiB      │
745                          ├──────────────────┼────────────┤
746                          │8 GiB - 16 GiB    │ 8 MiB      │
747                          ├──────────────────┼────────────┤
748                          │16 GiB - up       │ 16 MiB     │
749                          └──────────────────┴────────────┘
750
751       An  implementation  MAY  deviate from the block size rule when there is
752       good reason to do so. For example, if a file has been indexed at a cer‐
753       tain  block size and grows beyond 2000 blocks it may be retained at the
754       current block size for practical reasons. When there is  no  overriding
755       reason  to the contrary, such as when indexing a new file for the first
756       time, the block size rule above SHOULD be followed.
757
758       An implementation MUST therefore accept files with a block size differ‐
759       ing  from the above rule. This does not mean that arbitrary block sizes
760       are  allowed.  The  block  size  used  MUST  be  exactly  one  of   the
761       power-of-two block sizes listed in the table above.
762

EXAMPLE EXCHANGE

764                   ┌───┬───────────────────┬─────────────────────┐
765                   │#  │ A                 │ B                   │
766                   ├───┼───────────────────┼─────────────────────┤
767                   │1  │ ClusterConfigura‐ │ <-ClusterConfigura‐ │
768                   │   │ tion->            │ tion                │
769                   ├───┼───────────────────┼─────────────────────┤
770                   │2  │ Index->           │ <-Index             │
771                   ├───┼───────────────────┼─────────────────────┤
772                   │3  │ IndexUpdate->     │ <-IndexUpdate       │
773                   ├───┼───────────────────┼─────────────────────┤
774                   │4  │ IndexUpdate->     │                     │
775                   ├───┼───────────────────┼─────────────────────┤
776                   │5  │ Request->         │                     │
777                   ├───┼───────────────────┼─────────────────────┤
778                   │6  │ Request->         │                     │
779                   ├───┼───────────────────┼─────────────────────┤
780                   │7  │ Request->         │                     │
781                   ├───┼───────────────────┼─────────────────────┤
782                   │8  │ Request->         │                     │
783                   ├───┼───────────────────┼─────────────────────┤
784                   │9  │                   │ <-Response          │
785                   ├───┼───────────────────┼─────────────────────┤
786                   │10 │                   │ <-Response          │
787                   ├───┼───────────────────┼─────────────────────┤
788                   │11 │                   │ <-Response          │
789                   ├───┼───────────────────┼─────────────────────┤
790                   │12 │                   │ <-Response          │
791                   ├───┼───────────────────┼─────────────────────┤
792                   │13 │ Index Update->    │                     │
793                   └───┴───────────────────┴─────────────────────┘
794
795
796                   │…  │                   │                     │
797                   ├───┼───────────────────┼─────────────────────┤
798                   │14 │                   │ <-Ping              │
799                   ├───┼───────────────────┼─────────────────────┤
800                   │15 │ Ping->            │                     │
801                   └───┴───────────────────┴─────────────────────┘
802
803       The connection is established and at 1. both peers send ClusterConfigu‐
804       ration messages and then Index records. The Index records are  received
805       and both peers recompute their knowledge of the data in the cluster. In
806       this example, peer A has four missing or outdated blocks. At 5  through
807       8  peer A sends requests for these blocks. The requests are received by
808       peer B, who retrieves the data from the folder and  transmits  Response
809       records  (9  through  12).  Device  A updates their folder contents and
810       transmits an Index Update message (13). Both  peers  enter  idle  state
811       after 13. At some later time 14, the ping timer on device B expires and
812       a Ping message is sent. The same process occurs for device A at 15.
813

EXAMPLES OF STRONG CIPHER SUITES

815            ┌───────┬─────────────────────────────┬─────────────────────┐
816            │ID     │ Name                        │ Description         │
817            ├───────┼─────────────────────────────┼─────────────────────┤
818            │0x009F │ DHE-RSA-AES256-GCM-SHA384   │ TLSv1.2 DH RSA AES‐ │
819            │       │                             │ GCM(256) AEAD       │
820            ├───────┼─────────────────────────────┼─────────────────────┤
821            │0x006B │ DHE-RSA-AES256-SHA256       │ TLSv1.2   DH    RSA │
822            │       │                             │ AES(256) SHA256     │
823            ├───────┼─────────────────────────────┼─────────────────────┤
824            │0xC030 │ ECDHE-RSA-AES256-GCM-SHA384 │ TLSv1.2  ECDH   RSA │
825            │       │                             │ AESGCM(256) AEAD    │
826            ├───────┼─────────────────────────────┼─────────────────────┤
827            │0xC028 │ ECDHE-RSA-AES256-SHA384     │ TLSv1.2   ECDH  RSA │
828            │       │                             │ AES(256) SHA384     │
829            ├───────┼─────────────────────────────┼─────────────────────┤
830            │0x009E │ DHE-RSA-AES128-GCM-SHA256   │ TLSv1.2 DH RSA AES‐ │
831            │       │                             │ GCM(128) AEAD       │
832            ├───────┼─────────────────────────────┼─────────────────────┤
833            │0x0067 │ DHE-RSA-AES128-SHA256       │ TLSv1.2    DH   RSA │
834            │       │                             │ AES(128) SHA256     │
835            ├───────┼─────────────────────────────┼─────────────────────┤
836            │0xC02F │ ECDHE-RSA-AES128-GCM-SHA256 │ TLSv1.2  ECDH   RSA │
837            │       │                             │ AESGCM(128) AEAD    │
838            ├───────┼─────────────────────────────┼─────────────────────┤
839            │0xC027 │ ECDHE-RSA-AES128-SHA256     │ TLSv1.2   ECDH  RSA │
840            │       │                             │ AES(128) SHA256     │
841            └───────┴─────────────────────────────┴─────────────────────┘
842

AUTHOR

844       The Syncthing Authors
845
847       2014-2019, The Syncthing Authors
848
849
850
851
852v1                               Apr 13, 2019                 SYNCTHING-BEP(7)
Impressum