1DOVEADM-SYNC(1)                     Dovecot                    DOVEADM-SYNC(1)
2
3
4

NAME

6       doveadm-sync - Dovecot's two-way mailbox synchronization utility
7       doveadm-backup - Dovecot's one-way mailbox synchronization utility
8

SYNOPSIS

10       doveadm  [-Dv]  sync [-u user|-A|-F file] [-S socket_path] [-1fPRU] [-l
11       secs] [-r rawlog_path] [-m mailbox] [-g mailbox_guid] [-n namespace|-N]
12       [-x exclude] [-s state] [-T secs] [-t start date] [-e end date] -d|des‐
13       tination
14
15       doveadm [-Dv] backup [-u user|-A|-F file] [-S socket_path] [-fPRU]  [-l
16       secs] [-r rawlog_path] [-m mailbox] [-g mailbox_guid] [-n namespace|-N]
17       [-x exclude] [-s state] [-T secs] [-t start date] [-e end date] -d|des‐
18       tination
19

DESCRIPTION

21       dsync is Dovecot's mailbox synchronization utility.  It can be used for
22       several different use cases: Two-way synchronization of mailboxes, cre‐
23       ating backups of mails, and convert mailboxes from/to different mailbox
24       formats.  All of these can be used within the same  server  or  between
25       different  servers  (via  ssh(1) or tcp connections).  Remote mailboxes
26       can be accessed also via IMAP protocol, which allows  using  dsync  for
27       mailbox migration purposes.
28
29       You can run dsync in one of three modes:
30
31              ·      doveadm  backup  performs  one-way  synchronization.   If
32                     there are any changes in the  destination  they  will  be
33                     deleted,  so  the  destination will look exactly like the
34                     source.
35
36              ·      doveadm sync performs two-way synchronization.  It merges
37                     all  changes without losing anything.  Both the mailboxes
38                     will end up looking identical after  the  synchronization
39                     is finished.
40
41              ·      doveadm  sync -1 performs one-way synchronization, but it
42                     merges the changes in destination without  deleting  any‐
43                     thing.  This doesn't currently work perfectly, so its use
44                     should be limited.  Its main purpose is that during mail‐
45                     box  migration you can run doveadm backup multiple times,
46                     then switch mails to be delivered to the new mailbox  and
47                     run  doveadm  sync  -1 once more to transfer any last new
48                     mails from the old mailbox.
49
50                     The one-way algorithm is the same as two-way dsync  algo‐
51                     rithm  except  the  source  account  is  not modified. It
52                     fetches the message's GUID (Global UID), which is used to
53                     identify any conflicting UIDs in messages. As long as the
54                     source and destination side has matching UID<->GUID  map‐
55                     ping,  those  emails  are assumed to be synced correctly.
56                     Only after the first mismatch will changes begin.
57
58                     Example: Source mailbox has  messages  UID  1..5;  source
59                     mailbox  is  sync'd  using doveadm backup to the destina‐
60                     tion. Subsequently, UID 6  is  delivered  to  the  source
61                     mailbox  and UID 1 is expunged from the destination mail‐
62                     box. In this example, UID 1 is kept removed (in  destina‐
63                     tion)  because  UID  1..5  have identical Date+Message-ID
64                     headers. UID 6 is not seen in destination so it's copied.
65
66                     If both source and destination have UID 6, but  the  mes‐
67                     sages are different, the headers don't match and both the
68                     messages are kept in the destination  but  they're  given
69                     new  UIDs  7  and 8 just to be sure any client didn't get
70                     confused about what UID 11 actually  was.  Thus,  one-way
71                     sync  begins  to  quickly diverge from the source mailbox
72                     once changes start to occur on either side; one-way  sync
73                     should  therefore  normally  only  be used within a short
74                     period of time after a doveadm  backup  or  doveadm  sync
75                     command was used to synchronize the mailboxes.
76
77       There are also three different synchronization algorithms:
78
79              ·      Full synchronization (-f parameter) scans through all the
80                     messages in all  the  mailboxes.   This  guarantees  that
81                     everything  will  be synchronized, but it's unnecessarily
82                     slow for incremental synchronization.
83
84              ·      Fast synchronization (default)  first  attempts  to  find
85                     mailboxes  that have changed, and synchronize only those.
86                     This is done by checking the mailboxes' metadata (NEXTUID
87                     and  HIGHESTMODSEQ).  Usually this works fine, especially
88                     with  one-way  synchronization,  but  if  both  sides  do
89                     exactly  the same number of changes, the metadata may end
90                     up containing the same values even if  the  changes  were
91                     different.
92
93              ·      Stateful synchronization (-s parameter) is the most effi‐
94                     cient way to synchronize mailboxes.  It relies on  having
95                     the  earlier  dsync run's state saved somewhere and being
96                     passed to the next dsync run.  Based on this state  dsync
97                     can  send only the changes that happened after the previ‐
98                     ous dsync run.  As long as the  state  or  the  mailboxes
99                     aren't  corrupted  this  algorithm should work perfectly.
100                     The replicator process uses this  internally  to  perform
101                     most of the synchronization.
102
103       The  syncing is done as perfectly as possible: an IMAP or a POP3 client
104       shouldn't be able to notice any differences between the two  mailboxes.
105       Two-way syncing means that it's safe to do any kind of modifications in
106       both sides, and dsync will merge the changes without losing any changes
107       done  on  either side.  This is possible because dsync can access Dove‐
108       cot's index logs that keep track of changes.  It's of  course  possible
109       to  have  conflicts  during  merging, these are resolved in a safe way.
110       See the dsync design document for more information.
111
112       dsync uses the same configuration files as the  rest  of  Dovecot  (via
113       doveconf(1) binary).  The entire configuration can be changed by giving
114       -c parameter to another configuration file, or using  -o  parameter  to
115       override  specific  settings.  When executing a remote dsync program it
116       works the same way: it uses its own local configuration.
117
118       dsync can be run completely standalone.  It doesn't require any Dovecot
119       server  processes to be running, except when using -u parameter to do a
120       userdb lookup from auth process.
121
122       dsync can sync either one or multiple users using the -u or -A  parame‐
123       ters.   For  continuous  replication you can use the Dovecot replicator
124       process, which automatically runs dsync whenever messages have changed.
125

OPTIONS

127       Global doveadm(1) options:
128
129       -D     Enables verbosity and debug messages.
130
131       -o setting=value
132              Overrides  the  configuration  setting  from  /etc/dovecot/dove‐
133              cot.conf  and from the userdb with the given value.  In order to
134              override multiple settings, the -o option may be specified  mul‐
135              tiple times.
136
137       -v     Enables verbosity, including progress counter.
138
139       Command specific options:
140
141       -1     Do one-way synchronization instead of two-way synchronization.
142
143       -A     If  the  -A option is present, the command will be performed for
144              all users.  Using this option in combination with  system  users
145              from  userdb  { driver = passwd } is not recommended, because it
146              contains also users with a lower UID  than  the  one  configured
147              with the first_valid_uid setting.
148
149              When  the  SQL  userdb  module  is used make sure that the iter‐
150              ate_query setting in  /etc/dovecot/dovecot-sql.conf.ext  matches
151              your  database  layout.  When using the LDAP userdb module, make
152              sure that  the  iterate_attrs  and  iterate_filter  settings  in
153              /etc/dovecot/dovecot-ldap.conf.ext match your LDAP schema.  Oth‐
154              erwise doveadm(1) will be unable to iterate over all users.
155
156       -F file
157              Execute the command for all the users in the file.  This is sim‐
158              ilar  to the -A option, but instead of getting the list of users
159              from the userdb, they are read from the given  file.   The  file
160              contains one username per line.
161
162       -N     Synchronize  all  the  available  namespaces.   By  default only
163              namespaces that don't have explicit location  setting  are  syn‐
164              chronized.
165
166       -P     Run  a  doveadm-purge(1)  for  the  destination (remote) storage
167              after synchronization.
168
169       -R     Do a reverse sync. Normally, messages would be pushed  from  the
170              local  system  to the destination (remote). This option reverses
171              the flow, and will instead pull messages from the remote to  the
172              local storage.
173
174       -S socket_path
175              The option's argument is either an absolute path to a local UNIX
176              domain socket, or a hostname and port (hostname:port), in  order
177              to connect a remote host via a TCP socket.
178
179              This allows an administrator to execute doveadm(1) mail commands
180              through the given socket.
181
182       -T secs
183              Specify the time in seconds, how long doveadm(1) should wait for
184              stalled I/O operations.  The default timeout is 600 seconds.
185
186       -U     This  is  used  internally by replicator to have dsync notify it
187              when the synchronization is finished.
188
189       -d     Use the  default  destination,  which  is  looked  up  from  the
190              mail_replica userdb extra field.
191
192       -g mailbox_guid
193              Same  as -m, but find the mailbox to be synchronized by its GUID
194              instead of by name.
195
196       -l secs
197              Lock the dsync for this user.  Wait for maximum secs before giv‐
198              ing  up.  This parameter should be used to avoid broken synchro‐
199              nization if it's possible that dsync is being  run  concurrently
200              for the same user.
201
202       -m mailbox
203              Synchronize only this mailbox name.
204
205       -n namespace
206              Synchronize only the specified namespace.  This parameter can be
207              used multiple times.
208
209       -r rawlog_path
210              Running dsync remotely, write the remote input/output traffic to
211              the specified log file.
212
213       -s previous_state
214              Use stateful synchronization.  If the previous state is unknown,
215              use an empty string.  The new state is always printed  to  stan‐
216              dard output.
217
218       -u user/mask
219              Run  the command only for the given user.  It's also possible to
220              use '*' and '?' wildcards (e.g. -u *@example.org).
221              When neither the -A option, nor  the  -F file  option,  nor  the
222              -u user  was  specified,  the  command will be executed with the
223              environment of the currently logged in user.
224
225       -x mailbox_mask
226              Exclude the specified mailbox name/mask.  The mask  may  contain
227              "?"  and  "*"  wildcards.   This  parameter can be used multiple
228              times.
229

ARGUMENTS

231       destination
232              This argument specifies the synchronized destination.  It can be
233              one of:
234
235              location
236                     Same as mail_location setting, e.g. maildir:~/Maildir
237
238              remote:login@host
239                     Uses  dsync_remote_cmd  setting  to connect to the remote
240                     host (usually via ssh)
241
242              remoteprefix:login@host
243                     This is the same as  remote,  except  "user@domain\n"  is
244                     sent  before  dsync  protocol starts.  This allows imple‐
245                     menting  a  trusted  wrapper  script  that  runs  doveadm
246                     dsync-server by reading the username from the first line.
247
248              tcp:host[:port]
249                     Connects  to  remote doveadm server via TCP.  The default
250                     port is specified by doveadm_port setting.
251
252              tcps:host[:port]
253                     This is the same as tcp, but with SSL.
254

EXIT STATUS

256       dsync will exit with one of the following values:
257
258       0   Synchronization was done perfectly.
259
260       2   Synchronization was done without errors, but some changes  couldn't
261           be  done,  so  the mailboxes aren't perfectly synchronized. Running
262           dsync again usually fixes this. Typically this occurs  for  message
263           modification  sequences  with  newly created mailboxes. It can also
264           occur if one of the mailboxes change during the syncing.
265
266       1, >2
267           Synchronization failed.
268

EXAMPLE

270   SYNCHRONIZATION
271       Synchronize mailboxes with a remote server.  Any errors are written  to
272       stderr.
273
274              doveadm sync -u username@example.com remote:server-replica.example.com
275
276       If you need more complex parameters to ssh, you can use e.g.:
277
278              doveadm sync -u username@example.com ssh -i id_dsa.dovecot \
279              mailuser@example.com doveadm dsync-server -u username@example.com
280
281   CONVERTING
282       Assuming    that    the    mail_location    setting    in    /etc/dove‐
283       cot/conf.d/10-mail.conf is set to:  mail_location  =  mdbox:~/mdbox,  a
284       logged  in  system  user  may convert her/his mails from its Maildir in
285       her/his home directory to the mdbox mailbox format.  The  user  has  to
286       execute the command:
287
288              doveadm sync maildir:~/Maildir
289
290       If  you want to do this without any downtime, you can do the conversion
291       one user at a time.  Initially:
292
293           ·   Configuration uses mail_location = maildir:~/Maildir
294
295           ·   Set up the possibility of doing per-user  mail  location  using
296               userdb extra fields.
297
298       Then for each user:
299
300           1.  Run doveadm sync once to do the initial conversion.
301
302           2.  Run  doveadm  sync  again, because the initial conversion could
303               have taken a while and new changes could have  occurred  during
304               it.   This  second  time  only applies changes, so it should be
305               fast.
306
307           3.  Update mail extra field in userdb to mdbox:~/mdbox.  If  you're
308               using  auth  cache,  you  need  to flush it, e.g.  doveadm auth
309               cache flush.
310
311           4.  Wait for a few seconds and then kill (doveadm kick) the  user's
312               all  existing  imap  and  pop3  sessions  (that are still using
313               maildir).
314
315           5.  Run doveadm sync once more to apply  final  changes  that  were
316               possibly  done.   After  this  there  should  be  no changes to
317               Maildir, because the user's mail location has been changed  and
318               all existing processes using it have been killed.
319
320       Once  all users have been converted, you can set the default mail_loca‐
321       tion to mdbox and remove the per-user mail locations from userdb.
322

REPORTING BUGS

324       Report bugs, including doveconf -n output, to the Dovecot Mailing  List
325       <dovecot@dovecot.org>.   Information  about reporting bugs is available
326       at: http://dovecot.org/bugreport.html
327

SEE ALSO

329       doveadm(1), doveadm-auth(1), doveadm-kick(1),  doveadm-purge(1),  dove‐
330       conf(1)
331
332       Additional resources:
333
334       dsync design
335              http://wiki2.dovecot.org/Design/Dsync
336
337
338
339Dovecot v2.3                      2015-08-31                   DOVEADM-SYNC(1)
Impressum