1DOVEADM-SYNC(1) Dovecot DOVEADM-SYNC(1)
2
6 doveadm-sync - Dovecot's two-way mailbox synchronization utility
7 doveadm-backup - Dovecot's one-way mailbox synchronization utility
8
10 doveadm [-Dv] sync [-u user|-A|-F file] [-S socket_path] [-1fPRU] [-l
11 secs] [-r rawlog_path] [-m mailbox] [-g mailbox_guid] [-n namespace|-N]
12 [-x exclude] [-s state] [-T secs] [-t start date] [-e end date] -d|des‐
13 tination
14 doveadm [-Dv] backup [-u user|-A|-F file] [-S socket_path] [-fPRU] [-l
16 secs] [-r rawlog_path] [-m mailbox] [-g mailbox_guid] [-n namespace|-N]
17 [-x exclude] [-s state] [-T secs] [-t start date] [-e end date] -d|des‐
18 tination
19
21 dsync is Dovecot's mailbox synchronization utility. It can be used for
22 several different use cases: Two-way synchronization of mailboxes, cre‐
23 ating backups of mails, and convert mailboxes from/to different mailbox
24 formats. All of these can be used within the same server or between
25 different servers (via ssh(1) or tcp connections). Remote mailboxes
26 can be accessed also via IMAP protocol, which allows using dsync for
27 mailbox migration purposes.
28 You can run dsync in one of three modes:
30 · doveadm backup performs one-way synchronization. If
32 there are any changes in the destination they will be
33 deleted, so the destination will look exactly like the
34 source.
35 · doveadm sync performs two-way synchronization. It merges
37 all changes without losing anything. Both the mailboxes
38 will end up looking identical after the synchronization
39 is finished.
40 · doveadm sync -1 performs one-way synchronization, but it
42 merges the changes in destination without deleting any‐
43 thing. This doesn't currently work perfectly, so its use
44 should be limited. Its main purpose is that during mail‐
45 box migration you can run doveadm backup multiple times,
46 then switch mails to be delivered to the new mailbox and
47 run doveadm sync -1 once more to transfer any last new
48 mails from the old mailbox.
49 The one-way algorithm is the same as two-way dsync algo‐
51 rithm except the source account is not modified. It
52 fetches the message's GUID (Global UID), which is used to
53 identify any conflicting UIDs in messages. As long as the
54 source and destination side has matching UID<->GUID map‐
55 ping, those emails are assumed to be synced correctly.
56 Only after the first mismatch will changes begin.
57 Example: Source mailbox has messages UID 1..5; source
59 mailbox is sync'd using doveadm backup to the destina‐
60 tion. Subsequently, UID 6 is delivered to the source
61 mailbox and UID 1 is expunged from the destination mail‐
62 box. In this example, UID 1 is kept removed (in destina‐
63 tion) because UID 1..5 have identical Date+Message-ID
64 headers. UID 6 is not seen in destination so it's copied.
65 If both source and destination have UID 6, but the mes‐
67 sages are different, the headers don't match and both the
68 messages are kept in the destination but they're given
69 new UIDs 7 and 8 just to be sure any client didn't get
70 confused about what UID 11 actually was. Thus, one-way
71 sync begins to quickly diverge from the source mailbox
72 once changes start to occur on either side; one-way sync
73 should therefore normally only be used within a short
74 period of time after a doveadm backup or doveadm sync
75 command was used to synchronize the mailboxes.
76 There are also three different synchronization algorithms:
78 · Full synchronization (-f parameter) scans through all the
80 messages in all the mailboxes. This guarantees that
81 everything will be synchronized, but it's unnecessarily
82 slow for incremental synchronization.
83 · Fast synchronization (default) first attempts to find
85 mailboxes that have changed, and synchronize only those.
86 This is done by checking the mailboxes' metadata (NEXTUID
87 and HIGHESTMODSEQ). Usually this works fine, especially
88 with one-way synchronization, but if both sides do
89 exactly the same number of changes, the metadata may end
90 up containing the same values even if the changes were
91 different.
92 · Stateful synchronization (-s parameter) is the most effi‐
94 cient way to synchronize mailboxes. It relies on having
95 the earlier dsync run's state saved somewhere and being
96 passed to the next dsync run. Based on this state dsync
97 can send only the changes that happened after the previ‐
98 ous dsync run. As long as the state or the mailboxes
99 aren't corrupted this algorithm should work perfectly.
100 The replicator process uses this internally to perform
101 most of the synchronization.
102 The syncing is done as perfectly as possible: an IMAP or a POP3 client
104 shouldn't be able to notice any differences between the two mailboxes.
105 Two-way syncing means that it's safe to do any kind of modifications in
106 both sides, and dsync will merge the changes without losing any changes
107 done on either side. This is possible because dsync can access Dove‐
108 cot's index logs that keep track of changes. It's of course possible
109 to have conflicts during merging, these are resolved in a safe way.
110 See the dsync design document for more information.
111 dsync uses the same configuration files as the rest of Dovecot (via
113 doveconf(1) binary). The entire configuration can be changed by giving
114 -c parameter to another configuration file, or using -o parameter to
115 override specific settings. When executing a remote dsync program it
116 works the same way: it uses its own local configuration.
117 dsync can be run completely standalone. It doesn't require any Dovecot
119 server processes to be running, except when using -u parameter to do a
120 userdb lookup from auth process.
121 dsync can sync either one or multiple users using the -u or -A parame‐
123 ters. For continuous replication you can use the Dovecot replicator
124 process, which automatically runs dsync whenever messages have changed.
125
127 Global doveadm(1) options:
128 -D Enables verbosity and debug messages.
130 -o setting=value
132 Overrides the configuration setting from /etc/dovecot/dove‐
133 cot.conf and from the userdb with the given value. In order to
134 override multiple settings, the -o option may be specified mul‐
135 tiple times.
136 -v Enables verbosity, including progress counter.
138 Command specific options:
140 -1 Do one-way synchronization instead of two-way synchronization.
142 -A If the -A option is present, the command will be performed for
144 all users. Using this option in combination with system users
145 from userdb { driver = passwd } is not recommended, because it
146 contains also users with a lower UID than the one configured
147 with the first_valid_uid setting.
148 When the SQL userdb module is used make sure that the iter‐
150 ate_query setting in /etc/dovecot/dovecot-sql.conf.ext matches
151 your database layout. When using the LDAP userdb module, make
152 sure that the iterate_attrs and iterate_filter settings in
153 /etc/dovecot/dovecot-ldap.conf.ext match your LDAP schema. Oth‐
154 erwise doveadm(1) will be unable to iterate over all users.
155 -F file
157 Execute the command for all the users in the file. This is sim‐
158 ilar to the -A option, but instead of getting the list of users
159 from the userdb, they are read from the given file. The file
160 contains one username per line.
161 -N Synchronize all the available namespaces. By default only
163 namespaces that don't have explicit location setting are syn‐
164 chronized.
165 -P Run a doveadm-purge(1) for the destination (remote) storage
167 after synchronization.
168 -R Do a reverse sync. Normally, messages would be pushed from the
170 local system to the destination (remote). This option reverses
171 the flow, and will instead pull messages from the remote to the
172 local storage.
173 -S socket_path
175 The option's argument is either an absolute path to a local UNIX
176 domain socket, or a hostname and port (hostname:port), in order
177 to connect a remote host via a TCP socket.
178 This allows an administrator to execute doveadm(1) mail commands
180 through the given socket.
181 -T secs
183 Specify the time in seconds, how long doveadm(1) should wait for
184 stalled I/O operations. The default timeout is 600 seconds.
185 -U This is used internally by replicator to have dsync notify it
187 when the synchronization is finished.
188 -d Use the default destination, which is looked up from the
190 mail_replica userdb extra field.
191 -g mailbox_guid
193 Same as -m, but find the mailbox to be synchronized by its GUID
194 instead of by name.
195 -l secs
197 Lock the dsync for this user. Wait for maximum secs before giv‐
198 ing up. This parameter should be used to avoid broken synchro‐
199 nization if it's possible that dsync is being run concurrently
200 for the same user.
201 -m mailbox
203 Synchronize only this mailbox name.
204 -n namespace
206 Synchronize only the specified namespace. This parameter can be
207 used multiple times.
208 -r rawlog_path
210 Running dsync remotely, write the remote input/output traffic to
211 the specified log file.
212 -s previous_state
214 Use stateful synchronization. If the previous state is unknown,
215 use an empty string. The new state is always printed to stan‐
216 dard output.
217 -u user/mask
219 Run the command only for the given user. It's also possible to
220 use '*' and '?' wildcards (e.g. -u *@example.org).
221 When neither the -A option, nor the -F file option, nor the
222 -u user was specified, the command will be executed with the
223 environment of the currently logged in user.
224 -x mailbox_mask
226 Exclude the specified mailbox name/mask. The mask may contain
227 "?" and "*" wildcards. This parameter can be used multiple
228 times.
229
231 destination
232 This argument specifies the synchronized destination. It can be
233 one of:
234 location
236 Same as mail_location setting, e.g. maildir:~/Maildir
237 remote:login@host
239 Uses dsync_remote_cmd setting to connect to the remote
240 host (usually via ssh)
241 remoteprefix:login@host
243 This is the same as remote, except "user@domain\n" is
244 sent before dsync protocol starts. This allows imple‐
245 menting a trusted wrapper script that runs doveadm
246 dsync-server by reading the username from the first line.
247 tcp:host[:port]
249 Connects to remote doveadm server via TCP. The default
250 port is specified by doveadm_port setting.
251 tcps:host[:port]
253 This is the same as tcp, but with SSL.
254
256 dsync will exit with one of the following values:
257 0 Synchronization was done perfectly.
259 2 Synchronization was done without errors, but some changes couldn't
261 be done, so the mailboxes aren't perfectly synchronized. Running
262 dsync again usually fixes this. Typically this occurs for message
263 modification sequences with newly created mailboxes. It can also
264 occur if one of the mailboxes change during the syncing.
265 1, >2
267 Synchronization failed.
268
270 SYNCHRONIZATION
271 Synchronize mailboxes with a remote server. Any errors are written to
272 stderr.
273 doveadm sync -u username@example.com remote:server-replica.example.com
275 If you need more complex parameters to ssh, you can use e.g.:
277 doveadm sync -u username@example.com ssh -i id_dsa.dovecot \
279 mailuser@example.com doveadm dsync-server -u username@example.com
280 CONVERTING
282 Assuming that the mail_location setting in /etc/dove‐
283 cot/conf.d/10-mail.conf is set to: mail_location = mdbox:~/mdbox, a
284 logged in system user may convert her/his mails from its Maildir in
285 her/his home directory to the mdbox mailbox format. The user has to
286 execute the command:
287 doveadm sync maildir:~/Maildir
289 If you want to do this without any downtime, you can do the conversion
291 one user at a time. Initially:
292 · Configuration uses mail_location = maildir:~/Maildir
294 · Set up the possibility of doing per-user mail location using
296 userdb extra fields.
297 Then for each user:
299 1. Run doveadm sync once to do the initial conversion.
301 2. Run doveadm sync again, because the initial conversion could
303 have taken a while and new changes could have occurred during
304 it. This second time only applies changes, so it should be
305 fast.
306 3. Update mail extra field in userdb to mdbox:~/mdbox. If you're
308 using auth cache, you need to flush it, e.g. doveadm auth
309 cache flush.
310 4. Wait for a few seconds and then kill (doveadm kick) the user's
312 all existing imap and pop3 sessions (that are still using
313 maildir).
314 5. Run doveadm sync once more to apply final changes that were
316 possibly done. After this there should be no changes to
317 Maildir, because the user's mail location has been changed and
318 all existing processes using it have been killed.
319 Once all users have been converted, you can set the default mail_loca‐
321 tion to mdbox and remove the per-user mail locations from userdb.
322
324 Report bugs, including doveconf -n output, to the Dovecot Mailing List
325 <dovecot@dovecot.org>. Information about reporting bugs is available
326 at: http://dovecot.org/bugreport.html
327
329 doveadm(1), doveadm-auth(1), doveadm-kick(1), doveadm-purge(1), dove‐
330 conf(1)
331 Additional resources:
333 dsync design
335 http://wiki2.dovecot.org/Design/Dsync
336Dovecot v2.3 2015-08-31 DOVEADM-SYNC(1)