1DOVEADM-SYNC(1) Dovecot DOVEADM-SYNC(1)
2
6 doveadm-sync - Dovecot's two-way mailbox synchronization utility
7 doveadm-backup - Dovecot's one-way mailbox synchronization utility
8
10 doveadm [-Dv] sync [-u user|-A|-F file] [-S socket_path] [-1fPRU] [-l
11 secs] [-r rawlog path] [-m mailbox] [-g mailbox guid] [-n namespace|-N]
12 [-x exclude] [-a all mailbox] [-s state] [-T secs] [-t start date] [-e
13 end date] [-O sync flag] [-I max size] -d|destination
14 doveadm [-Dv] backup [-u user|-A|-F file] [-S socket_path] [-fPRU] [-l
16 secs] [-r rawlog path] [-m mailbox] [-g mailbox guid] [-n namespace|-N]
17 [-x exclude] [-a all mailbox] [-s state] [-T secs] [-t start date] [-e
18 end date] [-O sync flag] [-I max size] -d|destination
19
21 dsync is Dovecot's mailbox synchronization utility. It can be used for
22 several different use cases: Two-way synchronization of mailboxes, cre‐
23 ating backups of mails, and convert mailboxes from/to different mailbox
24 formats. All of these can be used within the same server or between
25 different servers (via ssh(1) or tcp connections). Remote mailboxes
26 can be accessed also via IMAP protocol, which allows using dsync for
27 mailbox migration purposes.
28 You can run dsync in one of three modes:
30 • doveadm backup performs one-way synchronization. If
32 there are any changes in the destination they will be
33 deleted, so the destination will look exactly like the
34 source.
35 • doveadm sync performs two-way synchronization. It merges
37 all changes without losing anything. Both the mailboxes
38 will end up looking identical after the synchronization
39 is finished.
40 • doveadm sync -1 performs one-way synchronization, but it
42 merges the changes in destination without deleting any‐
43 thing. This doesn't currently work perfectly, so its use
44 should be limited. Its main purpose is that during mail‐
45 box migration you can run doveadm backup multiple times,
46 then switch mails to be delivered to the new mailbox and
47 run doveadm sync -1 once more to transfer any last new
48 mails from the old mailbox.
49 The one-way algorithm is the same as two-way dsync algo‐
51 rithm except the source account is not modified. It
52 fetches the message's GUID (Global UID), which is used to
53 identify any conflicting UIDs in messages. As long as the
54 source and destination side has matching UID<->GUID map‐
55 ping, those emails are assumed to be synced correctly.
56 Only after the first mismatch will changes begin.
57 Example: Source mailbox has messages UID 1..5; source
59 mailbox is sync'd using doveadm backup to the destina‐
60 tion. Subsequently, UID 6 is delivered to the source
61 mailbox and UID 1 is expunged from the destination mail‐
62 box. In this example, UID 1 is kept removed (in destina‐
63 tion) because UID 1..5 have identical Date+Message-ID
64 headers. UID 6 is not seen in destination so it's copied.
65 If both source and destination have UID 6, but the mes‐
67 sages are different, the headers don't match and both the
68 messages are kept in the destination but they're given
69 new UIDs 7 and 8 just to be sure any client didn't get
70 confused about what UID 6 actually was. Thus, one-way
71 sync begins to quickly diverge from the source mailbox
72 once changes start to occur on either side; one-way sync
73 should therefore normally only be used within a short pe‐
74 riod of time after a doveadm backup or doveadm sync com‐
75 mand was used to synchronize the mailboxes.
76 There are also three different synchronization algorithms:
78 • Full synchronization (-f parameter) scans through all the
80 messages in all the mailboxes. This guarantees that ev‐
81 erything will be synchronized, but it's unnecessarily
82 slow for incremental synchronization.
83 • Fast synchronization (default) first attempts to find
85 mailboxes that have changed, and synchronize only those.
86 This is done by checking the mailboxes' metadata (NEXTUID
87 and HIGHESTMODSEQ). Usually this works fine, especially
88 with one-way synchronization, but if both sides do ex‐
89 actly the same number of changes, the metadata may end up
90 containing the same values even if the changes were dif‐
91 ferent.
92 • Stateful synchronization (-s parameter) is the most effi‐
94 cient way to synchronize mailboxes. It relies on having
95 the earlier dsync run's state saved somewhere and being
96 passed to the next dsync run. Based on this state dsync
97 can send only the changes that happened after the previ‐
98 ous dsync run. As long as the state or the mailboxes
99 aren't corrupted this algorithm should work perfectly.
100 The replicator process uses this internally to perform
101 most of the synchronization.
102 The syncing is done as perfectly as possible: an IMAP or a POP3 client
104 shouldn't be able to notice any differences between the two mailboxes.
105 Two-way syncing means that it's safe to do any kind of modifications in
106 both sides, and dsync will merge the changes without losing any changes
107 done on either side. This is possible because dsync can access Dove‐
108 cot's index logs that keep track of changes. It's of course possible
109 to have conflicts during merging, these are resolved in a safe way.
110 See the dsync design document for more information.
111 dsync uses the same configuration files as the rest of Dovecot (via
113 doveconf(1) binary). The entire configuration can be changed by giving
114 -c parameter to another configuration file, or using -o parameter to
115 override specific settings. When executing a remote dsync program it
116 works the same way: it uses its own local configuration.
117 dsync can be run completely standalone. It doesn't require any Dovecot
119 server processes to be running, except when using -u parameter to do a
120 userdb lookup from auth process.
121 dsync can sync either one or multiple users using the -u or -A parame‐
123 ters. For continuous replication you can use the Dovecot replicator
124 process, which automatically runs dsync whenever messages have changed.
125
127 Global doveadm(1) options:
128 -D Enables verbosity and debug messages.
130 -o setting=value
132 Overrides the configuration setting from /etc/dovecot/dove‐
133 cot.conf and from the userdb with the given value. In order to
134 override multiple settings, the -o option may be specified mul‐
135 tiple times.
136 -v Enables verbosity, including progress counter.
138 Command specific options:
140 -A If the -A option is present, the command will be performed for
142 all users. Using this option in combination with system users
143 from userdb { driver = passwd } is not recommended, because it
144 contains also users with a lower UID than the one configured
145 with the first_valid_uid setting.
146 When the SQL userdb module is used make sure that the iter‐
148 ate_query setting in /etc/dovecot/dovecot-sql.conf.ext matches
149 your database layout. When using the LDAP userdb module, make
150 sure that the iterate_attrs and iterate_filter settings in
151 /etc/dovecot/dovecot-ldap.conf.ext match your LDAP schema. Oth‐
152 erwise doveadm(1) will be unable to iterate over all users.
153 -F file
155 Execute the command for all the users in the file. This is sim‐
156 ilar to the -A option, but instead of getting the list of users
157 from the userdb, they are read from the given file. The file
158 contains one username per line.
159 -1 Do one-way synchronization instead of two-way synchronization.
161 -f Do full synchronization.
163 -N Synchronize all the available namespaces. By default only name‐
165 spaces that don't have explicit location setting are synchro‐
166 nized.
167 -P Run a doveadm-purge(1) for the destination (remote) storage af‐
169 ter synchronization.
170 -R Do a reverse sync. Normally, messages would be pushed from the
172 local system to the destination (remote). This option reverses
173 the flow, and will instead pull messages from the remote to the
174 local storage.
175 -S socket_path
177 The option's argument is either an absolute path to a local UNIX
178 domain socket, or a hostname and port (hostname:port), in order
179 to connect a remote host via a TCP socket.
180 This allows an administrator to execute doveadm(1) mail commands
182 through the given socket.
183 -T secs
185 Specify the time in seconds, how long doveadm(1) should wait for
186 stalled I/O operations. The default timeout is 600 seconds.
187 -U This is used internally by replicator to have dsync notify it
189 when the synchronization is finished.
190 -d Use the default destination, which is looked up from the
192 mail_replica userdb extra field.
193 -g mailbox_guid
195 Same as -m, but find the mailbox to be synchronized by its GUID
196 instead of by name.
197 -l secs
199 Lock the dsync for this user. Wait for maximum secs before giv‐
200 ing up. This parameter should be used to avoid broken synchro‐
201 nization if it's possible that dsync is being run concurrently
202 for the same user.
203 -m mailbox
205 Synchronize only this mailbox name.
206 -n namespace
208 Synchronize only the specified namespace. This parameter can be
209 used multiple times.
210 -a all mailbox
212 Name for the "All mails" virtual mailbox. If specified, mails
213 are attempted to be copied from this mailbox instead of being
214 saved separately. This may reduce the total disk space usage as
215 well as disk IO.
216 -t start date
218 Skip any mails whose received-timestamp is older than the speci‐
219 fied time.
220 -e end date
222 Skip any mails whose received-timestamp is newer than the speci‐
223 fied time.
224 -O sync flag
226 Sync only mails that have the specified flag. If the flag name
227 begins with "-", sync all mails except the ones with the speci‐
228 fied flag.
229 -I max size
231 Skip any mails larger than the specified size.
232 -r rawlog_path
234 Running dsync remotely, write the remote input/output traffic to
235 the specified log file.
236 -s previous_state
238 Use stateful synchronization. If the previous state is unknown,
239 use an empty string. The new state is always printed to stan‐
240 dard output.
241 -u user/mask
243 Run the command only for the given user. It's also possible to
244 use '*' and '?' wildcards (e.g. -u *@example.org).
245 When neither the -A option, nor the -F file option, nor the
246 -u user was specified, the command will be executed with the en‐
247 vironment of the currently logged in user.
248 -x mailbox_mask
250 Exclude the specified mailbox name/mask. The mask may contain
251 "?" and "*" wildcards. The mask can also be a special-use name
252 (e.g. \Trash). This parameter can be used multiple times.
253
255 destination
256 This argument specifies the synchronized destination. It can be
257 one of:
258 location
260 Same as mail_location setting, e.g. maildir:~/Maildir
261 remote:login@host
263 Uses dsync_remote_cmd setting to connect to the remote
264 host (usually via ssh)
265 remoteprefix:login@host
267 This is the same as remote, except "user@domain\n" is
268 sent before dsync protocol starts. This allows imple‐
269 menting a trusted wrapper script that runs doveadm
270 dsync-server by reading the username from the first line.
271 tcp:host[:port]
273 Connects to remote doveadm server via TCP. The default
274 port is specified by doveadm_port setting.
275 tcps:host[:port]
277 This is the same as tcp, but with SSL.
278 command [arg1 [, arg2, ...]]
280 Runs a local command that connects its standard input &
281 output to a dsync server.
282
284 dsync will exit with one of the following values:
285 0 Synchronization was done perfectly.
287 2 Synchronization was done without errors, but some changes couldn't
289 be done, so the mailboxes aren't perfectly synchronized. Running
290 dsync again usually fixes this. Typically this occurs for message
291 modification sequences with newly created mailboxes. It can also
292 occur if one of the mailboxes change during the syncing.
293 1, >2
295 Synchronization failed.
296
298 SYNCHRONIZATION
299 Synchronize mailboxes with a remote server. Any errors are written to
300 stderr.
301 doveadm sync -u username@example.com remote:server-replica.example.com
303 If you need more complex parameters to ssh, you can use e.g.:
305 doveadm sync -u username@example.com ssh -i id_dsa.dovecot \
307 mailuser@example.com doveadm dsync-server -u username@example.com
308 CONVERTING
310 Assuming that the mail_location setting in /etc/dove‐
311 cot/conf.d/10-mail.conf is set to: mail_location = mdbox:~/mdbox, a
312 logged in system user may convert her/his mails from its Maildir in
313 her/his home directory to the mdbox mailbox format. The user has to
314 execute the command:
315 doveadm sync maildir:~/Maildir
317 If you want to do this without any downtime, you can do the conversion
319 one user at a time. Initially:
320 • Configuration uses mail_location = maildir:~/Maildir
322 • Set up the possibility of doing per-user mail location using
324 userdb extra fields.
325 Then for each user:
327 1. Run doveadm sync once to do the initial conversion.
329 2. Run doveadm sync again, because the initial conversion could
331 have taken a while and new changes could have occurred during
332 it. This second time only applies changes, so it should be
333 fast.
334 3. Update mail extra field in userdb to mdbox:~/mdbox. If you're
336 using auth cache, you need to flush it, e.g. doveadm auth
337 cache flush.
338 4. Wait for a few seconds and then kill (doveadm kick) the user's
340 all existing imap and pop3 sessions (that are still using
341 maildir).
342 5. Run doveadm sync once more to apply final changes that were
344 possibly done. After this there should be no changes to
345 Maildir, because the user's mail location has been changed and
346 all existing processes using it have been killed.
347 Once all users have been converted, you can set the default mail_loca‐
349 tion to mdbox and remove the per-user mail locations from userdb.
350
352 Report bugs, including doveconf -n output, to the Dovecot Mailing List
353 <dovecot@dovecot.org>. Information about reporting bugs is available
354 at: http://dovecot.org/bugreport.html
355
357 doveadm(1), doveadm-auth(1), doveadm-kick(1), doveadm-purge(1), dove‐
358 conf(1)
359 Additional resources:
361 dsync design
363 http://wiki2.dovecot.org/Design/Dsync
364Dovecot v2.3 2015-08-31 DOVEADM-SYNC(1)