1RDIFFDIR(1) User Manuals RDIFFDIR(1)
2
6 duplicity - Encrypted backup using rsync algorithm
7
10 duplicity [options] source_directory target_url
11 duplicity [options] source_url target_directory
13 duplicity full [options] source_directory target_url
15 duplicity incremental [options] source_directory target_url
17 duplicity restore [options] source_url target_directory
19 duplicity verify [options] source_url target_directory
21 duplicity collection-status [options] target_url
23 duplicity list-current-files [options] target_url
25 duplicity cleanup [options] target_url
27 duplicity remove-older-than time [options] target_url
29 duplicity remove-all-but-n-full count [options] target_url
31
34 Duplicity incrementally backs up files and directory by encrypting tar-
35 format volumes with GnuPG and uploading them to a remote (or local)
36 file server. Currently local, ftp, ssh/scp, rsync, WebDAV, WebDAVs,
37 HSi and Amazon S3 backends are available. Because duplicity uses
38 librsync, the incremental archives are space efficient and only record
39 the parts of files that have changed since the last backup. Currently
40 duplicity supports deleted files, full unix permissions, directories,
41 symbolic links, fifos, etc., but not hard links.
42 Duplicity will read the PASSPHRASE environment variable to find the
44 passphrase to give to GnuPG. If this is not set, the user will be
45 prompted for the passphrase.
46 If you are backing up the root directory /, remember to --exclude
48 /proc, or else duplicity will probably crash on the weird stuff in
49 there.
50
53 Here is an example of a backup, using scp to back up /home/me to
54 some_dir on the other.host machine:
55 duplicity /home/me scp://uid@other.host/some_dir
57 If the above is run repeatedly, the first will be a full backup, and
59 subsequent ones will be incremental. To force a full backup, use the
60 full action:
61 duplicity full /home/me scp://uid@other.host/some_dir
63 Now suppose we accidentally delete /home/me and want to restore it the
65 way it was at the time of last backup:
66 duplicity scp://uid@other.host/some_dir /home/me
68 Duplicity enters restore mode because the URL comes before the local
70 directory. If we wanted to restore just the file "Mail/article" in
71 /home/me as it was three days ago into /home/me/restored_file:
72 duplicity -t 3D --file-to-restore Mail/article
74 scp://uid@other.host/some_dir /home/me/restored_file
75 The following command compares the files we backed up, so see what has
77 changed since then:
78 duplicity verify scp://uid@other.host/some_dir /home/me
80 Finally, duplicity recognizes several include/exclude options. For
82 instance, the following will backup the root directory, but exclude
83 /mnt, /tmp, and /proc:
84 duplicity --exclude /mnt --exclude /tmp --exclude /proc /
86 file:///usr/local/backup
87 Note that in this case the destination is the local directory
89 /usr/local/backup. The following will backup only the /home and /etc
90 directories under root:
91 duplicity --include /home --include /etc --exclude '**' /
93 file:///usr/local/backup
94 Duplicity can also access a repository via ftp. If a user name is
96 given, the environment variable FTP_PASSWORD is read to determine the
97 password:
98 FTP_PASSWORD=mypassword; duplicity /local/dir
100 ftp://user@other.host/some_dir
101
104 cleanup
105 Delete the extraneous duplicity files on the given backend.
106 Non-duplicity files, or files in complete data sets will not be
107 deleted. This should only be necessary after a duplicity ses‐
108 sion fails or is aborted prematurely.
109 collection-status
111 Summarize the status of the backup repository by printing the
112 chains and sets found, and the number of volumes in each.
113 full Indicate full backup. If this is set, perform full backup even
115 if signatures are available.
116 incr If this is requested an incremental backup will be performed.
118 Duplicity will abort if old signatures cannot be found. The
119 default is to switch to full backup under these conditions.
120 list-current-files
122 Lists the files currently backed up in the archive. The infor‐
123 mation will be extracted from the signature files, not the ar‐
124 chive data itself. Thus the whole archive does not have to be
125 downloaded, but on the other hand if the archive has been
126 deleted or corrupted, this command may not detect it.
127 remove-older-than time
129 Delete all backup sets older than the given time. If old backup
130 sets will not be deleted if backup sets newer than time depend
131 on them. See the TIME FORMATS section for more information.
132 Note, this action cannot be combined with backup or other
133 actions, such as cleanup.
134 remove-all-but-n-full count
136 Delete all backups sets that are older than the count:th last
137 full backup (in other words, keep the last count full backups
138 and associated incremental sets). count must be larger than
139 zero. A value of 1 means that only the single most recent backup
140 chain will be kept.
141 verify Enter verify mode instead of restore. If the --file-to-restore
143 option is given, restrict verify to that file or directory.
144 duplicity will exit with a non-zero error level if any files are
145 different. On verbosity level 4 or higher, log a message for
146 each file that has changed.
147
150 --allow-source-mismatch
151 Do not abort on attempts to use the same archive dir or remote
152 backend to back up different directories. duplicity will tell
153 you if you need this switch.
154 --archive-dir path
156 When backing up or restoring, specify the local archive direc‐
157 tory. This option is not necessary, but if hash data is found
158 locally in path it will be used in preference to the remote hash
159 data. Use of this option does not imply that the archive data is
160 no longer stored in the backup destination, nor that the local
161 archive directory need be kept safe. The local archive directory
162 is a performance optimization only, and may safely be discarded
163 at any time.
164 --encrypt-key key
166 When backing up, encrypt to the given public key, instead of
167 using symmetric (traditional) encryption. Can be specified mul‐
168 tiple times.
169 --exclude shell_pattern
171 Exclude the file or files matched by shell_pattern. If a direc‐
172 tory is matched, then files under that directory will also be
173 matched. See the FILE SELECTION section for more information.
174 --exclude-device-files
176 Exclude all device files. This can be useful for security/per‐
177 missions reasons or if rdiff-backup is not handling device files
178 correctly.
179 --exclude-filelist filename
181 Excludes the files listed in filename. See the FILE SELECTION
182 section for more information.
183 --exclude-filelist-stdin
185 Like --exclude-filelist, but the list of files will be read from
186 standard input. See the FILE SELECTION section for more infor‐
187 mation.
188 --exclude-globbing-filelist filename
190 Like --exclude-filelist but each line of the filelist will be
191 interpreted according to the same rules as --include and
192 --exclude.
193 --exclude-other-filesystems
195 Exclude files on file systems (identified by device number)
196 other than the file system the root of the source directory is
197 on.
198 --exclude-regexp regexp
200 Exclude files matching the given regexp. Unlike the --exclude
201 option, this option does not match files in a directory it
202 matches. See the FILE SELECTION section for more information.
203 --file-to-restore path
205 This option may be given in restore mode, causing only path to
206 be restored instead of the entire contents of the backup ar‐
207 chive. path should be given relative to the root of the direc‐
208 tory backed up.
209 --full-if-older-than time
211 Perform a full backup if an incremental backup is requested, but
212 the latest full backup in the collection is older than the given
213 time. See the TIME FORMATS section for more information.
214 --force
216 Proceed even if data loss might result. Duplicity will let the
217 user know when this option is required.
218 --ftp-passive
220 Use passive (PASV) data connections. The default is to use pas‐
221 sive, but to fallback to regular if the passive connection fails
222 or times out.
223 --ftp-regular
225 Use regular (PORT) data connections.
226 --gpg-options options
228 Allows you to pass options to gpg encryption. The options list
229 should be of the form "opt1=parm1 opt2=parm2" where the string
230 is quoted and the only spaces allowed are between options.
231 --include shell_pattern
233 Similar to --exclude but include matched files instead. Unlike
234 --exclude, this option will also match parent directories of
235 matched files (although not necessarily their contents). See
236 the FILE SELECTION section for more information.
237 --include-filelist filename
239 Like --exclude-filelist, but include the listed files instead.
240 See the FILE SELECTION section for more information.
241 --include-filelist-stdin
243 Like --include-filelist, but read the list of included files
244 from standard input.
245 --include-globbing-filelist filename
247 Like --include-filelist but each line of the filelist will be
248 interpreted according to the same rules as --include and
249 --exclude.
250 --include-regexp regexp
252 Include files matching the regular expression regexp. Only
253 files explicitly matched by regexp will be included by this
254 option. See the FILE SELECTION section for more information.
255 --no-encryption
257 Do not use GnuPG to encrypt files on remote system. Instead
258 just write gzipped volumes.
259 --no-print-statistics
261 By default duplicity will print statistics about the current
262 session after a successful backup. This switch disables that
263 behavior.
264 --null-separator
266 Use nulls (\0) instead of newlines (\n) as line separators,
267 which may help when dealing with filenames containing newlines.
268 This affects the expected format of the files specified by the
269 --{include|exclude}-filelist[-stdin] switches as well as the
270 format of the directory statistics file.
271 --num-retries number
273 Number of retries to make on errors before giving up.
274 --scp-command command
276 This option only matters when using the ssh/scp backend. The
277 command will be used instead of scp to send or receive files.
278 The default is "scp". To list and delete existing files, the
279 sftp command is used. See --ssh-options and --sftp-command.
280 --sftp-command command
282 This option only matters when using the ssh/scp backend. The
283 command will be used instead of sftp for listing and deleting
284 files. The default is "sftp". File transfers are done using the
285 scp command. See --ssh-options and --scp-command.
286 --sign-key key
288 This option can be used when backing up or restoring. When
289 backing up, all backup files will be signed with keyid key.
290 When restoring, duplicity will signal an error if any remote
291 file is not signed with the given keyid. key should be an 8
292 character hex string, like AA0E73D2.
293 --ssh-askpass
295 Tells the ssh/scp backend to use FTP_PASSWORD from the environ‐
296 ment, or, if that is not present, to prompt the user for the
297 remote system password.
298 --ssh-options options
300 Allows you to pass options to the ssh/scp/sftp backend. The
301 options list should be of the form "opt1=parm1 opt2=parm2" where
302 the option string is quoted and the only spaces allowed are
303 between options. The option string will be passed verbatim to
304 both scp and sftp, whose command line syntax differs slightly:
305 options passed with --ssh-options should therefore be given in
306 the long option format described in ssh_config(5), like in this
307 example:
308 duplicity --ssh-options="-oProtocol=2 -oIdentity‐
310 File=/my/backup/id" /home/me scp://uid@other.host/some_dir
311 --short-filenames
313 If this option is specified, the names of the files duplicity
314 writes will be shorter (about 30 chars) but less understandable.
315 This may be useful when backing up to MacOS or another OS or FS
316 that doesn't support long filenames.
317 --tempdir directory
319 Use this existing directory for duplicity temporary files
320 instead of the system default, which is usually the /tmp direc‐
321 tory. This option supercedes any environment variable.
322 -ttime, --restore-time time
324 When restoring, specify the time to restore to.
325 -v[0-9], --verbosity [0-9]
327 Specify verbosity level (0 is total silent, 3 is the default,
328 and 9 is noisiest).
329 --volsize number
331 Change the volume size to number Mb. Default is 5Mb.
332
336 Duplicity tries to maintain a standard URL format as much as possible.
337 The generic format for a URL is:
338 scheme://user[:password]@host[:port]/[/]path
340 It is not recommended to expose the password on the command line since
342 it could be revealed to anyone with permissions to do process listings,
343 however, it is permitted.
344 In protocols that support it, the path may be preceeded by a single
346 slash, '/path', to represent a relative path to the target home direc‐
347 tory, or preceeded by a double slash, '//path', to represent an abso‐
348 lute filesystem path.
349 Formats of each of the URL schemes follow:
351 ssh://user[:password]@other.host[:port]/some_dir
353 scp://user[:password]@other.host[:port]/some_dir
355 ftp://user[:password]@other.host[:port]/some_dir
357 hsi://user[:password]@other.host/some_dir
359 file:///some_dir
361 rsync://user[:password]@other.host[:port]::/module/some_dir
363 rsync://user[:password]@other.host[:port]/relative_path
365 rsync://user[:password]@other.host[:port]//absolute_path
367 s3://host/bucket_name[/prefix]
369 s3+http://bucket_name[/prefix]
371 webdav://user[:password]@other.host/some_dir
373 webdavs://user[:password]@other.host/some_dir
375
378 duplicity uses time strings in two places. Firstly, many of the files
379 duplicity creates will have the time in their filenames in the w3 date‐
380 time format as described in a w3 note at http://www.w3.org/TR/NOTE-
381 datetime. Basically they look like "2001-07-15T04:09:38-07:00", which
382 means what it looks like. The "-07:00" section means the time zone is
383 7 hours behind UTC.
384 Secondly, the -t, and --restore-time options take a time string, which
386 can be given in any of several formats:
387 1. the string "now" (refers to the current time)
389 2. a sequences of digits, like "123456890" (indicating the time in
391 seconds after the epoch)
392 3. A string like "2002-01-25T07:00:00+02:00" in datetime format
394 4. An interval, which is a number followed by one of the characters
396 s, m, h, D, W, M, or Y (indicating seconds, minutes, hours,
397 days, weeks, months, or years respectively), or a series of such
398 pairs. In this case the string refers to the time that preceded
399 the current time by the length of the interval. For instance,
400 "1h78m" indicates the time that was one hour and 78 minutes ago.
401 The calendar here is unsophisticated: a month is always 30 days,
402 a year is always 365 days, and a day is always 86400 seconds.
403 5. A date format of the form YYYY/MM/DD, YYYY-MM-DD, MM/DD/YYYY, or
405 MM-DD-YYYY, which indicates midnight on the day in question,
406 relative to the current timezone settings. For instance,
407 "2002/3/5", "03-05-2002", and "2002-3-05" all mean March 5th,
408 2002.
409
412 duplicity accepts the same file selection options rdiff-backup does,
413 including --exclude, --exclude-filelist-stdin, etc.
414 When duplicity is run, it searches through the given source directory
416 and backs up all the files specified by the file selection system. The
417 file selection system comprises a number of file selection conditions,
418 which are set using one of the following command line options:
419 --exclude, --exclude-device-files, --exclude-filelist, --exclude-
420 filelist-stdin, --exclude-globbing-filelist, --exclude-regexp,
421 --include, --include-filelist, --include-filelist-stdin, --include-
422 globbing-filelist, and --include-regexp. Each file selection condition
423 either matches or doesn't match a given file. A given file is excluded
424 by the file selection system exactly when the first matching file
425 selection condition specifies that the file be excluded; otherwise the
426 file is included.
427 For instance,
429 duplicity --include /usr --exclude /usr /usr
431 scp://user@host/backup
432 is exactly the same as
434 duplicity /usr scp://user@host/backup
436 because the include and exclude directives match exactly the same
438 files, and the --include comes first, giving it precedence. Similarly,
439 duplicity --include /usr/local/bin --exclude /usr/local /usr
441 scp://user@host/backup
442 would backup the /usr/local/bin directory (and its contents), but not
444 /usr/local/doc.
445 The include, exclude, include-globbing-filelist, and exclude-globbing-
447 filelist options accept extended shell globbing patterns. These pat‐
448 terns can contain the special patterns *, **, ?, and [...]. As in a
449 normal shell, * can be expanded to any string of characters not con‐
450 taining "/", ? expands to any character except "/", and [...] expands
451 to a single character of those characters specified (ranges are accept‐
452 able). The new special pattern, **, expands to any string of charac‐
453 ters whether or not it contains "/". Furthermore, if the pattern
454 starts with "ignorecase:" (case insensitive), then this prefix will be
455 removed and any character in the string can be replaced with an upper-
456 or lowercase version of itself.
457 Remember that you may need to quote these characters when typing them
459 into a shell, so the shell does not interpret the globbing patterns
460 before duplicity sees them.
461 The --exclude pattern option matches a file iff:
463 1. pattern can be expanded into the file's filename, or
465 2. the file is inside a directory matched by the option.
467 Conversely, --include pattern matches a file iff:
469 1. pattern can be expanded into the file's filename,
471 2. the file is inside a directory matched by the option, or
473 3. the file is a directory which contains a file matched by the
475 option.
476 For example,
478 --exclude /usr/local
480 matches /usr/local, /usr/local/lib, and /usr/local/lib/netscape. It is
482 the same as --exclude /usr/local --exclude '/usr/local/**'.
483 --include /usr/local
485 specifies that /usr, /usr/local, /usr/local/lib, and
487 /usr/local/lib/netscape (but not /usr/doc) all be backed up. Thus you
488 don't have to worry about including parent directories to make sure
489 that included subdirectories have somewhere to go. Finally,
490 --include ignorecase:'/usr/[a-z0-9]foo/*/**.py'
492 would match a file like /usR/5fOO/hello/there/world.py. If it did
494 match anything, it would also match /usr. If there is no existing file
495 that the given pattern can be expanded into, the option will not match
496 /usr.
497 The --include-filelist, --exclude-filelist, --include-filelist-stdin,
499 and --exclude-filelist-stdin options also introduce file selection con‐
500 ditions. They direct duplicity to read in a file, each line of which
501 is a file specification, and to include or exclude the matching files.
502 Lines are separated by newlines or nulls, depending on whether the
503 --null-separator switch was given. Each line in a filelist is inter‐
504 preted similarly to the way extended shell patterns are, with a few
505 exceptions:
506 1. Globbing patterns like *, **, ?, and [...] are not expanded.
508 2. Include patterns do not match files in a directory that is
510 included. So /usr/local in an include file will not match
511 /usr/local/doc.
512 3. Lines starting with "+ " are interpreted as include directives,
514 even if found in a filelist referenced by --exclude-filelist.
515 Similarly, lines starting with "- " exclude files even if they
516 are found within an include filelist.
517 For example, if file "list.txt" contains the lines:
519 /usr/local
521 - /usr/local/doc
522 /usr/local/bin
523 + /var
524 - /var
525 then "--include-filelist list.txt" would include /usr, /usr/local, and
527 /usr/local/bin. It would exclude /usr/local/doc,
528 /usr/local/doc/python, etc. It neither excludes nor includes
529 /usr/local/man, leaving the fate of this directory to the next specifi‐
530 cation condition. Finally, it is undefined what happens with /var. A
531 single file list should not contain conflicting file specifications.
532 The --include-globbing-filelist and --exclude-globbing-filelist options
534 also specify filelists, but each line in the filelist will be inter‐
535 preted as a globbing pattern the way --include and --exclude options
536 are interpreted (although "+ " and "- " prefixing is still allowed).
537 For instance, if the file "globbing-list.txt" contains the lines:
538 dir/foo
540 + dir/bar
541 - **
542 Then "--include-globbing-filelist globbing-list.txt" would be exactly
544 the same as specifying "--include dir/foo --include dir/bar --exclude
545 **" on the command line.
546 Finally, the --include-regexp and --exclude-regexp allow files to be
548 included and excluded if their filenames match a python regular expres‐
549 sion. Regular expression syntax is too complicated to explain here,
550 but is covered in Python's library reference. Unlike the --include and
551 --exclude options, the regular expression options don't match files
552 containing or contained in matched files. So for instance
553 --include '[0-9]{7}(?!foo)'
555 matches any files whose full pathnames contain 7 consecutive digits
557 which aren't followed by 'foo'. However, it wouldn't match /home even
558 if /home/ben/1234567 existed.
559
562 This section describes duplicity's basic operation and the format of
563 its data files. It should not necessary to read this section to use
564 duplicity.
565 The files used by duplicity to store backup data are tarfiles in GNU
567 tar format. They can be produced independently by rdiffdir(1). For
568 incremental backups, new files are saved normally in the tarfile. But
569 when a file changes, instead of storing a complete copy of the file,
570 only a diff is stored, as generated by rdiff(1). If a file is deleted,
571 a 0 length file is stored in the tar. It is possible to restore a
572 duplicity archive "manually" by using tar and then cp, rdiff, and rm as
573 necessary. These duplicity archives have the extension difftar.
574 Both full and incremental backup sets have the same format. In effect,
576 a full backup set is an incremental one generated from an empty signa‐
577 ture (see below). The files in full backup sets will start with
578 duplicity-full while the incremental sets start with duplicity-inc.
579 When restoring, duplicity applies patches in order, so deleting, for
580 instance, a full backup set may make related incremental backup sets
581 unuseable.
582 In order to determine which files have been deleted, and to calculate
584 diffs for changed files, duplicity needs to process information about
585 previous sessions. It stores this information in the form of tarfiles
586 where each entry's data contains the signature (as produced by rdiff)
587 of the file instead of the file's contents. These signature sets have
588 the extension sigtar.
589 Signature files are not required to restore a backup set, but without
591 an up-to-date signature, duplicity cannot append an incremental backup
592 to an existing archive.
593 To save bandwidth, duplicity generates full signature sets and incre‐
595 mental signature sets. A full signature set is generated for each full
596 backup, and an incremental one for each incremental backup. These
597 start with duplicity-full-signatures and duplicity-new-signatures
598 respectively. If --archive-dir is used, these signatures will be
599 stored both locally and remotely. The remote signatures will be
600 encrypted if encryption is enabled. The local signatures will not be
601 encrypted
602
605 TMPDIR, TEMP, TMP
606 In decreasing order of importance, specifies the directory to
607 use for temporary files (inherited from Python's tempfile mod‐
608 ule).
609
612 Hard links currently unsupported (they will be treated as non-linked
613 regular files).
614 Bad signatures will be treated as empty instead of logging appropriate
616 error message.
617
620 Original Author - Ben Escoto <bescoto@stanford.edu>
621 Current Maintainer - Kenneth Loafman <kenneth@loafman.com>
623
626 rdiffdir(1), python(1), rdiff(1), rdiff-backup(1).
627Version 0.4.11 May 2008 RDIFFDIR(1)