1DUPLICITY(1) User Manuals DUPLICITY(1)
2
6 duplicity - Encrypted incremental backup to local or remote storage.
7
10 For detailed descriptions for each command see chapter ACTIONS.
11 duplicity [full|incremental] [options] source_directory target_url
13 duplicity verify [options] [--compare-data] [--file-to-restore
15 <relpath>] [--time time] source_url target_directory
16 duplicity collection-status [options] [--file-changed <relpath>]
18 target_url
19 duplicity list-current-files [options] [--time time] target_url
21 duplicity [restore] [options] [--file-to-restore <relpath>] [--time
23 time] source_url target_directory
24 duplicity remove-older-than <time> [options] [--force] target_url
26 duplicity remove-all-but-n-full <count> [options] [--force] target_url
28 duplicity remove-all-inc-of-but-n-full <count> [options] [--force]
30 target_url
31 duplicity cleanup [options] [--force] [--extra-clean] target_url
33 duplicity replicate [options] [--time time] source_url target_url
35
38 Duplicity incrementally backs up files and folders into tar-format
39 volumes encrypted with GnuPG and places them to a remote (or local)
40 storage backend. See chapter URL FORMAT for a list of all supported
41 backends and how to address them. Because duplicity uses librsync,
42 incremental backups are space efficient and only record the parts of
43 files that have changed since the last backup. Currently duplicity
44 supports deleted files, full Unix permissions, uid/gid, directories,
45 symbolic links, fifos, etc., but not hard links.
46 If you are backing up the root directory /, remember to --exclude
48 /proc, or else duplicity will probably crash on the weird stuff in
49 there.
50
53 Here is an example of a backup, using sftp to back up /home/me to
54 some_dir on the other.host machine:
55 duplicity /home/me sftp://uid@other.host/some_dir
57 If the above is run repeatedly, the first will be a full backup, and
59 subsequent ones will be incremental. To force a full backup, use the
60 full action:
61 duplicity full /home/me sftp://uid@other.host/some_dir
63 or enforcing a full every other time via --full-if-older-than <time> ,
65 e.g. a full every month:
66 duplicity --full-if-older-than 1M /home/me
68 sftp://uid@other.host/some_dir
69 Now suppose we accidentally delete /home/me and want to restore it the
71 way it was at the time of last backup:
72 duplicity sftp://uid@other.host/some_dir /home/me
74 Duplicity enters restore mode because the URL comes before the local
76 directory. If we wanted to restore just the file "Mail/article" in
77 /home/me as it was three days ago into /home/me/restored_file:
78 duplicity -t 3D --file-to-restore Mail/article
80 sftp://uid@other.host/some_dir /home/me/restored_file
81 The following command compares the latest backup with the current
83 files:
84 duplicity verify sftp://uid@other.host/some_dir /home/me
86 Finally, duplicity recognizes several include/exclude options. For
88 instance, the following will backup the root directory, but exclude
89 /mnt, /tmp, and /proc:
90 duplicity --exclude /mnt --exclude /tmp --exclude /proc /
92 file:///usr/local/backup
93 Note that in this case the destination is the local directory
95 /usr/local/backup. The following will backup only the /home and /etc
96 directories under root:
97 duplicity --include /home --include /etc --exclude '**' /
99 file:///usr/local/backup
100 Duplicity can also access a repository via ftp. If a user name is
102 given, the environment variable FTP_PASSWORD is read to determine the
103 password:
104 FTP_PASSWORD=mypassword duplicity /local/dir
106 ftp://user@other.host/some_dir
107
110 Duplicity knows action commands, which can be finetuned with options.
111 The actions for backup (full,incr) and restoration (restore) can as
112 well be left out as duplicity detects in what mode it should switch to
113 by the order of target URL and local folder. If the target URL comes
114 before the local folder a restore is in order, is the local folder
115 before target URL then this folder is about to be backed up to the
116 target URL.
117 If a backup is in order and old signatures can be found duplicity
118 automatically performs an incremental backup.
119 Note: The following explanations explain some but not all options that
121 can be used in connection with that action command. Consult the
122 OPTIONS section for more detailed informations.
123 full <folder> <url>
126 Perform a full backup. A new backup chain is started even if
127 signatures are available for an incremental backup.
128 incr <folder> <url>
131 If this is requested an incremental backup will be performed.
132 Duplicity will abort if no old signatures can be found.
133 verify [--compare-data] [--time <time>] [--file-to-restore <rel_path>]
136 <url> <local_path>
137 Verify tests the integrity of the backup archives at the remote
138 location by downloading each file and checking both that it can
139 restore the archive and that the restored file matches the
140 signature of that file stored in the backup, i.e. compares the
141 archived file with its hash value from archival time. Verify
142 does not actually restore and will not overwrite any local
143 files. Duplicity will exit with a non-zero error level if any
144 files do not match the signature stored in the archive for that
145 file. On verbosity level 4 or higher, it will log a message for
146 each file that differs from the stored signature. Files must be
147 downloaded to the local machine in order to compare them.
148 Verify does not compare the backed-up version of the file to the
149 current local copy of the files unless the --compare-data option
150 is used (see below).
151 The --file-to-restore option restricts verify to that file or
152 folder. The --time option allows to select a backup to verify.
153 The --compare-data option enables data comparison (see below).
154 collection-status [--file-changed <relpath>]<url>
157 Summarize the status of the backup repository by printing the
158 chains and sets found, and the number of volumes in each.
159 list-current-files [--time <time>] <url>
162 Lists the files contained in the most current backup or backup
163 at time. The information will be extracted from the signature
164 files, not the archive data itself. Thus the whole archive does
165 not have to be downloaded, but on the other hand if the archive
166 has been deleted or corrupted, this command will not detect it.
167 restore [--file-to-restore <relpath>] [--time <time>] <url>
170 <target_folder>
171 You can restore the full monty or selected folders/files from a
172 specific time. Use the relative path as it is printed by list-
173 current-files. Usually not needed as duplicity enters restore
174 mode when it detects that the URL comes before the local folder.
175 remove-older-than <time> [--force] <url>
178 Delete all backup sets older than the given time. Old backup
179 sets will not be deleted if backup sets newer than time depend
180 on them. See the TIME FORMATS section for more information.
181 Note, this action cannot be combined with backup or other
182 actions, such as cleanup. Note also that --force will be needed
183 to delete the files instead of just listing them.
184 remove-all-but-n-full <count> [--force] <url>
187 Delete all backups sets that are older than the count:th last
188 full backup (in other words, keep the last count full backups
189 and associated incremental sets). count must be larger than
190 zero. A value of 1 means that only the single most recent backup
191 chain will be kept. Note that --force will be needed to delete
192 the files instead of just listing them.
193 remove-all-inc-of-but-n-full <count> [--force] <url>
196 Delete incremental sets of all backups sets that are older than
197 the count:th last full backup (in other words, keep only old
198 full backups and not their increments). count must be larger
199 than zero. A value of 1 means that only the single most recent
200 backup chain will be kept intact. Note that --force will be
201 needed to delete the files instead of just listing them.
202 cleanup [--force] [--extra-clean] <url>
205 Delete the extraneous duplicity files on the given backend.
206 Non-duplicity files, or files in complete data sets will not be
207 deleted. This should only be necessary after a duplicity
208 session fails or is aborted prematurely. Note that --force will
209 be needed to delete the files instead of just listing them.
210 replicate [--time time] <source_url> <target_url>
213 Replicate backup sets from source to target backend. Files will
214 be (re)-encrypted and (re)-compressed depending on normal
215 backend options. Signatures and volumes will not get recomputed,
216 thus options like --volsize or --max-blocksize have no effect.
217 When --time time is given, only backup sets older than time will
218 be replicated.
219
222 --allow-source-mismatch
223 Do not abort on attempts to use the same archive dir or remote
224 backend to back up different directories. duplicity will tell
225 you if you need this switch.
226 --archive-dir path
229 The archive directory. NOTE: This option changed in 0.6.0. The
230 archive directory is now necessary in order to manage
231 persistence for current and future enhancements. As such, this
232 option is now used only to change the location of the archive
233 directory. The archive directory should not be deleted, or
234 duplicity will have to recreate it from the remote repository
235 (which may require decrypting the backup contents).
236 When backing up or restoring, this option specifies that the
238 local archive directory is to be created in path. If the
239 archive directory is not specified, the default will be to
240 create the archive directory in ~/.cache/duplicity/.
241 The archive directory can be shared between backups to multiple
243 targets, because a subdirectory of the archive dir is used for
244 individual backups (see --name ).
245 The combination of archive directory and backup name must be
247 unique in order to separate the data of different backups.
248 The interaction between the --archive-dir and the --name options
250 allows for four possible combinations for the location of the
251 archive dir:
252 1. neither specified (default)
255 ~/.cache/duplicity/hash-of-url
256 2. --archive-dir=/arch, no --name
258 /arch/hash-of-url
259 3. no --archive-dir, --name=foo
261 ~/.cache/duplicity/foo
262 4. --archive-dir=/arch, --name=foo
264 /arch/foo
265 --asynchronous-upload
268 (EXPERIMENTAL) Perform file uploads asynchronously in the
269 background, with respect to volume creation. This means that
270 duplicity can upload a volume while, at the same time, preparing
271 the next volume for upload. The intended end-result is a faster
272 backup, because the local CPU and your bandwidth can be more
273 consistently utilized. Use of this option implies additional
274 need for disk space in the temporary storage location; rather
275 than needing to store only one volume at a time, enough storage
276 space is required to store two volumes.
277 --backend-retry-delay number
280 Specifies the number of seconds that duplicity waits after an
281 error has occured before attempting to repeat the operation.
282 --cf-backend backend
286 Allows the explicit selection of a cloudfiles backend. Defaults
287 to pyrax. Alternatively you might choose cloudfiles.
288 --compare-data
291 Enable data comparison of regular files on action verify. This
292 conducts a verify as described above to verify the integrity of
293 the backup archives, but additionally compares restored files to
294 those in target_directory. Duplicity will not replace any files
295 in target_directory. Duplicity will exit with a non-zero error
296 level if the files do not correctly verify or if any files from
297 the archive differ from those in target_directory. On verbosity
298 level 4 or higher, it will log a message for each file that
299 differs from its equivalent in target_directory.
300 --copy-links
303 Resolve symlinks during backup. Enabling this will resolve &
304 back up the symlink's file/folder data instead of the symlink
305 itself, potentially increasing the size of the backup.
306 --dry-run
309 Calculate what would be done, but do not perform any backend
310 actions
311 --encrypt-key key-id
314 When backing up, encrypt to the given public key, instead of
315 using symmetric (traditional) encryption. Can be specified
316 multiple times. The key-id can be given in any of the formats
317 supported by GnuPG; see gpg(1), section "HOW TO SPECIFY A USER
318 ID" for details.
319 --encrypt-secret-keyring filename
323 This option can only be used with --encrypt-key, and changes the
324 path to the secret keyring for the encrypt key to filename This
325 keyring is not used when creating a backup. If not specified,
326 the default secret keyring is used which is usually located at
327 .gnupg/secring.gpg
328 --encrypt-sign-key key-id
331 Convenience parameter. Same as --encrypt-key key-id --sign-key
332 key-id.
333 --exclude shell_pattern
336 Exclude the file or files matched by shell_pattern. If a
337 directory is matched, then files under that directory will also
338 be matched. See the FILE SELECTION section for more
339 information.
340 --exclude-device-files
343 Exclude all device files. This can be useful for
344 security/permissions reasons or if rdiff-backup is not handling
345 device files correctly.
346 --exclude-filelist filename
349 Excludes the files listed in filename, with each line of the
350 filelist interpreted according to the same rules as --include
351 and --exclude. See the FILE SELECTION section for more
352 information.
353 --exclude-if-present filename
356 Exclude directories if filename is present. Allows the user to
357 specify folders that they do not wish to backup by adding a
358 specified file (e.g. ".nobackup") instead of maintaining a
359 comprehensive exclude/include list. This option needs to come
360 before any other include or exclude options.
361 --exclude-older-than time
364 Exclude any files whose modification date is earlier than the
365 specified time. This can be used to produce a partial backup
366 that contains only recently changed files. See the TIME FORMATS
367 section for more information.
368 --exclude-other-filesystems
371 Exclude files on file systems (identified by device number)
372 other than the file system the root of the source directory is
373 on.
374 --exclude-regexp regexp
377 Exclude files matching the given regexp. Unlike the --exclude
378 option, this option does not match files in a directory it
379 matches. See the FILE SELECTION section for more information.
380 --extra-clean
383 When cleaning up, be more aggressive about saving space. For
384 example, this may delete signature files for old backup chains.
385 Caution: Without signature files those old backup chains are
387 unrestorable. Do not use --extra-clean unless you know what
388 you're doing.
389 See the cleanup argument for more information.
391 --file-changed path
394 This option may be given in collection-status mode, causing only
395 path status to be collect instead of the entire contents of the
396 backup archive. path should be given relative to the root of
397 the directory backed up.
398 --file-prefix, --file-prefix-manifest, --file-prefix-archive, --file-
402 prefix-signature
403 Adds a prefix to all files, manifest files, archive files,
404 and/or signature files.
405 The same set of prefixes must be passed in on backup and
407 restore.
408 If both global and type-specific prefixes are set, global prefix
410 will go before type-specific prefixes.
411 See also A NOTE ON FILENAME PREFIXES
413 --file-to-restore path
416 This option may be given in restore mode, causing only path to
417 be restored instead of the entire contents of the backup
418 archive. path should be given relative to the root of the
419 directory backed up.
420 --full-if-older-than time
423 Perform a full backup if an incremental backup is requested, but
424 the latest full backup in the collection is older than the given
425 time. See the TIME FORMATS section for more information.
426 --force
429 Proceed even if data loss might result. Duplicity will let the
430 user know when this option is required.
431 --ftp-passive
434 Use passive (PASV) data connections. The default is to use
435 passive, but to fallback to regular if the passive connection
436 fails or times out.
437 --ftp-regular
440 Use regular (PORT) data connections.
441 --gio Use the GIO backend and interpret any URLs as GIO would.
444 --hidden-encrypt-key key-id
447 Same as --encrypt-key, but it hides user's key id from encrypted
448 file. It uses the gpg's --hidden-recipient command to obfuscate
449 the owner of the backup. On restore, gpg will automatically try
450 all available secret keys in order to decrypt the backup. See
451 gpg(1) for more details.
452 --ignore-errors
456 Try to ignore certain errors if they happen. This option is only
457 intended to allow the restoration of a backup in the face of
458 certain problems that would otherwise cause the backup to fail.
459 It is not ever recommended to use this option unless you have a
460 situation where you are trying to restore from backup and it is
461 failing because of an issue which you want duplicity to ignore.
462 Even then, depending on the issue, this option may not have an
463 effect.
464 Please note that while ignored errors will be logged, there will
466 be no summary at the end of the operation to tell you what was
467 ignored, if anything. If this is used for emergency restoration
468 of data, it is recommended that you run the backup in such a way
469 that you can revisit the backup log (look for lines containing
470 the string IGNORED_ERROR).
471 If you ever have to use this option for reasons that are not
473 understood or understood but not your own responsibility, please
474 contact duplicity maintainers. The need to use this option under
475 production circumstances would normally be considered a bug.
476 --imap-full-address email_address
479 The full email address of the user name when logging into an
480 imap server. If not supplied just the user name part of the
481 email address is used.
482 --imap-mailbox option
485 Allows you to specify a different mailbox. The default is
486 "INBOX". Other languages may require a different mailbox than
487 the default.
488 --gpg-binary file_path
491 Allows you to force duplicity to use file_path as gpg command
492 line binary. Can be an absolute or relative file path or a file
493 name. Default value is 'gpg'. The binary will be localized via
494 the PATH environment variable.
495 --gpg-options options
498 Allows you to pass options to gpg encryption. The options list
499 should be of the form "--opt1 --opt2=parm" where the string is
500 quoted and the only spaces allowed are between options.
501 --include shell_pattern
504 Similar to --exclude but include matched files instead. Unlike
505 --exclude, this option will also match parent directories of
506 matched files (although not necessarily their contents). See
507 the FILE SELECTION section for more information.
508 --include-filelist filename
511 Like --exclude-filelist, but include the listed files instead.
512 See the FILE SELECTION section for more information.
513 --include-regexp regexp
516 Include files matching the regular expression regexp. Only
517 files explicitly matched by regexp will be included by this
518 option. See the FILE SELECTION section for more information.
519 --log-fd number
522 Write specially-formatted versions of output messages to the
523 specified file descriptor. The format used is designed to be
524 easily consumable by other programs.
525 --log-file filename
528 Write specially-formatted versions of output messages to the
529 specified file. The format used is designed to be easily
530 consumable by other programs.
531 --max-blocksize number
534 determines the number of the blocks examined for changes during
535 the diff process. For files < 1MB the blocksize is a constant
536 of 512. For files over 1MB the size is given by:
537 file_blocksize = int((file_len / (2000 * 512)) * 512)
539 return min(file_blocksize, globals.max_blocksize)
540 where globals.max_blocksize defaults to 2048. If you specify a
542 larger max_blocksize, your difftar files will be larger, but
543 your sigtar files will be smaller. If you specify a smaller
544 max_blocksize, the reverse occurs. The --max-blocksize option
545 should be in multiples of 512.
546 --name symbolicname
549 Set the symbolic name of the backup being operated on. The
550 intent is to use a separate name for each logically distinct
551 backup. For example, someone may use "home_daily_s3" for the
552 daily backup of a home directory to Amazon S3. The structure of
553 the name is up to the user, it is only important that the names
554 be distinct. The symbolic name is currently only used to affect
555 the expansion of --archive-dir , but may be used for additional
556 features in the future. Users running more than one distinct
557 backup are encouraged to use this option.
558 If not specified, the default value is a hash of the backend
560 URL.
561 --no-compression
564 Do not use GZip to compress files on remote system.
565 --no-encryption
568 Do not use GnuPG to encrypt files on remote system.
569 --no-print-statistics
572 By default duplicity will print statistics about the current
573 session after a successful backup. This switch disables that
574 behavior.
575 --null-separator
578 Use nulls (\0) instead of newlines (\n) as line separators,
579 which may help when dealing with filenames containing newlines.
580 This affects the expected format of the files specified by the
581 --{include|exclude}-filelist switches as well as the format of
582 the directory statistics file.
583 --numeric-owner
586 On restore always use the numeric uid/gid from the archive and
587 not the archived user/group names, which is the default
588 behaviour. Recommended for restoring from live cds which might
589 have the users with identical names but different uids/gids.
590 --num-retries number
593 Number of retries to make on errors before giving up.
594 --old-filenames
597 Use the old filename format (incompatible with Windows/Samba)
598 rather than the new filename format.
599 --par2-options options
602 Verbatim options to pass to par2.
603 --par2-redundancy percent
606 Adjust the level of redundancy in percent for Par2 recovery
607 files (default 10%).
608 --progress
611 When selected, duplicity will output the current upload progress
612 and estimated upload time. To annotate changes, it will perform
613 a first dry-run before a full or incremental, and then runs the
614 real operation estimating the real upload progress.
615 --progress-rate number
618 Sets the update rate at which duplicity will output the upload
619 progress messages (requires --progress option). Default is to
620 print the status each 3 seconds.
621 --rename <original path> <new path>
624 Treats the path orig in the backup as if it were the path new.
625 Can be passed multiple times. An example:
626 duplicity restore --rename Documents/metal Music/metal
628 sftp://uid@other.host/some_dir /home/me
629 --rsync-options options
632 Allows you to pass options to the rsync backend. The options
633 list should be of the form "opt1=parm1 opt2=parm2" where the
634 option string is quoted and the only spaces allowed are between
635 options. The option string will be passed verbatim to rsync,
636 after any internally generated option designating the remote
637 port to use. Here is a possibly useful example:
638 duplicity --rsync-options="--partial-dir=.rsync-partial"
640 /home/me rsync://uid@other.host/some_dir
641 --s3-european-buckets
644 When using the Amazon S3 backend, create buckets in Europe
645 instead of the default (requires --s3-use-new-style ). Also see
646 the EUROPEAN S3 BUCKETS section.
647 This option does not apply when using the newer boto3 backend,
649 which does not create buckets.
650 See also A NOTE ON AMAZON S3 below.
652 --s3-unencrypted-connection
655 Don't use SSL for connections to S3.
656 This may be much faster, at some cost to confidentiality.
658 With this option, anyone who can observe traffic between your
660 computer and S3 will be able to tell: that you are using
661 Duplicity, the name of the bucket, your AWS Access Key ID, the
662 increment dates and the amount of data in each increment.
663 This option affects only the connection, not the GPG encryption
665 of the backup increment files. Unless that is disabled, an
666 observer will not be able to see the file names or contents.
667 This option is not available when using the newer boto3 backend.
669 See also A NOTE ON AMAZON S3 below.
671 --s3-use-new-style
674 When operating on Amazon S3 buckets, use new-style subdomain
675 bucket addressing. This is now the preferred method to access
676 Amazon S3, but is not backwards compatible if your bucket name
677 contains upper-case characters or other characters that are not
678 valid in a hostname.
679 This option has no effect when using the newer boto3 backend,
681 which will always use new style subdomain bucket naming.
682 See also A NOTE ON AMAZON S3 below.
684 --s3-use-rrs
687 Store volumes using Reduced Redundancy Storage when uploading to
688 Amazon S3. This will lower the cost of storage but also lower
689 the durability of stored volumes to 99.99% instead the
690 99.999999999% durability offered by Standard Storage on S3.
691 --s3-use-ia
694 Store volumes using Standard - Infrequent Access when uploading
695 to Amazon S3. This storage class has a lower storage cost but a
696 higher per-request cost, and the storage cost is calculated
697 against a 30-day storage minimum. According to Amazon, this
698 storage is ideal for long-term file storage, backups, and
699 disaster recovery.
700 --s3-use-onezone-ia
703 Store volumes using One Zone - Infrequent Access when uploading
704 to Amazon S3. This storage is similar to Standard - Infrequent
705 Access, but only stores object data in one Availability Zone.
706 --s3-use-glacier
709 Store volumes using Glacier S3 when uploading to Amazon S3. This
710 storage class has a lower cost of storage but a higher per-
711 request cost along with delays of up to 12 hours from the time
712 of retrieval request. This storage cost is calculated against a
713 90-day storage minimum. According to Amazon this storage is
714 ideal for data archiving and long-term backup offering
715 99.999999999% durability. To restore a backup you will have to
716 manually migrate all data stored on AWS Glacier back to Standard
717 S3 and wait for AWS to complete the migration. Notice:
718 Duplicity will store the manifest.gpg files from full and
719 incremental backups on AWS S3 standard storage to allow quick
720 retrieval for later incremental backups, all other data is
721 stored in S3 Glacier.
722 --s3-use-deep-archive
725 Store volumes using Glacier Deep Archive S3 when uploading to
726 Amazon S3. This storage class has a lower cost of storage but a
727 higher per-request cost along with delays of up to 48 hours from
728 the time of retrieval request. This storage cost is calculated
729 against a 180-day storage minimum. According to Amazon this
730 storage is ideal for data archiving and long-term backup
731 offering 99.999999999% durability. To restore a backup you will
732 have to manually migrate all data stored on AWS Glacier Deep
733 Archive back to Standard S3 and wait for AWS to complete the
734 migration. Notice: Duplicity will store the manifest.gpg files
735 from full and incremental backups on AWS S3 standard storage to
736 allow quick retrieval for later incremental backups, all other
737 data is stored in S3 Glacier Deep Archive.
738 Glacier Deep Archive is only available when using the newer
740 boto3 backend.
741 --s3-use-multiprocessing
744 Allow multipart volumne uploads to S3 through multiprocessing.
745 This option requires Python 2.6 and can be used to make uploads
746 to S3 more efficient. If enabled, files duplicity uploads to S3
747 will be split into chunks and uploaded in parallel. Useful if
748 you want to saturate your bandwidth or if large files are
749 failing during upload.
750 This has no effect when using the newer boto3 backend. Boto3
752 always attempts to multiprocessing when it is believed it will
753 be more efficient.
754 See also A NOTE ON AMAZON S3 below.
756 --s3-use-server-side-encryption
759 Allow use of server side encryption in S3
760 --s3-multipart-chunk-size
763 Chunk size (in MB) used for S3 multipart uploads. Make this
764 smaller than --volsize to maximize the use of your bandwidth.
765 For example, a chunk size of 10MB with a volsize of 30MB will
766 result in 3 chunks per volume upload.
767 This has no effect when using the newer boto3 backend.
769 See also A NOTE ON AMAZON S3 below.
771 --s3-multipart-max-procs
774 Specify the maximum number of processes to spawn when performing
775 a multipart upload to S3. By default, this will choose the
776 number of processors detected on your system (e.g. 4 for a
777 4-core system). You can adjust this number as required to ensure
778 you don't overload your system while maximizing the use of your
779 bandwidth.
780 This has no effect when using the newer boto3 backend.
782 See also A NOTE ON AMAZON S3 below.
784 --s3-multipart-max-timeout
787 You can control the maximum time (in seconds) a multipart upload
788 can spend on uploading a single chunk to S3. This may be useful
789 if you find your system hanging on multipart uploads or if you'd
790 like to control the time variance when uploading to S3 to ensure
791 you kill connections to slow S3 endpoints.
792 This has no effect when using the newer boto3 backend.
794 See also A NOTE ON AMAZON S3 below.
796 --azure-blob-tier
799 Standard storage tier used for backup files (Hot|Cool|Archive).
800 --azure-max-single-put-size
803 Specify the number of the largest supported upload size where
804 the Azure library makes only one put call. If the content size
805 is known and below this value the Azure library will only
806 perform one put request to upload one block. The number is
807 expected to be in bytes.
808 --azure-max-block-size
811 Specify the number for the block size used by the Azure library
812 to upload blobs if it is split into multiple blocks. The
813 maximum block size the service supports is 104857600 (100MiB)
814 and the default is 4194304 (4MiB)
815 --azure-max-connections
818 Specify the number of maximum connections to transfer one blob
819 to Azure blob size exceeds 64MB. The default values is 2.
820 --scp-command command
823 (only ssh pexpect backend with --use-scp enabled) The command
824 will be used instead of "scp" to send or receive files. To list
825 and delete existing files, the sftp command is used.
826 See also A NOTE ON SSH BACKENDS section SSH pexpect backend.
827 --sftp-command command
830 (only ssh pexpect backend) The command will be used instead of
831 "sftp".
832 See also A NOTE ON SSH BACKENDS section SSH pexpect backend.
833 --short-filenames
836 If this option is specified, the names of the files duplicity
837 writes will be shorter (about 30 chars) but less understandable.
838 This may be useful when backing up to MacOS or another OS or FS
839 that doesn't support long filenames.
840 --sign-key key-id
843 This option can be used when backing up, restoring or verifying.
844 When backing up, all backup files will be signed with keyid key.
845 When restoring, duplicity will signal an error if any remote
846 file is not signed with the given key-id. The key-id can be
847 given in any of the formats supported by GnuPG; see gpg(1),
848 section "HOW TO SPECIFY A USER ID" for details. Should be
849 specified only once because currently only one signing key is
850 supported. Last entry overrides all other entries.
851 See also A NOTE ON SYMMETRIC ENCRYPTION AND SIGNING
852 --ssh-askpass
855 Tells the ssh backend to prompt the user for the remote system
856 password, if it was not defined in target url and no
857 FTP_PASSWORD env var is set. This password is also used for
858 passphrase-protected ssh keys.
859 --ssh-options options
862 Allows you to pass options to the ssh backend. Can be specified
863 multiple times or as a space separated options list. The
864 options list should be of the form "-oOpt1='parm1'
865 -oOpt2='parm2'" where the option string is quoted and the only
866 spaces allowed are between options. The option string will be
867 passed verbatim to both scp and sftp, whose command line syntax
868 differs slightly hence the options should therefore be given in
869 the long option format described in ssh_config(5).
870 example of a list:
872 duplicity --ssh-options="-oProtocol=2
874 -oIdentityFile='/my/backup/id'" /home/me
875 scp://user@host/some_dir
876 example with multiple parameters:
878 duplicity --ssh-options="-oProtocol=2" --ssh-
880 options="-oIdentityFile='/my/backup/id'" /home/me
881 scp://user@host/some_dir
882 NOTE: The ssh paramiko backend currently supports only the -i or
884 -oIdentityFile setting. If needed provide more host specific
885 options via ssh_config file.
886 --ssl-cacert-file file
889 (only webdav & lftp backend) Provide a cacert file for ssl
890 certificate verification.
891 See also A NOTE ON SSL CERTIFICATE VERIFICATION.
892 --ssl-cacert-path path/to/certs/
895 (only webdav backend and python 2.7.9+ OR lftp+webdavs and a
896 recent lftp) Provide a path to a folder containing cacert files
897 for ssl certificate verification.
898 See also A NOTE ON SSL CERTIFICATE VERIFICATION.
899 --ssl-no-check-certificate
902 (only webdav & lftp backend) Disable ssl certificate
903 verification.
904 See also A NOTE ON SSL CERTIFICATE VERIFICATION.
905 --swift-storage-policy
908 Use this storage policy when operating on Swift containers.
909 See also A NOTE ON SWIFT (OPENSTACK OBJECT STORAGE) ACCESS.
910 --metadata-sync-mode mode
913 This option defaults to 'partial', but you can set it to 'full'
914 Use 'partial' to avoid syncing metadata for backup chains that
915 you are not going to use. This saves time when restoring for
916 the first time, and lets you restore an old backup that was
917 encrypted with a different passphrase by supplying only the
918 target passphrase.
919 Use 'full' to sync metadata for all backup chains on the remote.
920 --tempdir directory
923 Use this existing directory for duplicity temporary files
924 instead of the system default, which is usually the /tmp
925 directory. This option supersedes any environment variable.
926 See also ENVIRONMENT VARIABLES.
927 -ttime, --time time, --restore-time time
930 Specify the time from which to restore or list files.
931 --time-separator char
934 Use char as the time separator in filenames instead of colon
935 (":").
936 --timeout seconds
939 Use seconds as the socket timeout value if duplicity begins to
940 timeout during network operations. The default is 30 seconds.
941 --use-agent
944 If this option is specified, then --use-agent is passed to the
945 GnuPG encryption process and it will try to connect to gpg-agent
946 before it asks for a passphrase for --encrypt-key or --sign-key
947 if needed.
948 Note: Contrary to previous versions of duplicity, this option
949 will also be honored by GnuPG 2 and newer versions. If GnuPG 2
950 is in use, duplicity passes the option --pinentry-mode=loopback
951 to the the gpg process unless --use-agent is specified on the
952 duplicity command line. This has the effect that GnuPG 2 uses
953 the agent only if --use-agent is given, just like GnuPG 1.
954 --verbosity level, -vlevel
957 Specify output verbosity level (log level). Named levels and
958 corresponding values are 0 Error, 2 Warning, 4 Notice (default),
959 8 Info, 9 Debug (noisiest).
960 level may also be
961 a character: e, w, n, i, d
962 a word: error, warning, notice, info, debug
963 The options -v4, -vn and -vnotice are functionally equivalent,
965 as are the mixed/upper-case versions -vN, -vNotice and -vNOTICE.
966 --version
969 Print duplicity's version and quit.
970 --volsize number
973 Change the volume size to number MB. Default is 200MB.
974
977 TMPDIR, TEMP, TMP
978 In decreasing order of importance, specifies the directory to
979 use for temporary files (inherited from Python's tempfile
980 module). Eventually the option --tempdir supercedes any of
981 these.
982 FTP_PASSWORD
984 Supported by most backends which are password capable. More
985 secure than setting it in the backend url (which might be
986 readable in the operating systems process listing to other users
987 on the same machine).
988 PASSPHRASE
990 This passphrase is passed to GnuPG. If this is not set, the user
991 will be prompted for the passphrase.
992 SIGN_PASSPHRASE
994 The passphrase to be used for --sign-key. If ommitted and sign
995 key is also one of the keys to encrypt against PASSPHRASE will
996 be reused instead. Otherwise, if passphrase is needed but not
997 set the user will be prompted for it.
998
1001 Duplicity uses the URL format (as standard as possible) to define data
1002 locations. The generic format for a URL is:
1003 scheme://[user[:password]@]host[:port]/[/]path
1005 It is not recommended to expose the password on the command line since
1007 it could be revealed to anyone with permissions to do process listings,
1008 it is permitted however. Consider setting the environment variable
1009 FTP_PASSWORD instead, which is used by most, if not all backends,
1010 regardless of it's name.
1011 In protocols that support it, the path may be preceded by a single
1013 slash, '/path', to represent a relative path to the target home
1014 directory, or preceded by a double slash, '//path', to represent an
1015 absolute filesystem path.
1016 Note:
1018 Scheme (protocol) access may be provided by more than one
1019 backend. In case the default backend is buggy or simply not
1020 working in a specific case it might be worth trying an
1021 alternative implementation. Alternative backends can be
1022 selected by prefixing the scheme with the name of the
1023 alternative backend e.g. ncftp+ftp:// and are mentioned below
1024 the scheme's syntax summary.
1025 Formats of each of the URL schemes follow:
1028 Amazon Drive Backend
1031 ad://some_dir
1033 See also A NOTE ON AMAZON DRIVE
1035 Azure
1037 azure://container-name
1039 See also A NOTE ON AZURE ACCESS
1041 B2
1043 b2://account_id[:application_key]@bucket_name/[folder/]
1045 Cloud Files (Rackspace)
1047 cf+http://container_name
1049 See also A NOTE ON CLOUD FILES ACCESS
1051 Dropbox
1053 dpbx:///some_dir
1055 Make sure to read A NOTE ON DROPBOX ACCESS first!
1057 Local file path
1059 file://[relative|/absolute]/local/path
1061 FISH (Files transferred over Shell protocol) over ssh
1063 fish://user[:password]@other.host[:port]/[relative|/absolute]_path
1065 FTP
1067 ftp[s]://user[:password]@other.host[:port]/some_dir
1069 NOTE: use lftp+, ncftp+ prefixes to enforce a specific backend,
1071 default is lftp+ftp://...
1072 Google Docs
1074 gdocs://user[:password]@other.host/some_dir
1076 NOTE: use pydrive+, gdata+ prefixes to enforce a specific
1078 backend, default is pydrive+gdocs://...
1079 Google Cloud Storage
1081 gs://bucket[/prefix]
1083 HSI
1085 hsi://user[:password]@other.host/some_dir
1087 hubiC
1089 cf+hubic://container_name
1091 See also A NOTE ON HUBIC
1093 IMAP email storage
1095 imap[s]://user[:password]@host.com[/from_address_prefix]
1097 See also A NOTE ON IMAP
1099 Mega cloud storage
1101 mega://user[:password]@mega.co.nz/some_dir
1103 OneDrive Backend
1105 onedrive://some_dir
1107 Par2 Wrapper Backend
1109 par2+scheme://[user[:password]@]host[:port]/[/]path
1111 See also A NOTE ON PAR2 WRAPPER BACKEND
1113 Rsync via daemon
1115 rsync://user[:password]@host.com[:port]::[/]module/some_dir
1117 Rsync over ssh (only key auth)
1119 rsync://user@host.com[:port]/[relative|/absolute]_path
1121 S3 storage (Amazon)
1123 s3://host[:port]/bucket_name[/prefix]
1125 s3+http://bucket_name[/prefix]
1126 boto3+s3://bucket_name[/prefix]
1127 See also A NOTE ON AMAZON S3 and A NOTE ON EUROPEAN S3 BUCKETS
1129 below.
1130 SCP/SFTP access
1132 scp://.. or
1134 sftp://user[:password]@other.host[:port]/[relative|/absolute]_path
1135 defaults are paramiko+scp:// and paramiko+sftp://
1137 alternatively try pexpect+scp://, pexpect+sftp://,
1138 lftp+sftp://
1139 See also --ssh-askpass, --ssh-options and A NOTE ON SSH
1140 BACKENDS.
1141 Swift (Openstack)
1143 swift://container_name[/prefix]
1145 See also A NOTE ON SWIFT (OPENSTACK OBJECT STORAGE)
1147 ACCESS
1148 Public Cloud Archive (OVH)
1150 pca://container_name[/prefix]
1152 See also A NOTE ON PCA ACCESS
1154 Tahoe-LAFS
1156 tahoe://alias/directory
1158 WebDAV
1160 webdav[s]://user[:password]@other.host[:port]/some_dir
1162 alternatively try lftp+webdav[s]://
1164 pydrive
1166 pydrive://<service account' email
1168 address>@developer.gserviceaccount.com/some_dir
1169 See also A NOTE ON PYDRIVE BACKEND below.
1171 multi
1173 multi:///path/to/config.json
1175 See also A NOTE ON MULTI BACKEND below.
1177 MediaFire
1179 mf://user[:password]@mediafire.com/some_dir
1181 See also A NOTE ON MEDIAFIRE BACKEND below.
1183
1186 duplicity uses time strings in two places. Firstly, many of the files
1187 duplicity creates will have the time in their filenames in the w3
1188 datetime format as described in a w3 note at http://www.w3.org/TR/NOTE-
1189 datetime. Basically they look like "2001-07-15T04:09:38-07:00", which
1190 means what it looks like. The "-07:00" section means the time zone is
1191 7 hours behind UTC.
1192 Secondly, the -t, --time, and --restore-time options take a time
1194 string, which can be given in any of several formats:
1195 1. the string "now" (refers to the current time)
1197 2. a sequences of digits, like "123456890" (indicating the time in
1199 seconds after the epoch)
1200 3. A string like "2002-01-25T07:00:00+02:00" in datetime format
1202 4. An interval, which is a number followed by one of the characters
1204 s, m, h, D, W, M, or Y (indicating seconds, minutes, hours,
1205 days, weeks, months, or years respectively), or a series of such
1206 pairs. In this case the string refers to the time that preceded
1207 the current time by the length of the interval. For instance,
1208 "1h78m" indicates the time that was one hour and 78 minutes ago.
1209 The calendar here is unsophisticated: a month is always 30 days,
1210 a year is always 365 days, and a day is always 86400 seconds.
1211 5. A date format of the form YYYY/MM/DD, YYYY-MM-DD, MM/DD/YYYY, or
1213 MM-DD-YYYY, which indicates midnight on the day in question,
1214 relative to the current time zone settings. For instance,
1215 "2002/3/5", "03-05-2002", and "2002-3-05" all mean March 5th,
1216 2002.
1217
1220 When duplicity is run, it searches through the given source directory
1221 and backs up all the files specified by the file selection system. The
1222 file selection system comprises a number of file selection conditions,
1223 which are set using one of the following command line options:
1224 --exclude
1225 --exclude-device-files
1226 --exclude-filelist
1227 --exclude-regexp
1228 --include
1229 --include-filelist
1230 --include-regexp
1231 Each file selection condition either matches or doesn't match a given
1232 file. A given file is excluded by the file selection system exactly
1233 when the first matching file selection condition specifies that the
1234 file be excluded; otherwise the file is included.
1235 For instance,
1237 duplicity --include /usr --exclude /usr /usr
1239 scp://user@host/backup
1240 is exactly the same as
1242 duplicity /usr scp://user@host/backup
1244 because the include and exclude directives match exactly the same
1246 files, and the --include comes first, giving it precedence. Similarly,
1247 duplicity --include /usr/local/bin --exclude /usr/local /usr
1249 scp://user@host/backup
1250 would backup the /usr/local/bin directory (and its contents), but not
1252 /usr/local/doc.
1253 The include, exclude, include-filelist, and exclude-filelist options
1255 accept some extended shell globbing patterns. These patterns can
1256 contain *, **, ?, and [...] (character ranges). As in a normal shell,
1257 * can be expanded to any string of characters not containing "/", ?
1258 expands to any character except "/", and [...] expands to a single
1259 character of those characters specified (ranges are acceptable). The
1260 new special pattern, **, expands to any string of characters whether or
1261 not it contains "/". Furthermore, if the pattern starts with
1262 "ignorecase:" (case insensitive), then this prefix will be removed and
1263 any character in the string can be replaced with an upper- or lowercase
1264 version of itself.
1265 Remember that you may need to quote these characters when typing them
1267 into a shell, so the shell does not interpret the globbing patterns
1268 before duplicity sees them.
1269 The --exclude pattern option matches a file if:
1271 1. pattern can be expanded into the file's filename, or
1273 2. the file is inside a directory matched by the option.
1274 Conversely, the --include pattern matches a file if:
1276 1. pattern can be expanded into the file's filename, or
1278 2. the file is inside a directory matched by the option, or
1279 3. the file is a directory which contains a file matched by the
1280 option.
1281 For example,
1283 --exclude /usr/local
1285 matches e.g. /usr/local, /usr/local/lib, and /usr/local/lib/netscape.
1287 It is the same as --exclude /usr/local --exclude '/usr/local/**'.
1288 On the other hand
1290 --include /usr/local
1292 specifies that /usr, /usr/local, /usr/local/lib, and
1294 /usr/local/lib/netscape (but not /usr/doc) all be backed up. Thus you
1295 don't have to worry about including parent directories to make sure
1296 that included subdirectories have somewhere to go.
1297 Finally,
1299 --include ignorecase:'/usr/[a-z0-9]foo/*/**.py'
1301 would match a file like /usR/5fOO/hello/there/world.py. If it did
1303 match anything, it would also match /usr. If there is no existing file
1304 that the given pattern can be expanded into, the option will not match
1305 /usr alone.
1306 The --include-filelist, and --exclude-filelist, options also introduce
1308 file selection conditions. They direct duplicity to read in a text
1309 file (either ASCII or UTF-8), each line of which is a file
1310 specification, and to include or exclude the matching files. Lines are
1311 separated by newlines or nulls, depending on whether the --null-
1312 separator switch was given. Each line in the filelist will be
1313 interpreted as a globbing pattern the way --include and --exclude
1314 options are interpreted, except that lines starting with "+ " are
1315 interpreted as include directives, even if found in a filelist
1316 referenced by --exclude-filelist. Similarly, lines starting with "- "
1317 exclude files even if they are found within an include filelist.
1318 For example, if file "list.txt" contains the lines:
1320 /usr/local
1322 - /usr/local/doc
1323 /usr/local/bin
1324 + /var
1325 - /var
1326 then --include-filelist list.txt would include /usr, /usr/local, and
1328 /usr/local/bin. It would exclude /usr/local/doc,
1329 /usr/local/doc/python, etc. It would also include /usr/local/man, as
1330 this is included within /user/local. Finally, it is undefined what
1331 happens with /var. A single file list should not contain conflicting
1332 file specifications.
1333 Each line in the filelist will also be interpreted as a globbing
1335 pattern the way --include and --exclude options are interpreted. For
1336 instance, if the file "list.txt" contains the lines:
1337 dir/foo
1339 + dir/bar
1340 - **
1341 Then --include-filelist list.txt would be exactly the same as
1343 specifying --include dir/foo --include dir/bar --exclude ** on the
1344 command line.
1345 Finally, the --include-regexp and --exclude-regexp options allow files
1347 to be included and excluded if their filenames match a python regular
1348 expression. Regular expression syntax is too complicated to explain
1349 here, but is covered in Python's library reference. Unlike the
1350 --include and --exclude options, the regular expression options don't
1351 match files containing or contained in matched files. So for instance
1352 --include '[0-9]{7}(?!foo)'
1354 matches any files whose full pathnames contain 7 consecutive digits
1356 which aren't followed by 'foo'. However, it wouldn't match /home even
1357 if /home/ben/1234567 existed.
1358
1361 1. The API Keys used for Amazon Drive have not been granted
1362 production limits. Amazon do not say what the development
1363 limits are and are not replying to to requests to whitelist
1364 duplicity. A related tool, acd_cli, was demoted to development
1365 limits, but continues to work fine except for cases of excessive
1366 usage. If you experience throttling and similar issues with
1367 Amazon Drive using this backend, please report them to the
1368 mailing list.
1369 2. If you previously used the acd+acdcli backend, it is strongly
1371 recommended to update to the ad backend instead, since it
1372 interfaces directly with Amazon Drive. You will need to setup
1373 the OAuth once again, but can otherwise keep your backups and
1374 config.
1375
1378 When backing up to Amazon S3, two backend implementations are
1379 available. The schemes "s3" and "s3+http" are implemented using the
1380 older boto library, which has been deprecated and is no longer
1381 supported. The "boto3+s3" scheme is based on the newer boto3 library.
1382 This new backend fixes several known limitations in the older backend,
1383 which have crept in as Amazon S3 has evolved while the deprecated boto
1384 library has not kept up.
1385 The boto3 backend should behave largely the same as the older S3
1387 backend, but there are some differences in the handling of some of the
1388 "S3" options. Additionally, there are some compatibility differences
1389 with the new backed. Because of these reasons, both backends have been
1390 retained for the time being. See the documentation for specific
1391 options regarding differences related to each backend.
1392 The boto3 backend does not support bucket creation. This is a
1394 deliberate choice which simplifies the code, and side steps problems
1395 related to region selection. Additionally, it is probably not a good
1396 practice to give your backup role bucket creation rights. In most
1397 cases the role used for backups should probably be limited to specific
1398 buckets.
1399 The boto3 backend only supports newer domain style buckets. Amazon is
1401 moving to deprecate the older bucket style, so migration is
1402 recommended. Use the older s3 backend for compatibility with backups
1403 stored in buckets using older naming conventions.
1404 The boto3 backend does not currently support initiating restores from
1406 the glacier storage class. When restoring a backup from glacier or
1407 glacier deep archive, the backup files must first be restored out of
1408 band. There are multiple options when restoring backups from cold
1409 storage, which vary in both cost and speed. See Amazon's documentation
1410 for details.
1411
1414 The Azure backend requires the Microsoft Azure Storage SDK for Python
1415 to be installed on the system. See REQUIREMENTS above.
1416 It uses environment variables for authentification: AZURE_ACCOUNT_NAME
1418 (required), AZURE_ACCOUNT_KEY (optional), AZURE_SHARED_ACCESS_SIGNATURE
1419 (optional). One of AZURE_ACCOUNT_KEY or AZURE_SHARED_ACCESS_SIGNATURE
1420 is required.
1421 A container name must be a valid DNS name, conforming to the following
1423 naming rules:
1424 1. Container names must start with a letter or number, and
1427 can contain only letters, numbers, and the dash (-)
1428 character.
1429 2. Every dash (-) character must be immediately preceded and
1431 followed by a letter or number; consecutive dashes are
1432 not permitted in container names.
1433 3. All letters in a container name must be lowercase.
1435 4. Container names must be from 3 through 63 characters
1437 long.
1438
1441 Pyrax is Rackspace's next-generation Cloud management API, including
1442 Cloud Files access. The cfpyrax backend requires the pyrax library to
1443 be installed on the system. See REQUIREMENTS above.
1444 Cloudfiles is Rackspace's now deprecated implementation of OpenStack
1446 Object Storage protocol. Users wishing to use Duplicity with Rackspace
1447 Cloud Files should migrate to the new Pyrax plugin to ensure support.
1448 The backend requires python-cloudfiles to be installed on the system.
1450 See REQUIREMENTS above.
1451 It uses three environment variables for authentification:
1453 CLOUDFILES_USERNAME (required), CLOUDFILES_APIKEY (required),
1454 CLOUDFILES_AUTHURL (optional)
1455 If CLOUDFILES_AUTHURL is unspecified it will default to the value
1457 provided by python-cloudfiles, which points to rackspace, hence this
1458 value must be set in order to use other cloud files providers.
1459
1462 1. First of all Dropbox backend requires valid authentication
1463 token. It should be passed via DPBX_ACCESS_TOKEN environment
1464 variable.
1465 To obtain it please create 'Dropbox API' application at:
1466 https://www.dropbox.com/developers/apps/create
1467 Then visit app settings and just use 'Generated access token'
1468 under OAuth2 section.
1469 Alternatively you can let duplicity generate access token
1470 itself. In such case temporary export DPBX_APP_KEY ,
1471 DPBX_APP_SECRET using values from app settings page and run
1472 duplicity interactively.
1473 It will print the URL that you need to open in the browser to
1474 obtain OAuth2 token for the application. Just follow on-screen
1475 instructions and then put generated token to DPBX_ACCESS_TOKEN
1476 variable. Once done, feel free to unset DPBX_APP_KEY and
1477 DPBX_APP_SECRET
1478 2. "some_dir" must already exist in the Dropbox folder. Depending
1481 on access token kind it may be:
1482 Full Dropbox: path is absolute and starts from 'Dropbox'
1483 root folder.
1484 App Folder: path is related to application folder.
1485 Dropbox client will show it in ~/Dropbox/Apps/<app-name>
1486 3. When using Dropbox for storage, be aware that all files,
1489 including the ones in the Apps folder, will be synced to all
1490 connected computers. You may prefer to use a separate Dropbox
1491 account specially for the backups, and not connect any computers
1492 to that account. Alternatively you can configure selective sync
1493 on all computers to avoid syncing of backup files
1494
1497 Amazon S3 provides the ability to choose the location of a bucket upon
1498 its creation. The purpose is to enable the user to choose a location
1499 which is better located network topologically relative to the user,
1500 because it may allow for faster data transfers.
1501 duplicity will create a new bucket the first time a bucket access is
1503 attempted. At this point, the bucket will be created in Europe if
1504 --s3-european-buckets was given. For reasons having to do with how the
1505 Amazon S3 service works, this also requires the use of the --s3-use-
1506 new-style option. This option turns on subdomain based bucket
1507 addressing in S3. The details are beyond the scope of this man page,
1508 but it is important to know that your bucket must not contain upper
1509 case letters or any other characters that are not valid parts of a
1510 hostname. Consequently, for reasons of backwards compatibility, use of
1511 subdomain based bucket addressing is not enabled by default.
1512 Note that you will need to use --s3-use-new-style for all operations on
1514 European buckets; not just upon initial creation.
1515 You only need to use --s3-european-buckets upon initial creation, but
1517 you may may use it at all times for consistency.
1518 Further note that when creating a new European bucket, it can take a
1520 while before the bucket is fully accessible. At the time of this
1521 writing it is unclear to what extent this is an expected feature of
1522 Amazon S3, but in practice you may experience timeouts, socket errors
1523 or HTTP errors when trying to upload files to your newly created
1524 bucket. Give it a few minutes and the bucket should function normally.
1525
1528 Filename prefixes can be used in multi backend with mirror mode to
1529 define affinity rules. They can also be used in conjunction with S3
1530 lifecycle rules to transition archive files to Glacier, while keeping
1531 metadata (signature and manifest files) on S3.
1532 Duplicity does not require access to archive files except when
1534 restoring from backup.
1535
1538 Support for Google Cloud Storage relies on its Interoperable Access,
1539 which must be enabled for your account. Once enabled, you can generate
1540 Interoperable Storage Access Keys and pass them to duplicity via the
1541 GS_ACCESS_KEY_ID and GS_SECRET_ACCESS_KEY environment variables.
1542 Alternatively, you can run gsutil config -a to have the Google Cloud
1543 Storage utility populate the ~/.boto configuration file.
1544 Enable Interoperable Access:
1546 https://code.google.com/apis/console#:storage
1547 Create Access Keys:
1548 https://code.google.com/apis/console#:storage:legacy
1549
1552 The hubic backend requires the pyrax library to be installed on the
1553 system. See REQUIREMENTS above. You will need to set your credentials
1554 for hubiC in a file called ~/.hubic_credentials, following this
1555 pattern:
1556 [hubic]
1558 email = your_email
1559 password = your_password
1560 client_id = api_client_id
1561 client_secret = api_secret_key
1562 redirect_uri = http://localhost/
1563
1566 An IMAP account can be used as a target for the upload. The userid may
1567 be specified and the password will be requested.
1568 The from_address_prefix may be specified (and probably should be). The
1570 text will be used as the "From" address in the IMAP server. Then on a
1571 restore (or list) command the from_address_prefix will distinguish
1572 between different backups.
1573
1576 The multi backend allows duplicity to combine the storage available in
1577 more than one backend store (e.g., you can store across a google drive
1578 account and a onedrive account to get effectively the combined storage
1579 available in both). The URL path specifies a JSON formated config file
1580 containing a list of the backends it will use. The URL may also specify
1581 "query" parameters to configure overall behavior. Each element of the
1582 list must have a "url" element, and may also contain an optional
1583 "description" and an optional "env" list of environment variables used
1584 to configure that backend.
1585 Query Parameters
1587 Query parameters come after the file URL in standard HTTP format for
1588 example:
1589 multi:///path/to/config.json?mode=mirror&onfail=abort
1590 multi:///path/to/config.json?mode=stripe&onfail=continue
1591 multi:///path/to/config.json?onfail=abort&mode=stripe
1592 multi:///path/to/config.json?onfail=abort
1593 Order does not matter, however unrecognized parameters are considered
1594 an error.
1595 mode=stripe
1597 This mode (the default) performs round-robin access to the list
1598 of backends. In this mode, all backends must be reliable as a
1599 loss of one means a loss of one of the archive files.
1600 mode=mirror
1602 This mode accesses backends as a RAID1-store, storing every file
1603 in every backend and reading files from the first-successful
1604 backend. A loss of any backend should result in no failure.
1605 Note that backends added later will only get new files and may
1606 require a manual sync with one of the other operating ones.
1607 onfail=continue
1609 This setting (the default) continues all write operations in as
1610 best-effort. Any failure results in the next backend tried.
1611 Failure is reported only when all backends fail a given
1612 operation with the error result from the last failure.
1613 onfail=abort
1615 This setting considers any backend write failure as a
1616 terminating condition and reports the error. Data reading and
1617 listing operations are independent of this and will try with the
1618 next backend on failure.
1619 JSON File Example
1621 [
1622 {
1623 "description": "a comment about the backend"
1624 "url": "abackend://myuser@domain.com/backup",
1625 "env": [
1626 {
1627 "name" : "MYENV",
1628 "value" : "xyz"
1629 },
1630 {
1631 "name" : "FOO",
1632 "value" : "bar"
1633 }
1634 ],
1635 "prefixes": ["prefix1_", "prefix2_"]
1636 },
1637 {
1638 "url": "file:///path/to/dir"
1639 }
1640 ]
1641
1644 Par2 Wrapper Backend can be used in combination with all other backends
1645 to create recovery files. Just add par2+ before a regular scheme (e.g.
1646 par2+ftp://user@host/dir or par2+s3+http://bucket_name ). This will
1647 create par2 recovery files for each archive and upload them all to the
1648 wrapped backend.
1649 Before restoring, archives will be verified. Corrupt archives will be
1651 repaired on the fly if there are enough recovery blocks available.
1652 Use --par2-redundancy percent to adjust the size (and redundancy) of
1654 recovery files in percent.
1655
1658 The pydrive backend requires Python PyDrive package to be installed on
1659 the system. See REQUIREMENTS above.
1660 There are two ways to use PyDrive: with a regular account or with a
1662 "service account". With a service account, a separate account is
1663 created, that is only accessible with Google APIs and not a web login.
1664 With a regular account, you can store backups in your normal Google
1665 Drive.
1666 To use a service account, go to the Google developers console at
1668 https://console.developers.google.com. Create a project, and make sure
1669 Drive API is enabled for the project. Under "APIs and auth", click
1670 Create New Client ID, then select Service Account with P12 key.
1671 Download the .p12 key file of the account and convert it to the .pem
1673 format:
1674 openssl pkcs12 -in XXX.p12 -nodes -nocerts > pydriveprivatekey.pem
1675 The content of .pem file should be passed to GOOGLE_DRIVE_ACCOUNT_KEY
1677 environment variable for authentification.
1678 The email address of the account will be used as part of URL. See URL
1680 FORMAT above.
1681 The alternative is to use a regular account. To do this, start as
1683 above, but when creating a new Client ID, select "Installed
1684 application" of type "Other". Create a file with the following content,
1685 and pass its filename in the GOOGLE_DRIVE_SETTINGS environment
1686 variable:
1687 client_config_backend: settings
1689 client_config:
1690 client_id: <Client ID from developers' console>
1691 client_secret: <Client secret from developers' console>
1692 save_credentials: True
1693 save_credentials_backend: file
1694 save_credentials_file: <filename to cache credentials>
1695 get_refresh_token: True
1696 In this scenario, the username and host parts of the URL play no role;
1698 only the path matters. During the first run, you will be prompted to
1699 visit an URL in your browser to grant access to your drive. Once
1700 granted, you will receive a verification code to paste back into
1701 Duplicity. The credentials are then cached in the file references above
1702 for future use.
1703
1706 The ssh backends support sftp and scp/ssh transport protocols. This is
1707 a known user-confusing issue as these are fundamentally different. If
1708 you plan to access your backend via one of those please inform yourself
1709 about the requirements for a server to support sftp or scp/ssh access.
1710 To make it even more confusing the user can choose between several ssh
1711 backends via a scheme prefix: paramiko+ (default), pexpect+, lftp+... .
1712 paramiko & pexpect support --use-scp, --ssh-askpass and --ssh-options.
1713 Only the pexpect backend allows to define --scp-command and --sftp-
1714 command.
1715 SSH paramiko backend (default) is a complete reimplementation of ssh
1717 protocols natively in python. Advantages are speed and maintainability.
1718 Minor disadvantage is that extra packages are needed as listed in
1719 REQUIREMENTS above. In sftp (default) mode all operations are done via
1720 the according sftp commands. In scp mode ( --use-scp ) though scp
1721 access is used for put/get operations but listing is done via ssh
1722 remote shell.
1723 SSH pexpect backend is the legacy ssh backend using the command line
1725 ssh binaries via pexpect. Older versions used scp for get and put
1726 operations and sftp for list and delete operations. The current
1727 version uses sftp for all four supported operations, unless the --use-
1728 scp option is used to revert to old behavior.
1729 SSH lftp backend is simply there because lftp can interact with the ssh
1731 cmd line binaries. It is meant as a last resort in case the above
1732 options fail for some reason.
1733 Why use sftp instead of scp? The change to sftp was made in order to
1735 allow the remote system to chroot the backup, thus providing better
1736 security and because it does not suffer from shell quoting issues like
1737 scp. Scp also does not support any kind of file listing, so sftp or
1738 ssh access will always be needed in addition for this backend mode to
1739 work properly. Sftp does not have these limitations but needs an sftp
1740 service running on the backend server, which is sometimes not an
1741 option.
1742
1745 Certificate verification as implemented right now [02.2016] only in the
1746 webdav and lftp backends. older pythons 2.7.8- and older lftp binaries
1747 need a file based database of certification authority certificates
1748 (cacert file).
1749 Newer python 2.7.9+ and recent lftp versions however support the system
1750 default certificates (usually in /etc/ssl/certs) and also giving an
1751 alternative ca cert folder via --ssl-cacert-path.
1752 The cacert file has to be a PEM formatted text file as currently
1754 provided by the CURL project. See
1755 http://curl.haxx.se/docs/caextract.html
1757 After creating/retrieving a valid cacert file you should copy it to
1759 either
1760 ~/.duplicity/cacert.pem
1762 ~/duplicity_cacert.pem
1763 /etc/duplicity/cacert.pem
1764 Duplicity searches it there in the same order and will fail if it can't
1766 find it. You can however specify the option --ssl-cacert-file <file>
1767 to point duplicity to a copy in a different location.
1768 Finally there is the --ssl-no-check-certificate option to disable
1770 certificate verification alltogether, in case some ssl library is
1771 missing or verification is not wanted. Use it with care, as even with
1772 self signed servers manually providing the private ca certificate is
1773 definitely the safer option.
1774
1777 Swift is the OpenStack Object Storage service.
1778 The backend requires python-switclient to be installed on the system.
1779 python-keystoneclient is also needed to use OpenStack's Keystone
1780 Identity service. See REQUIREMENTS above.
1781 It uses following environment variables for authentification:
1783 SWIFT_USERNAME (required), SWIFT_PASSWORD (required), SWIFT_AUTHURL
1784 (required), SWIFT_USERID (required, only for IBM Bluemix
1785 ObjectStorage), SWIFT_TENANTID (required, only for IBM Bluemix
1786 ObjectStorage), SWIFT_REGIONNAME (required, only for IBM Bluemix
1787 ObjectStorage), SWIFT_TENANTNAME (optional, the tenant can be included
1788 in the username)
1789 If the user was previously authenticated, the following environment
1791 variables can be used instead: SWIFT_PREAUTHURL (required),
1792 SWIFT_PREAUTHTOKEN (required)
1793 If SWIFT_AUTHVERSION is unspecified, it will default to version 1.
1795
1798 PCA is a long-term data archival solution by OVH. It runs a slightly
1799 modified version of Openstack Swift introducing latency in the data
1800 retrieval process. It is a good pick for a multi backend configuration
1801 where receiving volumes while an other backend is used to store
1802 manifests and signatures.
1803 The backend requires python-switclient to be installed on the system.
1805 python-keystoneclient is also needed to interact with OpenStack's
1806 Keystone Identity service. See REQUIREMENTS above.
1807 It uses following environment variables for authentification:
1809 PCA_USERNAME (required), PCA_PASSWORD (required), PCA_AUTHURL
1810 (required), PCA_USERID (optional), PCA_TENANTID (optional, but either
1811 the tenant name or tenant id must be supplied) PCA_REGIONNAME
1812 (optional), PCA_TENANTNAME (optional, but either the tenant name or
1813 tenant id must be supplied)
1814 If the user was previously authenticated, the following environment
1816 variables can be used instead: PCA_PREAUTHURL (required),
1817 PCA_PREAUTHTOKEN (required)
1818 If PCA_AUTHVERSION is unspecified, it will default to version 2.
1820
1823 This backend requires mediafire python library to be installed on the
1824 system. See REQUIREMENTS.
1825 Use URL escaping for username (and password, if provided via command
1827 line):
1828 mf://duplicity%40example.com@mediafire.com/some_folder
1831 The destination folder will be created for you if it does not exist.
1833
1836 Signing and symmetrically encrypt at the same time with the gpg binary
1837 on the command line, as used within duplicity, is a specifically
1838 challenging issue. Tests showed that the following combinations proved
1839 working.
1840 1. Setup gpg-agent properly. Use the option --use-agent and enter both
1842 passphrases (symmetric and sign key) in the gpg-agent's dialog.
1843 2. Use a PASSPHRASE for symmetric encryption of your choice but the
1845 signing key has an empty passphrase.
1846 3. The used PASSPHRASE for symmetric encryption and the passphrase of
1848 the signing key are identical.
1849
1852 Hard links currently unsupported (they will be treated as non-linked
1853 regular files).
1854 Bad signatures will be treated as empty instead of logging appropriate
1856 error message.
1857
1860 This section describes duplicity's basic operation and the format of
1861 its data files. It should not necessary to read this section to use
1862 duplicity.
1863 The files used by duplicity to store backup data are tarfiles in GNU
1865 tar format. They can be produced independently by rdiffdir(1). For
1866 incremental backups, new files are saved normally in the tarfile. But
1867 when a file changes, instead of storing a complete copy of the file,
1868 only a diff is stored, as generated by rdiff(1). If a file is deleted,
1869 a 0 length file is stored in the tar. It is possible to restore a
1870 duplicity archive "manually" by using tar and then cp, rdiff, and rm as
1871 necessary. These duplicity archives have the extension difftar.
1872 Both full and incremental backup sets have the same format. In effect,
1874 a full backup set is an incremental one generated from an empty
1875 signature (see below). The files in full backup sets will start with
1876 duplicity-full while the incremental sets start with duplicity-inc.
1877 When restoring, duplicity applies patches in order, so deleting, for
1878 instance, a full backup set may make related incremental backup sets
1879 unusable.
1880 In order to determine which files have been deleted, and to calculate
1882 diffs for changed files, duplicity needs to process information about
1883 previous sessions. It stores this information in the form of tarfiles
1884 where each entry's data contains the signature (as produced by rdiff)
1885 of the file instead of the file's contents. These signature sets have
1886 the extension sigtar.
1887 Signature files are not required to restore a backup set, but without
1889 an up-to-date signature, duplicity cannot append an incremental backup
1890 to an existing archive.
1891 To save bandwidth, duplicity generates full signature sets and
1893 incremental signature sets. A full signature set is generated for each
1894 full backup, and an incremental one for each incremental backup. These
1895 start with duplicity-full-signatures and duplicity-new-signatures
1896 respectively. These signatures will be stored both locally and
1897 remotely. The remote signatures will be encrypted if encryption is
1898 enabled. The local signatures will not be encrypted and stored in the
1899 archive dir (see --archive-dir ).
1900
1903 Duplicity requires a POSIX-like operating system with a python
1904 interpreter version 2.6+ installed. It is best used under GNU/Linux.
1905 Some backends also require additional components (probably available as
1907 packages for your specific platform):
1908 Amazon Drive backend
1910 python-requests - http://python-requests.org
1911 python-requests-oauthlib - https://github.com/requests/requests-
1912 oauthlib
1913 azure backend (Azure Blob Storage Service)
1915 Microsoft Azure Storage SDK for Python -
1916 https://pypi.python.org/pypi/azure-storage/
1917 boto backend (S3 Amazon Web Services, Google Cloud Storage)
1919 boto version 2.0+ - http://github.com/boto/boto
1920 cfpyrax backend (Rackspace Cloud) and hubic backend (hubic.com)
1922 Rackspace CloudFiles Pyrax API -
1923 http://docs.rackspace.com/sdks/guide/content/python.html
1924 dpbx backend (Dropbox)
1926 Dropbox Python SDK -
1927 https://www.dropbox.com/developers/reference/sdk
1928 gdocs gdata backend (legacy Google Docs backend)
1930 Google Data APIs Python Client Library -
1931 http://code.google.com/p/gdata-python-client/
1932 gdocs pydrive backend(default)
1934 see pydrive backend
1935 gio backend (Gnome VFS API)
1937 PyGObject - http://live.gnome.org/PyGObject
1938 D-Bus (dbus)- http://www.freedesktop.org/wiki/Software/dbus
1939 lftp backend (needed for ftp, ftps, fish [over ssh] - also supports
1941 sftp, webdav[s])
1942 LFTP Client - http://lftp.yar.ru/
1943 mega backend (mega.co.nz)
1945 megatools client - https://github.com/megous/megatools
1946 multi backend
1948 Multi -- store to more than one backend
1949 (also see A NOTE ON MULTI BACKEND ) below.
1950 ncftp backend (ftp, select via ncftp+ftp://)
1952 NcFTP - http://www.ncftp.com/
1953 OneDrive backend (Microsoft OneDrive)
1955 python-requests - http://python-requests.org
1956 python-requests-oauthlib - https://github.com/requests/requests-
1957 oauthlib
1958 Par2 Wrapper Backend
1960 par2cmdline - http://parchive.sourceforge.net/
1961 pydrive backend
1963 PyDrive -- a wrapper library of google-api-python-client -
1964 https://pypi.python.org/pypi/PyDrive
1965 (also see A NOTE ON PYDRIVE BACKEND ) below.
1966 rsync backend
1968 rsync client binary - http://rsync.samba.org/
1969 ssh paramiko backend (default)
1971 paramiko (SSH2 for python) -
1972 http://pypi.python.org/pypi/paramiko (downloads);
1973 http://github.com/paramiko/paramiko (project page)
1974 pycrypto (Python Cryptography Toolkit) -
1975 http://www.dlitz.net/software/pycrypto/
1976 ssh pexpect backend
1978 sftp/scp client binaries OpenSSH - http://www.openssh.com/
1979 Python pexpect module -
1980 http://pexpect.sourceforge.net/pexpect.html
1981 swift backend (OpenStack Object Storage)
1983 Python swiftclient module - https://github.com/openstack/python-
1984 swiftclient/
1985 Python keystoneclient module -
1986 https://github.com/openstack/python-keystoneclient/
1987 webdav backend
1989 certificate authority database file for ssl certificate
1990 verification of HTTPS connections -
1991 http://curl.haxx.se/docs/caextract.html
1992 (also see A NOTE ON SSL CERTIFICATE VERIFICATION).
1993 Python kerberos module for kerberos authentication -
1994 https://github.com/02strich/pykerberos
1995 MediaFire backend
1997 MediaFire Python Open SDK -
1998 https://pypi.python.org/pypi/mediafire/
1999
2002 Original Author - Ben Escoto <bescoto@stanford.edu>
2003 Current Maintainer - Kenneth Loafman <kenneth@loafman.com>
2005 Continuous Contributors
2007 Edgar Soldin, Mike Terry
2008 Most backends were contributed individually. Information about their
2010 authorship may be found in the according file's header.
2011 Also we'd like to thank everybody posting issues to the mailing list or
2013 on launchpad, sending in patches or contributing otherwise. Duplicity
2014 wouldn't be as stable and useful if it weren't for you.
2015 A special thanks goes to rsync.net, a Cloud Storage provider with
2017 explicit support for duplicity, for several monetary donations and for
2018 providing a special "duplicity friends" rate for their offsite backup
2019 service. Email info@rsync.net for details.
2020
2023 rdiffdir(1), python(1), rdiff(1), rdiff-backup(1).
2024Version 0.8.08 December 08, 2019 DUPLICITY(1)