1dd_rescue(1)           Data recovery and protection tool          dd_rescue(1)
2
3
4

NAME

6       dd_rescue - Data recovery and protection tool
7

SYNOPSIS

9       dd_rescue [options] infile outfile
10       dd_rescue [options] [-2/-3/-4/-z/-Z seed/seedfile] outfile
11       dd_rescue [options] [--shred2/--shred3/--shred4/--random/--frandom
12       seed/seedfile] outfile
13

DESCRIPTION

15       dd_rescue is a tool that copies data from a source (file, block device,
16       pipe, ...)  to one (or several) output file(s).
17
18       If input and output files are seekable (block devices or regular
19       files), dd_rescue does copy with large blocks (softbs) to increase
20       performance. When a read error is encountered, dd_rescue falls back to
21       reading smaller blocks (hardbs), to allow to recover the maximum amount
22       of data. If blocks can still not be read, dd_rescue by default skips
23       over them also in the output file, avoiding to overwrite data that
24       might have been copied there successfully in a previous run.  (Option
25       -A / --alwayswrite changes this.).
26
27       dd_rescue can copy in reverse direction as well, allowing to approach a
28       bad spot from both directions. As trying to read over a bad spot of
29       significant size can take very long (and potentially cause further
30       damage), this is an important optimization when recovering data. The
31       dd_rhelp tool takes advantage of this and automates data recovery.
32       dd_rescue does not (by default) truncate the output file.
33
34       dd_rescue by default reports on progress, and optionally also writes
35       into a logfile.  It has a progress bar and gives an estimate for the
36       remaining time.  dd_rescue has a wealth of options that influence its
37       behavior, such as the possibility to use direct IO for input/output, to
38       use fallocate() to preallocate space for the output file, using splice
39       copy (in kernel zerocopy) for efficiency, looking for empty blocks to
40       create sparse files, or using a pseudo random number generator (PRNG)
41       to quickly overwrite data with random numbers.
42
43       The modes to overwrite partitions or files with pseudo random numbers
44       make dd_rescue a tool that can be used for secure data deletion and
45       thus not just a data recovery and backup tool but also a data
46       protection tool.
47
48       You can use "-" as infile or outfile, meaning stdin or stdout. Note
49       that this means that either file is not seekable, limiting the
50       usefulness of some of dd_rescues features.
51

OPTIONS

53       When parsing numbers, dd_rescue assumes bytes. It accepts the following
54       suffixes:
55       b -- 512 size units (blocks)
56       k -- 1024 size units (binary kilobytes, kiB)
57       M -- 1024^2 size units (binary megabytes, MiB)
58       G -- 1024^3 size units (binary gigabytes, GiB)
59
60       The following options may be used to modify the behavior of dd_rescue .
61
62   General options
63       -h, --help
64               This option tells dd_rescue to output a list of options and
65               exit.
66
67       -V, --version
68               Display version number and exit.
69
70       -q, --quiet
71               tells dd_rescue to be less verbose.
72
73       -v, --verbose
74               makes dd_rescue more verbose.
75
76       -c 0/1, --color=0/1
77               controls whether dd_rescue uses colors. By default it does,
78               unless the terminal type from TERM is unknown or dumb or ends
79               in -m or -mono.
80
81       -f, --force
82               makes dd_rescue skip some sanity checks (e.g. automatically
83               setting reverse direction when input and output file are the
84               same and ipos < opos).
85
86       -i, --interactive
87               tells dd_rescue to ask before overwriting existing files.
88
89   Block sizes
90       -b softbs, --softbs=softbs, --bs=softbs
91               sets the (larger) block size to softbs bytes.  dd_rescue will
92               transfer chunks of that size unless a read error is encountered
93               (or the end of the input file or the maximum transfer size has
94               been reached).  The default value for this is 64k for buffered
95               I/O and 1M for direct I/O.
96
97       -B hardbs, --hardbs=hardbs, --block-size=hardbs
98               sets the (smaller) fallback block size to hardbs bytes. When
99               dd_rescue encounters read errors, it will fall back to copying
100               data in chunks of this size. This value defaults to 4k for
101               buffered I/O and 512 bytes for direct I/O.
102               hardbs should be equal to or smaller than softbs.  If both
103               block sizes are identical, no fallback mechanism (and thus no
104               retry) will take place on read errors.
105
106       -y syncsize, --syncfreq=syncsize
107               tells dd_rescue to call fsync() on the output file every
108               syncsize bytes (will be rounded to multiples of softbs sized
109               blocks). It will also update the progress indicator at least as
110               often. By default, syncsize is set to 0, meaning that fsync()
111               is only issued at the end of the copy operation.
112
113   Positions and length
114       -s ipos, --ipos=ipos, --input-position=ipos
115               sets the starting position of the infile to ipos.  Note that
116               ipos is specified in bytes (but suffixes can be used, see
117               above), not in terms of softbs or hardbs sized blocks.  The
118               default value for this is 0. When reverse direction copy is
119               used, an ipos of 0 is treated specially, meaning the end of
120               file.
121               Negative positions result in an error message.
122
123       -S opos, --opos=opos, --output-position=opos
124               sets the starting position of the outfile to opos.  If not
125               specified, opos is set to ipos, so the file offsets in input
126               and output file are the same.  For reverse direction copy, an
127               explicit opos of 0 will position at the end of the output file.
128
129       -x, --extend, --append
130               changes the interpretation of the output position to start at
131               the end of the existing output file, making appending to a file
132               convenient.  If the output file does not exist, an error will
133               be reported and dd_rescue aborted.
134
135       -m maxxfer, --maxxfer=maxxfer, --max-size=maxxfer
136               specifies the maximum number of bytes (suffixes apply, but it's
137               NOT counted in blocks) that dd_rescue copies. If EOF is
138               encountered before maxxfer bytes have been transferred, this
139               option will be silently ignored.
140
141       -M, --noextend
142               tells dd_rescue to not extend the output file. This option is
143               particularly helpful when overwriting a file with random data
144               or zeroes for safe data destruction. If the output file does
145               not exist, an error message will be generated and the program
146               be aborted.
147
148   Error handling
149       -e maxerr, --maxerr=maxerr
150               tells dd_rescue to exit, after maxerr read errors have been
151               encountered. By default, this is set to 0, resulting in
152               dd_rescue trying to move on until it hits EOF (or maxxfer bytes
153               have been transferred).
154
155       -w, --abort_we
156               makes dd_rescue abort on any write errors. By default, on
157               reported write errors, dd_rescue tries to rewrite the blocks
158               with small block size writes, so a small failure in a larger
159               block will not cause the whole block not to be written. Note
160               that this may be handled similarly by your Operating System
161               kernel with buffered writes without the user or dd_rescue
162               noticing; the write retry logic in dd_rescue is mostly useful
163               for direct I/O writes where write errors can be reliably
164               detected.
165               Write error detection with buffered writes is unreliable; the
166               kernel reports success and traces of the failing writeback
167               operations later may only appear in your syslog. dd_rescue does
168               try to notice the user by calling fsync() and carefully
169               checking the return values of fsync() and close() calls.
170               Note that dd_rescue does exit if writes to the output file
171               result in the Operating System reporting that no space is left.
172
173   Sparse files and write avoidance
174       -A, --alwayswrite
175               changes the behavior of dd_rescue to write zeroes to the output
176               file when the input file could not be read. By default, it just
177               skips over, leaving whatever content was in the output file at
178               the file position before. The default behavior may be desired,
179               if e.g. previous copy operations may have resulted in good data
180               being in place; it may be undesired if the output file may
181               contain garbage (or sensitive information) that should rather
182               be overwritten with zeroes.
183
184       -a, --sparse
185               will make dd_rescue look for empty blocks (of at least half of
186               softbs size), i.e. blocks filled with zeroes. Rather than
187               writing those zeroes to the output file, it will then skip
188               forward in the output file, resulting in a sparse file, saving
189               space in the output file system (if it supports sparse files).
190               Note that if the output file does already exist and already has
191               data stored at the location where zeroes are skipped over, this
192               will result in an incomplete copy in that the output file is
193               different from the input file at the location where blocks of
194               zeroes were skipped over.  dd_rescue tries to detect this and
195               issue a warning, but it does not prevent this from happening
196
197       -W, --avoidwrite
198               results in dd_rescue reading a block ( softbs sized) from the
199               output file prior to writing it. If it is already identical
200               with the data that would be written to it, the writes are
201               actually avoided.  This option may be useful for devices, where
202               e.g. writes should be avoided (e.g. because they may impact the
203               remaining lifetime or because they are very slow compared to
204               reads).
205
206   Other optimization
207       -R, --repeat
208               tells dd_rescue to only read one block ( softbs sized) and then
209               repeatedly write it to the output file.  Note that this results
210               in never hitting EOF on the input file and should be used with
211               a limit for the transfer size (options -m or -M) or when
212               filling up an output device completely.
213               This option is automatically set, if the input file name equals
214               "/dev/zero".
215
216       -u, --rmvtrim
217               instructs dd_rescue to remove the output file after writing to
218               it has completed and issue a FITRIM on the file system that
219               contains the output file. This makes only sense if writing
220               zeros (or random numbers) as opposed to useful content from
221               another file. (dd_rescue will ask for confirmation if this is
222               specified with a normal input file and no -f (--force) is
223               used.) This option may be used to ensure that all empty blocks
224               of a file system are filled with zeros (rather than containing
225               fragments of deleted files with possibly sensitive
226               information).
227               The FITRIM ioctl (on Linux) tells the flash storage to consider
228               the freed space as unused (like the fstrim tool or the discard
229               option) by issuing ATA TRIM commands. This will only succeed
230               with superuser privileges (but the error can otherwise be
231               safely ignored). This is useful to ensure full performance of
232               flash memory / SSDs. Note that FITRIM can take a while on large
233               file systems, especially if the file systems are not mounted
234               with the discard option and have not been trimmed (with e.g.
235               fstrim) for a while. Not all file systems and not all flash-
236               based storage support this.
237
238       -k, --splice
239               tells dd_rescue to use the Linux in-kernel zerocopy splice()
240               copy operation rather than reading blocks into a user space
241               buffer. Note that this operation mode does prevent the support
242               of a number of dd_rescue features that can normally be used,
243               such as falling back to smaller block sizes, avoiding writes,
244               sparse mode, repeat optimization, reverse direction copy. A
245               warning is issued to make the user aware.
246
247       -P, --fallocate
248               results in dd_rescue calling fallocate() on the output file,
249               telling the file system how much space to preallocate for the
250               output file. (The size is determined by the expected last
251               position, as inferred from the input file length and maxxfer ).
252               On file systems that support it, this results in them making
253               better allocation decisions, avoiding fragmentation. (Note that
254               it does not make sense to use sparse together with
255               fallocate().)
256               This option is only available if dd_rescue is compiled with
257               fallocate() support. For optimal support, it should be compiled
258               with the libfallocate library.
259
260       -C rate,  --ratecontrol=rate
261               limits the transfer speed of dd_rescue to the rate (per
262               second). The usual suffixes are allowed.  Note that this limits
263               the average speed; the current speed may be up to twice this
264               limit. Default is unlimited. Note that you will have to use
265               smaller softblocksizes if you want to go below 32k (kB/s).
266
267   Misc options
268       -r, --reverse
269               tells dd_rescue to copy in reverse direction, starting at ipos
270               (with special case 0 meaning EOF) and working towards the
271               beginning of the file. This is especially helpful if the input
272               file has a bad spot which can be extremely slow to skip over,
273               so approaching it from both directions saves a lot of time (and
274               may prevent further damage).
275               Note that dd_rescue does automatically switch to reverse
276               direction copy, if input and output file are identical and the
277               input position is smaller than the output position, similar to
278               the intelligence that memmove() uses to prevent loss of data
279               when overlapping areas are copied. The option -f / --force does
280               prevent this intelligence from happening.
281
282       -p, --preserve
283               When copying files, this option does result in file metadata
284               (timestamps, ownership, access rights, xattrs) to be copied,
285               similar to the option with the same name in the cp program.
286               Note that ACLs and xattrs will only be copied if dd_rescue has
287               been compiled with libxattr support and the library can be
288               dynamically loaded on the system. Also note that failing to
289               copy the attributes with -p is not considered a failure and
290               thus won't negatively affect the exit code of dd_rescue.
291
292       -t, --truncate
293               tells dd_rescue to open the output file with O_TRUNC, resulting
294               in the output file (if it is a regular file) to be truncated to
295               0 bytes before writing to it, removing all previous content
296               that the file may have contained.  By default, dd_rescue does
297               not remove previous content.
298
299       -T, --trunclast
300               tells dd_rescue to truncate the output file to the highest
301               copied position after the copy operation completed, thus
302               ensuring there's no data beyond the end of the data that has
303               been copied in this run.
304
305       -d, --odir_in
306               instructs dd_rescue to open infile with O_DIRECT, bypassing the
307               kernel buffers. While this option has a negative effect on
308               performance (the kernel does read-ahead for buffered I/O), it
309               will result in errors to be detected more quickly (kernel won't
310               retry) and allows for smaller I/O units (hardware sector size,
311               512bytes for most hard disks).
312               O_DIRECT may not be available on all platforms.
313
314       -D, --odir_out
315               tells dd_rescue to open outfile with O_DIRECT, bypassing kernel
316               buffers. This has a significant negative effect on performance,
317               as the program needs to wait for writes to hit the disks as
318               opposed to the asynchronous nature of buffered writeback.  On
319               the flip side, the return status from writing is reliable this
320               way and smaller I/O chunks (hardware sector size, 512bytes) are
321               possible.
322
323   Logging
324       -l logfile, --logfile=logfile
325               Unless in quiet mode, dd_rescue does produce constant updates
326               on the status of the copy operation to stderr. With this
327               option, these updates are also written to the specified
328               logfile.  The control characters (to move the cursor up to
329               overwrite the existing status lines) are not written to the
330               logfile.
331
332       -o bbfile, --bbfile=bbfile
333               instructs dd_rescue to write a list of bad blocks to bbfile.
334               The file will contain a list of numbers (ASCII), one per line,
335               where the numbers indicate the offset in terms of hardbs sized
336               blocks. The file format is compatible with that of badblocks.
337               Using dd_rescue on a block device (partition) and setting
338               hardbs to the block size of a file system that you want to
339               create, you should be able to feed the bbfile to mke2fs with
340               the option -l.
341
342   Multiple output files
343       -Y ofileX, --outfile=ofileX, --of=ofileX
344               If you want to copy data to multiple files simultaneously, you
345               can specify this option. It can be specified multiple times, so
346               many copies can be made.  Note that these files are secondary
347               output files; they share file position with the primary output
348               file outfile.  Errors when writing to a secondary output file
349               are ignored.
350
351   Data protection by overwriting with random numbers
352       -z RANDSEED, --random=RANDSEED
353       -Z RANDSEED, --frandom=RANDSEED
354       -2 RANDSEED, --shred2=RANDSEED
355       -3 RANDSEED, --shred3=RANDSEED
356       -4 RANDSEED, --shred4=RANDSEED
357               When you want to overwrite a file, partition or disk with
358               random data, using /dev/urandom (on Linux) as input is not a
359               very good idea; the interface has not been designed to yield a
360               high bandwidth. It's better to use a user space Pseudo Random
361               Number Generator (PRNG). With option -z / --random, the C
362               library's PRNG is used. With -Z / --frandom and the -2/-3/-4 /
363               --shred2/3/4 options, an RC4 based PRNG is used.
364               Note that in this mode, there is no infile so the first non-
365               option argument is the output file.
366               The PRNG needs seeding; the C libraries PRNG takes a 32bit
367               integer (4 bytes); the RC4 based PRNG takes 256 bytes. If
368               RANDSEED is an integer, the integer number will be used to seed
369               the C library's PRNG. For the RC4 method, the C library's PRNG
370               then generates the 256 bytes to seed it. This creates
371               repeatable PRNG data. The RANDSEED value of 0 is special; it
372               will create a seedval that's based on the current time and the
373               process' PID and should be different for multiple runs of
374               dd_rescue .
375               If RANDSEED is not an integer, it's assumed to be a file name
376               from which the seed values can be read.  dd_rescue will read 4
377               or 256 bytes from the file to seed the C library's or the RC4
378               PRNG. For good pseudo random numbers, using /dev/urandom to
379               seed is a good idea.
380               The modes -2/-3/-4 resp. --shred2/--shred3/--shred4 will
381               overwrite the output file multiple times; after each pass,
382               fsync() will ensure that the data does indeed hit the file. The
383               last pass for these modes will overwrite the file with zeroes.
384               The rationale behind doing this is to make it easier to hide
385               that important data may have been overwritten, to make it
386               easier for intelligent storage systems (such as SSDs) to
387               recycle the empty blocks and to allow for better compression of
388               a file system image containing such data.
389               With -2 / --shred2, one pass with RC4 generated PRNG is
390               happening and then zeroes are written. With -3 / --shred3,
391               there are two passes with RC4 PRNG generated random numbers and
392               a zero pass; the second PRNG pass writes the inverse (bit-wise
393               reversed) numbers from the first pass. -4 / --shred4 works like
394               -3 / --shred3, with an additional pass with independent random
395               numbers as third pass.
396
397   Plugins
398       Since version 1.42, dd_rescue has an interface for plugins. Plugins
399       have the ability to analyze the copied data or to transform it prior to
400       it being written.
401
402       -L plugin1[=param1[:param2[:..]]][,plugin2[=..][,..]]
403       --plugins=plugin1[=param1[:param2[:..]]][,plugin2[=..][,..]]
404               loads plugins plugin1 ... and passes parameters to it. All
405               plugins should support at least the help parameter and provide
406               information on their usage.
407               Plugins may impose limits on dd_rescue. Plugins that look at
408               the data can't work with splice, as this avoids copying data to
409               user space. Also the interface currently does not facilitate
410               reverse direction copy.  Some plugins may impose further
411               restrictions w.r.t. alignment of data in the file or not using
412               sparse detection.
413               See section PLUGINS for an overview of available plugins.
414
415

PLUGINS

417   null
418       The null plugin (ddr_null) does nothing, except if you specify the
419       [no]lnchange or the [no]change options in which case the plugin
420       indicates to others that it transforms the length of the output or the
421       data of the stream. (With the no prefix, it's reset to the default no-
422       change indication again.)  This may be helpful for testing or to
423       influence which file the hash plugin considers for reading/writing
424       extended attributes from/to and for plugins to change their behavior
425       with respect to hole detection.
426       ddr_null_ddr also allows you to specify debug in which case it just
427       reports the blocks that it passes on.
428
429   hash
430       When the hash plugin (subsequently referred to as ddr_hash) is loaded,
431       it will calculate a cryptographic hash and optionally also a HMAC over
432       the copied data and print the result at the end of the copy operations.
433       The hash algorithm can be chosen by specifying alg[o[rithm]]=ALG where
434       ALG is one of md5, sha1, sha256, sha224, sha512, sha384. (Specify
435       alg=help to get a list.)  To abbreviate the syntax, the alg= piece can
436       be omitted.
437       For backwards compatibility, the hash plugin can also be referred to
438       with the old MD5 name; it then defaults to the md5 algorithm.
439       The computed value should be identical to calling md5sum/sha256sum/...
440       on the target file (unless you only write part of the file), but saves
441       time by not accessing the (possibly large) file a second time.  The
442       hash plugin handles sparse writes and arbitrary offsets fine.
443
444       multipart=CHUNKSIZE tells ddr_hash to calculate multiple checksums for
445       file chunks of CHUNKSIZE each and then combine them into a combined
446       checksum by creating a checksum over the piece checksums. This is how
447       the checksum for S3 multipart objects is calculated (using the md5
448       hash); the output there is the combination checksum with a dash and the
449       number of parts appended.
450       Note that this feature is new in 1.99.6 and does not yet handle
451       situations cleanly, where offsets plus block sizes do not happen to
452       cleanly align with the CHUNKSIZE. The implementation for this will be
453       completed later.  Other features like the append/prepend/hmac pieces
454       also don't work well with multipart checksum calculation.
455
456       ddr_hash also supports the parameter append=STRING which appends the
457       given STRING to the output before computing the cryptographic hash.
458       Treating the STRING as a shared secret, this can actually be used to
459       protect against someone not knowing the secret altering the contents
460       (and recomputing the hash) without anyone noticing. It's thus a cheap
461       way of a cryptographic signature (but with preshared secrets as opposed
462       to public key cryptography). Use HMAC for a somewhat better way to sign
463       data with a shared secret.
464       ddr_hash also supports prepend=STRING which is likely harder to attack
465       with brute force than an appended string.  Note that ddr_hash always
466       prepends multiples of the hash algorithm's block size and pads the
467       STRING with 0 to match.
468
469       ddr_hash can be used to compute a HMAC (Hash-based Message
470       Authentication Code) instead of the plain hash. The HMAC uses a
471       password that's prepended and transformed twice to the data which is
472       then hashed twice.  HMAC is believed to protect somewhat better against
473       extension or collision attacks than a plain hash (with a plain
474       prepended secret), so it's a better way to authenticate data with a
475       shared secret. (You can use append/prepend in addition to HMAC, if you
476       have a need for a scheme with more than one secret.)
477       When HMAC is enabled with one of the following parameters, both the
478       plain hash and the HMAC are computed by ddr_hash. Both are output to
479       the console/log, but the HMAC is used instead of the hash value to be
480       written to a CHECKSUMS file or to an extended attribute or checked
481       against (see below).  hmacpwd=STRING sets the shared secret (password)
482       for computing the HMAC. Passing the secret on the command line has the
483       disadvantage that the shell may mistreat some bytes as special
484       characters and that the command line may be visible to all logged in
485       users on the system.  hmacpwdfd=INT sets a file descriptor from with
486       the secret (password) for HMAC computation will be read. Specifying 0
487       means standard input, in which case ddr_hash even prints a prompt for
488       you ... Other numbers may be useful if dd_rescue is called from another
489       program that opens a pipe to pass the secret.  hmacpwdnm=INNAME sets a
490       file from which the shared secret (password) is read. Note that all
491       bytes (up to 2048 of them) are read and used, including trailing white
492       space, 0-bytes or newlines.
493       Please note that the ddr_hash plugin at this point does NOT take a lot
494       of care to prevent the password/pre/appended secret from remaining in
495       memory or leaking into a swap/page file. (This will be improved once I
496       look into encryption plugins.)
497
498       ddr_hash accepts the parameter output , which will cause ddr_hash to
499       output the cryptographic hash to stdout in the same format that
500       md5sum/sha256sum/... use.  You can also specify outfd=INT to have the
501       plugin write the hash to a different file descriptor specified by the
502       integer number INT. Note that ddr_hash always processes data in binary
503       mode and correctly indicates this with a star (*) in the output
504       generated with output/outfd=.
505       The checksum can also be written to a file by giving the outnm=OUTNAME
506       parameter. Then a file with OUTNAME will be created and a
507       md5sum/sha256sum/...  compatible line will be printed to the file. If
508       the file exists and contains an entry for the file, it will be updated.
509       If the file exists and does not contain an entry for the file, one will
510       be appended. If OUTNAME is omitted, the file name CHECKSUMS.alg (or
511       HMACS.alg if HMAC is enabled) will be used (alg is replaced by the
512       chosen algorithm).  If the checksum can't be written, a warning will be
513       printed and the exit code of dd_rescue will become non-zero.
514
515       The checksum can be validated using checknm=CHKNAME .  The file will be
516       read and ddr_hash will look for an md5sum/sha256sum/...  compatible
517       line with a matching file name to take the checksum from and compare it
518       to the one computed. If NAME is omitted, the same default as described
519       above (in outnm=...) will be used. You can also read the checksum from
520       stdin if you prefer by specifying the check option.
521       Note that in any case, the check is only performed after the copy
522       operation is completed -- a faulty checksum will thus NOT result in the
523       copy not taking place. However, the exit code of dd_rescue will
524       indicate the error. (If you want to avoid copying data with a broken
525       checksum into the final target, use a temporary target that you delete
526       upon error and only move to the final location if dd_rescue's exit
527       value is 0; you can of course also copy to /dev/null for testing
528       beforehand, but it might be too costly reading the input file twice.)
529
530       You can store the cryptographic hash into the files by using the
531       set_xattr option. The hash will be stored into the extended attribute
532       user.checksum.ALG by default (user.hmac.ALG if HMAC is enabled), but
533       you can override the name of the attribute by specifying
534       set_xattr=XATTR.NAME instead. If the xattr can't be written, an error
535       will be reported, unless you also specify the fallb[ack][=CHKNAME]
536       option. In that case, ddr_hash tries to write the checksum to the
537       CHKNAME checksums file. (For the default for CHKNAME, see outnm= option
538       above.)
539       chk_xattr will validate that the computed hash matches the one read
540       from the extended attribute. The same default attribute name applies
541       and you can likewise override it with chk_xattr=XATTR.NAME .  A missing
542       attribute is considered an error (although the same fallback is tried
543       if you specify the fallback option). A broken checksum is of course
544       considered an error as well, but just like with checknm=CHKNAME won't
545       prevent the copy. See the discussion there.
546
547       Note that for output,outfd,outnm=,set_xattr ddr_hash will use the
548       output file name to attach the checksum to (be it by setting xattr or
549       the file name used in the checksum file), unless a plugin in the chain
550       after ddr_hash indicates that it changes the data.  In that case, it
551       will warn and associate the checksum with the input file name, unless
552       there's another plugin before ddr_hash in the chain which indicates
553       data transformation as well. In that case, there is no file that the
554       checksum could be associated with and ddr_hash will report an error.
555       Likewise for chknm=,check,chk_xattr ddr_hash will use the input file
556       name to get the checksum (be it by reading the xattr or by looking for
557       the input file name in a checksums file) unless there's a plugin in the
558       chain before ddr_hash that indicates that it changes the data. The
559       output file name will then be used, unless there's another plugin after
560       ddr_hash indicating data change as well, in which case there's no file
561       we could get the checksum for and thus an error is reported.
562
563       If your system supports extended attributes, those have the advantage
564       of traveling with the files; thus a rename or copy (with dd_rescue -p)
565       will maintain the checksum. Checksum files on the other hand can be
566       handled everywhere (including the transfer via ftp or http) and can be
567       cryptographically signed with PGP/GnuPG.
568
569       Please note that the md5 algorithm is NOT recommended any more for good
570       protection against malicious attempts to hide data modification; it's
571       not considered strong enough any more to prevent hash collisions.  sha1
572       is a bit better, but has been broken as well as of 2017.  The
573       recommendation is to use the SHA-2 family of hashes.  On 32bit
574       machines, I'd recommend sha256, while on 64bit machines, sha512 is
575       faster and thus the best choice.
576
577       ddr_hash also supports using the HMAC code and hashes for deriving keys
578       from passwords using the PKCS5 PBKDF2 (password-based key derivation
579       function) that allows you to improve the protection from mediocre
580       passwords by using a salt and a relatively expensive key stretching
581       operation. This is only meant for testing and may be removed in the
582       future. It's thus not documented in this man page. See the built-in
583       help function for a brief summary on the usage.
584
585   lzo
586       The lzo plugin allows to compress and decompress data using liblzo2.
587       lzo is an algorithm that is faster than most other algorithms but does
588       not compress as well.  See the ddr_lzo(1) man page for more details.
589
590   crypt
591       The crypt plugin allows to encrypt and decrypt data on the fly.  It
592       currently supports a variety of AES ciphers.  See the ddr_crypt(1) man
593       page for more details.
594

EXIT STATUS

596       On successful completion, dd_rescue returns an exit code of 0.  Any
597       other exit code indicates that the program has aborted because of an
598       error condition or that copying of the data has not been entirely
599       successful.
600

EXAMPLES

602       dd_rescue -k -P -p -t infile outfile
603              copies infile to outfile and does truncate the output file on
604              opening (so deleting any previous data in it), copies mode,
605              times, ownership at the end, uses fallocate to reserve the space
606              for the output file and uses efficient in kernel splice copy
607              method.
608
609       dd_rescue -A -d -D -b 512 /dev/sda /dev/sda
610              reads the contents of every sector of disk sda and writes it
611              back to the same location. Typical hard disks reallocate flaky
612              and faulty sectors on writes, so this operation may result in
613              the complete disk being usable again when there were errors
614              before. Unreadable blocks however will contain zeroes after
615              this.
616
617       dd_rescue -2 /dev/urandom -M outfile
618              overwrites the file outfile twice; once with good pseudo random
619              numbers and then with zeroes.
620
621       dd_rescue -t -a image1.raw image2.raw
622              copies a file system image and looks for empty blocks to create
623              a sparse output file to save disk space. (If the source file
624              system has been used a bit, on that file system creating a large
625              file with zeroes and removing it again prior to this operation
626              will result in more sectors with zeroes.
627              dd_rescue -u /dev/zero DUMMY will achieve this ...)
628
629       dd_rescue -ATL hash=md5:output,lzo=compress:bench,MD5:output in out.lzo
630              copies the file in to out.lzo with using lzo (lzo1x_1)
631              compression and calculating an md5 hash (checksum) on both
632              files. The md5 hashes for both are also written to stdout in the
633              md5sum output format.  Note that the compress parameter to lzo
634              is not strictly required here; the plugin could have deduced it
635              from the file names. This example shows that you can specify
636              multiple plugins with multiple parameters; the plugins are
637              forming a filter chain. You can specify the same plugin multiple
638              times.
639
640       dd_rescue -L hash=sha512:set_xattr:fallb,null=change infile /dev/null
641              reads the file infile and computes its sha512 hash. It stores it
642              in the input file's user.checksum.sha512 attribute (and falls
643              back to writing it to CHECKSUMS.sha512 if xattrs can't be
644              written). Note the use of the null plugin with faking data
645              change with the change parameter; this causes the hash plugin to
646              write to the input file which it would not normally have done.
647              Of course this will fail if you don't have the appropriate
648              privileges to write xattrs to infile nor to write the checksum
649              to CHECKSUMS.sha512.
650
651       See also README.dd_rescue and ddr_lzo(1) to learn about the
652       possibilities.
653

TESTING

655       Untested code is buggy, almost always.  I happen to have a damaged hard
656       disk that I use for testing dd_rescue from time to time. But to allow
657       for automated testing of error recovery, it's better to have
658       predictable failures for the program to deal with. So there is a fault
659       injection framework.
660       Specifying -F 5w/1,17r/3,42r/-1,80-84r/0 on the command-line will
661       result in in the 5th block (counted in hardblocksize) will fail to be
662       written once (from which dd_rescue should recover, as it tries a second
663       time for failed writes), block no 17 will fail to be read 3 times,
664       block no 42 will read fine once, but then fail afterwards, whereas
665       blocks 80 through 83 are completely unreadable (will fail infinite
666       times). Note that the range excludes the last block (80-84 means 4
667       blocks starting @ 80).
668       Block offsets are always counted in absolute positions, so starting in
669       the middle of a file with -s or reverse copying won't affect the
670       absolute position that is hit with the fault injection. (This has
671       changed since 1.98.)
672
673

BUGS/LIMITATIONS

675       The source code does use the 64bit functions provided by glibc for file
676       positioning. However, your kernel might not support it, so you might be
677       unable to copy partitions larger then 2GB into a file.
678       This program has been written using Linux and only tested on a couple
679       of Linux systems. People have reported to have successfully used it on
680       other Un*xish systems (such as xBSD or M*cOS), but these systems get
681       little regular test coverage; so please be advised to test properly
682       (possibly using the make check test suite included with the source
683       distribution) before relying on dd_rescue on non Linux based systems.
684       Currently, the escape sequence for moving the cursor up is hard coded
685       in the sources. It's fine for most terminal emulations (including vt100
686       and linux), but it should use the terminal description database
687       instead.
688       Since dd_rescue-1.10, non-seekable input or output files are supported,
689       but there's of course limitations to recover errors in such cases.
690
691       dd_rescue does not automate the recovery of faulty files or partitions
692       by automatically keeping a list of copied sectors and approaching bad
693       spots from both sides. There is a helper script dd_rhelp from LAB
694       Valentin that does this. Integration of such a mode into dd_rescue
695       itself is non-trivial and due to the complexity of the source code
696       might not happen.
697       There also is a tool, GNU ddrescue, that is a reimplementation of this
698       tool and which contains the capabilities to automate recovery of bad
699       files in the way dd_rhelp does. It does not have the feature richness
700       of dd_rescue, but is reported to be easier to operate for error
701       recovery than dd_rescue with dd_rhelp.
702
703       If your data is very valuable and you are considering sending your disk
704       to a data recovery company, you might be better off NOT trying to use
705       imaging tools like dd_rescue, dd_rhelp or GNU ddrescue. If you're
706       unlucky, the disk has suffered some mechanical damage (e.g. by having
707       been dropped), and continuing to use it may make the head damage the
708       surface further.  You may be able to detect this condition by quickly
709       raising error counts in the SMART attributes or by a clicking noise.
710
711       Please report bugs to me via email.
712
713   Data destruction considerations
714       The modes for overwriting data with pseudo random numbers to securely
715       delete sensitive data on purpose only implement a limited number of
716       overwrites. While Peter Gutmann's classic analysis concludes that the
717       then current hard disk technology requires more overwrites to be really
718       secure, the author believes that modern hard disk technology does not
719       allow data restoration of sectors that have been overwritten with the
720       --shred4 mode. This is in compliance with the recommendations from BSI
721       GSDS M7.15.
722       Overwriting whole partitions or disks with random numbers is a fairly
723       safe way to destroy data, unless the underlying storage device does too
724       much magic. SSDs are doing fancy stuff in their Flash Translation Layer
725       (FTL), so this tool might be insufficient to get rid of data. Use
726       SECURITY_ERASE (use hdparm) there or -- if available -- encrypt data
727       with AES256 and safely destroy the key.  Normal hard disks have a small
728       risk of leaking a few sectors due to reallocation of flaky sectors.
729       For securely destroying single files, your mileage may vary. The more
730       advanced your file system, the less likely dd_rescue's destruction will
731       be effective.  In particular, journaling file systems may carry old
732       data in the journal.  File systems that do copy-on-write (COW) such as
733       btrfs, are very likely to have old copies of your supposedly erased
734       file. It might help somewhat to fill the file systems with zeros
735       (dd_rescue -u /dev/zero /path/to/fs/DUMMYNAME) to force the file system
736       to release and overwrite non-current data after overwriting critical
737       files with random numbers. If you can, better destroy a whole partition
738       or disk.
739

SEE ALSO

741       README.dd_rescue README.dd_rhelp ddr_lzo(1)
742       wipe(1) shred(1) ddrescue(1) dd(1)
743

AUTHOR

745       Kurt Garloff <kurt@garloff.de>
746

CREDITS

748       Many little issues were reported by Valentin LAB, the author of
749       dd_rhelp .
750       The RC4 PRNG (frandom) is a port from Eli Billauer's kernel mode PRNG.
751       A number of recent ideas and suggestions came from Thomas.
752
754       This program is protected by the GNU General Public License (GPL) v2 or
755       v3 - at your option.
756

HISTORY

758       Since version 1.10, non seekable input and output files are supported.
759       Splice copy -k is supported since 1.15.
760       A progress bar exists since 1.17.
761       Support for preallocation (fallocate) -P exists since 1.19.
762       Since 1.23, we default to -y0, enhancing performance.
763       The Pseudo Random Number modes have been started with 1.29.
764       Write avoidance -W has been implemented in 1.30
765       Multiple output files -Y have been added in 1.32.
766       Long options and man page came with 1.33.
767       Optimized sparse detection (SSE2, armv6, armv8 asm, AVX2) has been
768       present since 1.35 and been enhanced until 1.43.
769       We support copying extended attributes since 1.40 using libxattr.
770       Removing and (fs)trimming the output file's file system exists since
771       1.41. Support for compilation with bionic (Android's C library) with
772       most features enabled also came with 1.41.
773       Plugins exist since 1.42, the MD5 plugin came with 1.42, the lzo plugin
774       with 1.43. 1.44 renamed the MD5 plugin to hash and added support for
775       the SHA-2 family of hashes. 1.45 added SHA-1 and the ability to store
776       and validate checksums.
777       1.98 brought encryption and the fault injection framework, 1.99 support
778       for ARMv8 crypto acceleration.  1.99.5 brought ratecontrol.  1.99.6
779       brought S3 style multipart checksums.
780
781       Some additional information can be found on
782       http://garloff.de/kurt/linux/ddrescue/
783       LAB Valentin's dd_rhelp can be found on
784       http://www.kalysto.org/utilities/dd_rhelp/index.en.html
785
786
787
788Kurt Garloff                      2017-06-23                      dd_rescue(1)
Impressum