1dd_rescue(1) Data recovery and protection tool dd_rescue(1)
2
6 dd_rescue - Data recovery and protection tool
7
9 dd_rescue [options] infile outfile
10 dd_rescue [options] [-2/-3/-4/-z/-Z seed/seedfile] outfile
11 dd_rescue [options] [--shred2/--shred3/--shred4/--random/--frandom
12 seed/seedfile] outfile
13
15 dd_rescue is a tool that copies data from a source (file, block device,
16 pipe, ...) to one (or several) output file(s).
17 If input and output files are seekable (block devices or regular
19 files), dd_rescue does copy with large blocks (softbs) to increase
20 performance. When a read error is encountered, dd_rescue falls back to
21 reading smaller blocks (hardbs), to allow to recover the maximum amount
22 of data. If blocks can still not be read, dd_rescue by default skips
23 over them also in the output file, avoiding to overwrite data that
24 might have been copied there successfully in a previous run. (Option
25 -A / --alwayswrite changes this.).
26 dd_rescue can copy in reverse direction as well, allowing to approach a
28 bad spot from both directions. As trying to read over a bad spot of
29 significant size can take very long (and potentially cause further
30 damage), this is an important optimization when recovering data. The
31 dd_rhelp tool takes advantage of this and automates data recovery.
32 dd_rescue does not (by default) truncate the output file.
33 dd_rescue by default reports on progress, and optionally also writes
35 into a logfile. It has a progress bar and gives an estimate for the
36 remaining time. dd_rescue has a wealth of options that influence its
37 behavior, such as the possibility to use direct IO for input/output, to
38 use fallocate() to preallocate space for the output file, using splice
39 copy (in kernel zerocopy) for efficiency, looking for empty blocks to
40 create sparse files, or using a pseudo random number generator (PRNG)
41 to quickly overwrite data with random numbers.
42 The modes to overwrite partitions or files with pseudo random numbers
44 make dd_rescue a tool that can be used for secure data deletion and
45 thus not just a data recovery and backup tool but also a data
46 protection tool.
47 You can use "-" as infile or outfile, meaning stdin or stdout. Note
49 that this means that either file is not seekable, limiting the
50 usefulness of some of dd_rescues features.
51
53 When parsing numbers, dd_rescue assumes bytes. It accepts the following
54 suffixes:
55 b -- 512 size units (blocks)
56 k -- 1024 size units (binary kilobytes, kiB)
57 M -- 1024^2 size units (binary megabytes, MiB)
58 G -- 1024^3 size units (binary gigabytes, GiB)
59 The following options may be used to modify the behavior of dd_rescue .
61 General options
63 -h, --help
64 This option tells dd_rescue to output a list of options and
65 exit.
66 -V, --version
68 Display version number and exit.
69 -q, --quiet
71 tells dd_rescue to be less verbose.
72 -v, --verbose
74 makes dd_rescue more verbose.
75 -c 0/1, --color=0/1
77 controls whether dd_rescue uses colors. By default it does,
78 unless the terminal type from TERM is unknown or dumb or ends
79 in -m or -mono.
80 -f, --force
82 makes dd_rescue skip some sanity checks (e.g. automatically
83 setting reverse direction when input and output file are the
84 same and ipos < opos).
85 -i, --interactive
87 tells dd_rescue to ask before overwriting existing files.
88 Block sizes
90 -b softbs, --softbs=softbs, --bs=softbs
91 sets the (larger) block size to softbs bytes. dd_rescue will
92 transfer chunks of that size unless a read error is encountered
93 (or the end of the input file or the maximum transfer size has
94 been reached). The default value for this is 64k for buffered
95 I/O and 1M for direct I/O.
96 -B hardbs, --hardbs=hardbs, --block-size=hardbs
98 sets the (smaller) fallback block size to hardbs bytes. When
99 dd_rescue encounters read errors, it will fall back to copying
100 data in chunks of this size. This value defaults to 4k for
101 buffered I/O and 512 bytes for direct I/O.
102 hardbs should be equal to or smaller than softbs. If both
103 block sizes are identical, no fallback mechanism (and thus no
104 retry) will take place on read errors.
105 -y syncsize, --syncfreq=syncsize
107 tells dd_rescue to call fsync() on the output file every
108 syncsize bytes (will be rounded to multiples of softbs sized
109 blocks). It will also update the progress indicator at least as
110 often. By default, syncsize is set to 0, meaning that fsync()
111 is only issued at the end of the copy operation.
112 Positions and length
114 -s ipos, --ipos=ipos, --input-position=ipos
115 sets the starting position of the infile to ipos. Note that
116 ipos is specified in bytes (but suffixes can be used, see
117 above), not in terms of softbs or hardbs sized blocks. The
118 default value for this is 0. When reverse direction copy is
119 used, an ipos of 0 is treated specially, meaning the end of
120 file.
121 Negative positions result in an error message.
122 -S opos, --opos=opos, --output-position=opos
124 sets the starting position of the outfile to opos. If not
125 specified, opos is set to ipos, so the file offsets in input
126 and output file are the same. For reverse direction copy, an
127 explicit opos of 0 will position at the end of the output file.
128 -x, --extend, --append
130 changes the interpretation of the output position to start at
131 the end of the existing output file, making appending to a file
132 convenient. If the output file does not exist, an error will
133 be reported and dd_rescue aborted.
134 -m maxxfer, --maxxfer=maxxfer, --max-size=maxxfer
136 specifies the maximum number of bytes (suffixes apply, but it's
137 NOT counted in blocks) that dd_rescue copies. If EOF is
138 encountered before maxxfer bytes have been transferred, this
139 option will be silently ignored.
140 -M, --noextend
142 tells dd_rescue to not extend the output file. This option is
143 particularly helpful when overwriting a file with random data
144 or zeroes for safe data destruction. If the output file does
145 not exist, an error message will be generated and the program
146 be aborted.
147 Error handling
149 -e maxerr, --maxerr=maxerr
150 tells dd_rescue to exit, after maxerr read errors have been
151 encountered. By default, this is set to 0, resulting in
152 dd_rescue trying to move on until it hits EOF (or maxxfer bytes
153 have been transferred).
154 -w, --abort_we
156 makes dd_rescue abort on any write errors. By default, on
157 reported write errors, dd_rescue tries to rewrite the blocks
158 with small block size writes, so a small failure in a larger
159 block will not cause the whole block not to be written. Note
160 that this may be handled similarly by your Operating System
161 kernel with buffered writes without the user or dd_rescue
162 noticing; the write retry logic in dd_rescue is mostly useful
163 for direct I/O writes where write errors can be reliably
164 detected.
165 Write error detection with buffered writes is unreliable; the
166 kernel reports success and traces of the failing writeback
167 operations later may only appear in your syslog. dd_rescue does
168 try to notice the user by calling fsync() and carefully
169 checking the return values of fsync() and close() calls.
170 Note that dd_rescue does exit if writes to the output file
171 result in the Operating System reporting that no space is left.
172 Sparse files and write avoidance
174 -A, --alwayswrite
175 changes the behavior of dd_rescue to write zeroes to the output
176 file when the input file could not be read. By default, it just
177 skips over, leaving whatever content was in the output file at
178 the file position before. The default behavior may be desired,
179 if e.g. previous copy operations may have resulted in good data
180 being in place; it may be undesired if the output file may
181 contain garbage (or sensitive information) that should rather
182 be overwritten with zeroes.
183 -a, --sparse
185 will make dd_rescue look for empty blocks (of at least half of
186 softbs size), i.e. blocks filled with zeroes. Rather than
187 writing those zeroes to the output file, it will then skip
188 forward in the output file, resulting in a sparse file, saving
189 space in the output file system (if it supports sparse files).
190 Note that if the output file does already exist and already has
191 data stored at the location where zeroes are skipped over, this
192 will result in an incomplete copy in that the output file is
193 different from the input file at the location where blocks of
194 zeroes were skipped over. dd_rescue tries to detect this and
195 issue a warning, but it does not prevent this from happening
196 -W, --avoidwrite
198 results in dd_rescue reading a block ( softbs sized) from the
199 output file prior to writing it. If it is already identical
200 with the data that would be written to it, the writes are
201 actually avoided. This option may be useful for devices, where
202 e.g. writes should be avoided (e.g. because they may impact the
203 remaining lifetime or because they are very slow compared to
204 reads).
205 Other optimization
207 -R, --repeat
208 tells dd_rescue to only read one block ( softbs sized) and then
209 repeatedly write it to the output file. Note that this results
210 in never hitting EOF on the input file and should be used with
211 a limit for the transfer size (options -m or -M) or when
212 filling up an output device completely.
213 This option is automatically set, if the input file name equals
214 "/dev/zero".
215 -u, --rmvtrim
217 instructs dd_rescue to remove the output file after writing to
218 it has completed and issue a FITRIM on the file system that
219 contains the output file. This makes only sense if writing
220 zeros (or random numbers) as opposed to useful content from
221 another file. (dd_rescue will ask for confirmation if this is
222 specified with a normal input file and no -f (--force) is
223 used.) This option may be used to ensure that all empty blocks
224 of a file system are filled with zeros (rather than containing
225 fragments of deleted files with possibly sensitive
226 information).
227 The FITRIM ioctl (on Linux) tells the flash storage to consider
228 the freed space as unused (like the fstrim tool or the discard
229 option) by issuing ATA TRIM commands. This will only succeed
230 with superuser privileges (but the error can otherwise be
231 safely ignored). This is useful to ensure full performance of
232 flash memory / SSDs. Note that FITRIM can take a while on large
233 file systems, especially if the file systems are not mounted
234 with the discard option and have not been trimmed (with e.g.
235 fstrim) for a while. Not all file systems and not all flash-
236 based storage support this.
237 -k, --splice
239 tells dd_rescue to use the Linux in-kernel zerocopy splice()
240 copy operation rather than reading blocks into a user space
241 buffer. Note that this operation mode does prevent the support
242 of a number of dd_rescue features that can normally be used,
243 such as falling back to smaller block sizes, avoiding writes,
244 sparse mode, repeat optimization, reverse direction copy. A
245 warning is issued to make the user aware.
246 -P, --fallocate
248 results in dd_rescue calling fallocate() on the output file,
249 telling the file system how much space to preallocate for the
250 output file. (The size is determined by the expected last
251 position, as inferred from the input file length and maxxfer ).
252 On file systems that support it, this results in them making
253 better allocation decisions, avoiding fragmentation. (Note that
254 it does not make sense to use sparse together with
255 fallocate().)
256 This option is only available if dd_rescue is compiled with
257 fallocate() support. For optimal support, it should be compiled
258 with the libfallocate library.
259 -C rate, --ratecontrol=rate
261 limits the transfer speed of dd_rescue to the rate (per
262 second). The usual suffixes are allowed. Note that this limits
263 the average speed; the current speed may be up to twice this
264 limit. Default is unlimited. Note that you will have to use
265 smaller softblocksizes if you want to go below 32k (kB/s).
266 Misc options
268 -r, --reverse
269 tells dd_rescue to copy in reverse direction, starting at ipos
270 (with special case 0 meaning EOF) and working towards the
271 beginning of the file. This is especially helpful if the input
272 file has a bad spot which can be extremely slow to skip over,
273 so approaching it from both directions saves a lot of time (and
274 may prevent further damage).
275 Note that dd_rescue does automatically switch to reverse
276 direction copy, if input and output file are identical and the
277 input position is smaller than the output position, similar to
278 the intelligence that memmove() uses to prevent loss of data
279 when overlapping areas are copied. The option -f / --force does
280 prevent this intelligence from happening.
281 -p, --preserve
283 When copying files, this option does result in file metadata
284 (timestamps, ownership, access rights, xattrs) to be copied,
285 similar to the option with the same name in the cp program.
286 Note that ACLs and xattrs will only be copied if dd_rescue has
287 been compiled with libxattr support and the library can be
288 dynamically loaded on the system. Also note that failing to
289 copy the attributes with -p is not considered a failure and
290 thus won't negatively affect the exit code of dd_rescue.
291 -t, --truncate
293 tells dd_rescue to open the output file with O_TRUNC, resulting
294 in the output file (if it is a regular file) to be truncated to
295 0 bytes before writing to it, removing all previous content
296 that the file may have contained. By default, dd_rescue does
297 not remove previous content.
298 -T, --trunclast
300 tells dd_rescue to truncate the output file to the highest
301 copied position after the copy operation completed, thus
302 ensuring there's no data beyond the end of the data that has
303 been copied in this run.
304 -d, --odir_in
306 instructs dd_rescue to open infile with O_DIRECT, bypassing the
307 kernel buffers. While this option has a negative effect on
308 performance (the kernel does read-ahead for buffered I/O), it
309 will result in errors to be detected more quickly (kernel won't
310 retry) and allows for smaller I/O units (hardware sector size,
311 512bytes for most hard disks).
312 O_DIRECT may not be available on all platforms.
313 -D, --odir_out
315 tells dd_rescue to open outfile with O_DIRECT, bypassing kernel
316 buffers. This has a significant negative effect on performance,
317 as the program needs to wait for writes to hit the disks as
318 opposed to the asynchronous nature of buffered writeback. On
319 the flip side, the return status from writing is reliable this
320 way and smaller I/O chunks (hardware sector size, 512bytes) are
321 possible.
322 Logging
324 -l logfile, --logfile=logfile
325 Unless in quiet mode, dd_rescue does produce constant updates
326 on the status of the copy operation to stderr. With this
327 option, these updates are also written to the specified
328 logfile. The control characters (to move the cursor up to
329 overwrite the existing status lines) are not written to the
330 logfile.
331 -o bbfile, --bbfile=bbfile
333 instructs dd_rescue to write a list of bad blocks to bbfile.
334 The file will contain a list of numbers (ASCII), one per line,
335 where the numbers indicate the offset in terms of hardbs sized
336 blocks. The file format is compatible with that of badblocks.
337 Using dd_rescue on a block device (partition) and setting
338 hardbs to the block size of a file system that you want to
339 create, you should be able to feed the bbfile to mke2fs with
340 the option -l.
341 Multiple output files
343 -Y ofileX, --outfile=ofileX, --of=ofileX
344 If you want to copy data to multiple files simultaneously, you
345 can specify this option. It can be specified multiple times, so
346 many copies can be made. Note that these files are secondary
347 output files; they share file position with the primary output
348 file outfile. Errors when writing to a secondary output file
349 are ignored.
350 Data protection by overwriting with random numbers
352 -z RANDSEED, --random=RANDSEED
353 -Z RANDSEED, --frandom=RANDSEED
354 -2 RANDSEED, --shred2=RANDSEED
355 -3 RANDSEED, --shred3=RANDSEED
356 -4 RANDSEED, --shred4=RANDSEED
357 When you want to overwrite a file, partition or disk with
358 random data, using /dev/urandom (on Linux) as input is not a
359 very good idea; the interface has not been designed to yield a
360 high bandwidth. It's better to use a user space Pseudo Random
361 Number Generator (PRNG). With option -z / --random, the C
362 library's PRNG is used. With -Z / --frandom and the -2/-3/-4 /
363 --shred2/3/4 options, an RC4 based PRNG is used.
364 Note that in this mode, there is no infile so the first non-
365 option argument is the output file.
366 The PRNG needs seeding; the C libraries PRNG takes a 32bit
367 integer (4 bytes); the RC4 based PRNG takes 256 bytes. If
368 RANDSEED is an integer, the integer number will be used to seed
369 the C library's PRNG. For the RC4 method, the C library's PRNG
370 then generates the 256 bytes to seed it. This creates
371 repeatable PRNG data. The RANDSEED value of 0 is special; it
372 will create a seedval that's based on the current time and the
373 process' PID and should be different for multiple runs of
374 dd_rescue .
375 If RANDSEED is not an integer, it's assumed to be a file name
376 from which the seed values can be read. dd_rescue will read 4
377 or 256 bytes from the file to seed the C library's or the RC4
378 PRNG. For good pseudo random numbers, using /dev/urandom to
379 seed is a good idea.
380 The modes -2/-3/-4 resp. --shred2/--shred3/--shred4 will
381 overwrite the output file multiple times; after each pass,
382 fsync() will ensure that the data does indeed hit the file. The
383 last pass for these modes will overwrite the file with zeroes.
384 The rationale behind doing this is to make it easier to hide
385 that important data may have been overwritten, to make it
386 easier for intelligent storage systems (such as SSDs) to
387 recycle the empty blocks and to allow for better compression of
388 a file system image containing such data.
389 With -2 / --shred2, one pass with RC4 generated PRNG is
390 happening and then zeroes are written. With -3 / --shred3,
391 there are two passes with RC4 PRNG generated random numbers and
392 a zero pass; the second PRNG pass writes the inverse (bit-wise
393 reversed) numbers from the first pass. -4 / --shred4 works like
394 -3 / --shred3, with an additional pass with independent random
395 numbers as third pass.
396 Plugins
398 Since version 1.42, dd_rescue has an interface for plugins. Plugins
399 have the ability to analyze the copied data or to transform it prior to
400 it being written.
401 -L plugin1[=param1[:param2[:..]]][,plugin2[=..][,..]]
403 --plugins=plugin1[=param1[:param2[:..]]][,plugin2[=..][,..]]
404 loads plugins plugin1 ... and passes parameters to it. All
405 plugins should support at least the help parameter and provide
406 information on their usage.
407 Plugins may impose limits on dd_rescue. Plugins that look at
408 the data can't work with splice, as this avoids copying data to
409 user space. Also the interface currently does not facilitate
410 reverse direction copy. Some plugins may impose further
411 restrictions w.r.t. alignment of data in the file or not using
412 sparse detection.
413 See section PLUGINS for an overview of available plugins.
414
417 null
418 The null plugin (ddr_null) does nothing, except if you specify the
419 [no]lnchange or the [no]change options in which case the plugin
420 indicates to others that it transforms the length of the output or the
421 data of the stream. (With the no prefix, it's reset to the default no-
422 change indication again.) This may be helpful for testing or to
423 influence which file the hash plugin considers for reading/writing
424 extended attributes from/to and for plugins to change their behavior
425 with respect to hole detection.
426 ddr_null_ddr also allows you to specify debug in which case it just
427 reports the blocks that it passes on.
428 hash
430 When the hash plugin (subsequently referred to as ddr_hash) is loaded,
431 it will calculate a cryptographic hash and optionally also a HMAC over
432 the copied data and print the result at the end of the copy operations.
433 The hash algorithm can be chosen by specifying alg[o[rithm]]=ALG where
434 ALG is one of md5, sha1, sha256, sha224, sha512, sha384. (Specify
435 alg=help to get a list.) To abbreviate the syntax, the alg= piece can
436 be omitted.
437 For backwards compatibility, the hash plugin can also be referred to
438 with the old MD5 name; it then defaults to the md5 algorithm.
439 The computed value should be identical to calling md5sum/sha256sum/...
440 on the target file (unless you only write part of the file), but saves
441 time by not accessing the (possibly large) file a second time. The
442 hash plugin handles sparse writes and arbitrary offsets fine.
443 multipart=CHUNKSIZE tells ddr_hash to calculate multiple checksums for
445 file chunks of CHUNKSIZE each and then combine them into a combined
446 checksum by creating a checksum over the piece checksums. This is how
447 the checksum for S3 multipart objects is calculated (using the md5
448 hash); the output there is the combination checksum with a dash and the
449 number of parts appended.
450 Note that this feature is new in 1.99.6 and does not yet handle
451 situations cleanly, where offsets plus block sizes do not happen to
452 cleanly align with the CHUNKSIZE. The implementation for this will be
453 completed later. Other features like the append/prepend/hmac pieces
454 also don't work well with multipart checksum calculation.
455 ddr_hash also supports the parameter append=STRING which appends the
457 given STRING to the output before computing the cryptographic hash.
458 Treating the STRING as a shared secret, this can actually be used to
459 protect against someone not knowing the secret altering the contents
460 (and recomputing the hash) without anyone noticing. It's thus a cheap
461 way of a cryptographic signature (but with preshared secrets as opposed
462 to public key cryptography). Use HMAC for a somewhat better way to sign
463 data with a shared secret.
464 ddr_hash also supports prepend=STRING which is likely harder to attack
465 with brute force than an appended string. Note that ddr_hash always
466 prepends multiples of the hash algorithm's block size and pads the
467 STRING with 0 to match.
468 ddr_hash can be used to compute a HMAC (Hash-based Message
470 Authentication Code) instead of the plain hash. The HMAC uses a
471 password that's prepended and transformed twice to the data which is
472 then hashed twice. HMAC is believed to protect somewhat better against
473 extension or collision attacks than a plain hash (with a plain
474 prepended secret), so it's a better way to authenticate data with a
475 shared secret. (You can use append/prepend in addition to HMAC, if you
476 have a need for a scheme with more than one secret.)
477 When HMAC is enabled with one of the following parameters, both the
478 plain hash and the HMAC are computed by ddr_hash. Both are output to
479 the console/log, but the HMAC is used instead of the hash value to be
480 written to a CHECKSUMS file or to an extended attribute or checked
481 against (see below). hmacpwd=STRING sets the shared secret (password)
482 for computing the HMAC. Passing the secret on the command line has the
483 disadvantage that the shell may mistreat some bytes as special
484 characters and that the command line may be visible to all logged in
485 users on the system. hmacpwdfd=INT sets a file descriptor from with
486 the secret (password) for HMAC computation will be read. Specifying 0
487 means standard input, in which case ddr_hash even prints a prompt for
488 you ... Other numbers may be useful if dd_rescue is called from another
489 program that opens a pipe to pass the secret. hmacpwdnm=INNAME sets a
490 file from which the shared secret (password) is read. Note that all
491 bytes (up to 2048 of them) are read and used, including trailing white
492 space, 0-bytes or newlines.
493 Please note that the ddr_hash plugin at this point does NOT take a lot
494 of care to prevent the password/pre/appended secret from remaining in
495 memory or leaking into a swap/page file. (This will be improved once I
496 look into encryption plugins.)
497 ddr_hash accepts the parameter output , which will cause ddr_hash to
499 output the cryptographic hash to stdout in the same format that
500 md5sum/sha256sum/... use. You can also specify outfd=INT to have the
501 plugin write the hash to a different file descriptor specified by the
502 integer number INT. Note that ddr_hash always processes data in binary
503 mode and correctly indicates this with a star (*) in the output
504 generated with output/outfd=.
505 The checksum can also be written to a file by giving the outnm=OUTNAME
506 parameter. Then a file with OUTNAME will be created and a
507 md5sum/sha256sum/... compatible line will be printed to the file. If
508 the file exists and contains an entry for the file, it will be updated.
509 If the file exists and does not contain an entry for the file, one will
510 be appended. If OUTNAME is omitted, the file name CHECKSUMS.alg (or
511 HMACS.alg if HMAC is enabled) will be used (alg is replaced by the
512 chosen algorithm). If the checksum can't be written, a warning will be
513 printed and the exit code of dd_rescue will become non-zero.
514 The checksum can be validated using chknm=CHKNAME . The file will be
516 read and ddr_hash will look for an md5sum/sha256sum/... compatible
517 line with a matching file name to take the checksum from and compare it
518 to the one computed. If NAME is omitted, the same default as described
519 above (in outnm=...) will be used. You can also read the checksum from
520 stdin if you prefer by specifying the check option.
521 Note that in any case, the check is only performed after the copy
522 operation is completed -- a faulty checksum will thus NOT result in the
523 copy not taking place. However, the exit code of dd_rescue will
524 indicate the error. (If you want to avoid copying data with a broken
525 checksum into the final target, use a temporary target that you delete
526 upon error and only move to the final location if dd_rescue's exit
527 value is 0; you can of course also copy to /dev/null for testing
528 beforehand, but it might be too costly reading the input file twice.)
529 If in addition to chknm (or chk_xattr ) the option chkadd is specified,
530 then a missing checksum will not be reported as error, but instead an
531 entry to the checksum file (or xattr) be added. A mismatch will still
532 be reported as error and the checksum file will not be updated.
533 You can store the cryptographic hash into the files by using the
535 set_xattr option. The hash will be stored into the extended attribute
536 user.checksum.ALG by default (user.hmac.ALG if HMAC is enabled), but
537 you can override the name of the attribute by specifying
538 set_xattr=XATTR.NAME instead. If the xattr can't be written, an error
539 will be reported, unless you also specify the fallb[ack][=CHKNAME]
540 option. In that case, ddr_hash tries to write the checksum to the
541 CHKNAME checksums file. (For the default for CHKNAME, see outnm= option
542 above.)
543 chk_xattr will validate that the computed hash matches the one read
544 from the extended attribute. The same default attribute name applies
545 and you can likewise override it with chk_xattr=XATTR.NAME . A missing
546 attribute is considered an error (although the same fallback is tried
547 if you specify the fallback option). A broken checksum is of course
548 considered an error as well, but just like with checknm=CHKNAME won't
549 prevent the copy. See the discussion there.
550 Note that for output,outfd,outnm=,set_xattr ddr_hash will use the
552 output file name to attach the checksum to (be it by setting xattr or
553 the file name used in the checksum file), unless a plugin in the chain
554 after ddr_hash indicates that it changes the data. In that case, it
555 will warn and associate the checksum with the input file name, unless
556 there's another plugin before ddr_hash in the chain which indicates
557 data transformation as well. In that case, there is no file that the
558 checksum could be associated with and ddr_hash will report an error.
559 Likewise for chknm=,check,chk_xattr ddr_hash will use the input file
560 name to get the checksum (be it by reading the xattr or by looking for
561 the input file name in a checksums file) unless there's a plugin in the
562 chain before ddr_hash that indicates that it changes the data. The
563 output file name will then be used, unless there's another plugin after
564 ddr_hash indicating data change as well, in which case there's no file
565 we could get the checksum for and thus an error is reported.
566 If your system supports extended attributes, those have the advantage
568 of traveling with the files; thus a rename or copy (with dd_rescue -p)
569 will maintain the checksum. Checksum files on the other hand can be
570 handled everywhere (including the transfer via ftp or http) and can be
571 cryptographically signed with PGP/GnuPG.
572 Please note that the md5 algorithm is NOT recommended any more for good
574 protection against malicious attempts to hide data modification; it's
575 not considered strong enough any more to prevent hash collisions. sha1
576 is a bit better, but has been broken as well as of 2017. The
577 recommendation is to use the SHA-2 family of hashes. On 32bit
578 machines, I'd recommend sha256, while on 64bit machines, sha512 is
579 faster and thus the best choice.
580 ddr_hash also supports using the HMAC code and hashes for deriving keys
582 from passwords using the PKCS5 PBKDF2 (password-based key derivation
583 function) that allows you to improve the protection from mediocre
584 passwords by using a salt and a relatively expensive key stretching
585 operation. This is only meant for testing and may be removed in the
586 future. It's thus not documented in this man page. See the built-in
587 help function for a brief summary on the usage.
588 lzo
590 The lzo plugin allows to compress and decompress data using liblzo2.
591 lzo is an algorithm that is faster than most other algorithms but does
592 not compress as well. See the ddr_lzo(1) man page for more details.
593 crypt
595 The crypt plugin allows to encrypt and decrypt data on the fly. It
596 currently supports a variety of AES ciphers. See the ddr_crypt(1) man
597 page for more details.
598
600 On successful completion, dd_rescue returns an exit code of 0. Any
601 other exit code indicates that the program has aborted because of an
602 error condition or that copying of the data has not been entirely
603 successful.
604
606 dd_rescue -k -P -p -t infile outfile
607 copies infile to outfile and does truncate the output file on
608 opening (so deleting any previous data in it), copies mode,
609 times, ownership at the end, uses fallocate to reserve the space
610 for the output file and uses efficient in kernel splice copy
611 method.
612 dd_rescue -A -d -D -b 512 /dev/sda /dev/sda
614 reads the contents of every sector of disk sda and writes it
615 back to the same location. Typical hard disks reallocate flaky
616 and faulty sectors on writes, so this operation may result in
617 the complete disk being usable again when there were errors
618 before. Unreadable blocks however will contain zeroes after
619 this.
620 dd_rescue -2 /dev/urandom -M outfile
622 overwrites the file outfile twice; once with good pseudo random
623 numbers and then with zeroes.
624 dd_rescue -t -a image1.raw image2.raw
626 copies a file system image and looks for empty blocks to create
627 a sparse output file to save disk space. (If the source file
628 system has been used a bit, on that file system creating a large
629 file with zeroes and removing it again prior to this operation
630 will result in more sectors with zeroes.
631 dd_rescue -u /dev/zero DUMMY will achieve this ...)
632 dd_rescue -ATL hash=md5:output,lzo=compress:bench,MD5:output in out.lzo
634 copies the file in to out.lzo with using lzo (lzo1x_1)
635 compression and calculating an md5 hash (checksum) on both
636 files. The md5 hashes for both are also written to stdout in the
637 md5sum output format. Note that the compress parameter to lzo
638 is not strictly required here; the plugin could have deduced it
639 from the file names. This example shows that you can specify
640 multiple plugins with multiple parameters; the plugins are
641 forming a filter chain. You can specify the same plugin multiple
642 times.
643 dd_rescue -L hash=sha512:set_xattr:fallb,null=change infile /dev/null
645 reads the file infile and computes its sha512 hash. It stores it
646 in the input file's user.checksum.sha512 attribute (and falls
647 back to writing it to CHECKSUMS.sha512 if xattrs can't be
648 written). Note the use of the null plugin with faking data
649 change with the change parameter; this causes the hash plugin to
650 write to the input file which it would not normally have done.
651 Of course this will fail if you don't have the appropriate
652 privileges to write xattrs to infile nor to write the checksum
653 to CHECKSUMS.sha512.
654 See also README.dd_rescue and ddr_lzo(1) to learn about the
656 possibilities.
657
659 Untested code is buggy, almost always. I happen to have a damaged hard
660 disk that I use for testing dd_rescue from time to time. But to allow
661 for automated testing of error recovery, it's better to have
662 predictable failures for the program to deal with. So there is a fault
663 injection framework.
664 Specifying -F 5w/1,17r/3,42r/-1,80-84r/0 on the command-line will
665 result in in the 5th block (counted in hardblocksize) will fail to be
666 written once (from which dd_rescue should recover, as it tries a second
667 time for failed writes), block no 17 will fail to be read 3 times,
668 block no 42 will read fine once, but then fail afterwards, whereas
669 blocks 80 through 83 are completely unreadable (will fail infinite
670 times). Note that the range excludes the last block (80-84 means 4
671 blocks starting @ 80).
672 Block offsets are always counted in absolute positions, so starting in
673 the middle of a file with -s or reverse copying won't affect the
674 absolute position that is hit with the fault injection. (This has
675 changed since 1.98.)
676
679 The source code does use the 64bit functions provided by glibc for file
680 positioning. However, your kernel might not support it, so you might be
681 unable to copy partitions larger then 2GB into a file.
682 This program has been written using Linux and only tested on a couple
683 of Linux systems. People have reported to have successfully used it on
684 other Un*xish systems (such as xBSD or M*cOS), but these systems get
685 little regular test coverage; so please be advised to test properly
686 (possibly using the make check test suite included with the source
687 distribution) before relying on dd_rescue on non Linux based systems.
688 Currently, the escape sequence for moving the cursor up is hard coded
689 in the sources. It's fine for most terminal emulations (including vt100
690 and linux), but it should use the terminal description database
691 instead.
692 Since dd_rescue-1.10, non-seekable input or output files are supported,
693 but there's of course limitations to recover errors in such cases.
694 dd_rescue does not automate the recovery of faulty files or partitions
696 by automatically keeping a list of copied sectors and approaching bad
697 spots from both sides. There is a helper script dd_rhelp from LAB
698 Valentin that does this. Integration of such a mode into dd_rescue
699 itself is non-trivial and due to the complexity of the source code
700 might not happen.
701 There also is a tool, GNU ddrescue, that is a reimplementation of this
702 tool and which contains the capabilities to automate recovery of bad
703 files in the way dd_rhelp does. It does not have the feature richness
704 of dd_rescue, but is reported to be easier to operate for error
705 recovery than dd_rescue with dd_rhelp.
706 If your data is very valuable and you are considering sending your disk
708 to a data recovery company, you might be better off NOT trying to use
709 imaging tools like dd_rescue, dd_rhelp or GNU ddrescue. If you're
710 unlucky, the disk has suffered some mechanical damage (e.g. by having
711 been dropped), and continuing to use it may make the head damage the
712 surface further. You may be able to detect this condition by quickly
713 raising error counts in the SMART attributes or by a clicking noise.
714 Please report bugs to me via email.
716 Data destruction considerations
718 The modes for overwriting data with pseudo random numbers to securely
719 delete sensitive data on purpose only implement a limited number of
720 overwrites. While Peter Gutmann's classic analysis concludes that the
721 then current hard disk technology requires more overwrites to be really
722 secure, the author believes that modern hard disk technology does not
723 allow data restoration of sectors that have been overwritten with the
724 --shred4 mode. This is in compliance with the recommendations from BSI
725 GSDS M7.15.
726 Overwriting whole partitions or disks with random numbers is a fairly
727 safe way to destroy data, unless the underlying storage device does too
728 much magic. SSDs are doing fancy stuff in their Flash Translation Layer
729 (FTL), so this tool might be insufficient to get rid of data. Use
730 SECURITY_ERASE (use hdparm) there or -- if available -- encrypt data
731 with AES256 and safely destroy the key. Normal hard disks have a small
732 risk of leaking a few sectors due to reallocation of flaky sectors.
733 For securely destroying single files, your mileage may vary. The more
734 advanced your file system, the less likely dd_rescue's destruction will
735 be effective. In particular, journaling file systems may carry old
736 data in the journal. File systems that do copy-on-write (COW) such as
737 btrfs, are very likely to have old copies of your supposedly erased
738 file. It might help somewhat to fill the file systems with zeros
739 (dd_rescue -u /dev/zero /path/to/fs/DUMMYNAME) to force the file system
740 to release and overwrite non-current data after overwriting critical
741 files with random numbers. If you can, better destroy a whole partition
742 or disk.
743
745 README.dd_rescue README.dd_rhelp ddr_lzo(1)
746 wipe(1) shred(1) ddrescue(1) dd(1)
747
749 Kurt Garloff <kurt@garloff.de>
750
752 Many little issues were reported by Valentin LAB, the author of
753 dd_rhelp .
754 The RC4 PRNG (frandom) is a port from Eli Billauer's kernel mode PRNG.
755 A number of recent ideas and suggestions came from Thomas.
756
758 This program is protected by the GNU General Public License (GPL) v2 or
759 v3 - at your option.
760
762 Since version 1.10, non seekable input and output files are supported.
763 Splice copy -k is supported since 1.15.
764 A progress bar exists since 1.17.
765 Support for preallocation (fallocate) -P exists since 1.19.
766 Since 1.23, we default to -y0, enhancing performance.
767 The Pseudo Random Number modes have been started with 1.29.
768 Write avoidance -W has been implemented in 1.30
769 Multiple output files -Y have been added in 1.32.
770 Long options and man page came with 1.33.
771 Optimized sparse detection (SSE2, armv6, armv8 asm, AVX2) has been
772 present since 1.35 and been enhanced until 1.43.
773 We support copying extended attributes since 1.40 using libxattr.
774 Removing and (fs)trimming the output file's file system exists since
775 1.41. Support for compilation with bionic (Android's C library) with
776 most features enabled also came with 1.41.
777 Plugins exist since 1.42, the MD5 plugin came with 1.42, the lzo plugin
778 with 1.43. 1.44 renamed the MD5 plugin to hash and added support for
779 the SHA-2 family of hashes. 1.45 added SHA-1 and the ability to store
780 and validate checksums.
781 1.98 brought encryption and the fault injection framework, 1.99 support
782 for ARMv8 crypto acceleration. 1.99.5 brought ratecontrol. 1.99.6
783 brought S3 style multipart checksums.
784 Some additional information can be found on
786 http://garloff.de/kurt/linux/ddrescue/
787 LAB Valentin's dd_rhelp can be found on
788 http://www.kalysto.org/utilities/dd_rhelp/index.en.html
789Kurt Garloff 2017-06-23 dd_rescue(1)