1DDPT(8) DDPT DDPT(8)
2
6 ddpt - copies data between files and storage devices. Support for
7 devices that understand the SCSI command set.
8
10 ddpt [bpt=BPT[,OBPC]] [bs=BS] [cdbsz=6|10|12|16] [coe=0|1]
11 [coe_limit=CL] [conv=CONVS] [count=COUNT] [ibs=IBS] if=IFILE
12 [iflag=FLAGS] [obs=OBS] [of=OFILE] [of2=OFILE2] [oflag=FLAGS]
13 [retries=RETR] [seek=SEEK] [skip=SKIP] [status=STAT] [verbose=VERB]
14 [--help] [--verbose] [--version] [--wscan]
15 For comparison here is the synopsis for GNU's dd command:
17 dd [bs=BS] [cbs=CBS] [conv=CONVS] [count=COUNT] [ibs=IBS] [if=IFILE]
19 [iflag=FLAGS] [obs=OBS] [of=OFILE] [oflag=FLAGS] [seek=SEEK]
20 [skip=SKIP] [status=STAT] [--help] [--version]
21
23 Copy data between files or read data from a file. Specialized for
24 "files" that are storage devices, especially those that can use the
25 SCSI command sets (e.g. SATA and SAS disks, plus DVD drives). Can issue
26 SCSI commands in pass-through ("pt") mode. Similar syntax and semantics
27 to the Unix dd(1) command.
28 For comparison, the SYNOPSIS section above shows both the ddpt command
30 line options followed by GNU's dd(1) command line options. Broadly
31 speaking ddpt can be considered a super-set of dd. See the section on
32 DD DIFFERENCES for significant differences between ddpt and dd.
33 ddpt does a segmented copy, first reading in BPT*IBS bytes from IFILE
35 (or less if near the end of the copy) into a copy buffer. In the
36 absence of the various options and conditions that bypass the write
37 operation, the copy buffer is then written out to OFILE. The copy
38 process continues working its way along IFILE and OFILE until either
39 COUNT is exhausted, an end of file is detected, or an error occurs. If
40 IBS and OBS are different, ddpt restricts the value of OBS such that
41 the copy buffer is an integral number output blocks (i.e. (((IBS*BPT) %
42 OBS) == 0) ). In the following descriptions, "segment" refers to all or
43 part of a copy buffer.
44 The term "pt device" is used for a pass-through device to which SCSI
46 commands like READ(10) and WRITE(10) may be sent. A pt device may only
47 be able to process SCSI commands in which case the "pt" flag is
48 assumed. The ability to recognize such a pt only device may vary
49 depending on the operating system (e.g. in Linux '/dev/sg2' is recog‐
50 nized). However if a device can process either normal UNIX
51 read()/write() calls or pass-through SCSI commands then the default is
52 to use UNIX read()/write() calls. That default can be overridden by
53 using the "pt" flag (e.g. "if=/dev/sdc iflag=pt"). When pt access is
54 specified any partition information is ignored. So "if=/dev/sdc2
55 iflag=pt skip=3" will start at logical block address 3 of '/dev/sdc'.
56 As a protection measure in version 0.92 ddpt will only accept that if
57 the force flag is given in addition to pt.
58
60 bpt=BPT[,OBPC]
61 where BPT is Blocks Per Transfer. The copy is made up of multi‐
62 ple transfers, each first reading BPT input blocks (i.e. BPT*IBS
63 bytes) from IFILE into the copy buffer and then from that copy
64 buffer writing BPT*IBS/OBS output blocks to OFILE. This contin‐
65 ues until the copy is finished, with the last transfer being
66 potentially shorter. The default BPT value varies depending on
67 IBS. When IBS < 8, BPT is 8192; when IBS < 64, BPT is 1024; when
68 IBS < 1024, BPT is 128; when IBS < 8192, BPT is 16; when IBS <
69 32768, BPT is 4; else BPT defaults to 1. If BPT is given as 0 it
70 is treated as the default value. For "bs=512", BPT defaults to
71 128 so that 64 KiB (or less) is read from IFILE into the copy
72 buffer.
73 The optional OBPC (Output Blocks Per Check) argument controls
74 controls the granularity of sparse writes, write sparing and
75 trim checks. The default granularity is the size of the copy
76 buffer (i.e. BPT*IBS bytes). That can be reduced by specifying
77 OBPC. The finest granularity is when OBPC is 1 which implies the
78 unit of each check is OBS bytes. When OBPC is 0, or not given,
79 the default granularity is used. Large OBPC values are rounded
80 down so that OBPC*OBS does not exceed the size of the copy buf‐
81 fer.
82 bs=BS where BS is the IFILE and OFILE block size in bytes. Conflicts
84 with either "ibs=" or "obs=" options. The value of BS is placed
85 in IBS and OBS. If IFILE or OFILE is a "pt" device then BS must
86 be the logical block size of the device. See the DD DIFFERENCES
87 section below. Default is 512 which to date has been correct for
88 hard disks. Other logical block sizes are 2048 bytes for DVDs
89 and 4096 bytes for the coming generation of hard disks.
90 cdbsz=6 | 10 | 12 | 16
92 size of SCSI READ and/or WRITE commands issued on pt devices.
93 Default is 10 byte SCSI command blocks (unless calculations
94 indicate that a 4 byte block number may be exceeded or BPT is
95 greater than 16 bits (65535), in which case it defaults to 16
96 byte SCSI commands).
97 coe=0 | 1
99 set to 1 for continue on error. Applies to errors on input and
100 output pt devices plus input from block devices or regular
101 files. Errors on other files will stop ddpt. Default is 0 which
102 implies stop on any error. See the 'coe' flag for more informa‐
103 tion.
104 coe_limit=CL
106 where CL is the maximum number of consecutive bad blocks stepped
107 over (due to "coe=1") on reads before the copy terminates. The
108 default is 0 which is interpreted as no limit. This option is
109 meant to stop the copy soon after unrecorded media is detected
110 while still offering "continue on error" capability.
111 conv=CONVS
113 see the CONVERSIONS section below.
114 count=COUNT
116 copy COUNT input blocks from IFILE to OFILE. If this option is
117 not given (or COUNT is '-1') then the COUNT may be deduced from
118 either IFILE or OFILE. See the COUNT section below.
119 ibs=IBS
121 where IBS is the IFILE block size in bytes. The default value is
122 BS or its default (512). Conflicts the "bs=" option (e.g. giving
123 both "bs=512 ibs=512" is considered a syntax error).
124 if=IFILE
126 read from IFILE. This option must be given (i.e. it is manda‐
127 tory). If IFILE is '-' then stdin is read. Starts reading at the
128 beginning of IFILE unless SKIP is given.
129 iflag=FLAGS
131 where FLAGS is a comma separated list of one or more flags out‐
132 lined in the FLAGS section below. These flags are associated
133 with IFILE and are ignored when IFILE is stdin.
134 obs=OBS
136 where OBS is the OFILE block size in bytes. The default value is
137 BS or its default (512). Conflicts the "bs=" option (e.g. giving
138 both "bs=512 obs=512" is considered a syntax error). If OBS is
139 given then it has the following restriction: the integer expres‐
140 sion (((IBS * BPT) % OBS) == 0) must be true. Stated another
141 way: the copy buffer size must be an integral multiple of OBS.
142 If of2=OFILE2 is given then OBS is its block size as well.
143 of=OFILE
145 write to OFILE. The default value is /dev/null . If OFILE is '-'
146 then writes to stdout. If OFILE is /dev/null then no actual
147 writes are performed. If OFILE is '.' (period) then it is
148 treated the same way as /dev/null . If OFILE exists then it is
149 _not_ truncated unless "oflag=trunc" is given. See section on DD
150 DIFFERENCES.
151 of2=OFILE2
153 write output to OFILE2. The default action is not to do this
154 additional write (i.e. when this option is not given). OFILE2 is
155 assumed to be a regular file or a fifo (i.e. a named pipe).
156 OFILE2 is opened for writing and is created if necessary. If
157 OFILE2 is a fifo (named pipe) then some other command should be
158 consuming that data (e.g. 'md5sum OFILE2'), otherwise this util‐
159 ity will block. The write to OFILE2 occurs before the write to
160 OFILE and prior to sparse writing and write sparing logic. So
161 everything read is written to OFILE2.
162 oflag=FLAGS
164 where FLAGS is a comma separated list of one or more flags out‐
165 lined in the FLAGS section. These flags are associated with
166 OFILE and are ignored when OFILE is /dev/null, '.' (period), or
167 stdout.
168 retries=RETR
170 sometimes retries at the host are useful, for example when there
171 is a transport error. When RETR is greater than zero then SCSI
172 READs and WRITEs are retried on error, RETR times. Default value
173 is zero. Only applies to errors on pt devices.
174 seek=SEEK
176 start writing SEEK blocks (each of OBS bytes) from the start of
177 OFILE. Default is block 0 (i.e. start of file). The SEEK value
178 may exceed the number of OBS-sized blocks in OFILE.
179 skip=SKIP
181 start reading SKIP blocks (each of IBS bytes) from the start of
182 IFILE. Default is block 0 (i.e. start of file). The SKIP value
183 must be less than or equal to the number of IBS-sized blocks in
184 IFILE.
185 status=STAT
187 the STAT value of 'noxfer' suppresses the throughput speed and
188 the copy time output at the end of the copy. The "status=noxfer"
189 option was recently introduced to GNU's dd command. The default
190 action of ddpt is to show the throughput (in megabytes per sec‐
191 ond) and the time taken to do the copy after the "records in"
192 and "records out" lines at the end of the copy. As a conve‐
193 nience the value 'null' is accepted for STAT and does nothing.
194 verbose=VERB
196 as VERB increases so does the amount of debug output sent to
197 stderr. Default value is zero which yields the minimum amount
198 of debug output. A value of 1 reports extra information that is
199 not repetitive. A value 2 reports cdbs and responses for SCSI
200 commands that are not repetitive (i.e. other that READ and
201 WRITE). Error processing is not considered repetitive. Values of
202 3 and 4 yield output for all SCSI commands, plus Unix read() and
203 write() calls, so there can be a lot of output. If VERB is "-1"
204 then output otherwise sent to stderr is redirected to /dev/null
205 .
206 -h, --help
208 outputs usage message and exits.
209 -v, --verbose
211 equivalent of verbose=1. If --verbose appears twice then that is
212 equivalent to verbose=2. Also -vv is equivalent to verbose=2.
213 -V, --version
215 outputs version number information and exits.
216 -w, --wscan
218 this option is available in Windows only. It lists storage
219 device names and the corresponding volumes, if any. When used
220 twice it adds the "bus type" of the closest transport (e.g. a
221 SATA disk in a USB connected enclosure has bus type Usb). When
222 used three times a SCSI adapter scan is added. When used four
223 times only a SCSI adapter scan is shown. See EXAMPLES section
224 below and the README.win32 file.
225
227 When the count=COUNT option is not given (or COUNT is '-1') then an
228 attempt is made to deduce COUNT as follows.
229 When both or either IFILE and OFILE are block devices, then the minimum
231 size, expressed in units of input blocks, is used. When both or either
232 IFILE and OFILE are pass-through devices, then the minimum size,
233 expressed in units of input blocks, is used.
234 If a regular file is used as input, its size, expressed in units of
236 input blocks (and rounded up if necessary) is used. Note that the
237 rounding up of the deduced COUNT may result in a partial read of the
238 last input block and a corresponding partial write to OFILE if it is a
239 regular file.
240 The size of pt devices is deduced from the SCSI READ CAPACITY command.
242 Block device sizes (or their partition sizes) are obtained from the
243 operating system, if available.
244 If skip=SKIP or skip=SEEK are given and the COUNT is deduced (i.e. not
246 explicitly given) then that size is scaled back so that the copy will
247 not overrun the file or device.
248 If COUNT is not given and IFILE is a fifo (and stdin is treated as a
250 fifo) then IFILE is read until an EOF is detected. If COUNT is not
251 given and IFILE is a /dev/zero (or equivalent) then zeros are read
252 until an error occurs (e.g. file system full).
253 If COUNT is not given and cannot be deduced then an error message is
255 issued and no copy takes place.
256
258 One or more conversions can be given to the "conv=" option. If more
259 than one is given, they should be comma separated. ddpt does not per‐
260 form the traditional dd conversions (e.g. ASCII to EBCDIC). Recently
261 added conversions overlap somewhat with the flags so some conversions
262 are now supported by ddpt.
263 fdatasync
265 equivalent to "oflag=fdatasync". Flushes data associated with
266 the OFILE to storage at the end of the copy. This conversion is
267 for compatibility with GNU's dd.
268 fsync equivalent to "oflag=fsync". Flushes data and meta-data associ‐
270 ated with the OFILE to storage at the end of the copy. This con‐
271 version is for compatibility with GNU's dd.
272 noerror
274 this conversion is very close to "iflag=coe" and is treated as
275 such. See the "coe" flag. Note that an error on OFILE will stop
276 the copy.
277 null has no affect, just a placeholder.
279 resume See "resume" in the FLAGS sections for more information.
281 sparing
283 See "sparing" in the FLAGS sections for more information.
284 sparse FreeBSD supports "conv=sparse" so the same syntax is supported
286 in ddpt. See "sparse" in the FLAGS sections for more informa‐
287 tion.
288 sync is ignored by ddpt. With dd it means supply zero fill (rather
290 than skip) and is typically used like this "conv=noerror,sync"
291 to have the same functionality as ddpt's "iflag=coe".
292 trunc if OFILE is a regular file then truncate it prior to starting
294 the copy. See "trunc" in the FLAGS section.
295
297 A list of flags and their meanings follow. The flag name is followed by
298 one or two indications in square brackets. The first indication is
299 either "[i]", "[o]" or "[io]" indicating this flag is active for the
300 IFILE, OFILE or both the IFILE and the OFILE. The second indication
301 contains some combination of "reg", "blk" or "pt" indicating whether
302 the flag applies to a regular file, a block device (accessed via Unix
303 read() and write() commands) or a pass-through device respectively.
304 append [o] [reg]
306 causes the O_APPEND flag to be added to the open of OFILE. For
307 regular files this will lead to data appended to the end of any
308 existing data. Conflicts the seek=SEEK option. The default
309 action of this utility is to overwrite any existing data from
310 the beginning of OFILE or, if SEEK is given, starting at block
311 SEEK. Note that attempting to 'append' to a device file (e.g. a
312 disk) will usually be ignored or may cause an error to be
313 reported.
314 coe [io] [pt], [i] [reg,blk]
316 continue on error. 'iflag=coe oflag=coe' and 'coe=1' are equiva‐
317 lent. Errors occurring on output regular or block files will
318 stop ddpt. Error messages are sent to stderr. This flag is sim‐
319 ilar to 'conv=noerror,sync' in the dd(1) utility. Unrecovered
320 errors are counted and output in the summary at the end of the
321 copy.
322 This paragraph is about coe on pt devices. A medium, hardware or
324 blank check error while reading will re-read blocks prior to the
325 bad block, then try to recover the bad block, supplying zeros if
326 that fails, and finally reread the blocks after the bad block. A
327 medium, hardware or blank check error while writing is noted and
328 ignored. SCSI disks may automatically try and remap faulty sec‐
329 tors (see the AWRE and ARRE in the read write error recovery
330 mode page (the sdparm utility can access these attributes)). If
331 bad LBAs are reported by the pass-through then the LBA of the
332 lowest and highest bad block is also output.
333 This paragraph is about coe on input regular files and block
335 devices. When a EIO or EREMOTEIO error is detected on a normal
336 segment read then the segment is re-read one block (i.e. IBS
337 bytes) at a time. Any block that yields a EIO or EREMOTEIO error
338 is replaced by zeros. Any other error, a short read or an end of
339 file will terminate the copy, usually after the data that has
340 been read is written to the output file.
341 direct [io] [reg,blk]
343 causes the O_DIRECT flag to be added to the open of IFILE and/or
344 OFILE. This flag requires some memory alignment on IO. Hence
345 user memory buffers are aligned to the page size. May have no
346 effect on pt devices. This flag will bypass caching/buffering
347 normally done by block layer. Beware of data coherency issues if
348 the same locations have been recently accessed via the block
349 layer in its normal mode (i.e. non-direct). See open(2) man
350 page.
351 dpo [io] [pt]
353 set the DPO bit (disable page out) in SCSI READ and WRITE com‐
354 mands. Not supported for 6 byte cdb variants of READ and WRITE.
355 Indicates that data is unlikely to be required to stay in device
356 (e.g. disk) cache. May speed media copy and/or cause a media
357 copy to have less impact on other device users.
358 errblk [i] [pt] [experimental]
360 attempts to create or append to a file called "errblk.txt" in
361 the current directory the logical block addresses of blocks that
362 cannot be read. The first (appended) line is "# start <time‐
363 stamp>". That is followed by the LBAs in hex (and prefixed with
364 "0x") of any block that cannot be read, one LBA per line. If the
365 sense data does not correctly identify the LBA of the first
366 error in the range it was asked to read then a LBA range is out‐
367 put in the form of the lowest and the highest LBA in the range
368 separated by a "-". At the end of the copy a line with "# stop
369 <timestamp>" is appended to "errblk.txt". Typically used with
370 "coe".
371 excl [io] [reg,blk]
373 causes the O_EXCL flag to be added to the open of IFILE and/or
374 OFILE. See open(2) man page.
375 fdatasync [o] [reg,blk]
377 Flushes data associated with the OFILE to storage at the end of
378 the copy.
379 flock [io] [reg,blk,pt]
381 after opening the associated file (i.e. IFILE and/or OFILE) an
382 attempt is made to get an advisory exclusive lock with the
383 flock() system call. The flock arguments are "FLOCK_EX |
384 FLOCK_NB" which will cause the lock to be taken if available
385 else a "temporarily unavailable" error is generated. An exit
386 status of 90 is produced in the latter case and no copy is done.
387 See flock(2) man page.
388 force [io] [pt]
390 override difference between given block size and the block size
391 found by the SCSI READ CAPACITY command. Use the given block
392 size. Without this flag the copy would not be performed. pt
393 access to what appears to be a block partition is aborted in
394 version 0.92; that can be overridden by the force flag. For
395 related reasons the 'norcap' flag requires this flag when
396 applied to a block device accessed via pt.
397 fsync [o] [reg,blk]
399 Flushes data and metadata (describing the file) associated with
400 the OFILE to storage at the end of the copy.
401 fua [io] [pt]
403 causes the FUA (force unit access) bit to be set in SCSI READ
404 and/or WRITE commands. The 6 byte variants of the SCSI READ and
405 WRITE commands do not support the FUA bit.
406 fua_nv [io] [pt]
408 causes the FUA_NV (force unit access non-volatile cache) bit to
409 be set in SCSI READ and/or WRITE commands. This only has an
410 effect with pt devices. The 6 byte variants of the SCSI READ
411 and WRITE commands do not support the FUA_NV bit.
412 nocache [io] [reg,blk]
414 use posix_fadvise() to advise corresponding file there is no
415 need to fill the file buffer with recently read or written
416 blocks. If used with "iflag=" it will increase the read ahead on
417 IFILE.
418 norcap [io] [pt]
420 do not perform SCSI READ CAPACITY command on the corresponding
421 pt device. If used on block device accessed via pt then 'force'
422 flag is also required. This is to warn about using pt access on
423 what may be a block device partition.
424 nowrite [o] [reg,blk,pt]
426 bypass writes to OFILE. The "records out" count is not incre‐
427 mented. OFILE is still opened but "oflag=trunc" if given is
428 ignored. Also the ftruncate call associated with the sparse flag
429 is ignored (i.e. bypassed). Commands such as trim and SCSI SYN‐
430 CHRONIZE CACHE are still sent.
431 null [io]
433 has no affect, just a placeholder.
434 pt [io] [blk,pt]
436 causes a device to be accessed in "pt" mode. In "pt" mode SCSI
437 READ and WRITE commands are sent to access blocks rather than
438 standard UNIX read() and write() commands. The "pt" mode may be
439 implicit if the device is only capable of passing through SCSI
440 commands (e.g. the /dev/sg devices in Linux). This flag is
441 needed for device nodes that can be accessed both via standard
442 UNIX read() and write() commands as well as SCSI commands. Such
443 devices default standard UNIX read() and write() commands in the
444 absence of this flag.
445 resume [o] [reg]
447 when a copy is interrupted (e.g. with Control-C from the key‐
448 board) then using the same invocation again with the addition of
449 "oflag=resume" will attempt to restart the copy from the point
450 of the interrupt (or just before that point). It is harmless to
451 use "oflag=resume" when OFILE doesn't exist or is zero length.
452 If the length of OFILE is greater than or equal to the length
453 implied by a ddpt invocation that includes "oflag=resume" then
454 no further data is copied.
455 self [io] [pt]
457 used together with trim flag to do a self trim (trim of segments
458 of a pt device that contain all zeros). If OFILE is not given,
459 then it is set to the same as IFILE. If SEEK is not given it set
460 to the same value as SKIP (possibly adjusted if IBS and OBS are
461 different). Implicitly sets "nowrite" flag.
462 sparing [o] [reg,blk,pt]
464 during the copy each IBS * BPT byte segment is read from IFILE
465 into a buffer. Then, instead of writing that buffer to OFILE,
466 the corresponding segment is read from OFILE into another buf‐
467 fer. If the two buffers are different, the former buffer is
468 written to the OFILE. If the two buffers compare equal then the
469 write to OFILE is not performed. Write sparing is useful when a
470 write operation is significantly slower than a read. Under some
471 conditions flash memory devices have slow writes plus an upper
472 limit on the number of times the same cell can be rewritten. The
473 granularity of the comparison can be reduced from the default
474 IBS * BPT byte segment with the the OBPC value given to the
475 "bpt=" option. The finest granularity is when OBPC is 1 which
476 implies OBS bytes.
477 sparse [o] [reg,blk,pt]
479 after each IBS * BPT byte segment is read from IFILE, it is
480 checked to see if it is all zeros. If so, that segment is not
481 written to OFILE. See the section on SPARSE WRITES below. The
482 granularity of the zero comparison can be reduced from the
483 default IBS * BPT byte segment with the OBPC value given to the
484 "bpt=" option.
485 ssync [o] [pt]
487 if OFILE is in "pt" mode then the SCSI SYNCHRONIZE CACHE command
488 is sent to OFILE at the end of the copy.
489 strunc [o] [reg]
491 perform a sparse copy with a ftruncate system call to extend the
492 length of the OFILE if required. See the sparse flag and the
493 section on SPARSE WRITES below.
494 sync [io] [reg,blk]
496 causes the O_SYNC flag to be added to the open of IFILE and/or
497 OFILE. See open(2) man page.
498 trim [io] [pt] [experimental]
500 similar logic to the "sparse" option. However instead of skip‐
501 ping segments that are full of zeros a "trim" command is sent to
502 OFILE. Usually set as an oflag argument but for self trim can be
503 used as an iflag argument (e.g. "iflag=self,trim"). Depending on
504 the usage this may require the device to support "deterministic
505 read zero after trim". See the TRIM, UNMAP AND WRITE SAME sec‐
506 tion below.
507 trunc [o] [reg]
509 if OFILE is a regular file then it is truncated prior to start‐
510 ing the copy. If SEEK is not given or 0 then OFILE is truncated
511 to zero length; when SEEK is larger than zero the truncation
512 takes place at file byte pointer SEEK*OBS. Ignored if
513 "oflag=append". Conflicts with "oflag=sparing".
514
516 Bypassing writes of blocks full of zeros can save a lot of IO. However
517 with regular files, bypassed writes at the end of the copy can lead to
518 an OFILE which is shorter than it would have been without sparse
519 writes. This can lead to integrity checking programs like md5sum and
520 sha1sum generating different values.
521 This utility has two ways of handling this file length problem: writing
523 the last block (even if it is full of zeros) or using the ftruncate
524 system call. A third approach is to ignore the problem (i.e. leaving
525 OFILE shorter). The ftruncate approach is used when "oflag=strunc"
526 while the last block is written when "oflag=sparse". To ignore the file
527 length issue use "oflag=sparse,sparse". Note that if OFILE's length is
528 already correct or longer than required, no action is taken.
529 The support for sparse writing of regular files may depend on the OS,
531 the file system and the settings of OFILE. POSIX makes few guarantees
532 when the ftruncate system call is used to extend a file's length, as
533 may occur when "oflag=strunc". Further, primitive file systems like
534 VFAT may not accept sparse writes or simulate the effect by writing
535 blocks of zeros. The latter approach will defeat any sparse writing
536 performance gain.
537
539 This is a new storage feature often associated with Solid State Disks
540 (SSDs) or disk arrays with "thin provisioning". In the ATA command set
541 (ACS-2) the relevant command is DATA SET MANAGEMENT with the TRIM bit
542 set. In the SCSI command set (SBC-3) it is either the UNMAP or WRITE
543 SAME command. Note there is no TRIM command however the term is fre‐
544 quently used in the technical press.
545 Trim is a way of telling a storage device that blocks are no longer
547 needed. Keeping the pool of unwritten blocks large is important for
548 the write performance of SSDs and the thrifty use of real storage in
549 thin provisioned arrays. Currently file systems in recent OSes may
550 issue trims associated with file deletes. The trim option in ddpt may
551 be useful when a partition or a whole SSD is to be "deleted". Note that
552 ddpt is bypassing file systems in that it only offers trim on
553 pass-through (pt) devices.
554 This utility issues SCSI commands to pt devices and for "trim" cur‐
556 rently issues a SCSI WRITE SAME(16) command with the UNMAP bit set. If
557 the pt device is a SSD with a ATA interface then recent versions of
558 Linux will translate the SCSI WRITE SAME to the ATA DATA SET MANAGEMENT
559 command with the TRIM bit set. The maximum size of each "trim" command
560 sent is the size of the copy buffer (i.e. IBS * BPT bytes). And that
561 maximum can be reduced with the OBPC argument of the "bpt=" option.
562 The trim can be used various ways. One way is a copy where the copy
564 buffer (or some part of it) is checked for zeros as is done by the
565 sparse oflag. When a zero segment is found, a trim "command" is sent to
566 the OFILE. For example:
567 ddpt if=dsk.img bs=512 of=/dev/sdc oflag=pt,trim
569 The copy buffer is 64 KiB (since BPT and OBPC default to 128 when
571 "bs=512") and it is checked for all zeros. If it is all zeros then a
572 trim command is sent to the corresponding location of /dev/sdc which is
573 accessed via the pt interface. If it is not all zeros then a SCSI WRITE
574 command is sent. Another way is to trim all or part of a disk. To trim
575 a whole disk (i.e. deleting all its data):
576 ddpt if=/dev/zero bs=512 of=/dev/sdc oflag=pt,trim
578 A third way is to "self-trim" which is to only trim those parts of a
580 disk that contain segments full of zeros:
581 ddpt if=/dev/sdc skip=0x2300 bs=512 iflag=pt,self,trim
583 count=0x1234f0
584 The "self" oflag automatically sets up the output side of the copy to
586 send trim commands (if required) back the the same device (i.e.
587 /dev/sdc). If this example was self-trimming a partition then the par‐
588 tition would start at LBA 0x2300 and be 0x1234f0 blocks long.
589 Some random product examples: the Intel X25-M G2 SSDs have trim with
591 recent firmware and they do deterministic read zero after trim. The
592 Seagate Pulsar SSD has an ATA interface which supports the determinis‐
593 tic reads of zero after the DATA SET MANAGEMENT command with the TRIM
594 option.
595
597 dd defaults "if=" and "of=" to stdin and stdout respectively. This fol‐
598 lows Unix filter conventions. However since dd and ddpt are often used
599 to read binary data for timing purposes, having to supply
600 "of=/dev/null" can be easily forgotten. Without it dd will potentially
601 spew binary data on the console. So ddpt has changed its defaults: the
602 "if=IFILE" is now mandatory and to read from stdin "if=-" can be used;
603 "of=OFILE" remains optional but its default changes to "/dev/null" (or
604 "NUL" in Windows). To send output to stdout ddpt accepts "of=-".
605 dd truncates OFILE unless "conv=notrunc" is given. When dd truncates,
607 it truncates to zero length unless SEEK is greater than zero. ddpt does
608 not truncate OFILE by default. If OFILE exists it will be overwritten.
609 The overwrite starts at block zero unless SEEK or "oflag=append" is
610 given. If OFILE is a regular file then "oflag=trunc" (or "conv=trunc")
611 will truncate OFILE prior to the copy.
612 Numeric arguments to ddpt can be given in hexadecimal, either with a
614 leading "0x" or "0X" or with a trailing "h". Note that dd accepts
615 "0x123" but interprets it as "0 * 123" (i.e. zero). ddpt will also
616 interpret "x" as multiplies unless the left operand is zero (e.g.
617 "0x123"). So both dd and ddpt will interpret "skip=2x123" as
618 "skip=246".
619 Terabyte size disks make it impractical to copy all the data into a
621 buffer before writing it out. Therefore both dd and ddpt read a rela‐
622 tively small amount of data into a copy (or transfer) buffer then write
623 it out to the destination, repeating this process until the COUNT is
624 exhausted.
625 A major difference in ddpt is the addition of BPT to control the size
627 of the copy buffer. With dd, IBS is the size of the copy buffer and the
628 unit of SKIP and COUNT. With ddpt, IBS * BPT is the size of the copy
629 buffer and IBS is the unit of SKIP and COUNT. This allows ddpt to have
630 its IBS set to the logical block size of IFILE without unduly restrict‐
631 ing the size of the copy buffer. And setting IBS (and OBS for OFILE)
632 accurately is required when the pass-through interface is used since
633 with the SCSI READ and WRITE commands the logical block size is
634 implicit.
635 The way dd handles its copy buffer (outlined in SUSv4 description of
637 dd) is relatively complex, especially when IBS and OBS are different
638 sizes. The restriction that ddpt places on IBS and OBS ( i.e.
639 (((IBS*BPT) % OBS) == 0) ) means that a single copy buffer can be used
640 since its size is a multiple of both IBS and OBS. Being able to pre‐
641 cisely define the copy buffer size in ddpt makes sparse writing, write
642 sparing and trim operations simpler to define and the user to control.
643 ddpt does not support dd's "cbs=" option (conversion block size). If
645 the "cbs=" option is given to ddpt then it is ignored.
646
648 A partial write is a write to the OFILE of less than OBS bytes. This
649 typically occurs at the end of a copy. dd can do partial writes. ddpt
650 does partial writes to regular files and fifos (including stdout). How‐
651 ever ddpt ignores partial writes when OFILE is a block device or a pt
652 device. When ddpt ignores a partial write, it sends a warning to the
653 console (stderr).
654 At the end of the copy two lines are output to the console:
656 <in_full>+<in_partial> records in
657 <out_full>+<out_partial> records out
658 The "records in" line is the number of full input blocks (each of IBS
660 bytes) that have been read plus the number of partial blocks ( usually
661 less than IBS bytes) that have been read. Following the lead of dd when
662 'iflag=coe' is active a block that cannot be read (and has zeros sub‐
663 stituted for its output) is regarded as a partial read. The "records
664 out" line is the number of full output blocks (each of OBS bytes) that
665 have been written plus the number of partial blocks (usually less than
666 OBS bytes) that have been written.
667 Block devices (e.g. /dev/sda and /dev/hda) can be given for IFILE. If
669 neither 'iflag=direct' nor 'iflag=pt' is given then normal block IO
670 involving buffering and caching is performed. If 'iflag=direct' is
671 given then the buffering and caching is bypassed (this is applicable to
672 both SCSI devices and ATA disks). When 'iflag=pt' is given SCSI com‐
673 mands are sent to the device which bypasses most of the actions per‐
674 formed by the block layer. The same applies for block devices given
675 for OFILE.
676 BPT, BS, COUNT, IBS, OBPC, OBS, SKIP and SEEK may include one of these
678 multiplicative suffixes: c C *1; w W *2; b B *512; k K KiB *1,024; KB
679 *1,000; m M MiB *1,048,576; MB *1,000,000 . This pattern continues for
680 "G", "T" and "P". The latter two suffixes can only be used for COUNT,
681 SKIP and SEEK. Also a suffix of the form "x<n>" multiplies the leading
682 number by <n>; however the combinations "0x" and "0X" are treated dif‐
683 ferently, see the next paragraph. These multiplicative suffixes are
684 compatible with GNU's dd command (since 2002) which claims compliance
685 with the SI and with IEC 60027-2 standards.
686 Alternatively numerical values can be given in hexadecimal preceded by
688 either "0x" or "0X" (or with a trailing "h" or "H"). When hex numbers
689 are given, multipliers cannot be used.
690 The COUNT, SKIP and SEEK arguments can take 64 bit values (i.e. very
692 big numbers). Other numerical values are limited to what can fit in a
693 signed 32 bit number.
694 All informative, warning and error output is sent to stderr so that
696 dd's output file can be stdout and remain unpolluted. If no options are
697 given, then the usage message is output and nothing else happens.
698 Disk partition information can often be found with fdisk(8) [the "-ul"
700 argument is useful in this respect]. Also parted(8) can be used like
701 this: 'parted /dev/sda unit s print' .
702 For pt devices this utility issues SCSI READ and WRITE (SBC) commands
704 which are appropriate for disks and reading from CD/DVD/BD drives.
705 Those commands are not formatted correctly for tape devices so ddpt
706 should not be used on tape devices. If the largest block address of the
707 requested transfer exceeds a 32 bit block number (i.e 0xffffffff) then
708 a warning is issued and the sg device is accessed via SCSI READ(16) and
709 WRITE(16) commands.
710 The attributes of a block device (e.g. partitions) are ignored when the
712 pt flag is used. Hence the whole device is read (rather than just the
713 second partition) by this invocation:
714 ddpt if=/dev/sdb2 iflag=pt of=t bs=512
716 Assuming /dev/sdb and /dev/sg2 refer to the same device, then after the
718 following two invocations, the contents of the files "t", "tt" and
719 "ttt" should be same:
720 ddpt if=/dev/sdb of=tt bs=512
722 ddpt if=/dev/sg2 of=ttt bs=512
724
726 The examples in this page use Linux device names. For suitable device
727 names in other supported Operating Systems see this web page:
728 http://sg.danny.cz/sg/device_name.html . The sg3_utils(8) man page in
729 the sg3_utils package also covers device naming.
730 ddpt usage looks quite similar to dd:
732 ddpt if=/dev/sg0 of=t bs=512 count=1MB
734 This will copy 1 million 512 byte blocks from the device associated
736 with /dev/sg0 (which should have 512 byte blocks) to a file called t.
737 Assuming /dev/sda and /dev/sg0 are the same device then the above is
738 equivalent to:
739 dd if=/dev/sda iflag=direct of=t bs=512 count=1000000
741 although dd's speed may improve if bs was larger and count was suitably
743 reduced. The use of the 'iflag=direct' option bypasses the buffering
744 and caching that is usually done on a block device.
745 The dd command's bs argument can be thought of as roughly equivalent to
747 ddpt's bs*bpt . dd almost assumes buffering on a block device and will
748 work as long as bs is a multiple of the actual logical block size.
749 Since ddpt can work at a lower level in some cases the bs argument must
750 be a disk's actual logical block size. Thus the bpt argument was intro‐
751 duced to make the copy more efficient. So these two invocations are
752 roughly equivalent:
753 dd if=/dev/sda of=t bs=8k count=64
755 ddpt if=/dev/sda of=t bs=512 bpt=16 count=1k
756 In both cases the total number of bytes moved is bs*count . And that
758 will be done by reading 8k (8192 bytes) into a buffer then writing out
759 that buffer to the file t. The read write sequence continues until the
760 count is complete or an error occurs.
761 The 'of2=' option can save time when the input would otherwise need to
763 be read twice. For example, to copy data and take a md5sum of it with‐
764 out needing to re-read the data:
765 mkfifo fif
767 md5sum fif &
768 ddpt if=/dev/sg3 iflag=coe of=sg3.img oflag=sparse of2=fif bs=512
769 This will image /dev/sg3 (e.g. an unmounted disk) and place the con‐
771 tents in the (sparse) file sg3.img . Without re-reading the data it
772 will also perform a md5sum calculation on the image.
773 Now we use sparse writing logic to get some idea of how many blocks on
775 a disk are full of zeros. After a SCSI FORMAT or an ATA SECURITY ERASE
776 command a disk may be all zeros.
777 ddpt if=/dev/sdc bs=512 oflag=sparse
779 Since no "of=" option is given, output goes to /dev/null so nothing is
781 actually written so the "records out" will be zero. However there will
782 be a count of "records in" and "bypassed records out". If /dev/sdc is
783 full of zeros then "records in" and "bypassed records out" will be the
784 same. Since the "bpt=" option is not given it defaults to "bpt=128,128"
785 so the copy buffer will be 64 KiB and the sparse check for zeros will
786 be done with 64 KiB (128 block) granularity.
787 For examples of the trim and self,trim options see the section above on
789 TRIM, UNMAP AND WRITE SAME.
790 Following is an example run on a Windows OS using the '--wscan' option
792 which shows the available device names (e.g. PD1) and the associated
793 volume name(s):
794 ddpt -w
796 PD0 [C] FUJITSU MHY2160BH 0000
797 PD1 [DF] WD 2500BEV External 1.05 WD-WXE90
798 CDROM0 [E] MATSHITA DVD/CDRW UJDA775 CB03
799 So, for example, volumes D: and F: reside on PhysicalDisk1 (abbreviated
801 to "PD1") which is manufactured by WD (Western Digital).
802 Further examples can be found on this web page:
804 http://sg.danny.cz/sg/ddpt.html . There is a text file called
805 ddpt_examples.txt in the "doc" directory of this package's distribution
806 tarball.
807
809 The signal handling has been borrowed from dd: SIGINT, SIGQUIT and SIG‐
810 PIPE output the number of remaining blocks to be transferred and the
811 records in + out counts; then they have their default action. SIGUSR1
812 causes the same information to be output and the copy continues. All
813 output caused by signals is sent to stderr.
814
816 To aid scripts that call ddpt, the exit status is set to indicate suc‐
817 cess (0) or failure (1 or more). Note that some of the lower values
818 correspond to the SCSI sense key values. The exit status values are:
819 0 success
821 1 syntax error. Either illegal command line options, options with
823 bad arguments or a combination of options that is not permitted.
824 2 the device reports that it is not ready for the operation
826 requested. The device may be in the process of becoming ready
827 (e.g. spinning up but not at speed) so the utility may work
828 after a wait.
829 3 the device reports a medium or hardware error (or a blank
831 check). For example an attempt to read a corrupted block on a
832 disk will yield this value.
833 5 the device reports an "illegal request" with an additional sense
835 code other than "invalid operation code". This is often a sup‐
836 ported command with a field set requesting an unsupported capa‐
837 bility.
838 6 the device reports a "unit attention" condition. This usually
840 indicates that something unrelated to the requested command has
841 occurred (e.g. a device reset) potentially before the current
842 SCSI command was sent. The requested command has not been exe‐
843 cuted by the device. Note that unit attention conditions are
844 usually only reported once by a device.
845 9 the device reports an illegal request with an additional sense
847 code of "invalid operation code" which means that it doesn't
848 support the requested command.
849 11 the device reports an aborted command. In some cases aborted
851 commands can be retried immediately (e.g. if the transport
852 aborted the command due to congestion).
853 15 the utility is unable to open, close or use the given IFILE or
855 OFILE. The given file name could be incorrect or there may be
856 permission problems. Adding the -v option may give more informa‐
857 tion.
858 20 the device reports it has a check condition but "no sense". It
860 is unlikely that this value will occur as an exit status.
861 21 the device reports a "recovered error". The requested command
863 was successful. Most likely a utility will report a recovered
864 error to stderr and continue, probably leaving the utility with
865 an exit status of 0 .
866 33 the command sent to device has timed out. This occurs in Linux
868 only; in other ports a command timeout will appear as a trans‐
869 port (or OS) error.
870 90 the flock flag has been given on a device and some other process
872 holds the advisory exclusive lock.
873 97 the response to a SCSI command failed sanity checks.
875 98 the device reports it has a check condition but the error
877 doesn't fit into any of the above categories.
878 99 any errors that can't be categorized into values 1 to 98 may
880 yield this value. This includes transport and operating system
881 errors after the command has been sent to the device.
882
884 Written by Doug Gilbert
885
887 Report bugs to <dgilbert at interlog dot com>.
888
890 Copyright © 2008-2011 Douglas Gilbert
891 This software is distributed under the GPL version 2. There is NO war‐
892 ranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PUR‐
893 POSE.
894
896 There is a web page discussing ddpt at http://sg.danny.cz/sg/ddpt.html
897 The lmbench package contains lmdd which is also interesting. For moving
899 data to and from tapes see dt which is found at http://www.scsi‐
900 faq.org/RMiller_Tools/index.html
901 To change mode parameters that effect a SCSI device's caching and error
903 recovery see sdparm(sdparm)
904 To scan and repair disk partitions see TestDisk (testdisk).
906 Additional references: dd(1), ddrescue(GNU), open(2), flock(2),
908 sg_dd,sg3_utils(sg3_utils)
909ddpt-0.92 February 2011 DDPT(8)