1DDPT(8) DDPT DDPT(8)
2
6 ddpt - copies data between files and storage devices. Support for
7 devices that understand the SCSI command set.
8
10 ddpt [bpt=BPT[,OBPC]] [bs=BS] [cdbsz={6|10|12|16|32}] [coe={0|1}]
11 [coe_limit=CL] [conv=CONVS] [count=COUNT] [delay=MS[,W_MS]] [ibs=IBS]
12 [id_usage=LIU] if=IFILE [iflag=FLAGS] [intio={0|1}] [iseek=SKIP]
13 [ito=ITO] [list_id=LID] [obs=OBS] [of=OFILE] [of2=OFILE2] [oflag=FLAGS]
14 [oseek=SEEK] [prio=PRIO] [protect=RDP[,WRP]] [retries=RETR] [rtf=RTF]
15 [rtype=RTYPE] [seek=SEEK] [skip=SKIP] [status=STAT] [to=TO] [ver‐
16 bose=VERB] [--help] [--job=JF] [--odx] [--verbose] [--version]
17 [--wscan] [--xcopy] [JF]
18 For comparison here is the synopsis for GNU's dd command:
20 dd [bs=BS] [cbs=CBS] [conv=CONVS] [count=COUNT] [ibs=IBS] [if=IFILE]
22 [iflag=FLAGS] [obs=OBS] [of=OFILE] [oflag=FLAGS] [seek=SEEK]
23 [skip=SKIP] [status=STAT] [--help] [--version]
24
26 Copies data between files or simply reads data from a file. This util‐
27 ity is specialized for "files" that are storage devices, especially
28 those that can use the SCSI command sets (e.g. SATA and SAS disks). It
29 can issue SCSI commands in pass-through ("pt") mode. Similar syntax and
30 semantics to the Unix dd(1) command.
31 For comparison, the SYNOPSIS section above shows both the ddpt command
33 line options followed by GNU's dd(1) command line options. Broadly
34 speaking ddpt can be considered a super-set of dd. See the section on
35 DD DIFFERENCES for significant differences between ddpt and dd.
36 This utility either does direct copies, based on read-write sequences,
38 or offloaded copies. In an offloaded copy the data being copied does
39 not necessarily pass through the memory of the the machine originating
40 the copy operation; this can save a significant amount of time and
41 lessen CPU usage.
42 When doing a direct copy, this utility breaks the copy into segments
44 since computer RAM is typically a scarce resource. First it reads in
45 BPT*IBS bytes from IFILE (or less if near the end of the copy) into a
46 copy buffer. In the absence of the various options and conditions that
47 bypass the write operation, the copy buffer is then written out to
48 OFILE. The copy process continues working its way along IFILE and
49 OFILE until either COUNT is exhausted, an end of file is detected, or
50 an error occurs. If IBS and OBS are different, ddpt restricts the value
51 of OBS such that the copy buffer is an integral number of output blocks
52 (i.e. (((IBS * BPT) % OBS) == 0) ). In the following descriptions,
53 "segment" refers to all or part of a copy buffer.
54 The term "pt device" is used for a pass-through device to which SCSI
56 commands like READ(10), WRITE(10) or POPULATE TOKEN may be sent. A pt
57 device may only be able to process SCSI commands in which case the "pt"
58 flag is assumed. The ability to recognize such a pt only device may
59 vary depending on the operating system (e.g. in Linux /dev/sg2 and
60 /dev/bsg/3:0:1:0 are recognized). However if a device can process
61 either normal UNIX read()/ write() calls or pass-through SCSI commands
62 then the default is to use UNIX read()/write() calls. That default can
63 be overridden by using the "pt" flag (e.g. "if=/dev/sdc iflag=pt").
64 When pt access is specified any partition information is ignored. So
65 "if=/dev/sdc2 iflag=pt skip=3" will start at logical block address 3 of
66 '/dev/sdc'. As a protection measure ddpt will only accept that if the
67 force flag is also given (i.e. 'iflag=pt,force').
68 This utility supports two types of offloaded copies. Both are based on
70 the EXTENDED COPY (XCOPY or xcopy) family of SCSI commands. The first
71 uses the XCOPY(LID1) command to do a disk to disk copy. LID1 stands for
72 List IDentifier length of 1 byte and the command is described in the
73 SPC-4 drafts and the earlier SPC-3 and SPC-2 standards. Recent SPC-4
74 drafts have added the XCOPY(LID4) sub-family of copy offloaded com‐
75 mands. There is a subset of XCOPY(LID4), specialized for offloaded disk
76 to disk copies, that is known by the market name: ODX. In the descrip‐
77 tions below "xcopy" refers to copies based on XCOPY(LID1) while "odx"
78 refers to either full or partial ODX copies. See the XCOPY and ODX
79 sections below for more information.
80
82 The dd-like options with the name=value syntax are listed first, sorted
83 by name. Following that, options starting with "-" are listed.
84 bpt=BPT[,OBPC]
86 where BPT is Blocks Per Transfer. A direct copy is made up of
87 multiple transfers, each first reading BPT input blocks (i.e.
88 BPT * IBS bytes) from IFILE into the copy buffer and then from
89 that copy buffer writing (BPT * IBS) / OBS output blocks to
90 OFILE. This continues until the copy is finished, with the last
91 transfer being potentially shorter. The default BPT value varies
92 depending on IBS. When IBS < 8, BPT is 8192; when IBS < 64, BPT
93 is 1024; when IBS < 1024, BPT is 128; when IBS < 8192, BPT is
94 16; when IBS < 32768, BPT is 4; else BPT defaults to 1. If BPT
95 is given as 0 it is treated as the default value. For "bs=512",
96 BPT defaults to 128 so that 64 KiB (or less) is read from IFILE
97 into the copy buffer. This option is treated differently in ODX
98 and is typically only needed for testing; see ODX section.
99 The optional OBPC (Output Blocks Per Check) argument controls
100 controls the granularity of sparse writes, write sparing and
101 trim checks. The default granularity is the size of the copy
102 buffer (i.e. BPT * IBS bytes). That can be reduced by specifying
103 OBPC. The finest granularity is when OBPC is 1 which implies the
104 unit of each check is OBS bytes. When OBPC is 0, or not given,
105 the default granularity is used. Large OBPC values are rounded
106 down so that OBPC*OBS does not exceed the size of the copy buf‐
107 fer.
108 odx: may be used to limit the data represented by each ROD.
109 Mainly for testing.
110 bs=BS where BS is the IFILE and OFILE block size in bytes. Conflicts
112 with either the "ibs=" or "obs=" options. The value of BS is
113 placed in IBS and OBS. If IFILE or OFILE is a "pt" device then
114 BS must be the logical block size of the device. See the DD DIF‐
115 FERENCES section below. The default is 512 bytes unless overrid‐
116 den by the DDPT_DEF_BS environment variable. Note that newer
117 disks use 4096 byte blocks with perhaps larger block sizes com‐
118 ing in the future. CD/DVD/BD media use a logical block size of
119 2048 bytes.
120 cdbsz={6|10|12|16|32}
122 size of SCSI READ and/or WRITE commands issued to pt devices.
123 The default is 10 byte SCSI command blocks unless calculations
124 indicate that a 4 byte block number may be exceeded or BPT is
125 greater than 16 bits (i.e. more than 65535 blocks), in which
126 case it defaults to 16 byte SCSI commands.
127 coe={0|1}
129 set to 1 for continue on error. Applies to errors on input and
130 output for pt devices but only on input from block devices or
131 regular files. Errors on other files will stop ddpt. Default is
132 0 which implies stop on any error. See the 'coe' flag for more
133 information.
134 coe_limit=CL
136 where CL is the maximum number of consecutive bad blocks stepped
137 over due to "coe=1" on reads before the copy terminates. The
138 default is 0 which is implies no limit. This option is meant to
139 stop the copy soon after unrecorded media is detected while
140 still offering "continue on error" capability for infrequent,
141 randomly distributed errors.
142 conv=CONVS
144 see the CONVERSIONS section below.
145 count=COUNT
147 copy COUNT input blocks from IFILE to OFILE. If this option is
148 not given (or COUNT is '-1') then the COUNT may be deduced from
149 either IFILE or OFILE. See the COUNT section below.
150 odx: if a gather list is given to skip=SKIP or a scatter list is
151 given to seek=SEEK then typically count=COUNT should not be sup‐
152 plied. This is because a scatter gather list implies a transfer
153 count. If both are given then ddpt will exit if they are
154 unequal, the force option can be used to override this action.
155 delay=MS[,W_MS]
157 after each segment is copied (typically every (IBS * BPT) bytes)
158 a delay (sleep) of MS milliseconds is performed. The default
159 value for MS is 0 which implies no delay. If W_MS is given and
160 greater than 0 (its default value) then there is an additional
161 delay of W_MS milliseconds associated with each actual write
162 operation that is performed. If MS is greater than 0 then there
163 is not a delay before the first copy segment (or after the
164 last); if W_MS is greater than 0 then there is not a delay
165 before the first write segment. These delays can be used for a
166 bandwidth limiting.
167 odx: the MS delay is implemented in the same fashion after each
168 ROD is copied, apart from the last. If W_MS is greater than 0
169 then that delay occurs before each WUT command, apart from the
170 first.
171 ibs=IBS
173 where IBS is the IFILE block size in bytes. The default value is
174 BS or its default (512). Conflicts the "bs=" option (i.e. giving
175 both "bs=512 ibs=512" is considered a syntax error).
176 id_usage=LIU
178 xcopy: SCSI EXTENDED COPY parameter list LIST ID USAGE field is
179 set to LIU. The default value is 0 or 2 . LIU can be a number
180 between 0 and 3 inclusive or a string. The strings can be
181 either: 'hold' for 0, 'discard' for 2 or 'disable' for 3.
182 if=IFILE
184 read from IFILE. This option must be given (apart from one odx
185 case). If IFILE is '-' then stdin is read. Starts reading at
186 the beginning of IFILE unless SKIP is given.
187 odx: the rtf=RTF option may replace the if=IFILE option as
188 input. See the ODX section.
189 iflag=FLAGS
191 where FLAGS is a comma separated list of one or more flags out‐
192 lined in the FLAGS section below. These flags are associated
193 with IFILE and are mostly ignored when IFILE is stdin.
194 intio={0|1}
196 set to 1 for allow signals (SIGINT, SIGPIPE and SIGUSR1 (or SIG‐
197 INFO)) to be received during IO from IFILE or IO to OFILE or
198 OFILE2. Default is 0 which causes these signals to be masked
199 during IO operations with a check for signals prior each IO. As
200 long as IO operations don't lock up (e.g. SCSI READ and WRITE
201 commands) the default is the safer option. Even if IO operations
202 do lock up it is best to let the kernel take care of that.
203 iseek=SKIP
205 start reading SKIP blocks (each of IBS bytes) from the start of
206 IFILE. Default is block 0 (i.e. start of file). This option is a
207 synonym for skip=SKIP, see its description.
208 ito=ITO
210 odx: ITO is the inactivity timeout whose units are seconds. The
211 default value is 0 which means the copy manager will take the
212 default inactivity timeout value from the Block Device ROD Token
213 Limits descriptor in the Third Party Copy VPD page. ITO is
214 ignored if it it exceeds the maximum inactivity timeout value in
215 the same descriptor (unless the force flag is given).
216 list_id=LID
218 LID is the xcopy LIST IDENTIFIER field. It is used to associate
219 an originating xcopy command with follow-up commands such as
220 RECEIVE ROD TOKEN INFORMATION. If given, the LID should not
221 clash with any other xcopy LID currently in use on this I_T
222 nexus.
223 xcopy: LID is a 1 byte (8 bit) value whose default value is 1
224 or, if id_usage=disable, 0 . LID must not exceed 255.
225 odx: LID is a 4 byte (32 bit) value whose default value is 257
226 (i.e. 0x101) and, if a second default is needed, 258 (0x102) is
227 used. If a clash is detected on the default list identifier
228 value then the next higher value is tried (stopping after 10
229 attempts).
230 obs=OBS
232 where OBS is the OFILE block size in bytes. The default value is
233 BS or its default (512). Conflicts the "bs=" option (e.g. giving
234 both "bs=512 obs=512" is considered a syntax error). If OBS is
235 given then it has the following restriction: the integer expres‐
236 sion (((IBS * BPT) % OBS) == 0) must be true. Stated another
237 way: the copy buffer size must be an integral multiple of OBS.
238 If of2=OFILE2 is given then OBS is its block size as well.
239 of=OFILE
241 write to OFILE. The default value is /dev/null . If OFILE is '-'
242 then writes to stdout. If OFILE is /dev/null then no actual
243 writes are performed. If OFILE is '.' (period) then it is
244 treated the same way as /dev/null . If OFILE exists then it is
245 _not_ truncated unless "oflag=trunc" is given. See section on DD
246 DIFFERENCES.
247 odx: if this option (of=OFILE) is not given and the rtf=RTF
248 option is given then the RTF file may be thought of as receiving
249 the output in the form of one or more ROD Tokens. See the ODX
250 section.
251 of2=OFILE2
253 write output to OFILE2. The default action is not to do this
254 additional write (i.e. when this option is not given). OFILE2 is
255 assumed to be a regular file or a fifo (i.e. a named pipe).
256 OFILE2 is opened for writing and is created if necessary. If
257 OFILE2 is a fifo (named pipe) then some other command should be
258 consuming that data (e.g. 'md5sum OFILE2'), otherwise this util‐
259 ity will block. The write to OFILE2 occurs before the write to
260 OFILE and prior to sparse writing and write sparing logic. So
261 everything read is written to OFILE2.
262 oflag=FLAGS
264 where FLAGS is a comma separated list of one or more flags out‐
265 lined in the FLAGS section. These flags are associated with
266 OFILE and are ignored when OFILE is /dev/null, '.' (period), or
267 stdout.
268 oseek=SEEK
270 start writing SEEK blocks (each of OBS bytes) from the start of
271 OFILE. Default is block 0 (i.e. start of file). This option is a
272 synonym for seek=SEEK, see its description.
273 prio=PRIO
275 xcopy: SCSI EXTENDED COPY parameter list PRIORITY field is set
276 to PRIO. The default value is 1 .
277 protect=RDP[,WRP]
279 where RDP is the RDPROTECT field in SCSI READ commands and WRP
280 is the WRPROTECT field in SCSI WRITE commands. The default value
281 for both is 0 which implies no additional protection information
282 will be transferred. Both RDP and WRP can be from 0 to 7. If
283 RDP is greater than 0 then IFILE must be a pt device. If WRP is
284 greater than 0 then OFILE must be a pt device. See the PROTEC‐
285 TION INFORMATION section below.
286 retries=RETR
288 sometimes retries at the host are useful, for example when there
289 is a transport error. When RETR is greater than zero then SCSI
290 READs and WRITEs are retried on error, RETR times. Default value
291 is zero. Only applies to errors on pt devices.
292 rtf=RTF
294 odx: where RTF is a filename. One or more ROD tokens are written
295 to RTF during a read to tokens variant or a full copy variant.
296 One or more ROD tokens are read from RTF during a write from
297 token variant. This option is not required on a full copy vari‐
298 ant. ROD Tokens are 512 bytes long and an extra 8 byte
299 (big-endian) integer containing the 'number of bytes repre‐
300 sented' is placed after each ROD Token if rtf_len is given.
301 rtype=RTYPE
303 odx: where RTYPE is the ROD Type. The default value (0) indi‐
304 cates that the copy manager (in the source) decides. RTYPE can
305 be a decimal number, a hex number (prefixed by 0x or with a "h"
306 appended) or one of "pit-def", "pit-vuln", "pit-pers", "pit-any"
307 or "zero". The final truncated word can be spelt out (e.g.
308 "pit-vulnerable"). The "pit-" prefix is a shortening of "point
309 in time" copy. The "zero" causes a special Block device zero
310 Token to be created.
311 seek=SEEK
313 start writing SEEK blocks (each of OBS bytes) from the start of
314 OFILE. Default is block 0 (i.e. start of file). The SEEK value
315 may exceed the number of OBS-sized blocks in OFILE.
316 odx: SEEK can be a scatter list: comma separated, in the form
317 seek=A1,N1[,A2,N2...] . The scatter list may alternatively be
318 read from a file using this form: seek=@<filename> or read from
319 stdin using this form: seek=- (or seek=@-) . A<n> and N<n> are
320 decimal (optionally with a suffix multiplier) unless a hex indi‐
321 cation is given. Hex values are indicated by either a leading
322 "0x" or a trailing "h". The address (i.e. A<n>) is a 64 bit
323 unsigned integer while the number of blocks (i.e. N<n>) is a 32
324 bit integer. Thus for a block size of 512 bytes, a single scat‐
325 ter gather list element cannot exceed 4 TB ((2**32 - 1) * 512).
326 Note that COUNT is a 64 bit unsigned integer and thus does not
327 have this restriction. There can be no more than 128 scatter
328 list elements.
329 skip=SKIP
331 start reading SKIP blocks (each of IBS bytes) from the start of
332 IFILE. Default is block 0 (i.e. start of file). The SKIP value
333 must be less than the number of IBS-sized blocks in IFILE.
334 odx: SKIP can be a gather list: comma separated, in the form
335 skip=A1,N1[,A2,N2...] . The gather list may alternatively be
336 read from a file using this form: skip=@<filename> or read from
337 stdin using this form: skip=- . See the odx section of the
338 seek=SEEK option for further details.
339 status=STAT
341 the STAT value of 'noxfer' suppresses the throughput speed and
342 the copy time reporting at the end of the copy. A STAT value of
343 'none' additionally suppresses the records in and out reporting
344 after the copy. So 'status=none' makes ddpt act like a tradi‐
345 tional Unix command in which "no news is good news". The
346 default action of ddpt is to show the throughput (in megabytes
347 per second) and the time taken to do the copy after the "records
348 in" and "records out" lines at the end of the copy. As a conve‐
349 nience the value 'null' is accepted for STAT and does nothing.
350 to=TO odx, xcopy: where TO is am xcopy originating command timeout in
352 seconds. The default value is 0 which is converted internally
353 to 600 seconds (10 minutes). Best to set this timeout value well
354 above the expected copy time. In a odx full copy this timeout
355 is applied to both the POPULATE TOKEN and WRITE USING TOKEN com‐
356 mands.
357 verbose=VERB
359 as VERB increases so does the amount of debug reporting sent to
360 stderr. Default value is zero which yields the minimum amount
361 of debug reporting. A value of 1 reports extra information that
362 is not repetitive. A value 2 reports cdbs and responses for SCSI
363 commands that are not repetitive (i.e. other that READ and
364 WRITE). Error processing is not considered repetitive. Values of
365 3 and 4 yield reporting for all SCSI commands, plus Unix read()
366 and write() calls, so there can be a lot of output. If VERB is
367 "-1" then reporting that would have been sent to stderr is redi‐
368 rected to /dev/null essentially throwing it away.
369 -h, --help
371 reports usage message then exits.
372 --job=JF
374 where JF is a file name. That file can contain options listed in
375 this section. The options within the file are processed in the
376 order they are found (i.e. parsing left to right, top (of file)
377 to bottom) and options may override earlier ones or accumulate.
378 For example '-v' on the command line followed by a job file con‐
379 taining '-vv' will result in a verbosity level of '-vvv' during
380 the copy phase. Job files can recurse (i.e. call another or the
381 same job file) up to a depth of 5. Empty lines and anything from
382 and including a '#' in a job file line is ignored.
383 -o, --odx
385 indicates to this utility that one of the four odx variants is
386 requested. See ODX section.
387 -v, --verbose
389 equivalent of verbose=1. If --verbose appears twice then that is
390 equivalent to verbose=2. Also -vv is equivalent to verbose=2.
391 -V, --version
393 reports version number information then exits.
394 -w, --wscan
396 this option is available in Windows only. It lists storage
397 device names and the corresponding volumes, if any. When used
398 twice it adds the "bus type" of the closest transport (e.g. a
399 SATA disk in a USB connected enclosure has bus type USB). When
400 used three times a SCSI adapter scan is added. When used four
401 times only a SCSI adapter scan is shown. See EXAMPLES section
402 below and the README.win32 file.
403 -x, --xcopy
405 this option will attempt to call the SCSI EXTENDED COPY(LID1)
406 command. In the absence of another indication the xcopy command
407 will be sent to the destination (i.e. OFILE). See the section on
408 ENVIRONMENT VARIABLES below.
409 JF a command line element that does not contain a '=' (i.e. a dd
411 style option) and does not start with '-' is treated as a job
412 file (i.e. JF). See the entry on --job=JF above. For sanity JF
413 is checked to make sure it is "regular" (i.e. not a device file
414 like /dev/sdc) and for non ASCII characters lest it be a binary
415 file by accident. Failing those checks may be overridden by
416 'iflag=force' or 'oflag=force'.
417
419 When the count=COUNT option is not given (or COUNT is '-1') then an
420 attempt is made to deduce COUNT as follows.
421 When both or either IFILE and OFILE are block devices, then the minimum
423 size, expressed in units of input blocks, is used. When both or either
424 IFILE and OFILE are pass-through devices, then the minimum size,
425 expressed in units of input blocks, is used.
426 If a regular file is used as input, its size, expressed in units of
428 input blocks (and rounded up if necessary) is used. Note that the
429 rounding up of the deduced COUNT may result in a partial read of the
430 last input block and a corresponding partial write to OFILE if it is a
431 regular file. After a regular file to regular file copy the length of
432 OFILE will be the same as IFILE unless OFILE existed and its length was
433 already greater than that of IFILE. To get a copy like the standard
434 Unix cp command, use oflag=trunc with ddpt.
435 The size of pt devices is deduced from the SCSI READ CAPACITY command.
437 Block device sizes (or their partition sizes) are obtained from the
438 operating system, if available.
439 If skip=SKIP or skip=SEEK are given and the COUNT is deduced (i.e. not
441 explicitly given) then that size is scaled back so that the copy will
442 not overrun the file or device.
443 If COUNT is not given and IFILE is a fifo (and stdin is treated as a
445 fifo) then IFILE is read until an EOF is detected. If COUNT is not
446 given and IFILE is a /dev/zero (or equivalent) then zeros are read
447 until an error occurs (e.g. file system full).
448 If COUNT is not given and cannot be deduced then an error message is
450 issued and no copy takes place.
451
453 One or more conversions can be given to the "conv=" option. If more
454 than one is given, they should be comma separated. ddpt does not per‐
455 form the traditional dd conversions (e.g. ASCII to EBCDIC). Recently
456 added conversions inherited from GNU's dd overlap somewhat with the
457 some of ddpt flags.
458 fdatasync
460 equivalent to "oflag=fdatasync". Flushes data associated with
461 the OFILE to storage at the end of the copy. This conversion is
462 for compatibility with GNU's dd.
463 fsync equivalent to "oflag=fsync". Flushes data and meta-data associ‐
465 ated with the OFILE to storage at the end of the copy. This con‐
466 version
467 no_del_tkn
469 equivalent to "oflag=no_del_tkn".
470 noerror
472 this conversion is very close to "iflag=coe" and is treated as
473 such. See the "coe" flag. Note that an error on a block device
474 or regular file OFILE will stop the copy.
475 notrunc
477 this conversion is accepted for compatibility with dd and
478 ignored since the default action of this utility is not to trun‐
479 cate OFILE.
480 null has no affect, just a placeholder.
482 resume See "resume" in the FLAGS sections for more information.
484 rtf_len
486 equivalent to "oflag=rtf_len".
487 sparing
489 See "sparing" in the FLAGS sections for more information.
490 sparse FreeBSD's dd supports "conv=sparse" and now GNU's dd does as
492 well so the same syntax is supported in ddpt. See "sparse" in
493 the FLAGS sections for more information.
494 sync is ignored by ddpt. With dd it means supply zero fill (rather
496 than skip) and is typically used like this "conv=noerror,sync"
497 to have the same functionality as ddpt's "iflag=coe".
498 trunc if OFILE is a regular file then truncate it prior to starting
500 the copy. See "trunc" in the FLAGS section.
501
503 A list of flags and their meanings follow. The flag name is followed by
504 one or two indications in square brackets. The first indication is
505 either "[i]", "[o]" or "[io]" indicating this flag is active for the
506 IFILE, OFILE or both the IFILE and the OFILE. The second indication
507 contains some combination of "reg", "blk" "pt", "odx", or "xcopy".
508 These indicate whether the flag applies to a regular file, a block
509 device (accessed via Unix read() and write() commands, a pass-through
510 device, an ODX offloaded copy or a XCOPY(LID1) offloaded copy respec‐
511 tively. Other special file types that are sometimes referred to are
512 "fifo" and "tape".
513 append [o] [reg], [io] [odx]
515 causes the O_APPEND flag to be added to the open of OFILE. For
516 regular files this will lead to data being appended to the end
517 of any existing data. Conflicts the seek=SEEK option. The
518 default action of this utility is to overwrite any existing data
519 from the beginning of OFILE or, if SEEK is given, starting at
520 block SEEK. Note that attempting to 'append' to a device file
521 (e.g. a disk) will usually be ignored or may cause an error to
522 be reported.
523 odx: if the rtf=RTF option is given, RTF exists, is a regular
524 file and this utility wants to write to RTF then new ROD Tokens
525 are appended to RTF. The default action is to truncate RTF
526 before new ROD Tokens are written to it.
527 atomic [o] [pt]
529 this flag changes the pass-through SCSI WRITE command to the
530 SCSI WRITE ATOMIC(16) command on OFILE (and the
531 cdbsz={6|10|12|16|32} option is ignored for OFILE). If this flag
532 is applied to IFILE or to a non pass-through file then it is
533 ignored.
534 block [io] [pt]
536 pass-through file opens are non-blocking by default and may
537 report the pt device is busy. Use this flag to open blocking so
538 utility may wait until another process locking (or with an
539 exclusive open) is complete before continuing.
540 bytchk [o] [pt]
542 only active when used together with oflag=verify. Sets the
543 BYTCHK field in the SCSI WRITE AND VERIFY command. Since that
544 field is two bits wide, this flag can be specified multiple
545 times (up to three) to place the coresponding value in the
546 field. T10 only allows BYTCHK=1 or 1 at this time.
547 cat [io] [xcopy]
549 xcopy: set CAT (residual data handling) bit in EXTENDED
550 COPY(LID1) parameter list segment descriptor header. May appear
551 in either flag list when xcopy is being used. Works with the PAD
552 bit for handling residual data on the destination side. See the
553 XCOPY section below.
554 coe [io] [pt], [i] [reg,blk]
556 continue on error. 'iflag=coe oflag=coe' and 'coe=1' are equiva‐
557 lent. Errors occurring on output regular or block files will
558 stop ddpt. Error messages are sent to stderr. This flag is sim‐
559 ilar to 'conv=noerror,sync' in the dd(1) utility. Unrecovered
560 errors are counted and reported in the summary at the end of the
561 copy.
562 This paragraph concerns coe on pt devices. A medium, hardware or
564 blank check error during a read operation will will cause the
565 following: first re-read blocks prior to the bad block, then try
566 to recover the bad block (supplying zeros if that fails), and
567 finally re-read the blocks after the bad block. A medium, hard‐
568 ware or blank check error while writing is reported but other‐
569 wise ignored. SCSI disks may automatically try and remap faulty
570 sectors (see the AWRE and ARRE in the read write error recovery
571 mode page (the sdparm utility can access these attributes)). If
572 bad LBAs are reported by the pass-through then the LBA of the
573 lowest and highest bad block is also reported.
574 This paragraph concerns coe on input regular files and block
576 devices. When a EIO or EREMOTEIO error is detected on a normal
577 segment read then the segment is re-read one block (i.e. IBS
578 bytes) at a time. Any block that yields a EIO or EREMOTEIO error
579 is replaced by zeros. Any other error, a short read or an end of
580 file will terminate the copy, usually after the data that has
581 been read is written to the output file.
582 dc [io] [blk,pt]
584 xcopy: set DC (destination counter) bit in EXTENDED COPY(LID1)
585 parameter list segment descriptor header. May appear in either
586 flag list when xcopy is being used.
587 direct [io] [reg,blk]
589 causes the O_DIRECT flag to be added to the open of IFILE and/or
590 OFILE. This flag requires some memory alignment on IO. Hence
591 user memory buffers are aligned to the page size. May have no
592 effect on pt devices. This flag will bypass caching/buffering
593 normally done by block layer. Beware of data coherency issues if
594 the same locations have been recently accessed via the block
595 layer in its normal mode (i.e. non-direct). See open(2) man
596 page.
597 dpo [io] [pt]
599 set the DPO bit (disable page out) in SCSI READ and WRITE com‐
600 mands. Not supported for 6 byte cdb variants of READ and WRITE.
601 Indicates that data is unlikely to be required to stay in device
602 (e.g. disk) cache. May speed media copy and/or cause a media
603 copy to have less impact on other device users.
604 errblk [i] [pt] [experimental]
606 attempts to create or append to a file called "errblk.txt" in
607 the current directory the logical block addresses of blocks that
608 cannot be read. The first (appended) line is "# start <time‐
609 stamp>". That is followed by the LBAs in hex (and prefixed with
610 "0x") of any block that cannot be read, one LBA per line. If the
611 sense data does not correctly identify the LBA of the first
612 error in the range it was asked to read then a LBA range is
613 reported in the form of the lowest and the highest LBA in the
614 range separated by a "-". At the end of the copy a line with "#
615 stop <timestamp>" is appended to "errblk.txt". Typically used
616 with "coe".
617 excl [io] [reg,blk]
619 causes the O_EXCL flag to be added to the open of IFILE and/or
620 OFILE. See open(2) man page.
621 fdatasync [o] [reg,blk]
623 Flushes data associated with the OFILE to storage at the end of
624 the copy.
625 ff [i] This replaces IFILE with a source of 0xff bytes. Can only be
627 used on input and an error is generated if the if=IFILE option
628 is also given. Zeros can easily be generated by using
629 "if=/dev/zero" or an equivalent.
630 flock [io] [reg,blk,pt]
632 after opening the associated file (i.e. IFILE and/or OFILE) an
633 attempt is made to get an advisory exclusive lock with the
634 flock() system call. The flock arguments are "FLOCK_EX |
635 FLOCK_NB" which will cause the lock to be taken if available
636 else a "temporarily unavailable" error is generated. An exit
637 status of 90 is produced in the latter case and no copy is done.
638 See flock(2) man page.
639 force [io] [pt] [xcopy,odx]
641 override difference between given block size and the block size
642 found by the SCSI READ CAPACITY command. Use the given block
643 size. Without this flag the copy would not be performed. pt
644 access to what appears to be a block partition is aborted in
645 version 0.92; that can be overridden by the force flag. For
646 related reasons the 'norcap' flag requires this flag when
647 applied to a block device accessed via pt.
648 xcopy and odx: various limits imposed by associated VPD pages or
649 the RECEIVE COPY OPERATING PARAMETERS command can be overridden
650 (i.e. exceeded) if this flag is given. Note that the copy man‐
651 ager will probably object.
652 fsync [o] [reg,blk]
654 Flushes data and metadata (describing the file) associated with
655 the OFILE to storage at the end of the copy.
656 fua [io] [pt]
658 causes the FUA (force unit access) bit to be set in SCSI READ
659 and/or WRITE commands. The 6 byte variants of the SCSI READ and
660 WRITE commands do not support the FUA bit.
661 fua_nv [io] [pt]
663 causes the FUA_NV (force unit access non-volatile cache) bit to
664 be set in SCSI READ and/or WRITE commands. This only has an
665 effect with pt devices. The 6 byte variants of the SCSI READ
666 and WRITE commands do not support the FUA_NV bit. The FUA_NV bit
667 was made obsolete in SBC-3 revision 35d.
668 ignoreew [o] [tape]
670 ignore the early warning indication (of end of tape) when writ‐
671 ing to tape. See TAPE section.
672 immed [io] [odx]
674 sets the IMMED bit in the POPULATE TOKEN (when [i]) or WRITE
675 USING TOKEN (when [o]) command. That command should return sta‐
676 tus promptly after starting the data transfer. The RECEIVE ROD
677 TOKEN INFORMATION command is then used to poll for completion.
678 SCSI command timeouts should not be exceeded, even for very
679 large RODs, if this flag is used.
680 nocache [io] [reg,blk]
682 use posix_fadvise(POSIX_FADV_DONTNEED) to advise corresponding
683 file there is no need to fill the file buffer with recently read
684 or written blocks. If used with "iflag=" it will increase the
685 read ahead on IFILE.
686 no_del_tkn [o] [odx]
688 will clear the DEL_TKN bit on the last WRITE USING TOKEN command
689 of each ROD Token in a odx full copy. In a large odx full copy
690 several ROD Tokens may be used (one after the other). The
691 default action is to set the DEL_TKN bit on the last WUT command
692 of each ROD. Either way it should not make much difference
693 because the copy manager deletes a ROD Token when its inactivity
694 time-out occurs.
695 nofm [o] [tape]
697 no File Mark (FM) on close when writing to tape. See TAPE sec‐
698 tion.
699 nopad [o] [tape]
701 when the block to be written to a tape drive contains less than
702 OBS bytes, then this option causes the partial block to be writ‐
703 ten as is. The default action for a tape in this case is to pad
704 the block. See TAPE section.
705 norcap [io] [pt]
707 do not perform SCSI READ CAPACITY command on the corresponding
708 pt device. If used on block device accessed via pt then 'force'
709 flag is also required. This is to warn about using pt access on
710 what may be a block device partition.
711 nowrite [o] [reg,blk,pt]
713 bypass writes to OFILE. The "records out" count is not incre‐
714 mented. OFILE is still opened but "oflag=trunc" if given is
715 ignored. Also the ftruncate call associated with the sparse flag
716 is ignored (i.e. bypassed). Commands such as trim and SCSI SYN‐
717 CHRONIZE CACHE are still sent.
718 null [io]
720 has no affect, just a placeholder.
721 odx [io] [odx]
723 indicates to this utility that one of the four variants of an
724 odx copy is requested. Using any of the --odx, rtf=RTF or
725 rtype=RTYPE options also indicates that odx is requested. See
726 the ODX section.
727 pad [o] [reg,blk,pt], [io] [xcopy]
729 when the block to be written (typically the last block) contains
730 less than OBS bytes, then this option causes the block to be
731 padded with zeros (i.e. bytes of binary zero). The default
732 action for a regular file and a fifo is to do a partial write.
733 The default action of a block and a pt device is to ignore the
734 partial write. The default action of a tape is to pad, so this
735 flag is not needed (see the nopad flag).
736 xcopy: sets the PAD bit in the CSCD descriptor of the associated
737 IFILE or OFILE. Is associated with residual data handling and
738 works together with the cat flag. See the XCOPY section below.
739 prealloc [o] [reg]
741 use the fallocate() call prior to starting a copy to set OFILE
742 to its expected size.
743 pt [io] [blk,pt]
745 causes a device to be accessed in "pt" mode. In "pt" mode SCSI
746 READ and WRITE commands are sent to access blocks rather than
747 standard UNIX read() and write() commands. The "pt" mode may be
748 implicit if the device is only capable of passing through SCSI
749 commands (e.g. the /dev/sg* and some /dev/bsg/* devices in
750 Linux). This flag is needed for device nodes that can be
751 accessed both via standard UNIX read() and write() commands as
752 well as SCSI commands. Such devices default standard UNIX read()
753 and write() commands in the absence of this flag.
754 rarc [i] [pt]
756 bit set in READ(10, 12, 16 and 32) to suppress RAID rebuild
757 functions when a bad (or recovered after difficulties) block is
758 detected.
759 resume [o] [reg]
761 when a copy is interrupted (e.g. with Control-C from the key‐
762 board) then using the same invocation again with the addition of
763 "oflag=resume" will attempt to restart the copy from the point
764 of the interrupt (or just before that point). It is harmless to
765 use "oflag=resume" when OFILE doesn't exist or is zero length.
766 If the length of OFILE is greater than or equal to the length
767 implied by a ddpt invocation that includes "oflag=resume" then
768 no further data is copied.
769 self [io] [pt]
771 used together with trim flag to do a self trim (trim of segments
772 of a pt device that contain all zeros). If OFILE is not given,
773 then it is set to the same as IFILE. If SEEK is not given it set
774 to the same value as SKIP (possibly adjusted if IBS and OBS are
775 different). Implicitly sets "nowrite" flag.
776 sparing [o] [reg,blk,pt]
778 during the copy each IBS * BPT byte segment is read from IFILE
779 into a buffer. Then, instead of writing that buffer to OFILE,
780 the corresponding segment is read from OFILE into another buf‐
781 fer. If the two buffers are different, the former buffer is
782 written to the OFILE. If the two buffers compare equal then the
783 write to OFILE is not performed. Write sparing is useful when a
784 write operation is significantly slower than a read. Under some
785 conditions flash memory devices have slow writes plus an upper
786 limit on the number of times the same cell can be rewritten. The
787 granularity of the comparison can be reduced from the default
788 IBS * BPT byte segment with the the OBPC value given to the
789 "bpt=" option. The finest granularity is when OBPC is 1 which
790 implies OBS bytes.
791 sparse [io] [reg,blk,pt]
793 after each IBS * BPT byte segment is read from IFILE, it is
794 checked to see if it is all zeros. If so, that segment is not
795 written to OFILE. See the section on SPARSE WRITES below. The
796 granularity of the zero comparison can be reduced from the
797 default IBS * BPT byte segment with the OBPC value given to the
798 "bpt=" option.
799 The sparse flag may be used on input when a file is only being
800 read (e.g. when of=OFILE is not given or OFILE is /dev/null) to
801 determine how many blocks are contained in sparse segments of
802 IFILE.
803 ssync [o] [pt]
805 if OFILE is in "pt" mode then the SCSI SYNCHRONIZE CACHE command
806 is sent to OFILE at the end of the copy.
807 strunc [o] [reg]
809 perform a sparse copy with a ftruncate system call to extend the
810 length of the OFILE if required. Sets the sparse flag internally
811 if this has not been specified on the command line. See the
812 sparse flag and the section on SPARSE WRITES below.
813 sync [io] [reg,blk]
815 causes the O_SYNC flag to be added to the open of IFILE and/or
816 OFILE. See open(2) man page.
817 rtf_len [io] [odx]
819 odx: with the 'read to tokens' variant, after 512 bytes of each
820 ROD Token are written to IRTF an additional 8 byte (big endian)
821 integer is written. That integer is the number of bytes the
822 associated ROD represents. The draft standards say for standard
823 ROD types the ROD Token holds this value. However vendor spe‐
824 cific ROD types may be used or the vendors may choose not to
825 comply. Either way the 'write from tokens' variant needs to know
826 the size of the ROD it is writing from.
827 trim [io] [pt] [experimental]
829 similar logic to the "sparse" option. However instead of skip‐
830 ping segments that are full of zeros a "trim" command is sent to
831 OFILE. Usually set as an oflag argument but for self trim can be
832 used as an iflag argument (e.g. "iflag=self,trim"). Depending on
833 the usage this may require the device to support "deterministic
834 read zero after trim". See the TRIM, UNMAP AND WRITE SAME sec‐
835 tion below.
836 trunc [o] [reg]
838 if OFILE is a regular file then it is truncated prior to start‐
839 ing the copy. If SEEK is not given or 0 then OFILE is truncated
840 to zero length; when SEEK is larger than zero the truncation
841 takes place at file byte pointer SEEK*OBS. Ignored if
842 "oflag=append". Conflicts with "oflag=sparing".
843 unmap [io] [pt]
845 same as the trim flag.
846 verify [o] [pt]
848 this causes SCSI WRITE AND VERIFY commands to be sent to OFILE
849 (instead of SCSI WRITE (or WRITE ATOMIC) commands). Note that
850 the fua flag is ignored when this flag is given. The BYTCHK
851 field in the SCSI WRITE AND VERIFY commands is set to zero
852 unless the bytchk flag is also given.
853 xcopy [io] [pt]
855 invoke SCSI XCOPY(LID1) logic and send the XCOPY command to the
856 either IFILE or OFILE depending on which flag this called. If
857 both are given (i.e. an invocation including 'iflag=xcopy
858 oflag=xcopy') then send the XCOPY(LID1) to OFILE.
859
861 This section describes XCOPY(LID1) support with this utility. For ODX
862 support (XCOPY(LID4) subset) see the ODX section.
863 A device (logical unit (LU)) that supports XCOPY operations should set
865 the 3PC field (3PC stands for Third Party Copy) in its standard INQUIRY
866 response. That is not checked when this utility does an xcopy operation
867 but if it fails, that is one thing that the user may want to check.
868 If the xcopy starts and fails while underway, then 'sg_copy_results -s'
870 may be useful to view the copy status. It might also be used from a
871 different process with the same I_T nexus (i.e. the same machine) to
872 check status during an xcopy operation.
873 The pad and cat flags control the handling of residual data. As the
875 data can be specified either in terms of source or target block size
876 and both might have different block sizes residual data is likely to
877 happen in these cases. If both block sizes are identical these bits
878 have no effect as residual data will not occur.
879 If neither of these flags are set, the EXTENDED COPY command will be
881 aborted with additional sense 'UNEXPECTED INEXACT SEGMENT'.
882 If only the cat flag is set the residual data will be retained and made
884 available for subsequent segment descriptors. Residual data will be
885 discarded for the last segment descriptor.
886 If the pad flag is set for the source descriptor only, any residual
888 data for both source or destination will be discarded.
889 If the pad flag is set for the target descriptor only any residual
891 source data will be handled as if the cat flag is set, but any residual
892 destination data will be padded to make a whole block transfer.
893 If the pad flag is set for both source and target any residual source
895 data will be discarded, and any residual destination data will be
896 padded.
897 There is a web page discussing ddpt, XCOPY and ODX at
899 http://sg.danny.cz/sg/ddpt_xcopy_odx.html
900
902 This section describes ODX support (an XCOPY(LID4) subset) for this
903 utility. ODX descriptions use the following command name abbrevia‐
904 tions: PT for the POPULATE TOKEN command, RRTI for the READ ROD TOKEN
905 INFORMATION command, and WUT for the WRITE USING TOKEN command.
906 A device (logical unit (LU)) that supports ODX operations is required
908 to set the 3PC field (3PC stands for Third Party Copy) in its standard
909 INQUIRY response and support the Third Party Copy VPD page. If this
910 utility generates errors noting the absence of these then the device in
911 question probably does not support ODX.
912 There a four variants of ODX supported by ddpt:
914 full copy : ddpt --odx if=/dev/sg3 bs=512 of=/dev/sg4
915 zero output blocks : ddpt if=/dev/null rtype=zero bs=512 of=/dev/sg4
916 read to tokens : ddpt if=/dev/sg3 bs=512 skip=@gath.lst rtf=a.rt
917 write from tokens : ddpt rtf=a.rt bs=512 of=/dev/sg4 seek=@scat.lst
918 The full copy will call PT and WUT commands repeatedly until the copy
920 is complete. More precisely the full copy will make the largest single
921 call to PT allowed by the input's Third Party Copy VPD page (and, if
922 given, allowed by the BPT argument in the bpt=BPT[,OBPC] option). Then
923 one or more WUT calls are made to write out from the ROD created by the
924 PT step. The largest single WUT call is constrained by the output's
925 Third Party Copy VPD page (and, if given, allowed by the OBPC argument
926 in the bpt=BPT[,OBPC] option). This sequence continues until the
927 requested copy is complete.
928 The zero output blocks variant is a special case of the full copy in
930 which only WUT calls are made. ODX defines a special ROD Token to zero
931 blocks. That special ROD Token has a fixed pattern (shown in SBC-3) and
932 does not need to be created by a PT command like normal ROD Tokens.
933 The read to tokens and the write from tokens variants are designed to
935 be the read (input) and write (output) sides respectively of a network
936 copy. Each can run on different machines by sending the RTF file from
937 the machine doing the read to the machine doing the write. The read to
938 tokens will make one or more PT calls and output the resulting ROD
939 Tokens to the RTF file. RTF might be a regular file or a named pipe.
940 All four variants can have the immed flag set. Then the PT and/or WUT
942 commands are issued with the IMMED bit set and the RRTI command is used
943 to poll for completion. The delay between the polls is as suggested by
944 the RRTI command (or if no suggestion is made, 500 milliseconds).
945 Either iflag=immed, oflag=immed or both can be given but are only
946 effective if the corresponding IFILE or OFILE sends a PT or WUT com‐
947 mand.
948 Typically there is no need to give the list_id=LID option. If this
950 option is not given then 257 is chosen. If that is busy then 258 is
951 tried. That continues until a usable LID is found or 10 LIDs have been
952 tried. In the latter case ddpt exits with status of 55 (operation in
953 progress). If the user gives list_id=LID option and LID is busy then
954 ddpt exits with exit status 55.
955 If the block size of the input and output are different (i.e. IBS is
957 not equal to OBS) then one must be a multiple of the other. So an input
958 block size of 512 bytes and an output block size of 4096 bytes (or vice
959 versa) is acceptable.
960 The four ODX variants are distinguished as follows: if OFILE is a
962 pass-through device, if=/dev/null (or equivalent) and rtype=zero then
963 the zero output blocks variant is selected. If both IFILE and OFILE are
964 pass-through devices and there is some indication of an ODX request
965 (e.g. the --odx option), then the full copy variant is selected. The
966 read to tokens and the write from token variants are indicated by the
967 absence of either a of=OFILE or a if=IFILE option, respectively, plus
968 the presence of a rtf=RTF option.
969 The helper utility ddptctl contains options to issue a single PT, RRTI,
971 WUT or COPY OPERATION ABORT command. It can also issue a series of
972 polling RRTI commands. It can decode information in ROD Tokens (which
973 is not as informative as it should be) and print the number of blocks
974 and block size of a disk, plus protection information if available. See
975 ddptctl.
976 There is a web page discussing ddpt, XCOPY and ODX at
978 http://sg.danny.cz/sg/ddpt_xcopy_odx.html
979
981 Bypassing writes of blocks full of zeros can save a lot of IO. However
982 with regular files, bypassed writes at the end of the copy can lead to
983 an OFILE which is shorter than it would have been without sparse
984 writes. This can lead to integrity checking programs like md5sum and
985 sha1sum generating different values.
986 This utility has two ways of handling this file length problem: writing
988 the last block (even if it is full of zeros) or using the ftruncate
989 system call. A third approach is to ignore the problem (i.e. leaving
990 OFILE shorter). The ftruncate approach is used when "oflag=strunc"
991 while the last block is written when "oflag=sparse". To ignore the file
992 length issue use "oflag=sparse,sparse". Note that if OFILE's length is
993 already correct or longer than required, no action is taken.
994 The support for sparse writing of regular files may depend on the OS,
996 the file system and the settings of OFILE. POSIX makes few guarantees
997 when the ftruncate system call is used to extend a file's length, as
998 may occur when "oflag=strunc". Further, primitive file systems like
999 VFAT may not accept sparse writes or simulate the effect by writing
1000 blocks of zeros. The latter approach will defeat any sparse writing
1001 performance gain.
1002
1004 This is a new storage feature often associated with Solid State Disks
1005 (SSDs) or disk arrays with "thin provisioning". In the ATA command set
1006 (ACS-2) the relevant command is DATA SET MANAGEMENT with the TRIM bit
1007 set. In the SCSI command set (SBC-3) it is either the UNMAP or WRITE
1008 SAME command. Note there is no TRIM command however the term is fre‐
1009 quently used in the technical press.
1010 Trim is a way of telling a storage device that blocks are no longer
1012 needed. Keeping the pool of unwritten blocks large is important for
1013 the write performance of SSDs and the thrifty use of real storage in
1014 thin provisioned arrays. Currently file systems in recent OSes may
1015 issue trims associated with file deletes. The trim option in ddpt may
1016 be useful when a partition or a whole SSD is to be "deleted". Note that
1017 ddpt is bypassing file systems in that it only offers trim on
1018 pass-through (pt) devices.
1019 This utility issues SCSI commands to pt devices and for "trim" cur‐
1021 rently issues a SCSI WRITE SAME(16) command with the UNMAP bit set. If
1022 the pt device is a SSD with a ATA interface then recent versions of
1023 Linux will translate the SCSI WRITE SAME to the ATA DATA SET MANAGEMENT
1024 command with the TRIM bit set. The maximum size of each "trim" command
1025 sent is the size of the copy buffer (i.e. IBS * BPT bytes). And that
1026 maximum can be reduced with the OBPC argument of the "bpt=" option.
1027 The trim can be used various ways. One way is a copy where the copy
1029 buffer (or some part of it) is checked for zeros as is done by the
1030 sparse oflag. When a zero segment is found, a trim "command" is sent to
1031 the OFILE. For example:
1032 ddpt if=dsk.img bs=512 of=/dev/sdc oflag=pt,trim
1034 The copy buffer is 64 KiB (since BPT and OBPC default to 128 when
1036 "bs=512") and it is checked for all zeros. If it is all zeros then a
1037 trim command is sent to the corresponding location of /dev/sdc which is
1038 accessed via the pt interface. If it is not all zeros then a SCSI WRITE
1039 command is sent. Another way is to trim all or part of a disk. To trim
1040 a whole disk (i.e. deleting all its data):
1041 ddpt if=/dev/zero bs=512 of=/dev/sdc oflag=pt,trim
1043 A third way is to "self-trim" which is to only trim those parts of a
1045 disk that contain segments full of zeros:
1046 ddpt if=/dev/sdc skip=0x2300 bs=512 iflag=pt,self,trim
1048 count=0x1234f0
1049 The "self" oflag automatically sets up the output side of the copy to
1051 send trim commands (if required) back the the same device (i.e.
1052 /dev/sdc). If this example was self-trimming a partition then the par‐
1053 tition would start at LBA 0x2300 and be 0x1234f0 blocks long.
1054 Some random product examples: the Intel X25-M G2 SSDs have trim with
1056 recent firmware and they do deterministic read zero after trim. The
1057 Seagate Pulsar SSD has an ATA interface which supports the determinis‐
1058 tic reads of zero after the DATA SET MANAGEMENT command with the TRIM
1059 option.
1060
1062 dd defaults "if=" and "of=" to stdin and stdout respectively. This fol‐
1063 lows Unix filter conventions. However since dd and ddpt are often used
1064 to read binary data for timing purposes, having to supply
1065 "of=/dev/null" can be easily forgotten. Without it dd will typically
1066 spew binary data on the console. So ddpt has changed its defaults: the
1067 "if=IFILE" is now mandatory for direct copies and to read from stdin
1068 "if=-" can be used; "of=OFILE" remains optional but its default changes
1069 to "/dev/null" (or "NUL" in Windows). To send output to stdout ddpt
1070 accepts "of=-".
1071 dd truncates OFILE unless "conv=notrunc" is given. When dd truncates,
1073 it truncates to zero length unless SEEK is greater than zero. ddpt does
1074 not truncate OFILE by default. If OFILE exists it will be overwritten.
1075 The overwrite starts at block zero unless SEEK or "oflag=append" is
1076 given. If OFILE is a regular file then "oflag=trunc" (or "conv=trunc")
1077 will truncate OFILE prior to the copy.
1078 Numeric arguments to ddpt can be given in hexadecimal, either with a
1080 leading "0x" or "0X" or with a trailing "h". Note that dd accepts
1081 "0x123" but interprets it as "0 * 123" (i.e. zero). ddpt will also
1082 interpret "x" as multiplies unless the left operand is zero (e.g.
1083 "0x123"). So both dd and ddpt will interpret "skip=2x123" as
1084 "skip=246".
1085 Terabyte size disks make it impractical to copy all the data into a
1087 buffer before writing it out. Therefore both dd and ddpt read a rela‐
1088 tively small amount of data into a copy (or transfer) buffer then write
1089 it out to the destination, repeating this process until the COUNT is
1090 exhausted.
1091 A major difference in ddpt is the addition of BPT to control the size
1093 of the copy buffer. With dd, IBS is the size of the copy buffer and the
1094 unit of SKIP and COUNT. With ddpt, IBS * BPT is the size of the copy
1095 buffer and IBS is the unit of SKIP and COUNT. This allows ddpt to have
1096 its IBS set to the logical block size of IFILE without unduly restrict‐
1097 ing the size of the copy buffer. And setting IBS (and OBS for OFILE)
1098 accurately is required when the pass-through interface is used since
1099 with the SCSI READ and WRITE commands the logical block size is
1100 implicit.
1101 The way dd handles its copy buffer (outlined in SUSv4 description of
1103 dd) is relatively complex, especially when IBS and OBS are different
1104 sizes. The restriction that ddpt places on IBS and OBS ( i.e. (((IBS *
1105 BPT) % OBS) == 0) ) means that a single copy buffer can be used since
1106 its size is a multiple of both IBS and OBS. Being able to precisely
1107 define the copy buffer size in ddpt makes sparse writing, write sparing
1108 and trim operations simpler to define and the user to control.
1109 ddpt does not support dd's "cbs=" option (conversion block size). If
1111 the "cbs=" option is given to ddpt then it is ignored.
1112 ddpt adds two types of disk to disk, offloaded copies: XCOPY(LID1)
1114 first introduced in SPC-2 (standardized in 2001), and ODX which is a
1115 subset of XCOPY(LID4) first introduced in SPC-4 draft (revision 34,
1116 2012).
1117
1119 This section is about protection information which is typically an
1120 extra 8 bytes associated with each logical block. Those 8 byte are
1121 divided into 3 fields: logical block guard (16 bit (2 byte) CRC), logi‐
1122 cal block application tag (2 bytes) and the logical block reference tag
1123 (4 bytes). The acronym DIF is sometimes used for protection informa‐
1124 tion.
1125 The feature to read and/or write protection information by using the
1127 protect=RDP[,WRP] option is currently experimental. It should be used
1128 with care and may not "play well" with some other features such as
1129 write sparing and sparse writing. It should be used to copy user data
1130 plus the associated protection information to or from a regular file.
1131 It could also be used for a device to device copy assuming the "pt"
1132 interface is used for both. Also only modern SCSI disks support protec‐
1133 tion information.
1134 When RDP or WRP is greater than 0 then a copy with associated protec‐
1136 tion information is active. In this state IBS and OBS must be the same
1137 and equal to the logical block size of the device(s) formatted with
1138 protection information. If a SCSI disk with 512 byte logical block size
1139 has protection information then the actual number of bytes transferred
1140 for each logical block is typically 520 bytes. For such a disk BS=512
1141 is required even when additional protection information is being trans‐
1142 ferred.
1143
1145 By default numeric arguments to options are assumed to be decimal.
1146 Almost all numeric arguments to options (e.g. COUNT in the count=COUNT
1147 option) may include one of these multiplicative suffixes: c C *1; w W
1148 *2; b B *512; k K KiB *1,024; KB *1,000; m M MiB *1,048,576; MB
1149 *1,000,000 . This pattern continues for "G", "T" and "P". The latter
1150 two suffixes can only be used for 64 bit values. Some numeric arguments
1151 are limited to 32 bit values (e.g. BSin the bs=BS option). Also a suf‐
1152 fix of the form "x<n>" multiplies the leading number by <n>; however
1153 the combinations "0x" and "0X" are treated differently, see the next
1154 paragraph. These multiplicative suffixes are compatible with GNU's dd
1155 command (since 2002) which claims compliance with the SI and with IEC
1156 60027-2 standards.
1157 Alternatively numerical values can be given in hexadecimal indicated by
1159 either a leading "0x" or "0X", or by a trailing "h" or "H". When hex
1160 numbers are given, suffix multipliers cannot be used.
1161 If a numeric argument is required to fit in 32 bits and is too large
1163 then an error is reported. Usually negative numbers are not permitted
1164 but "count=-1" is a special case and means "all available"; "ver‐
1165 bose=-1" is another special case.
1166
1168 Copying data behind an Operating System's back can cause problems. In
1169 the case of Linux, users should look at this link:
1170 http://linux-mm.org/Drop_Caches
1171 This command sequence may be useful:
1172 sync; echo 3 > /proc/sys/vm/drop_caches
1173 A partial write is a write to the OFILE of less than OBS bytes. This
1175 typically occurs at the end of a copy. dd can do partial writes. ddpt
1176 does partial writes to regular files and fifos (including stdout). How‐
1177 ever ddpt ignores partial writes when OFILE is a block device or a pt
1178 device. When ddpt ignores a partial write, it sends a warning to the
1179 console (stderr).
1180 At the end of the copy two lines are reported to the console:
1182 <in_full>+<in_partial> records in
1183 <out_full>+<out_partial> records out
1184 The "records in" line is the number of full input blocks (each of IBS
1186 bytes) that have been read plus the number of partial blocks ( usually
1187 less than IBS bytes) that have been read. Following the lead of dd when
1188 'iflag=coe' is active a block that cannot be read (and has zeros sub‐
1189 stituted for its output) is regarded as a partial read. The "records
1190 out" line is the number of full output blocks (each of OBS bytes) that
1191 have been written plus the number of partial blocks (usually less than
1192 OBS bytes) that have been written.
1193 Block devices (e.g. /dev/sda and /dev/hda) can be given for IFILE. If
1195 neither 'iflag=direct' nor 'iflag=pt' is given then normal block IO
1196 involving buffering and caching is performed. If 'iflag=direct' is
1197 given then the buffering and caching is bypassed (this is applicable to
1198 both SCSI devices and ATA disks). When 'iflag=pt' is given SCSI com‐
1199 mands are sent to the device which bypasses most of the actions per‐
1200 formed by the block layer. The same applies for block devices given
1201 for OFILE.
1202 All informative, warning and error reports are sent to stderr so that
1204 dd's output file can be stdout and remain unpolluted. If no options are
1205 given, then no copying (nor reading) takes place and a brief message is
1206 sent to stderr inviting the user to invoke ddpt again but with '--help'
1207 option to get the usage message.
1208 Disk partition information can often be found with fdisk(8) [the "-ul"
1210 argument is useful in this respect]. Also parted(8) can be used like
1211 this: 'parted /dev/sda unit s print' .
1212 For pt devices this utility issues SCSI READ and WRITE (SBC) commands
1214 which are appropriate for disks and reading from CD/DVD/BD drives.
1215 Those commands are not formatted correctly for tape drives so ddpt can‐
1216 not be used on tape drives via a pt device. If the largest block
1217 address of the requested transfer exceeds a 32 bit block number (i.e
1218 0xffffffff) then a warning is issued and the pt device is accessed via
1219 SCSI READ(16) and WRITE(16) commands.
1220 The attributes of a block device (e.g. partitions) are ignored when the
1222 pt flag is used. Hence the whole device is read (rather than just the
1223 second partition) by this invocation:
1224 ddpt if=/dev/sdb2 iflag=pt of=t bs=512
1226 Assuming /dev/sdb and /dev/sg2 refer to the same device, then after the
1228 following two invocations, the contents of the files "t", "tt" and
1229 "ttt" should be same:
1230 ddpt if=/dev/sdb of=tt bs=512
1232 ddpt if=/dev/sg2 of=ttt bs=512
1234 The SCSI READ(32) and WRITE(32) commands are restricted to media that
1236 is formatted with protection type 2. This is a T10 restriction.
1237
1239 The signal handling has been borrowed from GNU's dd: SIGINT, SIGQUIT
1240 and SIGPIPE report the number of remaining blocks to be transferred and
1241 the records in + out counts; then they have their default action.
1242 SIGUSR1 (or SIGINFO) causes the same information to be output and the
1243 copy continues. All output caused by signals is sent to stderr.
1244 Like GNU's dd, ddpt respects the signal disposition of "ignored"
1246 (SIG_IGN) set by the shell, script or other program that invokes ddpt.
1247 So in that case it will ignore such signals. Further dd ignores SIGUSR1
1248 if the environment variable POSIXLY_CORRECT is set because POSIX
1249 defines dd will only act on SIGINFO (and Linux has no such signal);
1250 ddpt ignores the POSIXLY_CORRECT environment variable. As recommended
1251 by Susv3, ddpt does not expect the signal (blocking) mask to be block‐
1252 ing SIGUSR1 (SIGINFO), SIGINT or SIGPIPE on entry.
1253 Unix system calls that do IO can be interrupted by signal processing,
1255 typically returning an EINTR error number. The dd utility (and many
1256 other Unix utilities) restart the IO operation that was interrupted.
1257 While this will work most of the time for disk IO it is problematic for
1258 tape drives because the implicit position pointer on the tape may have
1259 moved. So the default (i.e. "intio=0") in this utility is to mask
1260 those signals during IO operations and only check them prior to start‐
1261 ing an IO operation. Most low level IO (e.g. using SCSI command to
1262 write to a disk) will timeout if there is a low level error. However
1263 NFS (the Network File System) will potentially wait for a long time
1264 (e.g. expecting a network problem will soon be fixed) and in this case
1265 using "intio=1" may be best.
1266
1268 There is support for copies to and from tape drives in Linux. Only the
1269 st driver device names can be used (e.g. /dev/st0 and /dev/nst2). Hence
1270 use of Linux pass-through device names (e.g. /dev/sg2) for tape drives
1271 is not supported. On Debian-based distributions, it is suggested that
1272 the mt-st package is installed as it provides a more fully-featured
1273 version of the "mt" tape control program.
1274 Tape drives can operate in fixed- or variable-length block modes. In
1276 variable-block mode, each write to the tape writes a single block of
1277 that size. In fixed-block mode, each write to the tape must be a multi‐
1278 ple of the previously-selected block size.
1279 The block size/mode can be set with the mt command prior to invoking
1281 ddpt. For example:
1282 # mt -f /dev/nst0 setblk 0
1283 sets variable-block mode, and
1284 # mt -f /dev/nst0 setblk 32768
1285 sets fixed-block mode with block size 32768 bytes.
1286 Note that some tape drives support only fixed-block mode, and possibly
1288 even only one block size. (For example, QIC-150 tapes use a fixed block
1289 size of 512 bytes.) There may also be restrictions on the block size,
1290 e.g. it may have to be even.
1291 When using ddpt to write to tape, if the final read from the input is
1293 less than OBS, it is padded to OBS bytes before writing to tape to
1294 ensure that all blocks of the tape file are the same length. Having a
1295 shorter final block would fail if the drive is in fixed-block mode, and
1296 could create interchange problems. It is common to expect all blocks in
1297 a file on tape to be the same length. However, to tell ddpt to not pad
1298 the final block, use 'oflag=nopad'.
1299 The st tape driver normally writes a filemark when the file (e.g.
1301 /dev/nst0) is closed. To not have the filemark written, use
1302 'oflag=nofm'. One use case for that might be if using ddpt several
1303 times in succession to append more data to the same file on tape. In
1304 that case it is probably desirable to write the filemark at the end of
1305 the sequence. So either omit 'oflag=nofm' on the last ddpt invocation,
1306 or manually write a filemark using mt after ddpt exits:
1307 # mt -f /dev/nst0 weof 1
1308 For reading from an unknown tape where the block size(s) is not known,
1310 read in variable-block mode specifying a large IBS. The st driver
1311 returns a smaller amount of data if the size of the block read is
1312 smaller. Thus a command like:
1313 # ddpt if=/dev/nst0 of=output.bin bs=262144
1314 should read the file from tape regardless of the block size used
1315 (assuming no blocks are larger than 256KB). ddpt's verbose option will
1316 display what the actual block size(s) is.
1317
1319 If the command line invocation of an xcopy does not explicitly (and
1320 unambiguously) indicate whether the XCOPY SCSI command should be sent
1321 to IFILE (i.e. the source) or OFILE (i.e. the destination) then a
1322 check is made for the presence of the XCOPY_TO_SRC and XCOPY_TO_DST
1323 environment variables. If either one exists (but not both) then it
1324 indicates where the SCSI XCOPY command will be sent. By default the
1325 XCOPY command is sent to OFILE.
1326 The ODX write from tokens variant is very complex to implement if the
1328 amount of data held in each ROD is not known. The value should be found
1329 in the "number of bytes represented" field in the ROD Token but that is
1330 not well supported yet by vendors. So for such case that number can be
1331 appended as a big endian 8 byte integer following each ROD Token in the
1332 RTF file. The conv=rtf_len will cause that length to be appended. Spec‐
1333 ifying that option on each read to tokens and write from tokens invoca‐
1334 tion can be a nuisance. Setting the environment variable ODX_RTF_LEN
1335 will cause this utility to act as if the conv=rtf_len option has been
1336 given.
1337 Sometimes the default block size of 512 can be a nuisance. This can be
1339 overridden by the value associated with the DDPT_DEF_BS environment
1340 variable. If the environment variable is not found, the value cannot be
1341 decoded or is zero or less, then the default block size remains at 512
1342 bytes.
1343
1345 To aid scripts that call ddpt, the exit status is set to indicate suc‐
1346 cess (0) or failure (1 or more). Note that some of the lower values
1347 correspond to the SCSI sense key values. The exit status values are:
1348 0 success
1350 1 syntax error. Either illegal command line options, options with
1352 bad arguments or a combination of options that is not permitted.
1353 2 the device reports that it is not ready for the operation
1355 requested. The device may be in the process of becoming ready
1356 (e.g. spinning up but not at speed) so the utility may work
1357 after a wait.
1358 3 the device reports a medium or hardware error (or a blank
1360 check). For example an attempt to read a corrupted block on a
1361 disk will yield this value.
1362 5 the device reports an "illegal request" with an additional sense
1364 code other than "invalid operation code". This is often a sup‐
1365 ported command with a field set requesting an unsupported capa‐
1366 bility.
1367 6 the device reports a "unit attention" condition. This usually
1369 indicates that something unrelated to the requested command has
1370 occurred (e.g. a device reset) potentially before the current
1371 SCSI command was sent. The requested command has not been exe‐
1372 cuted by the device. Note that unit attention conditions are
1373 usually only reported once by a device.
1374 7 the device reports a "data protect" sense key. This implies some
1376 mechanism has blocked writes (or possibly all access to the
1377 media).
1378 9 the device reports an illegal request with an additional sense
1380 code of "invalid operation code" which means that it doesn't
1381 support the requested command.
1382 10 the device reports a "copy aborted". This implies another com‐
1384 mand or device problem has stopped and copy operation. The
1385 EXTENDED COPY family of commands (including WRITE USING TOKEN)
1386 may return this sense key.
1387 11 the device reports an aborted command. In some cases aborted
1389 commands can be retried immediately (e.g. if the transport
1390 aborted the command due to congestion).
1391 15 the utility is unable to open, close or use the given IFILE or
1393 OFILE. The given file name could be incorrect or there may be
1394 permission problems. Adding the -v option may give more informa‐
1395 tion.
1396 20 the device reports it has a check condition but "no sense". It
1398 is unlikely that this value will occur as an exit status.
1399 21 the device reports a "recovered error". The requested command
1401 was successful. Most likely a utility will report a recovered
1402 error to stderr and continue, probably leaving the utility with
1403 an exit status of 0 .
1404 24 the device reports a SCSI status of "reservation conflict". This
1406 means access to the device with the current command has been
1407 blocked because another machine (HBA or SCSI "initiator") holds
1408 a reservation on this device. On modern SCSI systems this is
1409 related to the use of the PERSISTENT RESERVATION family of com‐
1410 mands.
1411 33 the command sent to device has timed out. This occurs in Linux
1413 only; in other ports a command timeout will appear as a trans‐
1414 port (or OS) error.
1415 40 the command sent to a device has received an "aborted command"
1417 sense key with an additional sense code of 0x10. This group is
1418 related to problems with protection information (PI or DIF). For
1419 example this error may occur when reading a block on a drive
1420 that has never been written (or is unmapped) if that drive was
1421 formatted with type 1, 2 or 3 protection.
1422 51 a command received 'illegal field in parameter list'. This may
1424 occur with an odx copy if some combination of parameters is
1425 illegal or not supported (e.g. iflag=immed)
1426 55 a command received 'operation in progress'. This may occur with
1428 an odx copy when the given LID is already being used by another
1429 process (e.g. also using odx) on the same machine. Choose
1430 another LID.
1431 70 a command received 'invalid token operation, cause not
1433 reportable'. This may occur with an odx operation when the given
1434 ROD Token is invalid. One reason for that may be the inactivity
1435 timeout has been reached and the copy manager has cancelled the
1436 ROD Token.
1437 71-89 these status values provide more information than exit status
1439 70. See SPC-4 ASC and ASCQ assignments (currently in Annex F.2),
1440 specifically the entries for asc=23h . For example exit status
1441 72 corresponds to asc=23h, ascq=2h which implies the odx copy
1442 manager does not support copies between LUs in different tar‐
1443 gets. That is optional; an odx copy manager is required to sup‐
1444 port copies between LUs (that are block devices) in the same
1445 target.
1446 90 the flock flag has been given on a device and some other process
1448 holds the advisory exclusive lock.
1449 97 the response to a SCSI command failed sanity checks.
1451 98 the device reports it has a check condition but the error
1453 doesn't fit into any of the above categories.
1454 99 any errors that can't be categorized into values 1 to 98 may
1456 yield this value. This includes transport and operating system
1457 errors after the command has been sent to the device.
1458
1460 The examples in this page use Linux device names. For suitable device
1461 names in other supported Operating Systems see this web page:
1462 http://sg.danny.cz/sg/device_name.html . The sg3_utils(8) man page in
1463 the sg3_utils package also covers device naming.
1464 ddpt usage looks quite similar to dd:
1466 ddpt if=/dev/sg0 of=t bs=512 count=1MB
1468 This will copy 1 million 512 byte blocks from the device associated
1470 with /dev/sg0 (which should have 512 byte blocks) to a file called t.
1471 Assuming /dev/sda and /dev/sg0 are the same device then the above is
1472 equivalent to:
1473 dd if=/dev/sda iflag=direct of=t bs=512 count=1000000
1475 although dd's speed may improve if bs was larger and count was suitably
1477 reduced. The use of the 'iflag=direct' option bypasses the buffering
1478 and caching that is usually done on a block device.
1479 The dd command's bs argument can be thought of as roughly equivalent to
1481 ddpt's bs*bpt . dd almost assumes buffering on a block device and will
1482 work as long as bs is a multiple of the actual logical block size.
1483 Since ddpt can work at a lower level in some cases the bs argument must
1484 be a disk's actual logical block size. Thus the bpt argument was intro‐
1485 duced to make the copy more efficient. So these two invocations are
1486 roughly equivalent:
1487 dd if=/dev/sda of=t bs=8k count=64
1489 ddpt if=/dev/sda of=t bs=512 bpt=16 count=1k
1490 In both cases the total number of bytes moved is bs*count . And that
1492 will be done by reading 8k (8192 bytes) into a buffer then writing out
1493 that buffer to the file t. The read write sequence continues until the
1494 count is complete or an error occurs.
1495 The 'of2=' option can save time when the input would otherwise need to
1497 be read twice. For example, to copy data and take a md5sum of it with‐
1498 out needing to re-read the data:
1499 mkfifo fif
1501 md5sum fif &
1502 ddpt if=/dev/sg3 iflag=coe of=sg3.img oflag=sparse of2=fif bs=512
1503 This will image /dev/sg3 (e.g. an unmounted disk) and place the con‐
1505 tents in the (sparse) file sg3.img . Without re-reading the data it
1506 will also perform a md5sum calculation on the image.
1507 Now we use sparse writing logic to get some idea of how many blocks on
1509 a disk are full of zeros. After a SCSI FORMAT UNIT command or an ATA
1510 SECURITY ERASE command a disk may be all zeros.
1511 ddpt if=/dev/sdc bs=512 oflag=sparse
1513 Since no "of=" option is given, output goes to /dev/null so nothing is
1515 actually written so the "records out" will be zero. However there will
1516 be a count of "records in" and "bypassed records out". If /dev/sdc is
1517 full of zeros then "records in" and "bypassed records out" will be the
1518 same. Since the "bpt=" option is not given it defaults to "bpt=128,128"
1519 so the copy buffer will be 64 KiB and the sparse check for zeros will
1520 be done with 64 KiB (128 block) granularity.
1521 For examples of the trim and self,trim options see the section above on
1523 TRIM, UNMAP AND WRITE SAME.
1524 Following is an example run on a Windows OS using the '--wscan' option
1526 which shows the available device names (e.g. PD1) and the associated
1527 volume name(s):
1528 ddpt -w
1530 PD0 [C] FUJITSU MHY2160BH 0000
1531 PD1 [DF] WD 2500BEV External 1.05 WD-WXE90
1532 CDROM0 [E] MATSHITA DVD/CDRW UJDA775 CB03
1533 So, for example, volumes D: and F: reside on PhysicalDisk1 (abbreviated
1535 to "PD1") which is manufactured by WD (Western Digital).
1536 Further examples can be found on this web page:
1538 http://sg.danny.cz/sg/ddpt.html . There is a text file containing exam‐
1539 ples called ddpt_examples.txt in the "doc" directory of this package's
1540 distribution tarball. The ddpt_examples.txt file contains some examples
1541 of using job files.
1542
1544 Written by Doug Gilbert
1545
1547 Report bugs to <dgilbert at interlog dot com>.
1548
1550 Copyright © 2008-2014 Douglas Gilbert
1551 This software is distributed under the GPL version 2. There is NO war‐
1552 ranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PUR‐
1553 POSE.
1554
1556 This utility has a companion/helper utility called ddptctl(8)
1557 There is a web page discussing ddpt at http://sg.danny.cz/sg/ddpt.html
1558 The lmbench package contains lmdd which is also interesting. For moving
1560 data to and from tapes see dt which is found at http://www.scsi‐
1561 faq.org/RMiller_Tools/index.html
1562 To change mode parameters that effect a SCSI device's caching and error
1564 recovery see sdparm(sdparm)
1565 To verify the data on the media or to verify it against some other copy
1567 of the data see sg_verify(sg3_utils)
1568 To scan and repair disk partitions see TestDisk (testdisk).
1570 Additional references: dd(1), open(2), flock(2),
1572 sg_xcopy,sg_copy_results, sg_dd(sg3_utils)
1573ddpt-0.95 December 2014 DDPT(8)