1DDPT_SGL(8) DDPT DDPT_SGL(8)
2
6 ddpt_sgl - helper for ddpt utility's scatter gather lists
7
9 ddpt_sgl [--action=ACT] [--a-sgl=SGL] [--b-sgl=SGL] [--chs=CHS] [--de‐
10 gen=DV] [--document] [--elem=SE[,LE]] [--extension=FNE] [--flexible]
11 [--help] [--hex] [--iaf=IAF] [--index=IA] [--interleave=IL]
12 [--non-overlap] [--out=O_SGL] [--quiet] [--round=RB] [--sort-cmp=SC]
13 [--stats] [--verbose] [--version]
14
16 This utility is a scatter gather list helper for the ddpt utility which
17 copies data between or within SCSI devices (logical units). While
18 ddpt's command line syntax is modelled on that of the POSIX dd command,
19 this utility has a more standard Unix command line syntax with both
20 short and long variants of each option.
21 Scatter gather lists (sgl_s) are made up of scatter gather elements.
23 Each element is made up a starting logical block address (LBA) and a
24 number of blocks (NUM) from and including that LBA.
25 The scatter gather lists can also be viewed as arrays in which elements
27 can be accessed by an index. Multiple sgl elements can be accessed with
28 an array of indexes, hence index arrays. Indexes in this utility start
29 at 0 and run to (n - 1) where n is the number of elements in the sgl.
30 Also negative indexes are permitted where -1 is the index of the last
31 sgl element, -2 is the index of the second last sgl element, etc.
32 For "twin" actions there is an assumed relationship between a-sgl and
34 b-sgl as there is between two sgl_s used as the gather list (e.g.
35 skip=) and the scatter list (e.g. seek=) in the ddpt utility. Breaking
36 it down to individual logical blocks: LBAr0 is read and its data is
37 written to LBAw0, LBAr1-->LBAw1, LBAr2-->LBAw2, etc; or more generally
38 LBAr_n-->LNAw_n. Many actions will change the order in which those
39 "read-write" items are performed, the twin portion of the action at‐
40 tempts to maintain the LBAr_n-->LNAw_n mapping. Generally speaking,
41 copies are the same no matter what order the LBAs are read and written.
42 One exception is an overlapping scatter list (i.e. on the write side)
43 in which case the order of writes onto the same LBA does matter, hence
44 there is an option to check sgl_s are well-formed in that respect:
45 --non-overlap.
46 For background on scatter gather lists see the section of that name in
48 the ddpt(8) manpage found in this package. There is a web page at
49 https://sg.danny.cz/sg/ddpt.html .
50
52 Arguments to long options are mandatory for short options as well.
53 -a, --action=ACT
55 ACT is some action to perform on the given scatter gather
56 list(s). To list the available actions set ACT to 'xxx' (or
57 'enum'). The available actions are listed in the ACTIONS section
58 below.
59 -A, --a-sgl=SGL
61 SGL is a scatter gather list, a sequence of comma separated un‐
62 signed integers (up to 64 bits each). SGL has several forms, the
63 simplest is: LBA0,NUM0,LBA1,NUM1,LBA2,NUM2... and there should
64 be an even number of values with the exception of LBA0 appearing
65 by itself. In this case NUM0 is assumed to be 0. Other SGL forms
66 are '@<filename>' and 'H@<filename>' where the contents of the
67 <filename> is parsed as a scatter gather list. Since there are
68 two options for inputting SGLs, this one is termed as the
69 'a-sgl'.
70 See the section on FILE FORMATS below and the section on SCATTER
71 GATHER LISTS in the ddpt(8) manpage for more information on
72 sgl_s and their associated terminology.
73 -B, --b-sgl=SGL
75 SGL is a scatter gather list, a second list termed as the
76 'b-sgl' to differentiate it from the other sgl (a-sgl).
77 -C, --chs=CHS
79 CHS is a 3 element tuple, separated by commas. Currently 16 bit
80 values from 1 to 0xffff are accepted (the cylinder can be one
81 more: 0x10000 (or 65536)). The first value is the number of
82 cylinders, the second value is the number of heads (limited to
83 16), and the final value is the number of sectors per track
84 (limited to 255). Sectors are counted origin 1 according to CHS
85 conventions (cf. normal LBAs which nearly always count from 0).
86 -D, --degen=DV
88 DV of 0 (the default) means that all degenerate elements (apart
89 from the last) are ignored (and dropped from the internal repre‐
90 sentation which may later be written to an output file). If DV
91 is odd then a degenerate element's LBA is taken into account
92 when calculating the highest and lowest LBA in a sgl (and may be
93 included in an output file). If DV is even (apart from a DV of
94 0) then a degenerate element's LBA it taken into account when
95 determining if a sgl is monotonic increasing, decreasing or nei‐
96 ther (and may be included in an output file).
97 -d, --document
99 this option causes information about the a sgl or index array to
100 be written as comments (i.e. lines starting with '#') to the be‐
101 ginning of output file(s) created by this utility.
102 If this option is given twice then the command line that caused
103 the output is added to the file as a comment (before any numbers
104 are output).
105 -E, --elem=SE[,LE]
107 this option allows a single sgl element (at position SE (start‐
108 ing element index)) to be output to O_SGL or O_SGL.FNE (or IAF).
109 SE is origin 0. If LE (last element index) is given then a range
110 of sgl elements are output starting at index SE to index LE in‐
111 clusive. If a "twin" operation is being performed then this op‐
112 tion only applies to the "a" side output, not the "twin" side.
113 This option is ignored by the output of the split_n and tsplit_n
114 actions.
115 Negative values for either SE or LE count from the end of sgl.
117 For example --elem=0,-1 refers to the whole of the list.
118 If LE is less than SE (after any negative indexes are converted
119 to their equivalent positive index) then that range is output in
120 reverse.
121 -e, --extension=FNE
123 FNE is the filename extension used when output filenames are
124 generated. For non-split actions the generated filenames are of
125 the form: O_SGL.FNE . For the split_n action the generated file‐
126 names are of the form: O_SGL[1..n].FNE . For the tsplit_n action
127 the a-sg is named as per the previous sentence, while for the
128 b-sgl the generated filenames are of the form: O_SGL[1..n]_t.FNE
129 .
130 If O_SGL is '-' (by itself) then all output is sent to stdout
131 and this option is ignored.
132 -f, --flexible
134 this option effects the parsing (reading) of sgl_s and index ar‐
135 rays that are in files which are in hexadecimal. Such files
136 should have a leading line (i.e. before any numbers) with 'HEX'
137 on it. Without this option any such file must be invoked with
138 'H@' before the filename; in other words the 'H' in the invoca‐
139 tion needs to match the 'HEX' in the file. With this option a
140 file can be invoked with '@' and if a line with 'HEX' is parsed
141 before any numbers then it switches to hexadecimal mode; so that
142 all the parsed numbers are assumed to be in hexadecimal.
143 -h, --help
145 outputs the usage message summarizing command line options then
146 exits.
147 -H, --hex
149 used to define the numeric format of sgl and index array ele‐
150 ments written to output (often a file named O_SGL or stdout). If
151 not given then only decimal values are written to output. If
152 this option is given once then hexadecimal values, prefixed with
153 '0x', are written. If this option is given twice then a line
154 with the string 'HEX' is written to output, before any values,
155 and those values are implicitly hexadecimal (i.e. no leading
156 '0x' nor 'h' suffix).
157 -I, --iaf=IAF
159 where IAF is a filename (or '-' for stdout) to write an index
160 array to. The only action that generates an index array cur‐
161 rently is --action=sort (and tsort). This option can be together
162 used with, or in place of, the --out=O_SGL option.
163 The --document, --elem=SE[,LE] and --hex options effect what is
164 written. See the section on FILE FORMATS below.
165 -x, --index=IA
167 where IA is one or more indexes, comma separated or, if prefixed
168 by "@" or "H@", a filename containing a list of indexes. These
169 indexes are used by the --action=select and --action=tselect to
170 select elements from the 'a-sgl'. Positive and negative indexes
171 that are too large (in absolute terms) are ignored and create
172 noise if the --verbose option is given. See the section on FILE
173 FORMATS below.
174 -i, --interleave=IL
176 IL is an integer, starting from 0. When IL is 0 (the default)
177 there is no interleave. The interleave only effects the split_n
178 and tsplit_n actions and when greater than zero is the maximum
179 number of logical blocks written in each segment in the output
180 file, prior to moving to the next output file.
181 For the case where IL is 1 and --action=split_1 is given then
182 the output file will have every LBA (given by the a-sgl) as a
183 separate sgl element (and thus each will have a NUM of 1).
184 For the tsplit_n action the interleave is only applied to the
185 a-sgl but it does effect the twin sgl files.
186 -N, --non-overlap
188 Checks any given sgl and any resulting sgl (from an action) to
189 see if any portion of the sgl overlaps. This is done by first
190 sorting each sgl by the LBA field, then checking every element
191 against the previous one to determine if there is overlap. SCSI
192 commands that accept sgl_s process degenerate elements without
193 error but if two elements in a WRITE command overlap then it is
194 the storage device's choice which one to WRITE first. The last
195 one to be written will be the one read in subsequent read opera‐
196 tions.
197 If no errors are detected then if (all) are non-overlapping then
198 0 is returned. If no errors are detected then if (any) are over‐
199 lapping then 36 is returned.
200 -o, --out=O_SGL
202 O_SGL is the name of a file to write a resultant scatter gather
203 list to. If O_SGL is '-' then the output is directed to stdout.
204 If O_SGL starts with '+' then the output is appended to the file
205 whose name follows the '+'.
206 For the split and tsplit actions, the leading '+' is interpreted
207 as appended to all files that meet the template and exist, oth‐
208 erwise the file is created. If '-' is given then all output is
209 directed to stdout (and the --extension=FNE option, if given, is
210 ignored).
211 -q, --quiet
213 suppresses warning and messages announcing an action has suc‐
214 ceeded. When this option is given, actions that have a logical
215 (boolean) result don't output messages but still yield an in‐
216 dicative exit status. The exit status will typically be either 0
217 for true or 36 for false. are typically sent to stderr.
218 -r, --round=RB
220 RB is the number of round blocks. Without the option the split_n
221 action will divide the number of blocks to be split by '<n>' (or
222 use IL) to get a nominal value. This value is the number of
223 blocks taken from the a-sgl before moving to the next output
224 file. The RB value (default 0) is the maximum number of blocks
225 the nominal value may be changed by to align with an existing
226 element boundary in the a-sgl.
227 If the number of blocks in 'a-sgl is less than 10 or RB is
228 greater than one third of the nominal value, then RB is ignored
229 (with a notification written to stderr).
230 For the tsplit_n action this option only applies to the a-sgl.
231 -S, --sort-cmp=SC
233 where SC is a value indicating what the sort action's comparison
234 will be. When SC is 0 (the default) the sort is ascending based
235 on the LBA; when it is 1 the sort is descending based on LBA.
236 When SC is 2 the sort is ascending based on NUM; when it is 3
237 the sort is descending based on NUM. Any other value is mapped
238 to 0. All sorts are stable which means that sgl elements with
239 the same LBA (in the case of SC being 0 or 1) keep their same
240 relative position. A side effect of this is that the ascending
241 and descending sorts are not always reversals of one another.
242 -s, --stats
244 print out sgl statistics on any given sgl and any resultant sgl.
245 -v, --verbose
247 increase the level of verbosity, (i.e. debug output).
248 -V, --version
250 print the version string and then exit.
251
253 Actions are given on the command line as part of the --action=ACT op‐
254 tion. Currently only one action is allowed per invocation. If more are
255 allowed in the future, they will be comma separated and performed in
256 the order in which they appear (i.e. left to right).
257 If no action is given and the --a-sgl=SGL and --out=O_SGL options (with
259 no --b-sgl=SGL option) are given then the a-sgl is copied to O_SGL (or
260 O_SGL.FNE if the --extension=FNE option is given).
261 The actions are listed below in alphabetical order.
263 append-b2a
265 appends the b-sgl to the end of the a-sgl and outputs the result
266 to O_SGL (or O_SGL.FNE if the --extension=FNE option is given).
267 Requires the --a-sgl=SGL, --b-sgl=SGL and --out=O_SGL options.
268 divisible<n>[,L|N] or divisible_<n>[,L|N]
270 where <n> is an integer, 1 or higher. This action checks if each
271 LBA and NUM in a-sgl is divisible by <n> (where 'is divisible'
272 is equivalent to having a remainder of zero). If all are divisi‐
273 ble then true is returned (i.e. the exit status 0); otherwise
274 false is returned (i.e. exit status 36).
275 If the optional ",L" suffix (or ",LBA") is given then only each
276 LBA element in a-sgl is checked for divisibility. If the op‐
277 tional ",N" suffix (or ",NUM") then only each NUM element in
278 a-sgl is checked for divisibility.
279 The output of the string to stderr announcing divisibility, or
280 lack of it, can be suppressed by the --quiet option.
281 enum prints out the list of supported actions then exits. Giving the
283 action 'xxx' has the same effect.
284 equal this action compares the sgl_s given to --a-sgl=SGL and
286 --b-sgl=SGL. If the same LBAs are in the same order with the
287 same overall number of blocks (but not necessarily the same num‐
288 ber of elements) then true is returned (i.e. the exit status 0);
289 otherwise false is returned (i.e. exit status 36). For example
290 the two element sgl "0x10,0x5, 0x15,0x2" is 'equal' to the one
291 element sgl "0x10, 0x7".
292 The output of the string to stderr announcing equality, or lack
293 of it, can be suppressed by the --quiet option.
294 none this action does nothing. This is the default action. If
296 --a-sgl=SGL and --out=O_SGL options are given and no other ac‐
297 tion, then a-sgl is copied to O_SGL.
298 It is a placeholder.
299 part-equal
301 this action is similar to the equal action but relaxes the con‐
302 dition that both lists must have the same overall number of
303 blocks. For example the two element sgl "0x10,0x5, 0x15,0x2" is
304 'part-equal' to the one element sgl "0x10, 0x12".
305 part-same
307 this action is similar to the same action but relaxes the condi‐
308 tion that both lists must have the same overall number of
309 blocks. For example the two element sgl "0x15,0x2,0x10,0x5" is
310 'part-same' as the one element sgl "0x10, 0x12".
311 same this action is similar to the equal action but relaxes the con‐
313 dition that both lists must be in the same order. The implemen‐
314 tation sorts both given lists before comparing them. For exam‐
315 ple the two element sgl "0x15,0x2, 0x10,0x5" is the 'same' as
316 the one element sgl "0x10, 0x7".
317 scale<n> or scale_<n>
319 where <n> is an integer, positive or negative but not zero. When
320 <n> is positive then the starting LBA and the NUM in each a-sgl
321 element is multiplied by <n> . The new (scaled) sgl is written
322 to O_SGL (or O_SGL.FNE if the --extension=FNE option is given).
323 When <n> is negative then the absolute value of <n> is used as a
324 divisor for each starting LBA and NUM in each a-sgl element.
325 As an example: converting a 512 byte logical block (LB) size sgl
326 to a 4096 byte LB size and vice versa is relatively common. To
327 convert from 4096 --> 512 byte LB size then --action=scale_8 is
328 appropriate. To convert from 512 --> 4096 byte LB size then
329 --action=scale_-8 is appropriate.
330 Note: because an integer division is used (that rounds 'towards
331 zero') when <n> is negative then LBs or NUMs may be "lost" in
332 this conversion. This can be checked beforehand with the --ac‐
333 tion=divisible<n>[,L|N] option. For example: for 512 --> 4096
334 conversions: --action=divisible_8 will report if any starting
335 LBAs or NUMs are not divisible be 8 and hence are not able to be
336 precisely represented as 4096 byte LB addresses or number of
337 4096 byte blocks.
338 select this action can be used to select a subset (or superset) of the
340 a-sgl in the specified order. Alternatively it can be seen as
341 re-ordering the elements in a-sgl such as is done toward the end
342 of a sort operation. Assuming all the indexes in IA are valid,
343 then the O_SGL file will have the same number of elements as
344 there are indexes in IA.
345 This option requires non-empty --a-sgl=SGL and --index=IA op‐
346 tions, plus the --out=O_SGL option.
347 sort this action will sort the sgl given by --a-sgl=SGL in ascending
349 order by LBA. The resulting sgl is output to O_SGL (or O_SGL.FNE
350 if the --extension=FNE option is given).
351 The sort is "stable", so if two elements have the same starting
352 LBA then they will appear in the same relative order in the out‐
353 put.
354 split<n> or split_<n>
356 where <n> is an integer, 1 or higher. This action divides
357 --a-sgl=SGL into <n> roughly equal length (i.e. number of
358 blocks) output sgl_s. The output files are named "O_SGL<1..n>"
359 or "O_SGL<1..n>.FNE". Both the --interleave=IL and --round=RB
360 options are taken into account during the split process.
361 to-chs this action takes the 'flat' LBA SGL given to --a-sgl=SGL and
363 converts it into CHS (cylinder/head/sector) based SGL which is
364 written out as directed to --out=O_SGL. This action requires the
365 --chs=CHS option as well as the --a-sgl=SGL and --out=O_SGL op‐
366 tions.
367 tselect
369 this is a "twin select" action that selects from --a-sgl=SGL
370 (a-sgl) then re-orders --b-sgl=SGL (b-sgl) in unison. The select
371 from a-sgl is the same as described under the select action
372 above. Additionally b-sgl is is broken up so it has "breaks" at
373 the same positions (i.e. number of blocks from the start of the
374 sgl) as a-sgl does; plus the "breaks" b-sgl has already got. So
375 the "broken up" b-sgl will have at least as many elements as
376 a-sgl. The output of the re-ordered b-sgl is then written to
377 O_SGL_t or O_SGL_t.FNE if the --extension=FNE option is given.
378 tsort this is a "twin sort" action that sorts --a-sgl=SGL (a-sgl) and
380 re-orders --b-sgl=SGL (b-sgl) in unison. The sort of a-sgl is
381 the same as described under the sort action above. Additionally
382 b-sgl is is broken up so it has "breaks" at the same positions
383 (i.e. number of blocks from the start of the sgl) as a-sgl does;
384 plus the "breaks" b-sgl has already got. So the "broken up"
385 b-sgl will have at least as many elements as a-sgl. The re-or‐
386 dering vector generated by the stable sort of a-sgl is then ap‐
387 plied to the broken up b-sgl. The output of the re-ordered b-sgl
388 is then written to O_SGL_t or O_SGL_t.FNE if the --extension=FNE
389 option is given.
390 tsplit<n> or tsplit_<n>
392 this is a "twin split" action that splits the --a-sgl=SGL and
393 --b-sgl=SGL into separate series of output files. These separate
394 series maintain the LBA to LBA correspondence of the original
395 a_sgl and b_sgl lists. <n> is an integer, 1 or higher. This ac‐
396 tion divides --a-sgl=SGL into <n> roughly equal length (i.e.
397 number of blocks) output sgl_s. The "roughly equal length" is
398 influenced by the --interleave=IL and --round=RB options. The
399 output filenames are generated the same way as described for the
400 split action. The sgl from --a-sgl=SGL is expected to be a
401 "hard" sgl which means its last element should not be degenerate
402 (i.e. have a NUM of 0).
403 The second half of the "twin split" is to split the --b-sgl=SGL
404 sgl. The same number of output files are used as for the 'A'
405 side but the filenames have a slightly different form:
406 "O_SGL<1..n>_t" or "O_SGL<1..n>_t.FNE" (if the --extension=FNE
407 option is given). The critical point of this split is that it
408 moves in lockstep with the 'A' side split in the sense that
409 whatever block count an 'A' side segment uses, the following 'B'
410 side segment split uses the same block count. The sgl from
411 --b-sgl=SGL may be a "hard" or "soft" sgl. In the simplest case
412 the 'B' side sgl can be just '0' which gets expanded to '0,0'
413 (i.e. degenerate list starting at LBA 0); this will use the
414 overall block count from the 'A' side.
415
417 Both sgl_s and index arrays can be read from, or written to, files.
418 The options that supply sgl_s or index arrays to be read (e.g.
419 --a-sgl=SGL, --b-sgl=SGL and --index=IA) by default allow them to be
420 given directly on the command line. These will typically be comma sepa‐
421 rated lists (although space and tab could be used as separators if they
422 were appropriately escaped). So with these options when reading sgl_s
423 and index arrays, a leading "@" or "H@" is needed to indicate that a
424 file name follows.
425 By default, numbers given in this utility and other utilities in this
427 package are assumed to be in decimal. Hexadecimal (hex) numbers can be
428 given with either a leading "0x" or trailing "h". A whole file can be
429 flagged as containing hex numbers (and thus not needing a leading "0x"
430 nor trailing "h" on each number) by using "H@" on the command line be‐
431 fore the filename. The file itself may contain a line with 'HEX' in it,
432 prior to any numbers that are to be parsed. If the --flexible option is
433 given then "@" can be used before the filename and when 'HEX' is de‐
434 tected in the file (before any numbers) the code switches to hex mode.
435 Without the --flexible option "H@" must be use before the filename. As
436 a convenience the 'HEX' string may appear after hex numbers have been
437 decoded and it will be ignored. This is to allow hex sgl_s files to be
438 concatenated together and still be parsed without error.
439 A file being parsed may contain comments following a "#" symbols. Ev‐
441 erything from and include the hash mark to the end of a line is ig‐
442 nored. Blank lines and "whitespace" (spaces, tabs, CRs and LFs) are
443 also ignored.
444 If large sgl_s or index arrays are being used it is better to have one
446 element per line in the file to be read. This is because a line is not
447 expected to be over 1024 bytes long with more than 254 parsable items
448 on it. This utility imposes no limit on the number of lines a file to
449 be parsed may have.
450 Files to be written out by this utility have their names specified by
452 the --out=O_SGL (optionally together with --extension=FNE) and the
453 --iaf=IAF options. Unlike the file reading options, no "@" character
454 should be placed in front of the filename on the command line. If a
455 filename of "-" is given then output is written to stdout instead of a
456 file. stdout is normally the console. If the filename starts with "+"
457 then that character is skipped the output will be appended to that
458 file, if it exists. If the filename starts with "+" and the file does
459 not exist then it is created. If "+" is not given and the file already
460 exists then it is truncated (to 0) then overwritten. Some output file
461 names have numbers (e.g. as a result of the --action=spilt_<n> option)
462 or "_t" (e.g. as a result of "twin" actions) appended to them (before
463 the extension, if any). Sgl elements are output one per line, with a
464 comma separating the LBA and the NUM. Index arrays are output one ele‐
465 ment (an index) per line. The --hex option controls the form of those
466 numbers output. If --hex is not given, the numbers are output in deci‐
467 mal. If the --hex option is given one the number are output in hex with
468 a "0x" prefix. If the --hex option is given twice then the line 'HEX'
469 is written to the file before any numbers and those numbers are in hex
470 without any adornment (i.e. with no leading "0x").
471 If the --document option is given then some information including a
473 date timestamp of generation is placed as comments at the beginning of
474 files that are written out by this utility. If the --document option is
475 given twice then the invocation line of this utility that caused the
476 output is placed in the written file as an additional comment.
477 The written file format is compatible with the read file format. So,
479 for example, a sgl generated by a invocation of this utility can later
480 be used as a file to be read by another invocation of this utility.
481
483 The exit status of ddpt_sgl is 0 when it is successful. Note that some
484 options and actions that return a boolean value return 0 for true and
485 36 for false. Otherwise the exit status for this utility is the same as
486 that for ddpt. See the EXIT STATUS section in the ddpt man page.
487
489 Examples are needed. See testing/test_sgl.sh script in this package.
490 That script can be run without root permissions and places its work
491 file (sgl_s) in the /tmp directory.
492
494 Written by Douglas Gilbert.
495
497 Report bugs to <dgilbert at interlog dot com>.
498
500 Copyright © 2020-2021 Douglas Gilbert
501 This software is distributed under a FreeBSD license. There is NO war‐
502 ranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PUR‐
503 POSE.
504
506 ddpt(8), ddptctl(8)
507ddpt-0.97 April 2021 DDPT_SGL(8)