1JIGDO-FILE(1) JIGDO-FILE(1)
2
6 jigdo-file - Prepare files for Jigsaw Download (distribution of huge
7 files, e.g. CD images).
8
10 jigdo-file COMMAND
11 [ --image=cdrom.iso ] [ --jigdo=cdrom.jigdo ] [ --template=cdrom.tem‐
12 plate ] [ --force ] [ MORE OPTIONS ] [ [ FILES... ] [ --files-from=f ]
13 ]
14 Common COMMANDs: make-template, make-image, verify
16
18 Jigsaw Download, or short jigdo, is a scheme developed primarily to
19 make it easy to distribute huge filesystem images (e.g. CD (ISO9660) or
20 DVD (UDF) images) over the internet, but it could also be used for
21 other data which is awkward to handle due to its size, like audio/video
22 files or large software packages.
23 jigdo tries to ensure that the large file (always called image from now
25 on) is downloaded in small parts which can be stored on different
26 servers. People who want to download the image do so by telling the
27 jigdo(1) (NOT IMPLEMENTED YET) download tool to process one `.jigdo'
28 file; using it, jigdo downloads the parts and reassembles the image.
29 jigdo-file is used to prepare the files for download.
30 What makes jigdo special is that the parts that are used to reconstruct
32 the image can have any size and content - they only need to be con‐
33 tained in a contiguous region anywhere in the image.
34 For example, if you wish to distribute an ISO9660 image which contains
36 a snapshot of an FTP server, you can instruct jigdo-file to prepare the
37 download data in such a way that when people use jigdo to download the
38 image, jigdo actually fetches the individual files from the FTP server
39 and assembles them into an exact copy of your image - during the down‐
40 load! (If the image is not a filesystem dump, you can use split(1) to
41 create the small parts that the image will be reassembled from.)
42 You are completely free to choose where the individual parts of the im‐
44 age are stored: They may be in entirely different directories on dif‐
45 ferent servers (e.g. because of storage/bandwidth constraints), but
46 this is invisible to the people downloading your image. The information
47 about available servers only needs to be added to the `.jigdo' file by
48 you before distributing it.
49 The `DETAILS' section below contains technical details on how jigdo
51 works. The `EXAMPLES' section lists a number of common scenarios and
52 may help you to get an idea of what jigdo is useful for.
53
55 Many options are specific to a particular COMMAND; the ones below are
56 general or used by several commands. Further options are listed below
57 with the individual commands. All options are silently ignored if they
58 are not applicable to the current command. For any BYTES parameters to
59 options, you can append one of the letters `k', `M' or `G' to the
60 amount you specify, to indicate kilobytes, megabytes or gigabytes.
61 -h --help
63 Output short summary of commands and options.
64 -H --help-all
66 Output complete summary of commands and options.
67 -v --version
69 Output program version.
70 -i --image=cdrom.iso
72 Specify location of the file containing the image. The image is
73 the large file that you want to distribute.
74 -j --jigdo=cdrom.jigdo
76 Specify location of the Jigsaw Download description file. The
77 jigdo file is a human-readable file generated by jigdo-file, to
78 which you add information about all the servers you are going to
79 upload the files to. jigdo will download this file as the first
80 step of retrieving the image.
81 -t --template=cdrom.template
83 Specify location of the image `template' file. The template file
84 is a binary file generated by jigdo-file, it contains informa‐
85 tion on how to reassemble the image and also (in compressed
86 form) all the data from the image which was not found in any of
87 the parts.
88 Depending on the command, each of these three files is used
90 sometimes for input, sometimes for output. If the file is to be
91 used for output for a particular command and the output file al‐
92 ready exists, jigdo-file exits with an error, unless --force is
93 present.
94 In most cases, you will only need to specify one out of -i -j
96 -t, because any missing filenames will be deduced from the one
97 you specify. This is done by first stripping any extension from
98 the supplied name and then appending nothing (if deducing --im‐
99 age), `.jigdo' or `.template'.
100 -r --report=default|noprogress|quiet|grep
102 Control how verbose the program is, and what format the output
103 has: noprogress is the same as default except that no `x% done'
104 progress messages are printed. quiet restricts the output to
105 what is absolutely necessary, mostly error messages. grep is
106 only different from default for the make-template command: It
107 enables output in a simple `<offset> <file>' format which is
108 useful when searching for binary files in other binary files.
109 -f --force
111 Overwrite existent output files without complaining.
112 --no-force
114 This is the default. Refuse to overwrite existent output files.
115 -c --cache=jigdo-cache.db
117 jigdo-file usually needs to read the entire contents of all the
118 FILES you specify. If you use it repeatedly (e.g. because you
119 make a new CD image available daily), caching the file informa‐
120 tion will increase the program's speed significantly. The cache
121 file is automatically created if it is not yet present. Data is
122 usually both read from and written to it.
123 --no-cache
125 This is the default. Do not use a cache.
126 --cache-expiry=SECONDS
128 Set maximum age of cache entries. Any entries older than this
129 will be removed from the cache. The default is 30 days. You can
130 append one of the letters `h', `d', `w', `m', `y' to denote
131 hours, days, weeks, months or years, respectively. A value of
132 `0' or `off' disables expiry, so that all entries will stay in
133 the cache forever. See the section `CACHE FILES' below for more
134 information.
135 --readbuffer=BYTES
137 Set size of internal buffers. The default is 128k - if you have
138 a fast disc, increasing this value may make jigdo-file faster,
139 but in general, changing it is not necessary.
140 -C --checksum-algorithm=ALGO
142 Choice of checksum algorithm to use in describing the image and
143 matched diles. Valid options are md5 and sha256. sha256 is a
144 more secure algorithm, but will make the template file and jigdo
145 file slightly larger. It will also create template and jigdo
146 files that are not compatible with older versions of jigdo, be‐
147 fore 0.8.0. Default is currently md5, but this may change in fu‐
148 ture.
149 --checksum-block-size
151 Uninteresting internal parameter. Set size of blocks into which
152 files are subdivided. The default is 128k. If you change it, any
153 cache file will have to be regenerated. Internally, jigdo-file
154 may choose to use a slightly larger or smaller value.
155 -T --files-from=file
157 Read file and directory names from the specified file. If file
158 is `-', read names from standard input. Each line in the file is
159 taken as a name, so the names may contain spaces, but not new‐
160 line characters. An empty line causes jigdo-file to stop reading
161 from the file.
162 find(1) is a powerful tool for generating file lists, but make
164 sure to use `find -type f' if possible - otherwise, if you in‐
165 struct find to output both a filename and a symlink to that
166 filename, jigdo-file will read the file contents twice.
167 --hex Output checksums in hexadecimal instead of Base64-like format.
169 This should not be used with the make-template command, because
170 the resulting `.jigdo' file violates the `.jigdo' file format.
171 Its intended use is to make jigdo-file more interoperable with
172 other Unix shell utilities like md5sum(1).
173 --no-hex
175 This is the default. Use jigdo's own Base64-like encoding of
176 checksums.
177 --debug[=help|=all|=UNIT,~UNIT...
179 Switch on or off debugging output. Just `--debug' is equivalent
180 to `--debug=all'. The argument is a comma-separated list of unit
181 names for which debugging output is to be enabled, or disabled
182 if the name is preceded by `~'. The special name `all' means all
183 units. By default, debugging output is switched off except for
184 the units `assert' and `general'. The exact list of available
185 units for which debugging can be switched on depends on whether
186 jigdo was compiled with debugging support - the list can be
187 printed with `--debug=help'.
188 FILES Names of files or directories to use as input. These are the
190 parts that are contained in the image. In case one of the names
191 is a directory, the program recursively scans the directory and
192 adds all files contained in it. While doing this, it follows
193 symbolic links, but avoids symlink loops.
194 If one of the filenames starts with the character `-', you must
196 precede the list of files with `--'. A value of `-' has no spe‐
197 cial meaning in this list, it stands for a file whose name is a
198 single hyphen.
199
201 The command name is the first non-option argument passed to jigdo-file.
202 Most commands have short abbreviations as well as long names. The short
203 command names should not be used in scripts - there may be incompatible
204 changes to them in the future!
205 MAKE-TEMPLATE, MT
207 Reads image and FILES, creates `.jigdo' and `.template'. This is the
208 main functionality of jigdo-file.
209 It is possible to specify both --image=- and --files-from=-. In this
211 case, first the list of files is read from standard input until an
212 empty line is encountered. Everything following it is assumed to be the
213 image data. This can be useful if you use mkisofs(1) or similar pro‐
214 grams that can output the complete image on their standard output, be‐
215 cause there is no need to store the image on disc temporarily.
216 If a FILES argument contains the characters `//' (Unix) or `\.\' (Win‐
218 dows), this has special meaning. In the final jigdo file that users
219 will download, each of the parts is referenced in the `[Parts]' section
220 with a URI of the form `Label:some/filename'. (See `FORMAT OF .JIGDO
221 FILES' below for a detailed description.) The `[Servers]' section gives
222 a mapping of labels to servers on the internet, with lines like `La‐
223 bel=http://myserver.org/jigdofiles/'. Using this information, jigdo
224 will create the final download URI for the part, `http://my‐
225 server.org/jigdofiles/some/filename'. Specifying `//' (or `\.\') in a
226 file or directory name serves to `cut off' the names at the right di‐
227 rectory level. For example, if the Unix path of one of your FILES is
228 `/path/some/filename', you can tell jigdo-file to cut off after the
229 `/path' by passing it the argument `/path//some/filename', or `/path//'
230 if you want the whole directory scanned. The path names need not be ab‐
231 solute; `somedirectory//' is also possible.
232 --label Label=/path
234 Specify a name to use as the label name for a path on disc. (In‐
235 fluences the output jigdo file.) If you used `//' in the FILES
236 arguments as described above, jigdo-file will by default pick
237 label names automatically (`A', `B' etc.). With this option, you
238 can give labels more meaningful names. Note that the label name
239 will only be used if one or more FILES begin with `/path//'.
240 Try to use label names that start with uppercase characters, to
242 disambiguate them clearly from protocol names like `http',
243 `ftp'.
244 --uri Label=http://some.server.org/
246 By default, using --label as described above will cause lines of
247 the form `Label=file:/path/' to be written to the `[Servers]'
248 section of the output jigdo file. If you want to override the
249 `file:' URI so that the line reads `La‐
250 bel=http://some.server.org/', you can do so by specifying --uri
251 along with --label. Giving just --uri Label=... without the cor‐
252 responding --label Label=... has no effect, and even if you
253 specify both, an entry is only added to the `[Servers]' section
254 if the label is referenced by at least one `[Parts]' entry.
255 The supplied value is not quoted by the program; if it contains
257 characters such as space or any of the characters #"'\ then you
258 must quote it. (Under Unix, you may need to quote the value
259 twice to also protect it from the shell, e.g. \\\\ or '\\' to
260 get a single backslash in the URI.)
261 The mapping specified with an --uri option is ignored if it is
263 already present in the output jigdo file.
264 Users of the Windows version may notice that the `\' directory
266 separators are converted into `/' in the `file:' URIs that are
267 generated by default. This is done to increase cross-platform
268 compatibility of `file:' - the print-missing command of the Win‐
269 dows version will automatically re-convert the characters when
270 it prints the URIs. In case you supply your own `file:' URIs un‐
271 der Windows using --uri, you must also exchange `/' and `\'.
272 -0 to -9
274 Set amount of compression in the output template file, from -0
275 (no compression) to -9 (maximum compression). The default is -9,
276 which can make the template generation quite slow. By default,
277 the compression algorithm used is the same as for gzip(1).
278 --gzip and --bzip2
280 Choose between the gzip and bzip2 compression algorithms. The
281 default is gzip. Bzip2 usually gives a better compression ratio,
282 but compression is significantly slower than with gzip.
283 --min-length=BYTES
285 Set minimum length of a part for jigdo-file to look for it in
286 the image. The default is 1k. Parts smaller than this will
287 never be found in the image, so their data will be included in
288 the template file. The search algorithm used requires such a
289 minimum length, otherwise template generation could become ex‐
290 tremely slow. If you know for sure that all your FILES are
291 larger than a certain amount, you can increase jigdo-file's
292 speed slightly by specifying the amount with this option. There
293 is a hard-wired absolute minimum of 256 bytes - anything lower
294 will silently be set to 256.
295 --merge=FILE
297 Include the contents of FILE in the output `.jigdo' file. The
298 file can contain data which you want added to the output (for
299 example, a `[Servers]' section with a list of your servers as
300 entries), or it can be the jigdo file output by an earlier run
301 of jigdo-file.
302 It is possible to specify the same file for input with --merge
304 and for output with --jigdo. However, you will also need to use
305 --force to make the program overwrite the old version of the
306 jigdo file with the new one. FILE can be `-' for standard in‐
307 put.
308 When adding new information to the supplied file, jigdo-file
310 will not insert new lines into the `[Parts]' section if an entry
311 for the same checksum (but not necessarily with the same URI!)
312 already exists, and it will not insert new lines into the
313 `[Servers]' section if a completely identical entry already ex‐
314 ists.
315 When reading in the existing FILE, the behaviour is slightly
317 different: The program preserves entries in the `[Parts]' sec‐
318 tion with identical checksum, but different URIs. For completely
319 identical entries (same checksum and URI), only one entry is
320 preserved and the duplicates are removed. The `[Servers]' sec‐
321 tion is left untouched.
322 --image-section
324 This is the default. Causes jigdo-file to add an `[Image]' sec‐
325 tion to the `.jigdo' file.
326 As an exception, a new `[Image]' section is not added if you use
328 --merge and the file to merge contains an `[Image]' section with
329 a line which reads `Template-MD5Sum=' (end of line after the
330 `='). In this case, the generated template data's MD5 checksum
331 value is just added after the `=' of the first line of this form
332 in the file - no whole new `[Image]' section is appended. This
333 behaviour is useful because it allows you to pass via --merge an
334 `[Image]' section with arbitrary content and then have the MD5
335 checksum automatically added by jigdo-file. The section `FORMAT
336 OF .JIGDO FILES' below explains the `[Image]' section contents
337 in greater detail.
338 --no-image-section
340 Do not include an `[Image]' section in the `.jigdo' file. You
341 need to add one yourself if you use this option. However, doing
342 that is not easy (you also need to add a `Template-MD5Sum' line
343 with the correct checksum, or jigdo will complain), so use of
344 this option is discouraged.
345 --servers-section
347 This is the default. Causes jigdo-file to add a `[Servers]' sec‐
348 tion to the `.jigdo' file. This default section uses `file:'
349 URIs, which allows for immediate reassembly of the image from
350 the local filesystem, and is also useful if you want to edit the
351 file manually and replace the `file:' URIs with other URIs.
352 --no-servers-section
354 Do not add a `[Servers]' section at the end of the `.jigdo'
355 file. Useful e.g. if you are going to append the section with a
356 script.
357 --match-exec=SHELLCOMMAND
359 Whenever a file is found in the image, execute the supplied com‐
360 mand string by passing it to a shell. jigdo-file sets up a num‐
361 ber of environment variables with information about the file
362 match. For example, if the file `/path//a/b/file' was found in
363 the image and `Label:a/b/file' is going to be written to the
364 `.jigdo' file:
365 • LABEL="Label" - Name of the label for the file. The example
367 assumes that `--label Label=/path' was specified by you. In
368 the absence of such an option, LABEL will be set but empty.
369 • LABELPATH="/path/" - The path corresponding to the label, or
371 in other words, the prefix of the matched file's path that
372 will not appear in the output `.jigdo' file. Is set even with‐
373 out any `--label' option present. Ends with a slash.
374 • MATCHPATH="a/b/" - The rest of the path, without the leafname
376 of the matched file. Is either empty or ends with a slash.
377 • LEAF="file" - The leafname of the matched file.
379 • MD5SUM="lNVdUSqbo2yqm33webrhnw" - The md5sum of the matched
381 file, in Base64-like format.
382 • SHA256SUM="QXBJ8VZKeh0NXH0uOhdhgguPPE5tT1wvYO27sLx9Fsc" - The
384 sha256sum of the matched file, in Base64-like format.
385 • FILE="/path//a/b/file" - For convenience, the complete path of
387 the file. The variable is always set to $LABELPATH$MATCH‐
388 PATH$LEAF.
389 Please be careful to correctly quote the string passed to this option,
391 otherwise your supplied command will not work with filenames that con‐
392 tain spaces. As an example, to create a backup of hard links to the
393 matched files, use the following option: --match-exec='mkdir -p "${LA‐
394 BEL:-.}/$MATCHPATH" && ln -f "$FILE" "${LABEL:-.}/$MATCHPATH$LEAF"'
395 By default, no command is executed. Use --match-exec="" to remove a
397 command string which was set with an earlier use of this option.
398 --greedy-matching
400 This is the default. Imagine that your image contains a .tar
401 file which in turn contains another file x, and that you provide
402 both the .tar and the files inside it on the command line. When
403 jigdo-file scans the image, it encounters the beginning of the
404 .tar file, and then the file x.
405 At this point, a decision must be made: Should the smaller file
407 x be recorded as matched, or should it be ignored in favour of
408 the larger (and thus better) match of the .tar file? Unfortu‐
409 nately, at this point it is not clear whether there will actu‐
410 ally be a full match of the .tar, so by default, the program
411 prefers the small match.
412 --no-greedy-matching
414 In the case where a large partial match is present and a shorter
415 match has been confirmed, ignore the small match. (See the op‐
416 tion above.)
417 MAKE-IMAGE, MI
419 Reads `.template' and FILES, creates image (or `imagename.tmp'). Pro‐
420 vides a rudimentary way of reassembling images - jigdo is usually bet‐
421 ter suited for this task. However, in contrast to jigdo, no `.jigdo'
422 file is required.
423 If the image is to be written to a file (and not to standard output),
425 it is possible to create the image in several steps, with several invo‐
426 cations of `jigdo-file make-image', as follows: You first invoke jigdo-
427 file, specifying as many files as are available at this time. The pro‐
428 gram scans the files, and those that are contained in the image are
429 copied to a temporary file, whose name is formed by appending `.tmp' to
430 the image filename.
431 For all further files which could be parts of the image, you repeat
433 this process. As soon as all parts are present, the temporary file will
434 be truncated slightly (to delete some administrative data that jigdo-
435 file appends at the end) and renamed to the final image name. The pos‐
436 sibility of reassembling the image in several steps is especially use‐
437 ful for gathering files from removable media, e.g. several older CDs.
438 Scripts using make-image can detect whether image creation is complete
440 by checking the exit status: 0 signals successful creation, whereas 1
441 means that more files need to be supplied. Other errors result in an
442 exit status of 2 (`recoverable', e.g. file not found) or 3 (non-recov‐
443 erable, e.g. write error).
444 --check-files
446 This is the default. Whenever any part is copied to the image,
447 re-check its checksum against the checksum stored in the tem‐
448 plate. It is recommended that you leave this switched on, even
449 if it slows down image creation a bit.
450 --no-check-files
452 Do not check files' checksums when copying them to the image.
453 This can be safely used when no cache file is used (which means
454 that files will be written to the image immediately after being
455 scanned) or the whole image is checked later with the verify
456 command.
457 PRINT-MISSING, PM
459 Reads `.jigdo', `.template' and (if present) `imagename.tmp', outputs a
460 list of URIs still needed to completely reassemble the image.
461 Together with the make-image command, this provides most of the func‐
463 tionality of jigdo on the command line.
464 For each part that is not yet present in the temporary image file, the
466 file checksum is looked up in the `[Parts]' section of the jigdo file.
467 Any label in the corresponding entry is then expanded according to the
468 label definitions in the `[Servers]' section and printed on standard
469 output. jigdo allows you to specify several alternative locations for
470 each label in this section, but print-missing will only output the
471 first one for each missing part.
472 If the checksum cannot be found in the `[Parts]' section (this Should
474 Not Happen unless you deleted that section), a lookup is instead made
475 for `MD5Sum:<checksum>' and `SHA256Sum:<checksum>', just like with
476 jigdo. (Thus, if you want to get rid of the `[Parts]' section, you can
477 do so if you rename each part to its own checksum.)
478 --uri Label=http://some.server.org/
480 Override the entries in the `.jigdo' file for any label with a
481 URI of your choice. With the example above, a `[Parts]' entry of
482 `Label:some/filename' will cause the line
483 `http://some.server.org/some/filename' to be printed.
484 The supplied value is not quoted by the program; if it contains
486 characters such as space or any of the characters #"'\ then you
487 must quote it. (Under Unix, you may need to quote the value
488 twice to also protect it from the shell, e.g. \\\\ or '\\' to
489 get a single backslash in the URI.)
490 PRINT-MISSING-ALL, PMA
492 Just like print-missing, this command outputs a list of URIs still
493 needed to completely reassemble the image. However, all alternative
494 download locations are printed instead of just one. In the output, the
495 URIs for a file are separated from other files' URIs with blank lines.
496 The --uri option has the same effect as for print-missing.
497 VERIFY, VER
499 Reads image (presumably generated with make-image) and `.template',
500 checks for correct checksum of image.
501 The template data does not only contain checksums of the individual
503 parts, but also of the image as a whole. make-image already performs a
504 number of internal checks, but if you like, you can additionally check
505 the image with this command.
506 SCAN, SC
508 Reads all the FILES and enters them into the cache, unless they are al‐
509 ready cached. The --cache option must be present for this command.
510 --no-scan-whole-file
512 This is the default. This only causes the first --checksum-
513 block-size bytes of each file to be read. If the cache is used
514 later by jigdo-file make-image, the rest of the file will be
515 read once these first bytes are recognized in the input image.
516 --scan-whole-file
518 Immediately read the entire file contents and store them in the
519 cache.
520 MD5SUM, MD5
522 Reads all the FILES and prints out MD5 checksums of their contents.
523 This command is quite similar to md5sum(1), except that the checksum is
524 output in the Base64-like encoding which is also used elsewhere by
525 jigdo-file.
526 The FILES arguments are processed in the same way as with the other
528 commands, which means that recursion automatically takes place for any
529 arguments that are directories, and that symbolic links are not listed
530 except when the file(s) they point to are not reachable directly.
531 In the checksum list printed on standard output, only the part of the
533 filename following any `//' (or `\.\' on Windows) is printed. Any
534 --cache will be used for querying files' MD5 checksums and/or writing
535 the checksums of scanned files.
536 SHA256SUM, SHA256
538 Reads all the FILES and prints out SHA256 checksums of their contents.
539 This command is quite similar to sha256sum(1), except that the checksum
540 is output in the Base64-like encoding which is also used elsewhere by
541 jigdo-file.
542 The FILES arguments are processed in the same way as with the other
544 commands, which means that recursion automatically takes place for any
545 arguments that are directories, and that symbolic links are not listed
546 except when the file(s) they point to are not reachable directly.
547 In the checksum list printed on standard output, only the part of the
549 filename following any `//' (or `\.\' on Windows) is printed. Any
550 --cache will be used for querying files' SHA256 checksums and/or writ‐
551 ing the checksums of scanned files.
552 LIST-TEMPLATE, LS
554 Reads a `.template' file and outputs low-level information about the
555 image and all parts contained in it, including offset, length and
556 checksum.
557 You can also use this command with temporary image files (by specifying
559 something like --template=imagename.tmp) - in that case, the output
560 also distinguishes between parts that have been written to the image
561 and parts that haven't.
562 The exact output format may change incompatibly between different jigdo
564 releases. The following different types of lines can be output. `have-
565 file' only occurs for `.tmp' files, indicating a file that has already
566 been successfully written to the temporary file:
567 in-template offset-in-image length
569 need-file-md5 offset-in-image length file-md5sum filestart-rsyncsum
570 have-file-sha256 offset-in-image length file-sha256sum filestart-rsyncsum
571 image-info-sha256 image-length rsyncsum-size image-sha1sum
572
575 Jigsaw Download was created with the format of ISO9660 CD images in
576 mind - however, the following also applies to many other filesystem
577 formats, as well as to `tar' archives and uncompressed `zip' archives.
578 A CD image contains both information for organizing the filesystem
579 (header with disc name etc., ISO9660 directory data, data of extensions
580 such as Joliet or RockRidge, zero padding) and the files contained on
581 the CD. An important property that jigdo relies on is that each file is
582 stored in one contiguous section of the image; it is not split into two
583 or more parts.
584 When jigdo-file is given a number of files that might be contained in
586 an image, it detects whether any of the files are present using a
587 `rolling checksum' inspired by the one used by rsync(1). The resulting
588 data is written to the `.template' file: If a section of the image
589 could not be matched (e.g. it was directory information), the data is
590 compressed and written directly to the template. However, if a matching
591 file was found, its data is omitted from the template. Instead, only a
592 reference (an MD5 or SHA256 checksum of the file) is inserted in the
593 template.
594 Note that the template data only contains binary data, it does not con‐
596 tain any filenames or URIs, since it cannot be easily edited in case
597 any of these values need to be changed. All that information is stored
598 in the `.jigdo' file, a text file to which you can add URLs for your
599 server(s). The jigdo file provides a mapping for each MD5/SHA256 check‐
600 sum to one or more alternative download locations for the corresponding
601 part.
602 Apart from the mapping of MD5/SHA256 sums to URIs, the jigdo file also
604 contains an URI pointing to a download location for the template file.
605 This way, the jigdo download tool only needs to be given one URI (that
606 of the `.jigdo' file) to be able to download and reassemble the com‐
607 plete image.
608
610 The overall format of `.jigdo' files follows that of `.ini' files, as
611 also used by the Gnome and KDE projects for some data. The file is or‐
612 ganized into sections, each of which is preceded by a line reading
613 `[Sectionname]'. Within each section, lines have the form `La‐
614 bel=Value'. Such lines are also called `entries' below. All `.jigdo'
615 files use UTF-8 as their character encoding.
616 Comments are introduced with the `#' character and extend to the end of
618 the line. Whitespace is ignored at line start and end as well as to the
619 left and right of section names and the `=' in entries. Furthermore,
620 the jigdo utilities split up the text of the entry value (i.e. the part
621 after the `=') into whitespace-separated words, much like the Unix
622 shell. Single '' and double "" quotes can be used to prevent that e.g.
623 URIs containing whitespace are split apart. Similarly, characters with
624 special meaning (the characters '"#\ and space/tab) must be quoted with
625 \ to appear in the value. As with the shell, there is a difference be‐
626 tween ' ' and " ": Within ' ', the characters "#\ and whitespace lose
627 their special meaning and become ordinary characters, whereas within
628 " ", only the characters '# and whitespace lose their special meaning -
629 in other words, backslash escapes still work inside " ", but not ' '.
630 `.jigdo' files can optionally be compressed with gzip(1). jigdo-file
632 always outputs uncompressed files, which you can compress yourself.
633 jigdo-lite supports single uncompressed and compressed files.
634 (Behaviour which may change in the future and which should not be re‐
636 lied upon: jigdo additionally supports any number of concatenated
637 plaintext and gzipped parts in the files - for example, you can com‐
638 press a `.jigdo' file and then add a couple of lines of uncompressed
639 data to the end.)
640 In all cases, the `.gz' extension should be removed from the filename -
642 the tools will determine automatically from the file contents whether a
643 file is compressed or not.
644 Below is a description of the individual section names used by jigdo.
646 JIGDO SECTION
648 [Jigdo]
649 Version=1.1
650 Generator=jigdo-file/1.0.0
651 Information about the version of the jigdo file format used, and the
654 program that generated it. There should be one such section per
655 `.jigdo' file.
656 IMAGE SECTION
658 [Image]
659 Filename="filename for saving on user's disc"
660 Template="URI where to fetch template file"
661 Template-MD5Sum=OQ8riqT1BuyzsrT9964A7g
662 Template-SHA256Sum=MVJIxGifflRF9K8ERdbqoyns4Ucw9Xy1ubdnE6CtMbo
663 ShortInfo=single-line description of the image (200 characters max.)
664 Info=long description (5000 characters max.)
665 The value for the `Template' entry can be either an URL (absolute or
668 relative to the URL of the jigdo file) or a string of the form `La‐
669 bel:pathname' (UNIMPLEMENTED), as described below.
670 The `Template-MD5Sum' and/or `Template-MD5Sum' entries are added by
672 jigdo-file and specify the checksum of the generated `.template' file.
673 It is used by jigdo to detect cases where the downloaded template data
674 is corrupted or belongs to a different image.
675 Unlike other entry values, the values of the `ShortInfo' and `Info' en‐
677 tries are not split up into words, instead all quoting is preserved.
678 The value of the `Info' entry is special in that jigdo(1) can option‐
680 ally parse XML markup it contains. If the markup has errors such as un‐
681 balanced/unsupported tags, the string is displayed literally, without
682 XML parsing. Supported tags are <b></b> (bold), <i></i> (italic),
683 <tt></tt> (typewriter font), <u></u> (underline), <big></big> (larger
684 font), <small></small> (smaller font) and <br/> (linebreak). Supported
685 entities include < (`<'), > (`>') and & (`&'). Note that the
686 whole `Info' entry must be on one line in the jigdo file.
687 This section may occur multiple times, but all except the first one
689 will be ignored. This is useful e.g. when creating a `.jigdo' file for
690 a DVD image when you already have `.jigdo' files for CDs with the same
691 content: You can simply `[Include]' (see below) the CDs' jigdo files at
692 the end of the DVD jigdo file, after its `[Image]' section.
693 PARTS SECTION
695 [Parts]
696 xJNkjrq8NYMraeGavUpllw=LabelA:part0
697 GoTResP2EC6Lb_2wTsqOoQ=LabelA:part1
698 kyfebwu6clbYqqWUdFIyaw=LabelB:some/path/part2
699 -J9UAimo0Bqg9c0oOXI1mQ=http://some.where.com/part3
700 All lines in the section, which provides the mapping from checksums to
703 URIs, have the same format: On the left side of the `=' the checksum
704 (encoded with a Base64-like encoding) is given, and on the right a
705 string corresponding to the part with this checksum; either a complete
706 URI or a string of the form `Label:pathname', which is expanded into
707 one or more URIs by looking up the definition(s) for the Label in the
708 `[Servers]' section.
709 In case a particular checksum cannot be found in any `[Parts]' section
711 by jigdo, the program will perform a lookup for `MD5Sum:<checksum>' and
712 `SHA256Sum:<checksum>', e.g. for `MD5Sum:xJNkjrq8NYMraeGavUpllw' if you
713 deleted the line for `part0' above.
714 A checksum appearing multiple times in this section indicates alterna‐
716 tive download locations for the part.
717 There may be any number of `[Parts]' sections in the file; they are all
719 considered when looking up checksums.
720 jigdo-file always puts the `[Parts]' section at the end of the file,
722 and it even rearranges any file specified with --merge to have only one
723 such section at the end. This is done to allow jigdo to display the in‐
724 formation from the `[Image]' section while the rest of that file is
725 still being downloaded.
726 SERVERS SECTION
728 [Servers]
729 LabelA=http://myserver.org/
730 LabelA=ftp://mirror.myserver.org/
731 LabelB=LabelC:subdirectory/
732 LabelC=http://some.where.com/jigdo/
733 All lines in the section, which provides the mapping from server labels
736 to server locations, have the same format: On the left side of the `='
737 the label name is given, and on the right the value to expand the label
738 name to.
739 A label name appearing multiple times in this section indicates alter‐
741 native download locations for the parts that use the label in the
742 `[Parts]' section. This notation makes it very easy to add mirrors to
743 the jigdo file.
744 As shown by the example above, the label values may themselves refer‐
746 ence other labels. In this case, the entry `LabelB:some/path/part2' in
747 the `[Parts]' section will expand to `http://some.where.com/jigdo/sub‐
748 directory/some/path/part2'. Loops in the label definitions result in
749 undefined behaviour and must be avoided.
750 There may be any number of `[Servers]' sections in the file; they are
752 all considered when looking up labels. Either of `[Parts]' or
753 `[Servers]', but not both, can be omitted from the jigdo file.
754 COMMENT SECTION
756 [Comment]
757 Any text, except that lines must not begin with `['.
758 All text following a `[Comment]' or `[comment]' line is ignored, up to
761 the next line with a section label.
762 INCLUDE DIRECTIVE
764 [Include http://some.url/file.jigdo]
765 Lines of this form cause the content of the specified jigdo file to be
768 downloaded and parsed just like the main jigdo file. The effect will be
769 the same as copying the included file's contents into the file which
770 contains the include directive. (Exception: Any relative URLs are al‐
771 ways resolved using the URL of the `.jigdo' file that contains that
772 relative URL.)
773 The URL argument can be an absolute or relative URL. Relative URLs are
775 assumed to be relative to the URL of the jigdo file which contains the
776 include directive. Includes can be nested, but it is an error to create
777 a loop of include directives. It is not possible to use URLs of the
778 form `Label:pathname'.
779 The URL cannot be quoted with "". Any `]' characters in the argument
781 must be escaped as `%5D', and any spaces as `%20'.
782 Include directives are only supported by jigdo, they are ignored by
784 jigdo-lite.
785 An include directive terminates any previous section, but it does not
787 start a new one. In other words, a new section must always be started
788 after the include line, jigdo does not allow normal entries to appear
789 below the `[Include]'.
790
792 Any file specified with the --cache option is used to store information
793 about the FILES presented to jigdo-file. When querying the cache, a
794 file is considered unchanged (and the cached data is used) only if
795 filename, file size and last modification time (mtime) match exactly.
796 For the filename match, not the entire file name is used, but only the
797 part following any `//', so that any changes to the part before the
798 `//' will not invalidate the cache.
799 Old cache entries are removed from the cache if they have not been read
801 from or written to for the amount of time specified with --cache-ex‐
802 piry. Entries are not immediately removed from the cache if the file
803 they refer to no longer exists - this makes it possible to cache infor‐
804 mation about files on removable media.
805 Cache expiry only takes place after jigdo-file has done its main work -
807 if any old entries are accessed before expiry takes place, they will be
808 kept. For example, if the program is run using the default expiry time
809 of 30 days, but accesses a cache file with entries generated 2 months
810 ago, then entries in that cache will be considered, and only those
811 cache entries that were not needed during the program run will be ex‐
812 pired.
813 Due to a peculiarity of the underlying database library (libdb3), cache
815 files never shrink, they only grow. If a large number of entries was
816 expired from your cache file and you want it to shrink, you can either
817 just delete it (of course then everything will have to be regenerated)
818 or use the utilities accompanying libdb3 to dump and restore the data‐
819 base, with a command like `db3_dump old-cache.db | db3_load new-
820 cache.db'. For Debian, these programs are supplied in the package
821 `libdb3-util'.
822 If a different --md5-block-size is specified, the entire file needs to
824 be re-read to update its cache entry. If a different --min-length is
825 specified, only the first `md5-block-size' bytes of the file need to be
826 re-read.
827
829 PREPARING YOUR CD IMAGE FOR DISTRIBUTION
830 You have created a CD image `image.iso' from some of the files stored
831 in the directory `/home/ftp' on your harddisc, which is also available
832 online as `ftp://mysite.org'. As you don't want to waste space by ef‐
833 fectively hosting the same data twice (once as files on the FTP server,
834 once inside the image), and you are fed up with users' downloads abort‐
835 ing after 200MB and their restarting the download dozens of times, you
836 decide to use jigdo. How do you prepare the image for download?
837 In fact, only one command is necessary:
839 jigdo-file make-template --image=image.iso --jigdo=/home/ftp/im‐
842 age.jigdo --template=/home/ftp/image.template /home/ftp// --la‐
843 bel Mysite=/home/ftp --uri Mysite=ftp://mysite.org/
844 People can now point jigdo at `ftp://mysite.org/image.jigdo' to down‐
847 load your image. The template file needs to be accessible as
848 `ftp://mysite.org/image.template'.
849 Note that nothing prevents you from doing the same for an FTP server
851 that isn't administrated by you - in that case, you only need to host
852 the `.jigdo' and `.template' files on your own server/homepage.
853 PREPARING AN ARBITRARY LARGE FILE FOR DISTRIBUTION
855 We assume that you have a large file that is not a filesystem, e.g.
856 `movie.mpeg'. Because of space problems, you want to distribute the
857 data on two servers.
858 In this case, the parts of the image need to be generated artificially
860 with the split command. For example, to create chunks of 4MB each, use
861 `split -b 4m movie.mpeg part'. Copy the resulting files `partXX' into
862 two directories `1' and `2' that you create, according to how you want
863 the files distributed between the servers. Next, create the jigdo and
864 template files with `jigdo-file make-template --image=movie.mpeg 1//
865 2//'. You will need to edit the `.jigdo' file and provide the right
866 URIs for the two servers that you are going to upload the `partXX'
867 files to.
868 CUSTOMIZED VERSIONS OF IMAGES
870 Because it is possible to assign a different URI for each part of an
871 image if necessary, jigdo is very flexible. Only one example is the
872 possibility of customized versions of images: Suppose that someone is
873 distributing a CD image, and that you want to make a few small changes
874 to it and redistribute your own version. You download the `offi‐
875 cial.iso' CD image with jigdo (passing it the URL of `official.jigdo'),
876 write it to CD-R, make your changes (say, adding files from the `my‐
877 files' directory on your harddisc) and produce your own version,
878 `myversion.iso'. Next, you instruct jigdo-file to create the jigdo and
879 template files for your modified image, using the command
880 jigdo-file make-template --image=myversion.iso /mnt/cdrom/ my‐
883 files// --label My=myfiles/ --uri My=http://my.homepage.net/
884 --merge=official.jigdo
885 while `official.iso' is mounted under `/mnt/cdrom'. By using --merge,
887 you have told jigdo-file to take the contents of `official.jigdo', add
888 to it a new `[Image]' section for `myversion.iso' and write the result‐
889 ing jigdo file to `myversion.jigdo' - so now `myversion.jigdo' offers
890 two images for download, the original version and your modified ver‐
891 sion. (If you do not want it to offer the official version, edit it and
892 remove the `[Image]' section that lists `official.iso'.)
893 Now you can upload the `.jigdo' file, the `.template' file and also the
895 files in `myfiles' to `http://my.homepage.net/'. Thus, for people to
896 download your modified image, you do not need to upload the complete
897 image contents to your web space, but only the changes you made!
898 (In case you only made very few changes, you could also omit the `my‐
900 files' parameter in the command above, then all your changes end up in
901 the new template file.)
902 COMBINING MANY JIGDO-MANAGED IMAGES INTO ONE
904 It is also no problem to combine data from several sources that use
905 jigdo. For example, if of five different and unrelated servers each one
906 distributes a different CD image via jigdo, you can create a customized
907 DVD image that contains the data from all these CDs. When people use
908 jigdo to download your image, the individual files on the DVD are
909 fetched from the same sources as the original CDs.
910 Consequently, even though you will be distributing a 3.2GB file via
912 your web space, the actual amount of data that is stored on your server
913 will only be in the order of several MBs.
914
916 For certain contents of one of the input files, most notably a sequence
917 of zero bytes longer than --min-length at the start of the file and an
918 area of zeros preceding the file data in the image, jigdo-file make-
919 template may fail to find the file in the image. Unfortunately, this
920 restriction cannot be avoided because the program could become very
921 slow otherwise. If you use the --debug option, all instances of jigdo-
922 file discarding possible matches are indicated by lines containing the
923 word `DROPPED'.
924 In fact, not only all-zeroes files trigger this behaviour, but also
926 files which contain at their start a long sequence of short identical
927 strings. For example, both a file containing only `a' characters and
928 one containing `abcabcabcabc...' are problematic.
929
931 jigdo(1) (NOT YET IMPLEMENTED), jigdo-lite(1), jigdo-mirror(1),
932 split(1) (or `info split'), find(1) (or `info find'), mkisofs(1),
933 md5sum(1), sha256sum(1), jigit-mkimage(1)
934
936 Jigsaw Download <URL:http://atterer.org/jigdo/> was written by Richard
937 Atterer <jigdo atterer.org>, to make downloading of CD ROM images for
938 the Debian Linux distribution more convenient.
939 Steve McIntyre <93sam@debian.org> picked up later development of jigdo
941 after Richard had moved on - see the git repo work
942 <URL:https://git.einval.com/cgi-bin/gitweb.cgi?p=jigdo.git;a=summary>
943 or packages in Debian for more recent releases.
944 Oct 31, 2021 JIGDO-FILE(1)