1JIGDO-FILE(1) JIGDO-FILE(1)
2
6 jigdo-file - Prepare files for Jigsaw Download (distribution of huge
7 files, e.g. CD images).
8
10 jigdo-file COMMAND
11 [ --image=cdrom.iso ] [ --jigdo=cdrom.jigdo ] [ --template=cdrom.tem‐
12 plate ] [ --force ] [ MORE OPTIONS ] [ FILES ... | --files-from=f ]
13 Common COMMANDs: make-template, make-image, verify
14
17 Jigsaw Download, or short jigdo, is a scheme developed primarily to
18 make it easy to distribute huge filesystem images (e.g. CD (ISO9660) or
19 DVD (UDF) images) over the internet, but it could also be used for
20 other data which is awkward to handle due to its size, like audio/video
21 files or large software packages.
22 jigdo tries to ensure that the large file (always called image from now
24 on) is downloaded in small parts which can be stored on different
25 servers. People who want to download the image do so by telling the
26 jigdo(1) (NOT IMPLEMENTED YET) download tool to process one `.jigdo'
27 file; using it, jigdo downloads the parts and reassembles the image.
28 jigdo-file is used to prepare the files for download.
29 What makes jigdo special is that the parts that are used to reconstruct
31 the image can have any size and content - they only need to be con‐
32 tained in a contiguous region anywhere in the image.
33 For example, if you wish to distribute an ISO9660 image which contains
35 a snapshot of an FTP server, you can instruct jigdo-file to prepare the
36 download data in such a way that when people use jigdo to download the
37 image, jigdo actually fetches the individual files from the FTP server
38 and assembles them into an exact copy of your image - during the down‐
39 load! (If the image is not a filesystem dump, you can use split(1) to
40 create the small parts that the image will be reassembled from.)
41 You are completely free to choose where the individual parts of the
43 image are stored: They may be in entirely different directories on dif‐
44 ferent servers (e.g. because of storage/bandwidth constraints), but
45 this is invisible to the people downloading your image. The information
46 about available servers only needs to be added to the `.jigdo' file by
47 you before distributing it.
48 The `DETAILS' section below contains technical details on how jigdo
50 works. The `EXAMPLES' section lists a number of common scenarios and
51 may help you to get an idea of what jigdo is useful for.
52
54 Many options are specific to a particular COMMAND; the ones below are
55 general or used by several commands. Further options are listed below
56 with the individual commands. All options are silently ignored if they
57 are not applicable to the current command. For any BYTES parameters to
58 options, you can append one of the letters `k', `M' or `G' to the
59 amount you specify, to indicate kilobytes, megabytes or gigabytes.
60 -h --help
62 Output short summary of commands and options.
63 -H --help-all
65 Output complete summary of commands and options.
66 -v --version
68 Output program version.
69 -i --image=cdrom.iso
71 Specify location of the file containing the image. The image is
72 the large file that you want to distribute.
73 -j --jigdo=cdrom.jigdo
75 Specify location of the Jigsaw Download description file. The
76 jigdo file is a human-readable file generated by jigdo-file, to
77 which you add information about all the servers you are going to
78 upload the files to. jigdo will download this file as the first
79 step of retrieving the image.
80 -t --template=cdrom.template
82 Specify location of the image `template' file. The template file
83 is a binary file generated by jigdo-file, it contains informa‐
84 tion on how to reassemble the image and also (in compressed
85 form) all the data from the image which was not found in any of
86 the parts.
87 Depending on the command, each of these three files is used
89 sometimes for input, sometimes for output. If the file is to be
90 used for output for a particular command and the output file
91 already exists, jigdo-file exits with an error, unless --force
92 is present.
93 In most cases, you will only need to specify one out of -i -j
95 -t, because any missing filenames will be deduced from the one
96 you specify. This is done by first stripping any extension from
97 the supplied name and then appending nothing (if deducing
98 --image), `.jigdo' or `.template'.
99 -r --report=default|noprogress|quiet|grep
101 Control how verbose the program is, and what format the output
102 has: noprogress is the same as default except that no `x% done'
103 progress messages are printed. quiet restricts the output to
104 what is absolutely necessary, mostly error messages. grep is
105 only different from default for the make-template command: It
106 enables output in a simple `<offset> <file>' format which is
107 useful when searching for binary files in other binary files.
108 -f --force
110 Overwrite existent output files without complaining.
111 --no-force
113 This is the default. Refuse to overwrite existent output files.
114 -c --cache=jigdo-cache.db
116 jigdo-file usually needs to read the entire contents of all the
117 FILES you specify. If you use it repeatedly (e.g. because you
118 make a new CD image available daily), caching the file informa‐
119 tion will increase the program's speed significantly. The cache
120 file is automatically created if it is not yet present. Data is
121 usually both read from and written to it.
122 --no-cache
124 This is the default. Do not use a cache.
125 --cache-expiry=SECONDS
127 Set maximum age of cache entries. Any entries older than this
128 will be removed from the cache. The default is 30 days. You can
129 append one of the letters `h', `d', `w', `m', `y' to denote
130 hours, days, weeks, months or years, respectively. A value of
131 `0' or `off' disables expiry, so that all entries will stay in
132 the cache forever. See the section `CACHE FILES' below for more
133 information.
134 --readbuffer=BYTES
136 Set size of internal buffers. The default is 128k - if you have
137 a fast disc, increasing this value may make jigdo-file faster,
138 but in general, changing it is not necessary.
139 --md5-block-size=BYTES
141 Uninteresting internal parameter. Set size of blocks into which
142 files are subdivided. The default is 128k. If you change it, any
143 cache file will have to be regenerated. Internally, jigdo-file
144 may choose to use a slightly larger or smaller value.
145 -T --files-from=file
147 Read file and directory names from the specified file. If file
148 is `-', read names from standard input. Each line in the file is
149 taken as a name, so the names may contain spaces, but not new‐
150 line characters. An empty line causes jigdo-file to stop reading
151 from the file.
152 find(1) is a powerful tool for generating file lists, but make
154 sure to use `find -type f' if possible - otherwise, if you
155 instruct find to output both a filename and a symlink to that
156 filename, jigdo-file will read the file contents twice.
157 --hex Output checksums in hexadecimal instead of Base64-like format.
159 This should not be used with the make-template command, because
160 the resulting `.jigdo' file violates the `.jigdo' file format.
161 Its intended use is to make jigdo-file more interoperable with
162 other Unix shell utilities like md5sum(1).
163 --no-hex
165 This is the default. Use jigdo's own Base64-like encoding of
166 checksums.
167 --debug[=help|=all|=UNIT,~UNIT... ]
169 Switch on or off debugging output. Just `--debug' is equivalent
170 to `--debug=all'. The argument is a comma-separated list of unit
171 names for which debugging output is to be enabled, or disabled
172 if the name is preceded by `~'. The special name `all' means all
173 units. By default, debugging output is switched off except for
174 the units `assert' and `general'. The exact list of available
175 units for which debugging can be switched on depends on whether
176 jigdo was compiled with debugging support - the list can be
177 printed with `--debug=help'.
178 FILES Names of files or directories to use as input. These are the
180 parts that are contained in the image. In case one of the names
181 is a directory, the program recursively scans the directory and
182 adds all files contained in it. While doing this, it follows
183 symbolic links, but avoids symlink loops.
184 If one of the filenames starts with the character `-', you must
186 precede the list of files with `--'. A value of `-' has no spe‐
187 cial meaning in this list, it stands for a file whose name is a
188 single hyphen.
189
191 The command name is the first non-option argument passed to jigdo-file.
192 Most commands have short abbreviations as well as long names. The short
193 command names should not be used in scripts - there may be incompatible
194 changes to them in the future!
195 MAKE-TEMPLATE, MT
197 Reads image and FILES, creates `.jigdo' and `.template'. This is the
198 main functionality of jigdo-file.
199 It is possible to specify both --image=- and --files-from=-. In this
201 case, first the list of files is read from standard input until an
202 empty line is encountered. Everything following it is assumed to be the
203 image data. This can be useful if you use mkisofs(1) or similar pro‐
204 grams that can output the complete image on their standard output,
205 because there is no need to store the image on disc temporarily.
206 If a FILES argument contains the characters `//' (Unix) or `\.\' (Win‐
208 dows), this has special meaning. In the final jigdo file that users
209 will download, each of the parts is referenced in the `[Parts]' section
210 with a URI of the form `Label:some/filename'. (See `FORMAT OF .JIGDO
211 FILES' below for a detailed description.) The `[Servers]' section gives
212 a mapping of labels to servers on the internet, with lines like
213 `Label=http://myserver.org/jigdofiles/'. Using this information, jigdo
214 will create the final download URI for the part,
215 `http://myserver.org/jigdofiles/some/filename'. Specifying `//' (or
216 `\.\') in a file or directory name serves to `cut off' the names at the
217 right directory level. For example, if the Unix path of one of your
218 FILES is `/path/some/filename', you can tell jigdo-file to cut off
219 after the `/path' by passing it the argument `/path//some/filename', or
220 `/path//' if you want the whole directory scanned. The path names need
221 not be absolute; `somedirectory//' is also possible.
222 --label Label=/path
224 Specify a name to use as the label name for a path on disc.
225 (Influences the output jigdo file.) If you used `//' in the
226 FILES arguments as described above, jigdo-file will by default
227 pick label names automatically (`A', `B' etc.). With this
228 option, you can give labels more meaningful names. Note that the
229 label name will only be used if one or more FILES begin with
230 `/path//'.
231 Try to use label names that start with uppercase characters, to
233 disambiguate them clearly from protocol names like `http',
234 `ftp'.
235 --uri Label=http://some.server.org/
237 By default, using --label as described above will cause lines of
238 the form `Label=file:/path/' to be written to the `[Servers]'
239 section of the output jigdo file. If you want to override the
240 `file:' URI so that the line reads
241 `Label=http://some.server.org/', you can do so by specifying
242 --uri along with --label. Giving just --uri Label=... without
243 the corresponding --label Label=... has no effect, and even if
244 you specify both, an entry is only added to the `[Servers]' sec‐
245 tion if the label is referenced by at least one `[Parts]' entry.
246 The supplied value is not quoted by the program; if it contains
248 characters such as space or any of the characters #"'\ then you
249 must quote it. (Under Unix, you may need to quote the value
250 twice to also protect it from the shell, e.g. \\\\ or '\\' to
251 get a single backslash in the URI.)
252 The mapping specified with an --uri option is ignored if it is
254 already present in the output jigdo file.
255 Users of the Windows version may notice that the `\' directory
257 separators are converted into `/' in the `file:' URIs that are
258 generated by default. This is done to increase cross-platform
259 compatibility of `file:' - the print-missing command of the Win‐
260 dows version will automatically re-convert the characters when
261 it prints the URIs. In case you supply your own `file:' URIs
262 under Windows using --uri, you must also exchange `/' and `\'.
263 -0 to -9
265 Set amount of compression in the output template file, from -0
266 (no compression) to -9 (maximum compression). The default is -9,
267 which can make the template generation quite slow. By default,
268 the compression algorithm used is the same as for gzip(1).
269 --gzip and --bzip2
271 Choose between the gzip and bzip2 compression algorithms. The
272 default is gzip. Bzip2 usually gives a better compression ratio,
273 but compression is significantly slower than with gzip.
274 --min-length=BYTES
276 Set minimum length of a part for jigdo-file to look for it in
277 the image. The default is 1k. Parts smaller than this will
278 never be found in the image, so their data will be included in
279 the template file. The search algorithm used requires such a
280 minimum length, otherwise template generation could become
281 extremely slow. If you know for sure that all your FILES are
282 larger than a certain amount, you can increase jigdo-file's
283 speed slightly by specifying the amount with this option. There
284 is a hard-wired absolute minimum of 256 bytes - anything lower
285 will silently be set to 256.
286 --merge=FILE
288 Include the contents of FILE in the output `.jigdo' file. The
289 file can contain data which you want added to the output (for
290 example, a `[Servers]' section with a list of your servers as
291 entries), or it can be the jigdo file output by an earlier run
292 of jigdo-file.
293 It is possible to specify the same file for input with --merge
295 and for output with --jigdo. However, you will also need to use
296 --force to make the program overwrite the old version of the
297 jigdo file with the new one. FILE can be `-' for standard
298 input.
299 When adding new information to the supplied file, jigdo-file
301 will not insert new lines into the `[Parts]' section if an entry
302 for the same MD5 checksum (but not necessarily with the same
303 URI!) already exists, and it will not insert new lines into the
304 `[Servers]' section if a completely identical entry already
305 exists.
306 When reading in the existing FILE, the behaviour is slightly
308 different: The program preserves entries in the `[Parts]' sec‐
309 tion with identical checksum, but different URIs. For completely
310 identical entries (same checksum and URI), only one entry is
311 preserved and the duplicates are removed. The `[Servers]' sec‐
312 tion is left untouched.
313 --image-section
315 This is the default. Causes jigdo-file to add an `[Image]' sec‐
316 tion to the `.jigdo' file.
317 As an exception, a new `[Image]' section is not added if you use
319 --merge and the file to merge contains an `[Image]' section with
320 a line which reads `Template-MD5Sum=' (end of line after the
321 `='). In this case, the generated template data's MD5 checksum
322 value is just added after the `=' of the first line of this form
323 in the file - no whole new `[Image]' section is appended. This
324 behaviour is useful because it allows you to pass via --merge an
325 `[Image]' section with arbitrary content and then have the MD5
326 checksum automatically added by jigdo-file. The section `FORMAT
327 OF `[Image]' section contents in greater detail.
328 --no-image-section
330 Do not include an `[Image]' section in the `.jigdo' file. You
331 need to add one yourself if you use this option. However, doing
332 that is not easy (you also need to add a `Template-MD5Sum' line
333 with the correct checksum, or jigdo will complain), so use of
334 this option is discouraged.
335 --servers-section
337 This is the default. Causes jigdo-file to add a `[Servers]' sec‐
338 tion to the `.jigdo' file. This default section uses `file:'
339 URIs, which allows for immediate reassembly of the image from
340 the local filesystem, and is also useful if you want to edit the
341 file manually and replace the `file:' URIs with other URIs.
342 --no-servers-section
344 Do not add a `[Servers]' section at the end of the `.jigdo'
345 file. Useful e.g. if you are going to append the section with a
346 script.
347 --match-exec=SHELLCOMMAND
349 Whenever a file is found in the image, execute the supplied com‐
350 mand string by passing it to a shell. jigdo-file sets up a num‐
351 ber of environment variables with information about the file
352 match. For example, if the file `/path//a/b/file' was found in
353 the image and `Label:a/b/file' is going to be written to the
354 `.jigdo' file:
355 · LABEL="Label" - Name of the label for the file. The example
357 assumes that `--label Label=/path' was specified by you. In
358 the absence of such an option, LABEL will be set but empty.
359 · LABELPATH="/path/" - The path corresponding to the label, or
361 in other words, the prefix of the matched file's path that
362 will not appear in the output `.jigdo' file. Is set even with‐
363 out any `--label' option present. Ends with a slash.
364 · MATCHPATH="a/b/" - The rest of the path, without the leafname
366 of the matched file. Is either empty or ends with a slash.
367 · LEAF="file" - The leafname of the matched file.
369 · MD5SUM="lNVdUSqbo2yqm33webrhnw" - The md5sum of the matched
371 file, in Base64-like format.
372 · FILE="/path//a/b/file" - For convenience, the complete path of
374 the file. The variable is always set to $LABELPATH$MATCH‐
375 PATH$LEAF.
376 Please be careful to correctly quote the string passed to this option,
378 otherwise your supplied command will not work with filenames that con‐
379 tain spaces. As an example, to create a backup of hard links to the
380 matched files, use the following option: --match-exec='mkdir -p
381 "${LABEL:-.}/$MATCHPATH" && ln -f "$FILE" "${LABEL:-.}/$MATCH‐
382 PATH$LEAF"'
383 By default, no command is executed. Use --match-exec="" to remove a
385 command string which was set with an earlier use of this option.
386 --greedy-matching
388 This is the default. Imagine that your image contains a .tar
389 file which in turn contains another file x, and that you provide
390 both the .tar and the files inside it on the command line. When
391 jigdo-file scans the image, it encounters the beginning of the
392 .tar file, and then the file x.
393 At this point, a decision must be made: Should the smaller file
395 x be recorded as matched, or should it be ignored in favour of
396 the larger (and thus better) match of the .tar file? Unfortu‐
397 nately, at this point it is not clear whether there will actu‐
398 ally be a full match of the .tar, so by default, the program
399 prefers the small match.
400 --no-greedy-matching
402 In the case where a large partial match is present and a shorter
403 match has been confirmed, ignore the small match. (See the
404 option above.)
405 MAKE-IMAGE, MI
407 Reads `.template' and FILES, creates image (or `imagename.tmp'). Pro‐
408 vides a rudimentary way of reassembling images - jigdo is usually bet‐
409 ter suited for this task. However, in contrast to jigdo, no `.jigdo'
410 file is required.
411 If the image is to be written to a file (and not to standard output),
413 it is possible to create the image in several steps, with several invo‐
414 cations of `jigdo-file make-image', as follows: You first invoke jigdo-
415 file, specifying as many files as are available at this time. The pro‐
416 gram scans the files, and those that are contained in the image are
417 copied to a temporary file, whose name is formed by appending `.tmp' to
418 the image filename.
419 For all further files which could be parts of the image, you repeat
421 this process. As soon as all parts are present, the temporary file will
422 be truncated slightly (to delete some administrative data that jigdo-
423 file appends at the end) and renamed to the final image name. The pos‐
424 sibility of reassembling the image in several steps is especially use‐
425 ful for gathering files from removable media, e.g. several older CDs.
426 Scripts using make-image can detect whether image creation is complete
428 by checking the exit status: 0 signals successful creation, whereas 1
429 means that more files need to be supplied. Other errors result in an
430 exit status of 2 (`recoverable', e.g. file not found) or 3 (non-recov‐
431 erable, e.g. write error).
432 --check-files
434 This is the default. Whenever any part is copied to the image,
435 re-check its checksum against the checksum stored in the tem‐
436 plate. It is recommended that you leave this switched on, even
437 if it slows down image creation a bit.
438 --no-check-files
440 Do not check files' checksums when copying them to the image.
441 This can be safely used when no cache file is used (which means
442 that files will be written to the image immediately after being
443 scanned) or the whole image is checked later with the verify
444 command.
445 PRINT-MISSING, PM
447 Reads `.jigdo', `.template' and (if present) `imagename.tmp', outputs a
448 list of URIs still needed to completely reassemble the image.
449 Together with the make-image command, this provides most of the func‐
451 tionality of jigdo on the command line.
452 For each part that is not yet present in the temporary image file, the
454 file checksum is looked up in the `[Parts]' section of the jigdo file.
455 Any label in the corresponding entry is then expanded according to the
456 label definitions in the `[Servers]' section and printed on standard
457 output. jigdo allows you to specify several alternative locations for
458 each label in this section, but print-missing will only output the
459 first one for each missing part.
460 If the checksum cannot be found in the `[Parts]' section (this Should
462 Not Happen unless you deleted that section), a lookup is instead made
463 for `MD5Sum:<checksum>', just like with jigdo. (Thus, if you want to
464 get rid of the `[Parts]' section, you can do so if you rename each part
465 to its own checksum.)
466 --uri Label=http://some.server.org/
468 Override the entries in the `.jigdo' file for any label with a
469 URI of your choice. With the example above, a `[Parts]' entry of
470 `Label:some/filename' will cause the line
471 `http://some.server.org/some/filename' to be printed.
472 The supplied value is not quoted by the program; if it contains
474 characters such as space or any of the characters #"'\ then you
475 must quote it. (Under Unix, you may need to quote the value
476 twice to also protect it from the shell, e.g. \\\\ or '\\' to
477 get a single backslash in the URI.)
478 PRINT-MISSING-ALL, PMA
480 Just like print-missing, this command outputs a list of URIs still
481 needed to completely reassemble the image. However, all alternative
482 download locations are printed instead of just one. In the output, the
483 URIs for a file are separated from other files' URIs with blank lines.
484 The --uri option has the same effect as for print-missing.
485 VERIFY, VER
487 Reads image (presumably generated with make-image) and `.template',
488 checks for correct checksum of image.
489 The template data does not only contain checksums of the individual
491 parts, but also of the image as a whole. make-image already performs a
492 number of internal checks, but if you like, you can additionally check
493 the image with this command.
494 SCAN, SC
496 Reads all the FILES and enters them into the cache, unless they are
497 already cached. The --cache option must be present for this command.
498 --no-scan-whole-file
500 This is the default. This only causes the first --md5-block-size
501 bytes of each file to be read. If the cache is used later by
502 jigdo-file make-image, the rest of the file will be read once
503 these first bytes are recognized in the input image.
504 --scan-whole-file
506 Immediately read the entire file contents and store them in the
507 cache.
508 MD5SUM, MD5
510 Reads all the FILES and prints out MD5 checksums of their contents.
511 This command is quite similar to md5sum(1), except that the checksum is
512 output in the Base64-like encoding which is also used elsewhere by
513 jigdo-file.
514 The FILES arguments are processed in the same way as with the other
516 commands, which means that recursion automatically takes place for any
517 arguments that are directories, and that symbolic links are not listed
518 except when the file(s) they point to are not reachable directly.
519 In the checksum list printed on standard output, only the part of the
521 filename following any `//' (or `\.\' on Windows) is printed. Any
522 --cache will be used for querying files' MD5 checksums and/or writing
523 the checksums of scanned files.
524 LIST-TEMPLATE, LS
526 Reads a `.template' file and outputs low-level information about the
527 image and all parts contained in it, including offset, length and
528 checksum.
529 You can also use this command with temporary image files (by specifying
531 something like --template=imagename.tmp) - in that case, the output
532 also distinguishes between parts that have been written to the image
533 and parts that haven't.
534 The exact output format may change incompatibly between different jigdo
536 releases. The following different types of lines can be output. `have-
537 file' only occurs for `.tmp' files, indicating a file that has already
538 been successfully written to the temporary file:
539 in-template offset-in-image length
541 need-file offset-in-image length file-md5sum filestart-rsyncsum
542 have-file offset-in-image length file-md5sum filestart-rsyncsum
543 image-info image-length image-md5sum rsyncsum-size
544
546 Jigsaw Download was created with the format of ISO9660 CD images in
547 mind - however, the following also applies to many other filesystem
548 formats, as well as to `tar' archives and uncompressed `zip' archives.
549 A CD image contains both information for organizing the filesystem
550 (header with disc name etc., ISO9660 directory data, data of extensions
551 such as Joliet or RockRidge, zero padding) and the files contained on
552 the CD. An important property that jigdo relies on is that each file is
553 stored in one contiguous section of the image; it is not split into two
554 or more parts.
555 When jigdo-file is given a number of files that might be contained in
557 an image, it detects whether any of the files are present using a
558 `rolling checksum' inspired by the one used by rsync(1). The resulting
559 data is written to the `.template' file: If a section of the image
560 could not be matched (e.g. it was directory information), the data is
561 compressed and written directly to the template. However, if a matching
562 file was found, its data is omitted from the template. Instead, only a
563 reference (an MD5 checksum of the file) is inserted in the template.
564 Note that the template data only contains binary data, it does not con‐
566 tain any filenames or URIs, since it cannot be easily edited in case
567 any of these values need to be changed. All that information is stored
568 in the `.jigdo' file, a text file to which you can add URLs for your
569 server(s). The jigdo file provides a mapping for each MD5 checksum to
570 one or more alternative download locations for the corresponding part.
571 Apart from the mapping of MD5 sums to URIs, the jigdo file also con‐
573 tains an URI pointing to a download location for the template file.
574 This way, the jigdo download tool only needs to be given one URI (that
575 of the `.jigdo' file) to be able to download and reassemble the com‐
576 plete image.
577
579 The overall format of `.jigdo' files follows that of `.ini' files, as
580 also used by the Gnome and KDE projects for some data. The file is
581 organized into sections, each of which is preceded by a line reading
582 `[Sectionname]'. Within each section, lines have the form
583 `Label=Value'. Such lines are also called `entries' below. All `.jigdo'
584 files use UTF-8 as their character encoding.
585 Comments are introduced with the `#' character and extend to the end of
587 the line. Whitespace is ignored at line start and end as well as to the
588 left and right of section names and the `=' in entries. Furthermore,
589 the jigdo utilities split up the text of the entry value (i.e. the part
590 after the `=') into whitespace-separated words, much like the Unix
591 shell. Single '' and double "" quotes can be used to prevent that e.g.
592 URIs containing whitespace are split apart. Similarly, characters with
593 special meaning (the characters '"#\ and space/tab) must be quoted with
594 \ to appear in the value. As with the shell, there is a difference
595 between ' ' and " ": Within ' ', the characters "#\ and whitespace lose
596 their special meaning and become ordinary characters, whereas within
597 " ", only the characters '# and whitespace lose their special meaning -
598 in other words, backslash escapes still work inside " ", but not ' '.
599 `.jigdo' files can optionally be compressed with gzip(1). jigdo-file
601 always outputs uncompressed files, which you can compress yourself.
602 jigdo-lite supports single uncompressed and compressed files.
603 (Behaviour which may change in the future and which should not be
605 relied upon: jigdo additionally supports any number of concatenated
606 plaintext and gzipped parts in the files - for example, you can com‐
607 press a `.jigdo' file and then add a couple of lines of uncompressed
608 data to the end.)
609 In all cases, the `.gz' extension should be removed from the filename -
611 the tools will determine automatically from the file contents whether a
612 file is compressed or not.
613 Below is a description of the individual section names used by jigdo.
615 JIGDO SECTION
617 [Jigdo]
618 Version=1.1
619 Generator=jigdo-file/1.0.0
620 Information about the version of the jigdo file format used, and the
622 program that generated it. There should be one such section per
623 `.jigdo' file.
624 IMAGE SECTION
626 [Image]
627 Filename="filename for saving on user's disc"
628 Template="URI where to fetch template file"
629 Template-MD5Sum=OQ8riqT1BuyzsrT9964A7g
630 ShortInfo=single-line description of the image (200 characters max.)
631 Info=long description (5000 characters max.)
632 The value for the `Template' entry can be either an URL (absolute or
634 relative to the URL of the jigdo file) or a string of the form
635 `Label:pathname' (UNIMPLEMENTED), as described below.
636 The `Template-MD5Sum' entry is added by jigdo-file and specifies the
638 MD5 checksum of the generated `.template' file. It is used by jigdo to
639 detect cases where the downloaded template data is corrupted or belongs
640 to a different image.
641 Unlike other entry values, the values of the `ShortInfo' and `Info'
643 entries are not split up into words, instead all quoting is preserved.
644 The value of the `Info' entry is special in that jigdo(1) can option‐
646 ally parse XML markup it contains. If the markup has errors such as
647 unbalanced/unsupported tags, the string is displayed literally, without
648 XML parsing. Supported tags are <b></b> (bold), <i></i> (italic),
649 <tt></tt> (typewriter font), <u></u> (underline), <big></big> (larger
650 font), <small></small> (smaller font) and <br/> (linebreak). Supported
651 entities include < (`<'), > (`>') and & (`&'). Note that the
652 whole `Info' entry must be on one line in the jigdo file.
653 This section may occur multiple times, but all except the first one
655 will be ignored. This is useful e.g. when creating a `.jigdo' file for
656 a DVD image when you already have `.jigdo' files for CDs with the same
657 content: You can simply `[Include]' (see below) the CDs' jigdo files at
658 the end of the DVD jigdo file, after its `[Image]' section.
659 PARTS SECTION
661 [Parts]
662 xJNkjrq8NYMraeGavUpllw=LabelA:part0
663 GoTResP2EC6Lb_2wTsqOoQ=LabelA:part1
664 kyfebwu6clbYqqWUdFIyaw=LabelB:some/path/part2
665 -J9UAimo0Bqg9c0oOXI1mQ=http://some.where.com/part3
666 All lines in the section, which provides the mapping from MD5 checksums
668 to URIs, have the same format: On the left side of the `=' the checksum
669 (encoded with a Base64-like encoding) is given, and on the right a
670 string corresponding to the part with this checksum; either a complete
671 URI or a string of the form `Label:pathname', which is expanded into
672 one or more URIs by looking up the definition(s) for the Label in the
673 `[Servers]' section.
674 In case a particular MD5 checksum cannot be found in any `[Parts]' sec‐
676 tion by jigdo, the program will perform a lookup for `MD5Sum:<check‐
677 sum>', e.g. for `MD5Sum:xJNkjrq8NYMraeGavUpllw' if you deleted the line
678 for `part0' above.
679 A checksum appearing multiple times in this section indicates alterna‐
681 tive download locations for the part.
682 There may be any number of `[Parts]' sections in the file; they are all
684 considered when looking up MD5 checksums.
685 jigdo-file always puts the `[Parts]' section at the end of the file,
687 and it even rearranges any file specified with --merge to have only one
688 such section at the end. This is done to allow jigdo to display the
689 information from the `[Image]' section while the rest of that file is
690 still being downloaded.
691 SERVERS SECTION
693 [Servers]
694 LabelA=http://myserver.org/
695 LabelA=ftp://mirror.myserver.org/
696 LabelB=LabelC:subdirectory/
697 LabelC=http://some.where.com/jigdo/
698 All lines in the section, which provides the mapping from server labels
700 to server locations, have the same format: On the left side of the `='
701 the label name is given, and on the right the value to expand the label
702 name to.
703 A label name appearing multiple times in this section indicates alter‐
705 native download locations for the parts that use the label in the
706 `[Parts]' section. This notation makes it very easy to add mirrors to
707 the jigdo file.
708 As shown by the example above, the label values may themselves refer‐
710 ence other labels. In this case, the entry `LabelB:some/path/part2' in
711 the `[Parts]' section will expand to `http://some.where.com/jigdo/sub‐
712 directory/some/path/part2'. Loops in the label definitions result in
713 undefined behaviour and must be avoided.
714 There may be any number of `[Servers]' sections in the file; they are
716 all considered when looking up labels. Either of `[Parts]' or
717 `[Servers]', but not both, can be omitted from the jigdo file.
718 COMMENT SECTION
720 [Comment]
721 Any text, except that lines must not begin with `['.
722 All text following a `[Comment]' or `[comment]' line is ignored, up to
724 the next line with a section label.
725 INCLUDE DIRECTIVE
727 [Include http://some.url/file.jigdo]
728 Lines of this form cause the content of the specified jigdo file to be
730 downloaded and parsed just like the main jigdo file. The effect will be
731 the same as copying the included file's contents into the file which
732 contains the include directive. (Exception: Any relative URLs are
733 always resolved using the URL of the `.jigdo' file that contains that
734 relative URL.)
735 The URL argument can be an absolute or relative URL. Relative URLs are
737 assumed to be relative to the URL of the jigdo file which contains the
738 include directive. Includes can be nested, but it is an error to create
739 a loop of include directives. It is not possible to use URLs of the
740 form `Label:pathname'.
741 The URL cannot be quoted with "". Any `]' characters in the argument
743 must be escaped as `%5D', and any spaces as `%20'.
744 Include directives are only supported by jigdo, they are ignored by
746 jigdo-lite.
747 An include directive terminates any previous section, but it does not
749 start a new one. In other words, a new section must always be started
750 after the include line, jigdo does not allow normal entries to appear
751 below the `[Include]'.
752
754 Any file specified with the --cache option is used to store information
755 about the FILES presented to jigdo-file. When querying the cache, a
756 file is considered unchanged (and the cached data is used) only if
757 filename, file size and last modification time (mtime) match exactly.
758 For the filename match, not the entire file name is used, but only the
759 part following any `//', so that any changes to the part before the
760 `//' will not invalidate the cache.
761 Old cache entries are removed from the cache if they have not been read
763 from or written to for the amount of time specified with --cache-
764 expiry. Entries are not immediately removed from the cache if the file
765 they refer to no longer exists - this makes it possible to cache infor‐
766 mation about files on removable media.
767 Cache expiry only takes place after jigdo-file has done its main work -
769 if any old entries are accessed before expiry takes place, they will be
770 kept. For example, if the program is run using the default expiry time
771 of 30 days, but accesses a cache file with entries generated 2 months
772 ago, then entries in that cache will be considered, and only those
773 cache entries that were not needed during the program run will be
774 expired.
775 Due to a peculiarity of the underlying database library (libdb3), cache
777 files never shrink, they only grow. If a large number of entries was
778 expired from your cache file and you want it to shrink, you can either
779 just delete it (of course then everything will have to be regenerated)
780 or use the utilities accompanying libdb3 to dump and restore the data‐
781 base, with a command like `db3_dump old-cache.db | db3_load new-
782 cache.db'. For Debian, these programs are supplied in the package
783 `libdb3-util'.
784 If a different --md5-block-size is specified, the entire file needs to
786 be re-read to update its cache entry. If a different --min-length is
787 specified, only the first `md5-block-size' bytes of the file need to be
788 re-read.
789
791 PREPARING YOUR CD IMAGE FOR DISTRIBUTION
792 You have created a CD image `image.iso' from some of the files stored
793 in the directory `/home/ftp' on your harddisc, which is also available
794 online as `ftp://mysite.org'. As you don't want to waste space by
795 effectively hosting the same data twice (once as files on the FTP
796 server, once inside the image), and you are fed up with users' down‐
797 loads aborting after 200MB and their restarting the download dozens of
798 times, you decide to use jigdo. How do you prepare the image for down‐
799 load?
800 In fact, only one command is necessary:
802 jigdo-file make-template --image=image.iso
805 --jigdo=/home/ftp/image.jigdo --template=/home/ftp/image.tem‐
806 plate /home/ftp// --label Mysite=/home/ftp --uri
807 Mysite=ftp://mysite.org/
808 People can now point jigdo at `ftp://mysite.org/image.jigdo' to down‐
810 load your image. The template file needs to be accessible as
811 `ftp://mysite.org/image.template'.
812 Note that nothing prevents you from doing the same for an FTP server
814 that isn't administrated by you - in that case, you only need to host
815 the `.jigdo' and `.template' files on your own server/homepage.
816 PREPARING AN ARBITRARY LARGE FILE FOR DISTRIBUTION
818 We assume that you have a large file that is not a filesystem, e.g.
819 `movie.mpeg'. Because of space problems, you want to distribute the
820 data on two servers.
821 In this case, the parts of the image need to be generated artificially
823 with the split command. For example, to create chunks of 4MB each, use
824 `split -b 4m movie.mpeg part'. Copy the resulting files `partXX' into
825 two directories `1' and `2' that you create, according to how you want
826 the files distributed between the servers. Next, create the jigdo and
827 template files with `jigdo-file make-template --image=movie.mpeg 1//
828 2//'. You will need to edit the `.jigdo' file and provide the right
829 URIs for the two servers that you are going to upload the `partXX'
830 files to.
831 CUSTOMIZED VERSIONS OF IMAGES
833 Because it is possible to assign a different URI for each part of an
834 image if necessary, jigdo is very flexible. Only one example is the
835 possibility of customized versions of images: Suppose that someone is
836 distributing a CD image, and that you want to make a few small changes
837 to it and redistribute your own version. You download the `offi‐
838 cial.iso' CD image with jigdo (passing it the URL of `official.jigdo'),
839 write it to CD-R, make your changes (say, adding files from the
840 `myfiles' directory on your harddisc) and produce your own version,
841 `myversion.iso'. Next, you instruct jigdo-file to create the jigdo and
842 template files for your modified image, using the command
843 jigdo-file make-template --image=myversion.iso /mnt/cdrom/
846 myfiles// --label My=myfiles/ --uri My=http://my.homepage.net/
847 --merge=official.jigdo
848 while `official.iso' is mounted under `/mnt/cdrom'. By using --merge,
849 you have told jigdo-file to take the contents of `official.jigdo', add
850 to it a new `[Image]' section for `myversion.iso' and write the result‐
851 ing jigdo file to `myversion.jigdo' - so now `myversion.jigdo' offers
852 two images for download, the original version and your modified ver‐
853 sion. (If you do not want it to offer the official version, edit it and
854 remove the `[Image]' section that lists `official.iso'.)
855 Now you can upload the `.jigdo' file, the `.template' file and also the
857 files in `myfiles' to `http://my.homepage.net/'. Thus, for people to
858 download your modified image, you do not need to upload the complete
859 image contents to your web space, but only the changes you made!
860 (In case you only made very few changes, you could also omit the
862 `myfiles' parameter in the command above, then all your changes end up
863 in the new template file.)
864 COMBINING MANY JIGDO-MANAGED IMAGES INTO ONE
866 It is also no problem to combine data from several sources that use
867 jigdo. For example, if of five different and unrelated servers each one
868 distributes a different CD image via jigdo, you can create a customized
869 DVD image that contains the data from all these CDs. When people use
870 jigdo to download your image, the individual files on the DVD are
871 fetched from the same sources as the original CDs.
872 Consequently, even though you will be distributing a 3.2GB file via
874 your web space, the actual amount of data that is stored on your server
875 will only be in the order of several MBs.
876
878 For certain contents of one of the input files, most notably a sequence
879 of zero bytes longer than --min-length at the start of the file and an
880 area of zeros preceding the file data in the image, jigdo-file make-
881 template may fail to find the file in the image. Unfortunately, this
882 restriction cannot be avoided because the program could become very
883 slow otherwise. If you use the --debug option, all instances of jigdo-
884 file discarding possible matches are indicated by lines containing the
885 word `DROPPED'.
886 In fact, not only all-zeroes files trigger this behaviour, but also
888 files which contain at their start a long sequence of short identical
889 strings. For example, both a file containing only `a' characters and
890 one containing `abcabcabcabc...' are problematic.
891
893 jigdo(1) (NOT YET IMPLEMENTED), jigdo-lite(1), jigdo-mirror(1),
894 split(1) (or `info split'), find(1) (or `info find'), mkisofs(1),
895 md5sum(1)
896
898 Jigsaw Download <URL:http://atterer.net/jigdo/> was written by Richard
899 Atterer <jigdo atterer.net>, to make downloading of CD ROM images for
900 the Debian Linux distribution more convenient.
901 19 May 2006 JIGDO-FILE(1)