1XZ(1) XZ Utils XZ(1)
2
6 xz, unxz, xzcat, lzma, unlzma, lzcat - Compress or decompress .xz and
7 .lzma files
8
10 xz [option...] [file...]
11
13 unxz is equivalent to xz --decompress.
14 xzcat is equivalent to xz --decompress --stdout.
15 lzma is equivalent to xz --format=lzma.
16 unlzma is equivalent to xz --format=lzma --decompress.
17 lzcat is equivalent to xz --format=lzma --decompress --stdout.
18 When writing scripts that need to decompress files, it is recommended
20 to always use the name xz with appropriate arguments (xz -d or xz -dc)
21 instead of the names unxz and xzcat.
22
24 xz is a general-purpose data compression tool with command line syntax
25 similar to gzip(1) and bzip2(1). The native file format is the .xz
26 format, but the legacy .lzma format used by LZMA Utils and raw com‐
27 pressed streams with no container format headers are also supported.
28 xz compresses or decompresses each file according to the selected oper‐
30 ation mode. If no files are given or file is -, xz reads from standard
31 input and writes the processed data to standard output. xz will refuse
32 (display an error and skip the file) to write compressed data to stan‐
33 dard output if it is a terminal. Similarly, xz will refuse to read
34 compressed data from standard input if it is a terminal.
35 Unless --stdout is specified, files other than - are written to a new
37 file whose name is derived from the source file name:
38 · When compressing, the suffix of the target file format (.xz or
40 .lzma) is appended to the source filename to get the target file‐
41 name.
42 · When decompressing, the .xz or .lzma suffix is removed from the
44 filename to get the target filename. xz also recognizes the suf‐
45 fixes .txz and .tlz, and replaces them with the .tar suffix.
46 If the target file already exists, an error is displayed and the file
48 is skipped.
49 Unless writing to standard output, xz will display a warning and skip
51 the file if any of the following applies:
52 · File is not a regular file. Symbolic links are not followed, and
54 thus they are not considered to be regular files.
55 · File has more than one hard link.
57 · File has setuid, setgid, or sticky bit set.
59 · The operation mode is set to compress and the file already has a
61 suffix of the target file format (.xz or .txz when compressing to
62 the .xz format, and .lzma or .tlz when compressing to the .lzma for‐
63 mat).
64 · The operation mode is set to decompress and the file doesn't have a
66 suffix of any of the supported file formats (.xz, .txz, .lzma, or
67 .tlz).
68 After successfully compressing or decompressing the file, xz copies the
70 owner, group, permissions, access time, and modification time from the
71 source file to the target file. If copying the group fails, the per‐
72 missions are modified so that the target file doesn't become accessible
73 to users who didn't have permission to access the source file. xz
74 doesn't support copying other metadata like access control lists or
75 extended attributes yet.
76 Once the target file has been successfully closed, the source file is
78 removed unless --keep was specified. The source file is never removed
79 if the output is written to standard output.
80 Sending SIGINFO or SIGUSR1 to the xz process makes it print progress
82 information to standard error. This has only limited use since when
83 standard error is a terminal, using --verbose will display an automati‐
84 cally updating progress indicator.
85 Memory usage
87 The memory usage of xz varies from a few hundred kilobytes to several
88 gigabytes depending on the compression settings. The settings used
89 when compressing a file determine the memory requirements of the decom‐
90 pressor. Typically the decompressor needs 5 % to 20 % of the amount of
91 memory that the compressor needed when creating the file. For example,
92 decompressing a file created with xz -9 currently requires 65 MiB of
93 memory. Still, it is possible to have .xz files that require several
94 gigabytes of memory to decompress.
95 Especially users of older systems may find the possibility of very
97 large memory usage annoying. To prevent uncomfortable surprises, xz
98 has a built-in memory usage limiter, which is disabled by default.
99 While some operating systems provide ways to limit the memory usage of
100 processes, relying on it wasn't deemed to be flexible enough (e.g.
101 using ulimit(1) to limit virtual memory tends to cripple mmap(2)).
102 The memory usage limiter can be enabled with the command line option
104 --memlimit=limit. Often it is more convenient to enable the limiter by
105 default by setting the environment variable XZ_DEFAULTS, e.g.
106 XZ_DEFAULTS=--memlimit=150MiB. It is possible to set the limits sepa‐
107 rately for compression and decompression by using --memlimit-com‐
108 press=limit and --memlimit-decompress=limit. Using these two options
109 outside XZ_DEFAULTS is rarely useful because a single run of xz cannot
110 do both compression and decompression and --memlimit=limit (or -M
111 limit) is shorter to type on the command line.
112 If the specified memory usage limit is exceeded when decompressing, xz
114 will display an error and decompressing the file will fail. If the
115 limit is exceeded when compressing, xz will try to scale the settings
116 down so that the limit is no longer exceeded (except when using --for‐
117 mat=raw or --no-adjust). This way the operation won't fail unless the
118 limit is very small. The scaling of the settings is done in steps that
119 don't match the compression level presets, e.g. if the limit is only
120 slightly less than the amount required for xz -9, the settings will be
121 scaled down only a little, not all the way down to xz -8.
122 Concatenation and padding with .xz files
124 It is possible to concatenate .xz files as is. xz will decompress such
125 files as if they were a single .xz file.
126 It is possible to insert padding between the concatenated parts or
128 after the last part. The padding must consist of null bytes and the
129 size of the padding must be a multiple of four bytes. This can be use‐
130 ful e.g. if the .xz file is stored on a medium that measures file sizes
131 in 512-byte blocks.
132 Concatenation and padding are not allowed with .lzma files or raw
134 streams.
135
137 Integer suffixes and special values
138 In most places where an integer argument is expected, an optional suf‐
139 fix is supported to easily indicate large integers. There must be no
140 space between the integer and the suffix.
141 KiB Multiply the integer by 1,024 (2^10). Ki, k, kB, K, and KB are
143 accepted as synonyms for KiB.
144 MiB Multiply the integer by 1,048,576 (2^20). Mi, m, M, and MB are
146 accepted as synonyms for MiB.
147 GiB Multiply the integer by 1,073,741,824 (2^30). Gi, g, G, and GB
149 are accepted as synonyms for GiB.
150 The special value max can be used to indicate the maximum integer value
152 supported by the option.
153 Operation mode
155 If multiple operation mode options are given, the last one takes
156 effect.
157 -z, --compress
159 Compress. This is the default operation mode when no operation
160 mode option is specified and no other operation mode is implied
161 from the command name (for example, unxz implies --decompress).
162 -d, --decompress, --uncompress
164 Decompress.
165 -t, --test
167 Test the integrity of compressed files. This option is equiva‐
168 lent to --decompress --stdout except that the decompressed data
169 is discarded instead of being written to standard output. No
170 files are created or removed.
171 -l, --list
173 Print information about compressed files. No uncompressed out‐
174 put is produced, and no files are created or removed. In list
175 mode, the program cannot read the compressed data from standard
176 input or from other unseekable sources.
177 The default listing shows basic information about files, one
179 file per line. To get more detailed information, use also the
180 --verbose option. For even more information, use --verbose
181 twice, but note that this may be slow, because getting all the
182 extra information requires many seeks. The width of verbose
183 output exceeds 80 characters, so piping the output to e.g.
184 less -S may be convenient if the terminal isn't wide enough.
185 The exact output may vary between xz versions and different
187 locales. For machine-readable output, --robot --list should be
188 used.
189 Operation modifiers
191 -k, --keep
192 Don't delete the input files.
193 -f, --force
195 This option has several effects:
196 · If the target file already exists, delete it before compress‐
198 ing or decompressing.
199 · Compress or decompress even if the input is a symbolic link
201 to a regular file, has more than one hard link, or has the
202 setuid, setgid, or sticky bit set. The setuid, setgid, and
203 sticky bits are not copied to the target file.
204 · When used with --decompress --stdout and xz cannot recognize
206 the type of the source file, copy the source file as is to
207 standard output. This allows xzcat --force to be used like
208 cat(1) for files that have not been compressed with xz. Note
209 that in future, xz might support new compressed file formats,
210 which may make xz decompress more types of files instead of
211 copying them as is to standard output. --format=format can
212 be used to restrict xz to decompress only a single file for‐
213 mat.
214 -c, --stdout, --to-stdout
216 Write the compressed or decompressed data to standard output
217 instead of a file. This implies --keep.
218 --single-stream
220 Decompress only the first .xz stream, and silently ignore possi‐
221 ble remaining input data following the stream. Normally such
222 trailing garbage makes xz display an error.
223 xz never decompresses more than one stream from .lzma files or
225 raw streams, but this option still makes xz ignore the possible
226 trailing data after the .lzma file or raw stream.
227 This option has no effect if the operation mode is not --decom‐
229 press or --test.
230 --no-sparse
232 Disable creation of sparse files. By default, if decompressing
233 into a regular file, xz tries to make the file sparse if the
234 decompressed data contains long sequences of binary zeros. It
235 also works when writing to standard output as long as standard
236 output is connected to a regular file and certain additional
237 conditions are met to make it safe. Creating sparse files may
238 save disk space and speed up the decompression by reducing the
239 amount of disk I/O.
240 -S .suf, --suffix=.suf
242 When compressing, use .suf as the suffix for the target file
243 instead of .xz or .lzma. If not writing to standard output and
244 the source file already has the suffix .suf, a warning is dis‐
245 played and the file is skipped.
246 When decompressing, recognize files with the suffix .suf in
248 addition to files with the .xz, .txz, .lzma, or .tlz suffix. If
249 the source file has the suffix .suf, the suffix is removed to
250 get the target filename.
251 When compressing or decompressing raw streams (--format=raw),
253 the suffix must always be specified unless writing to standard
254 output, because there is no default suffix for raw streams.
255 --files[=file]
257 Read the filenames to process from file; if file is omitted,
258 filenames are read from standard input. Filenames must be ter‐
259 minated with the newline character. A dash (-) is taken as a
260 regular filename; it doesn't mean standard input. If filenames
261 are given also as command line arguments, they are processed
262 before the filenames read from file.
263 --files0[=file]
265 This is identical to --files[=file] except that each filename
266 must be terminated with the null character.
267 Basic file format and compression options
269 -F format, --format=format
270 Specify the file format to compress or decompress:
271 auto This is the default. When compressing, auto is equiva‐
273 lent to xz. When decompressing, the format of the input
274 file is automatically detected. Note that raw streams
275 (created with --format=raw) cannot be auto-detected.
276 xz Compress to the .xz file format, or accept only .xz files
278 when decompressing.
279 lzma, alone
281 Compress to the legacy .lzma file format, or accept only
282 .lzma files when decompressing. The alternative name
283 alone is provided for backwards compatibility with LZMA
284 Utils.
285 raw Compress or uncompress a raw stream (no headers). This
287 is meant for advanced users only. To decode raw streams,
288 you need use --format=raw and explicitly specify the fil‐
289 ter chain, which normally would have been stored in the
290 container headers.
291 -C check, --check=check
293 Specify the type of the integrity check. The check is calcu‐
294 lated from the uncompressed data and stored in the .xz file.
295 This option has an effect only when compressing into the .xz
296 format; the .lzma format doesn't support integrity checks. The
297 integrity check (if any) is verified when the .xz file is decom‐
298 pressed.
299 Supported check types:
301 none Don't calculate an integrity check at all. This is usu‐
303 ally a bad idea. This can be useful when integrity of
304 the data is verified by other means anyway.
305 crc32 Calculate CRC32 using the polynomial from IEEE-802.3
307 (Ethernet).
308 crc64 Calculate CRC64 using the polynomial from ECMA-182. This
310 is the default, since it is slightly better than CRC32 at
311 detecting damaged files and the speed difference is neg‐
312 ligible.
313 sha256 Calculate SHA-256. This is somewhat slower than CRC32
315 and CRC64.
316 Integrity of the .xz headers is always verified with CRC32. It
318 is not possible to change or disable it.
319 --ignore-check
321 Don't verify the integrity check of the compressed data when
322 decompressing. The CRC32 values in the .xz headers will still
323 be verified normally.
324 Do not use this option unless you know what you are doing. Pos‐
326 sible reasons to use this option:
327 · Trying to recover data from a corrupt .xz file.
329 · Speeding up decompression. This matters mostly with SHA-256
331 or with files that have compressed extremely well. It's rec‐
332 ommended to not use this option for this purpose unless the
333 file integrity is verified externally in some other way.
334 -0 ... -9
336 Select a compression preset level. The default is -6. If mul‐
337 tiple preset levels are specified, the last one takes effect.
338 If a custom filter chain was already specified, setting a com‐
339 pression preset level clears the custom filter chain.
340 The differences between the presets are more significant than
342 with gzip(1) and bzip2(1). The selected compression settings
343 determine the memory requirements of the decompressor, thus
344 using a too high preset level might make it painful to decom‐
345 press the file on an old system with little RAM. Specifically,
346 it's not a good idea to blindly use -9 for everything like it
347 often is with gzip(1) and bzip2(1).
348 -0 ... -3
350 These are somewhat fast presets. -0 is sometimes faster
351 than gzip -9 while compressing much better. The higher
352 ones often have speed comparable to bzip2(1) with compa‐
353 rable or better compression ratio, although the results
354 depend a lot on the type of data being compressed.
355 -4 ... -6
357 Good to very good compression while keeping decompressor
358 memory usage reasonable even for old systems. -6 is the
359 default, which is usually a good choice e.g. for dis‐
360 tributing files that need to be decompressible even on
361 systems with only 16 MiB RAM. (-5e or -6e may be worth
362 considering too. See --extreme.)
363 -7 ... -9
365 These are like -6 but with higher compressor and decom‐
366 pressor memory requirements. These are useful only when
367 compressing files bigger than 8 MiB, 16 MiB, and 32 MiB,
368 respectively.
369 On the same hardware, the decompression speed is approximately a
371 constant number of bytes of compressed data per second. In
372 other words, the better the compression, the faster the decom‐
373 pression will usually be. This also means that the amount of
374 uncompressed output produced per second can vary a lot.
375 The following table summarises the features of the presets:
377 Preset DictSize CompCPU CompMem DecMem
379 -0 256 KiB 0 3 MiB 1 MiB
380 -1 1 MiB 1 9 MiB 2 MiB
381 -2 2 MiB 2 17 MiB 3 MiB
382 -3 4 MiB 3 32 MiB 5 MiB
383 -4 4 MiB 4 48 MiB 5 MiB
384 -5 8 MiB 5 94 MiB 9 MiB
385 -6 8 MiB 6 94 MiB 9 MiB
386 -7 16 MiB 6 186 MiB 17 MiB
387 -8 32 MiB 6 370 MiB 33 MiB
388 -9 64 MiB 6 674 MiB 65 MiB
389 Column descriptions:
391 · DictSize is the LZMA2 dictionary size. It is waste of memory
393 to use a dictionary bigger than the size of the uncompressed
394 file. This is why it is good to avoid using the presets -7
395 ... -9 when there's no real need for them. At -6 and lower,
396 the amount of memory wasted is usually low enough to not mat‐
397 ter.
398 · CompCPU is a simplified representation of the LZMA2 settings
400 that affect compression speed. The dictionary size affects
401 speed too, so while CompCPU is the same for levels -6 ... -9,
402 higher levels still tend to be a little slower. To get even
403 slower and thus possibly better compression, see --extreme.
404 · CompMem contains the compressor memory requirements in the
406 single-threaded mode. It may vary slightly between xz ver‐
407 sions. Memory requirements of some of the future multi‐
408 threaded modes may be dramatically higher than that of the
409 single-threaded mode.
410 · DecMem contains the decompressor memory requirements. That
412 is, the compression settings determine the memory require‐
413 ments of the decompressor. The exact decompressor memory
414 usage is slightly more than the LZMA2 dictionary size, but
415 the values in the table have been rounded up to the next full
416 MiB.
417 -e, --extreme
419 Use a slower variant of the selected compression preset level
420 (-0 ... -9) to hopefully get a little bit better compression
421 ratio, but with bad luck this can also make it worse. Decom‐
422 pressor memory usage is not affected, but compressor memory
423 usage increases a little at preset levels -0 ... -3.
424 Since there are two presets with dictionary sizes 4 MiB and
426 8 MiB, the presets -3e and -5e use slightly faster settings
427 (lower CompCPU) than -4e and -6e, respectively. That way no two
428 presets are identical.
429 Preset DictSize CompCPU CompMem DecMem
431 -0e 256 KiB 8 4 MiB 1 MiB
432 -1e 1 MiB 8 13 MiB 2 MiB
433 -2e 2 MiB 8 25 MiB 3 MiB
434 -3e 4 MiB 7 48 MiB 5 MiB
435 -4e 4 MiB 8 48 MiB 5 MiB
436 -5e 8 MiB 7 94 MiB 9 MiB
437 -6e 8 MiB 8 94 MiB 9 MiB
438 -7e 16 MiB 8 186 MiB 17 MiB
439 -8e 32 MiB 8 370 MiB 33 MiB
440 -9e 64 MiB 8 674 MiB 65 MiB
441 For example, there are a total of four presets that use 8 MiB
443 dictionary, whose order from the fastest to the slowest is -5,
444 -6, -5e, and -6e.
445 --fast
447 --best These are somewhat misleading aliases for -0 and -9, respec‐
448 tively. These are provided only for backwards compatibility
449 with LZMA Utils. Avoid using these options.
450 --block-size=size
452 When compressing to the .xz format, split the input data into
453 blocks of size bytes. The blocks are compressed independently
454 from each other, which helps with multi-threading and makes lim‐
455 ited random-access decompression possible. This option is typi‐
456 cally used to override the default block size in multi-threaded
457 mode, but this option can be used in single-threaded mode too.
458 In multi-threaded mode about three times size bytes will be
460 allocated in each thread for buffering input and output. The
461 default size is three times the LZMA2 dictionary size or 1 MiB,
462 whichever is more. Typically a good value is 2-4 times the size
463 of the LZMA2 dictionary or at least 1 MiB. Using size less than
464 the LZMA2 dictionary size is waste of RAM because then the LZMA2
465 dictionary buffer will never get fully used. The sizes of the
466 blocks are stored in the block headers, which a future version
467 of xz will use for multi-threaded decompression.
468 In single-threaded mode no block splitting is done by default.
470 Setting this option doesn't affect memory usage. No size infor‐
471 mation is stored in block headers, thus files created in single-
472 threaded mode won't be identical to files created in multi-
473 threaded mode. The lack of size information also means that a
474 future version of xz won't be able decompress the files in
475 multi-threaded mode.
476 --block-list=sizes
478 When compressing to the .xz format, start a new block after the
479 given intervals of uncompressed data.
480 The uncompressed sizes of the blocks are specified as a comma-
482 separated list. Omitting a size (two or more consecutive com‐
483 mas) is a shorthand to use the size of the previous block.
484 If the input file is bigger than the sum of sizes, the last
486 value in sizes is repeated until the end of the file. A special
487 value of 0 may be used as the last value to indicate that the
488 rest of the file should be encoded as a single block.
489 If one specifies sizes that exceed the encoder's block size
491 (either the default value in threaded mode or the value speci‐
492 fied with --block-size=size), the encoder will create additional
493 blocks while keeping the boundaries specified in sizes. For
494 example, if one specifies --block-size=10MiB
495 --block-list=5MiB,10MiB,8MiB,12MiB,24MiB and the input file is
496 80 MiB, one will get 11 blocks: 5, 10, 8, 10, 2, 10, 10, 4, 10,
497 10, and 1 MiB.
498 In multi-threaded mode the sizes of the blocks are stored in the
500 block headers. This isn't done in single-threaded mode, so the
501 encoded output won't be identical to that of the multi-threaded
502 mode.
503 --flush-timeout=timeout
505 When compressing, if more than timeout milliseconds (a positive
506 integer) has passed since the previous flush and reading more
507 input would block, all the pending input data is flushed from
508 the encoder and made available in the output stream. This can
509 be useful if xz is used to compress data that is streamed over a
510 network. Small timeout values make the data available at the
511 receiving end with a small delay, but large timeout values give
512 better compression ratio.
513 This feature is disabled by default. If this option is speci‐
515 fied more than once, the last one takes effect. The special
516 timeout value of 0 can be used to explicitly disable this fea‐
517 ture.
518 This feature is not available on non-POSIX systems.
520 This feature is still experimental. Currently xz is unsuitable
522 for decompressing the stream in real time due to how xz does
523 buffering.
524 --memlimit-compress=limit
526 Set a memory usage limit for compression. If this option is
527 specified multiple times, the last one takes effect.
528 If the compression settings exceed the limit, xz will adjust the
530 settings downwards so that the limit is no longer exceeded and
531 display a notice that automatic adjustment was done. Such
532 adjustments are not made when compressing with --format=raw or
533 if --no-adjust has been specified. In those cases, an error is
534 displayed and xz will exit with exit status 1.
535 The limit can be specified in multiple ways:
537 · The limit can be an absolute value in bytes. Using an inte‐
539 ger suffix like MiB can be useful. Example: --memlimit-com‐
540 press=80MiB
541 · The limit can be specified as a percentage of total physical
543 memory (RAM). This can be useful especially when setting the
544 XZ_DEFAULTS environment variable in a shell initialization
545 script that is shared between different computers. That way
546 the limit is automatically bigger on systems with more mem‐
547 ory. Example: --memlimit-compress=70%
548 · The limit can be reset back to its default value by setting
550 it to 0. This is currently equivalent to setting the limit
551 to max (no memory usage limit). Once multithreading support
552 has been implemented, there may be a difference between 0 and
553 max for the multithreaded case, so it is recommended to use 0
554 instead of max until the details have been decided.
555 See also the section Memory usage.
557 --memlimit-decompress=limit
559 Set a memory usage limit for decompression. This also affects
560 the --list mode. If the operation is not possible without
561 exceeding the limit, xz will display an error and decompressing
562 the file will fail. See --memlimit-compress=limit for possible
563 ways to specify the limit.
564 -M limit, --memlimit=limit, (old alias --memory=limit)
566 This is equivalent to specifying --memlimit-compress=limit
567 --memlimit-decompress=limit.
568 --no-adjust
570 Display an error and exit if the compression settings exceed the
571 memory usage limit. The default is to adjust the settings down‐
572 wards so that the memory usage limit is not exceeded. Automatic
573 adjusting is always disabled when creating raw streams (--for‐
574 mat=raw).
575 -T threads, --threads=threads
577 Specify the number of worker threads to use. Setting threads to
578 a special value 0 makes xz use as many threads as there are CPU
579 cores on the system. The actual number of threads can be less
580 than threads if the input file is not big enough for threading
581 with the given settings or if using more threads would exceed
582 the memory usage limit.
583 Currently the only threading method is to split the input into
585 blocks and compress them independently from each other. The
586 default block size depends on the compression level and can be
587 overriden with the --block-size=size option.
588 Threaded decompression hasn't been implemented yet. It will
590 only work on files that contain multiple blocks with size infor‐
591 mation in block headers. All files compressed in multi-threaded
592 mode meet this condition, but files compressed in single-
593 threaded mode don't even if --block-size=size is used.
594 Custom compressor filter chains
596 A custom filter chain allows specifying the compression settings in
597 detail instead of relying on the settings associated to the presets.
598 When a custom filter chain is specified, preset options (-0 ... -9 and
599 --extreme) earlier on the command line are forgotten. If a preset
600 option is specified after one or more custom filter chain options, the
601 new preset takes effect and the custom filter chain options specified
602 earlier are forgotten.
603 A filter chain is comparable to piping on the command line. When com‐
605 pressing, the uncompressed input goes to the first filter, whose output
606 goes to the next filter (if any). The output of the last filter gets
607 written to the compressed file. The maximum number of filters in the
608 chain is four, but typically a filter chain has only one or two fil‐
609 ters.
610 Many filters have limitations on where they can be in the filter chain:
612 some filters can work only as the last filter in the chain, some only
613 as a non-last filter, and some work in any position in the chain.
614 Depending on the filter, this limitation is either inherent to the fil‐
615 ter design or exists to prevent security issues.
616 A custom filter chain is specified by using one or more filter options
618 in the order they are wanted in the filter chain. That is, the order
619 of filter options is significant! When decoding raw streams (--for‐
620 mat=raw), the filter chain is specified in the same order as it was
621 specified when compressing.
622 Filters take filter-specific options as a comma-separated list. Extra
624 commas in options are ignored. Every option has a default value, so
625 you need to specify only those you want to change.
626 To see the whole filter chain and options, use xz -vv (that is, use
628 --verbose twice). This works also for viewing the filter chain options
629 used by presets.
630 --lzma1[=options]
632 --lzma2[=options]
633 Add LZMA1 or LZMA2 filter to the filter chain. These filters
634 can be used only as the last filter in the chain.
635 LZMA1 is a legacy filter, which is supported almost solely due
637 to the legacy .lzma file format, which supports only LZMA1.
638 LZMA2 is an updated version of LZMA1 to fix some practical
639 issues of LZMA1. The .xz format uses LZMA2 and doesn't support
640 LZMA1 at all. Compression speed and ratios of LZMA1 and LZMA2
641 are practically the same.
642 LZMA1 and LZMA2 share the same set of options:
644 preset=preset
646 Reset all LZMA1 or LZMA2 options to preset. Preset con‐
647 sist of an integer, which may be followed by single-let‐
648 ter preset modifiers. The integer can be from 0 to 9,
649 matching the command line options -0 ... -9. The only
650 supported modifier is currently e, which matches
651 --extreme. If no preset is specified, the default values
652 of LZMA1 or LZMA2 options are taken from the preset 6.
653 dict=size
655 Dictionary (history buffer) size indicates how many bytes
656 of the recently processed uncompressed data is kept in
657 memory. The algorithm tries to find repeating byte
658 sequences (matches) in the uncompressed data, and replace
659 them with references to the data currently in the dictio‐
660 nary. The bigger the dictionary, the higher is the
661 chance to find a match. Thus, increasing dictionary size
662 usually improves compression ratio, but a dictionary big‐
663 ger than the uncompressed file is waste of memory.
664 Typical dictionary size is from 64 KiB to 64 MiB. The
666 minimum is 4 KiB. The maximum for compression is cur‐
667 rently 1.5 GiB (1536 MiB). The decompressor already sup‐
668 ports dictionaries up to one byte less than 4 GiB, which
669 is the maximum for the LZMA1 and LZMA2 stream formats.
670 Dictionary size and match finder (mf) together determine
672 the memory usage of the LZMA1 or LZMA2 encoder. The same
673 (or bigger) dictionary size is required for decompressing
674 that was used when compressing, thus the memory usage of
675 the decoder is determined by the dictionary size used
676 when compressing. The .xz headers store the dictionary
677 size either as 2^n or 2^n + 2^(n-1), so these sizes are
678 somewhat preferred for compression. Other sizes will get
679 rounded up when stored in the .xz headers.
680 lc=lc Specify the number of literal context bits. The minimum
682 is 0 and the maximum is 4; the default is 3. In addi‐
683 tion, the sum of lc and lp must not exceed 4.
684 All bytes that cannot be encoded as matches are encoded
686 as literals. That is, literals are simply 8-bit bytes
687 that are encoded one at a time.
688 The literal coding makes an assumption that the highest
690 lc bits of the previous uncompressed byte correlate with
691 the next byte. E.g. in typical English text, an upper-
692 case letter is often followed by a lower-case letter, and
693 a lower-case letter is usually followed by another lower-
694 case letter. In the US-ASCII character set, the highest
695 three bits are 010 for upper-case letters and 011 for
696 lower-case letters. When lc is at least 3, the literal
697 coding can take advantage of this property in the uncom‐
698 pressed data.
699 The default value (3) is usually good. If you want maxi‐
701 mum compression, test lc=4. Sometimes it helps a little,
702 and sometimes it makes compression worse. If it makes it
703 worse, test e.g. lc=2 too.
704 lp=lp Specify the number of literal position bits. The minimum
706 is 0 and the maximum is 4; the default is 0.
707 Lp affects what kind of alignment in the uncompressed
709 data is assumed when encoding literals. See pb below for
710 more information about alignment.
711 pb=pb Specify the number of position bits. The minimum is 0
713 and the maximum is 4; the default is 2.
714 Pb affects what kind of alignment in the uncompressed
716 data is assumed in general. The default means four-byte
717 alignment (2^pb=2^2=4), which is often a good choice when
718 there's no better guess.
719 When the aligment is known, setting pb accordingly may
721 reduce the file size a little. E.g. with text files hav‐
722 ing one-byte alignment (US-ASCII, ISO-8859-*, UTF-8),
723 setting pb=0 can improve compression slightly. For
724 UTF-16 text, pb=1 is a good choice. If the alignment is
725 an odd number like 3 bytes, pb=0 might be the best
726 choice.
727 Even though the assumed alignment can be adjusted with pb
729 and lp, LZMA1 and LZMA2 still slightly favor 16-byte
730 alignment. It might be worth taking into account when
731 designing file formats that are likely to be often com‐
732 pressed with LZMA1 or LZMA2.
733 mf=mf Match finder has a major effect on encoder speed, memory
735 usage, and compression ratio. Usually Hash Chain match
736 finders are faster than Binary Tree match finders. The
737 default depends on the preset: 0 uses hc3, 1-3 use hc4,
738 and the rest use bt4.
739 The following match finders are supported. The memory
741 usage formulas below are rough approximations, which are
742 closest to the reality when dict is a power of two.
743 hc3 Hash Chain with 2- and 3-byte hashing
745 Minimum value for nice: 3
746 Memory usage:
747 dict * 7.5 (if dict <= 16 MiB);
748 dict * 5.5 + 64 MiB (if dict > 16 MiB)
749 hc4 Hash Chain with 2-, 3-, and 4-byte hashing
751 Minimum value for nice: 4
752 Memory usage:
753 dict * 7.5 (if dict <= 32 MiB);
754 dict * 6.5 (if dict > 32 MiB)
755 bt2 Binary Tree with 2-byte hashing
757 Minimum value for nice: 2
758 Memory usage: dict * 9.5
759 bt3 Binary Tree with 2- and 3-byte hashing
761 Minimum value for nice: 3
762 Memory usage:
763 dict * 11.5 (if dict <= 16 MiB);
764 dict * 9.5 + 64 MiB (if dict > 16 MiB)
765 bt4 Binary Tree with 2-, 3-, and 4-byte hashing
767 Minimum value for nice: 4
768 Memory usage:
769 dict * 11.5 (if dict <= 32 MiB);
770 dict * 10.5 (if dict > 32 MiB)
771 mode=mode
773 Compression mode specifies the method to analyze the data
774 produced by the match finder. Supported modes are fast
775 and normal. The default is fast for presets 0-3 and nor‐
776 mal for presets 4-9.
777 Usually fast is used with Hash Chain match finders and
779 normal with Binary Tree match finders. This is also what
780 the presets do.
781 nice=nice
783 Specify what is considered to be a nice length for a
784 match. Once a match of at least nice bytes is found, the
785 algorithm stops looking for possibly better matches.
786 Nice can be 2-273 bytes. Higher values tend to give bet‐
788 ter compression ratio at the expense of speed. The
789 default depends on the preset.
790 depth=depth
792 Specify the maximum search depth in the match finder.
793 The default is the special value of 0, which makes the
794 compressor determine a reasonable depth from mf and nice.
795 Reasonable depth for Hash Chains is 4-100 and 16-1000 for
797 Binary Trees. Using very high values for depth can make
798 the encoder extremely slow with some files. Avoid set‐
799 ting the depth over 1000 unless you are prepared to
800 interrupt the compression in case it is taking far too
801 long.
802 When decoding raw streams (--format=raw), LZMA2 needs only the
804 dictionary size. LZMA1 needs also lc, lp, and pb.
805 --x86[=options]
807 --powerpc[=options]
808 --ia64[=options]
809 --arm[=options]
810 --armthumb[=options]
811 --sparc[=options]
812 Add a branch/call/jump (BCJ) filter to the filter chain. These
813 filters can be used only as a non-last filter in the filter
814 chain.
815 A BCJ filter converts relative addresses in the machine code to
817 their absolute counterparts. This doesn't change the size of
818 the data, but it increases redundancy, which can help LZMA2 to
819 produce 0-15 % smaller .xz file. The BCJ filters are always
820 reversible, so using a BCJ filter for wrong type of data doesn't
821 cause any data loss, although it may make the compression ratio
822 slightly worse.
823 It is fine to apply a BCJ filter on a whole executable; there's
825 no need to apply it only on the executable section. Applying a
826 BCJ filter on an archive that contains both executable and non-
827 executable files may or may not give good results, so it gener‐
828 ally isn't good to blindly apply a BCJ filter when compressing
829 binary packages for distribution.
830 These BCJ filters are very fast and use insignificant amount of
832 memory. If a BCJ filter improves compression ratio of a file,
833 it can improve decompression speed at the same time. This is
834 because, on the same hardware, the decompression speed of LZMA2
835 is roughly a fixed number of bytes of compressed data per sec‐
836 ond.
837 These BCJ filters have known problems related to the compression
839 ratio:
840 · Some types of files containing executable code (e.g. object
842 files, static libraries, and Linux kernel modules) have the
843 addresses in the instructions filled with filler values.
844 These BCJ filters will still do the address conversion, which
845 will make the compression worse with these files.
846 · Applying a BCJ filter on an archive containing multiple simi‐
848 lar executables can make the compression ratio worse than not
849 using a BCJ filter. This is because the BCJ filter doesn't
850 detect the boundaries of the executable files, and doesn't
851 reset the address conversion counter for each executable.
852 Both of the above problems will be fixed in the future in a new
854 filter. The old BCJ filters will still be useful in embedded
855 systems, because the decoder of the new filter will be bigger
856 and use more memory.
857 Different instruction sets have have different alignment:
859 Filter Alignment Notes
861 x86 1 32-bit or 64-bit x86
862 PowerPC 4 Big endian only
863 ARM 4 Little endian only
864 ARM-Thumb 2 Little endian only
865 IA-64 16 Big or little endian
866 SPARC 4 Big or little endian
867 Since the BCJ-filtered data is usually compressed with LZMA2,
869 the compression ratio may be improved slightly if the LZMA2
870 options are set to match the alignment of the selected BCJ fil‐
871 ter. For example, with the IA-64 filter, it's good to set pb=4
872 with LZMA2 (2^4=16). The x86 filter is an exception; it's usu‐
873 ally good to stick to LZMA2's default four-byte alignment when
874 compressing x86 executables.
875 All BCJ filters support the same options:
877 start=offset
879 Specify the start offset that is used when converting
880 between relative and absolute addresses. The offset must
881 be a multiple of the alignment of the filter (see the ta‐
882 ble above). The default is zero. In practice, the
883 default is good; specifying a custom offset is almost
884 never useful.
885 --delta[=options]
887 Add the Delta filter to the filter chain. The Delta filter can
888 be only used as a non-last filter in the filter chain.
889 Currently only simple byte-wise delta calculation is supported.
891 It can be useful when compressing e.g. uncompressed bitmap
892 images or uncompressed PCM audio. However, special purpose
893 algorithms may give significantly better results than Delta +
894 LZMA2. This is true especially with audio, which compresses
895 faster and better e.g. with flac(1).
896 Supported options:
898 dist=distance
900 Specify the distance of the delta calculation in bytes.
901 distance must be 1-256. The default is 1.
902 For example, with dist=2 and eight-byte input A1 B1 A2 B3
904 A3 B5 A4 B7, the output will be A1 B1 01 02 01 02 01 02.
905 Other options
907 -q, --quiet
908 Suppress warnings and notices. Specify this twice to suppress
909 errors too. This option has no effect on the exit status. That
910 is, even if a warning was suppressed, the exit status to indi‐
911 cate a warning is still used.
912 -v, --verbose
914 Be verbose. If standard error is connected to a terminal, xz
915 will display a progress indicator. Specifying --verbose twice
916 will give even more verbose output.
917 The progress indicator shows the following information:
919 · Completion percentage is shown if the size of the input file
921 is known. That is, the percentage cannot be shown in pipes.
922 · Amount of compressed data produced (compressing) or consumed
924 (decompressing).
925 · Amount of uncompressed data consumed (compressing) or pro‐
927 duced (decompressing).
928 · Compression ratio, which is calculated by dividing the amount
930 of compressed data processed so far by the amount of uncom‐
931 pressed data processed so far.
932 · Compression or decompression speed. This is measured as the
934 amount of uncompressed data consumed (compression) or pro‐
935 duced (decompression) per second. It is shown after a few
936 seconds have passed since xz started processing the file.
937 · Elapsed time in the format M:SS or H:MM:SS.
939 · Estimated remaining time is shown only when the size of the
941 input file is known and a couple of seconds have already
942 passed since xz started processing the file. The time is
943 shown in a less precise format which never has any colons,
944 e.g. 2 min 30 s.
945 When standard error is not a terminal, --verbose will make xz
947 print the filename, compressed size, uncompressed size, compres‐
948 sion ratio, and possibly also the speed and elapsed time on a
949 single line to standard error after compressing or decompressing
950 the file. The speed and elapsed time are included only when the
951 operation took at least a few seconds. If the operation didn't
952 finish, e.g. due to user interruption, also the completion per‐
953 centage is printed if the size of the input file is known.
954 -Q, --no-warn
956 Don't set the exit status to 2 even if a condition worth a warn‐
957 ing was detected. This option doesn't affect the verbosity
958 level, thus both --quiet and --no-warn have to be used to not
959 display warnings and to not alter the exit status.
960 --robot
962 Print messages in a machine-parsable format. This is intended
963 to ease writing frontends that want to use xz instead of
964 liblzma, which may be the case with various scripts. The output
965 with this option enabled is meant to be stable across xz
966 releases. See the section ROBOT MODE for details.
967 --info-memory
969 Display, in human-readable format, how much physical memory
970 (RAM) xz thinks the system has and the memory usage limits for
971 compression and decompression, and exit successfully.
972 -h, --help
974 Display a help message describing the most commonly used
975 options, and exit successfully.
976 -H, --long-help
978 Display a help message describing all features of xz, and exit
979 successfully
980 -V, --version
982 Display the version number of xz and liblzma in human readable
983 format. To get machine-parsable output, specify --robot before
984 --version.
985
987 The robot mode is activated with the --robot option. It makes the out‐
988 put of xz easier to parse by other programs. Currently --robot is sup‐
989 ported only together with --version, --info-memory, and --list. It
990 will be supported for compression and decompression in the future.
991 Version
993 xz --robot --version will print the version number of xz and liblzma in
994 the following format:
995 XZ_VERSION=XYYYZZZS
997 LIBLZMA_VERSION=XYYYZZZS
998 X Major version.
1000 YYY Minor version. Even numbers are stable. Odd numbers are alpha
1002 or beta versions.
1003 ZZZ Patch level for stable releases or just a counter for develop‐
1005 ment releases.
1006 S Stability. 0 is alpha, 1 is beta, and 2 is stable. S should be
1008 always 2 when YYY is even.
1009 XYYYZZZS are the same on both lines if xz and liblzma are from the same
1011 XZ Utils release.
1012 Examples: 4.999.9beta is 49990091 and 5.0.0 is 50000002.
1014 Memory limit information
1016 xz --robot --info-memory prints a single line with three tab-separated
1017 columns:
1018 1. Total amount of physical memory (RAM) in bytes
1020 2. Memory usage limit for compression in bytes. A special value of
1022 zero indicates the default setting, which for single-threaded mode
1023 is the same as no limit.
1024 3. Memory usage limit for decompression in bytes. A special value of
1026 zero indicates the default setting, which for single-threaded mode
1027 is the same as no limit.
1028 In the future, the output of xz --robot --info-memory may have more
1030 columns, but never more than a single line.
1031 List mode
1033 xz --robot --list uses tab-separated output. The first column of every
1034 line has a string that indicates the type of the information found on
1035 that line:
1036 name This is always the first line when starting to list a file. The
1038 second column on the line is the filename.
1039 file This line contains overall information about the .xz file. This
1041 line is always printed after the name line.
1042 stream This line type is used only when --verbose was specified. There
1044 are as many stream lines as there are streams in the .xz file.
1045 block This line type is used only when --verbose was specified. There
1047 are as many block lines as there are blocks in the .xz file.
1048 The block lines are shown after all the stream lines; different
1049 line types are not interleaved.
1050 summary
1052 This line type is used only when --verbose was specified twice.
1053 This line is printed after all block lines. Like the file line,
1054 the summary line contains overall information about the .xz
1055 file.
1056 totals This line is always the very last line of the list output. It
1058 shows the total counts and sizes.
1059 The columns of the file lines:
1061 2. Number of streams in the file
1062 3. Total number of blocks in the stream(s)
1063 4. Compressed size of the file
1064 5. Uncompressed size of the file
1065 6. Compression ratio, for example 0.123. If ratio is over
1066 9.999, three dashes (---) are displayed instead of the
1067 ratio.
1068 7. Comma-separated list of integrity check names. The follow‐
1069 ing strings are used for the known check types: None, CRC32,
1070 CRC64, and SHA-256. For unknown check types, Unknown-N is
1071 used, where N is the Check ID as a decimal number (one or
1072 two digits).
1073 8. Total size of stream padding in the file
1074 The columns of the stream lines:
1076 2. Stream number (the first stream is 1)
1077 3. Number of blocks in the stream
1078 4. Compressed start offset
1079 5. Uncompressed start offset
1080 6. Compressed size (does not include stream padding)
1081 7. Uncompressed size
1082 8. Compression ratio
1083 9. Name of the integrity check
1084 10. Size of stream padding
1085 The columns of the block lines:
1087 2. Number of the stream containing this block
1088 3. Block number relative to the beginning of the stream (the
1089 first block is 1)
1090 4. Block number relative to the beginning of the file
1091 5. Compressed start offset relative to the beginning of the
1092 file
1093 6. Uncompressed start offset relative to the beginning of the
1094 file
1095 7. Total compressed size of the block (includes headers)
1096 8. Uncompressed size
1097 9. Compression ratio
1098 10. Name of the integrity check
1099 If --verbose was specified twice, additional columns are included on
1101 the block lines. These are not displayed with a single --verbose,
1102 because getting this information requires many seeks and can thus be
1103 slow:
1104 11. Value of the integrity check in hexadecimal
1105 12. Block header size
1106 13. Block flags: c indicates that compressed size is present,
1107 and u indicates that uncompressed size is present. If the
1108 flag is not set, a dash (-) is shown instead to keep the
1109 string length fixed. New flags may be added to the end of
1110 the string in the future.
1111 14. Size of the actual compressed data in the block (this
1112 excludes the block header, block padding, and check fields)
1113 15. Amount of memory (in bytes) required to decompress this
1114 block with this xz version
1115 16. Filter chain. Note that most of the options used at com‐
1116 pression time cannot be known, because only the options that
1117 are needed for decompression are stored in the .xz headers.
1118 The columns of the summary lines:
1120 2. Amount of memory (in bytes) required to decompress this file
1121 with this xz version
1122 3. yes or no indicating if all block headers have both com‐
1123 pressed size and uncompressed size stored in them
1124 Since xz 5.1.2alpha:
1125 4. Minimum xz version required to decompress the file
1126 The columns of the totals line:
1128 2. Number of streams
1129 3. Number of blocks
1130 4. Compressed size
1131 5. Uncompressed size
1132 6. Average compression ratio
1133 7. Comma-separated list of integrity check names that were
1134 present in the files
1135 8. Stream padding size
1136 9. Number of files. This is here to keep the order of the ear‐
1137 lier columns the same as on file lines.
1138 If --verbose was specified twice, additional columns are included on
1140 the totals line:
1141 10. Maximum amount of memory (in bytes) required to decompress
1142 the files with this xz version
1143 11. yes or no indicating if all block headers have both com‐
1144 pressed size and uncompressed size stored in them
1145 Since xz 5.1.2alpha:
1146 12. Minimum xz version required to decompress the file
1147 Future versions may add new line types and new columns can be added to
1149 the existing line types, but the existing columns won't be changed.
1150
1152 0 All is good.
1153 1 An error occurred.
1155 2 Something worth a warning occurred, but no actual errors
1157 occurred.
1158 Notices (not warnings or errors) printed on standard error don't affect
1160 the exit status.
1161
1163 xz parses space-separated lists of options from the environment vari‐
1164 ables XZ_DEFAULTS and XZ_OPT, in this order, before parsing the options
1165 from the command line. Note that only options are parsed from the
1166 environment variables; all non-options are silently ignored. Parsing
1167 is done with getopt_long(3) which is used also for the command line
1168 arguments.
1169 XZ_DEFAULTS
1171 User-specific or system-wide default options. Typically this is
1172 set in a shell initialization script to enable xz's memory usage
1173 limiter by default. Excluding shell initialization scripts and
1174 similar special cases, scripts must never set or unset
1175 XZ_DEFAULTS.
1176 XZ_OPT This is for passing options to xz when it is not possible to set
1178 the options directly on the xz command line. This is the case
1179 e.g. when xz is run by a script or tool, e.g. GNU tar(1):
1180 XZ_OPT=-2v tar caf foo.tar.xz foo
1182 Scripts may use XZ_OPT e.g. to set script-specific default com‐
1184 pression options. It is still recommended to allow users to
1185 override XZ_OPT if that is reasonable, e.g. in sh(1) scripts one
1186 may use something like this:
1187 XZ_OPT=${XZ_OPT-"-7e"}
1189 export XZ_OPT
1190
1192 The command line syntax of xz is practically a superset of lzma,
1193 unlzma, and lzcat as found from LZMA Utils 4.32.x. In most cases, it
1194 is possible to replace LZMA Utils with XZ Utils without breaking exist‐
1195 ing scripts. There are some incompatibilities though, which may some‐
1196 times cause problems.
1197 Compression preset levels
1199 The numbering of the compression level presets is not identical in xz
1200 and LZMA Utils. The most important difference is how dictionary sizes
1201 are mapped to different presets. Dictionary size is roughly equal to
1202 the decompressor memory usage.
1203 Level xz LZMA Utils
1205 -0 256 KiB N/A
1206 -1 1 MiB 64 KiB
1207 -2 2 MiB 1 MiB
1208 -3 4 MiB 512 KiB
1209 -4 4 MiB 1 MiB
1210 -5 8 MiB 2 MiB
1211 -6 8 MiB 4 MiB
1212 -7 16 MiB 8 MiB
1213 -8 32 MiB 16 MiB
1214 -9 64 MiB 32 MiB
1215 The dictionary size differences affect the compressor memory usage too,
1217 but there are some other differences between LZMA Utils and XZ Utils,
1218 which make the difference even bigger:
1219 Level xz LZMA Utils 4.32.x
1221 -0 3 MiB N/A
1222 -1 9 MiB 2 MiB
1223 -2 17 MiB 12 MiB
1224 -3 32 MiB 12 MiB
1225 -4 48 MiB 16 MiB
1226 -5 94 MiB 26 MiB
1227 -6 94 MiB 45 MiB
1228 -7 186 MiB 83 MiB
1229 -8 370 MiB 159 MiB
1230 -9 674 MiB 311 MiB
1231 The default preset level in LZMA Utils is -7 while in XZ Utils it is
1233 -6, so both use an 8 MiB dictionary by default.
1234 Streamed vs. non-streamed .lzma files
1236 The uncompressed size of the file can be stored in the .lzma header.
1237 LZMA Utils does that when compressing regular files. The alternative
1238 is to mark that uncompressed size is unknown and use end-of-payload
1239 marker to indicate where the decompressor should stop. LZMA Utils uses
1240 this method when uncompressed size isn't known, which is the case for
1241 example in pipes.
1242 xz supports decompressing .lzma files with or without end-of-payload
1244 marker, but all .lzma files created by xz will use end-of-payload
1245 marker and have uncompressed size marked as unknown in the .lzma
1246 header. This may be a problem in some uncommon situations. For exam‐
1247 ple, a .lzma decompressor in an embedded device might work only with
1248 files that have known uncompressed size. If you hit this problem, you
1249 need to use LZMA Utils or LZMA SDK to create .lzma files with known
1250 uncompressed size.
1251 Unsupported .lzma files
1253 The .lzma format allows lc values up to 8, and lp values up to 4. LZMA
1254 Utils can decompress files with any lc and lp, but always creates files
1255 with lc=3 and lp=0. Creating files with other lc and lp is possible
1256 with xz and with LZMA SDK.
1257 The implementation of the LZMA1 filter in liblzma requires that the sum
1259 of lc and lp must not exceed 4. Thus, .lzma files, which exceed this
1260 limitation, cannot be decompressed with xz.
1261 LZMA Utils creates only .lzma files which have a dictionary size of 2^n
1263 (a power of 2) but accepts files with any dictionary size. liblzma
1264 accepts only .lzma files which have a dictionary size of 2^n or 2^n +
1265 2^(n-1). This is to decrease false positives when detecting .lzma
1266 files.
1267 These limitations shouldn't be a problem in practice, since practically
1269 all .lzma files have been compressed with settings that liblzma will
1270 accept.
1271 Trailing garbage
1273 When decompressing, LZMA Utils silently ignore everything after the
1274 first .lzma stream. In most situations, this is a bug. This also
1275 means that LZMA Utils don't support decompressing concatenated .lzma
1276 files.
1277 If there is data left after the first .lzma stream, xz considers the
1279 file to be corrupt unless --single-stream was used. This may break
1280 obscure scripts which have assumed that trailing garbage is ignored.
1281
1283 Compressed output may vary
1284 The exact compressed output produced from the same uncompressed input
1285 file may vary between XZ Utils versions even if compression options are
1286 identical. This is because the encoder can be improved (faster or bet‐
1287 ter compression) without affecting the file format. The output can
1288 vary even between different builds of the same XZ Utils version, if
1289 different build options are used.
1290 The above means that once --rsyncable has been implemented, the result‐
1292 ing files won't necessarily be rsyncable unless both old and new files
1293 have been compressed with the same xz version. This problem can be
1294 fixed if a part of the encoder implementation is frozen to keep rsynca‐
1295 ble output stable across xz versions.
1296 Embedded .xz decompressors
1298 Embedded .xz decompressor implementations like XZ Embedded don't neces‐
1299 sarily support files created with integrity check types other than none
1300 and crc32. Since the default is --check=crc64, you must use
1301 --check=none or --check=crc32 when creating files for embedded systems.
1302 Outside embedded systems, all .xz format decompressors support all the
1304 check types, or at least are able to decompress the file without veri‐
1305 fying the integrity check if the particular check is not supported.
1306 XZ Embedded supports BCJ filters, but only with the default start off‐
1308 set.
1309
1311 Basics
1312 Compress the file foo into foo.xz using the default compression level
1313 (-6), and remove foo if compression is successful:
1314 xz foo
1316 Decompress bar.xz into bar and don't remove bar.xz even if decompres‐
1318 sion is successful:
1319 xz -dk bar.xz
1321 Create baz.tar.xz with the preset -4e (-4 --extreme), which is slower
1323 than e.g. the default -6, but needs less memory for compression and
1324 decompression (48 MiB and 5 MiB, respectively):
1325 tar cf - baz | xz -4e > baz.tar.xz
1327 A mix of compressed and uncompressed files can be decompressed to stan‐
1329 dard output with a single command:
1330 xz -dcf a.txt b.txt.xz c.txt d.txt.lzma > abcd.txt
1332 Parallel compression of many files
1334 On GNU and *BSD, find(1) and xargs(1) can be used to parallelize com‐
1335 pression of many files:
1336 find . -type f \! -name '*.xz' -print0 \
1338 | xargs -0r -P4 -n16 xz -T1
1339 The -P option to xargs(1) sets the number of parallel xz processes.
1341 The best value for the -n option depends on how many files there are to
1342 be compressed. If there are only a couple of files, the value should
1343 probably be 1; with tens of thousands of files, 100 or even more may be
1344 appropriate to reduce the number of xz processes that xargs(1) will
1345 eventually create.
1346 The option -T1 for xz is there to force it to single-threaded mode,
1348 because xargs(1) is used to control the amount of parallelization.
1349 Robot mode
1351 Calculate how many bytes have been saved in total after compressing
1352 multiple files:
1353 xz --robot --list *.xz | awk '/^totals/{print $5-$4}'
1355 A script may want to know that it is using new enough xz. The follow‐
1357 ing sh(1) script checks that the version number of the xz tool is at
1358 least 5.0.0. This method is compatible with old beta versions, which
1359 didn't support the --robot option:
1360 if ! eval "$(xz --robot --version 2> /dev/null)" ||
1362 [ "$XZ_VERSION" -lt 50000002 ]; then
1363 echo "Your xz is too old."
1364 fi
1365 unset XZ_VERSION LIBLZMA_VERSION
1366 Set a memory usage limit for decompression using XZ_OPT, but if a limit
1368 has already been set, don't increase it:
1369 NEWLIM=$((123 << 20)) # 123 MiB
1371 OLDLIM=$(xz --robot --info-memory | cut -f3)
1372 if [ $OLDLIM -eq 0 -o $OLDLIM -gt $NEWLIM ]; then
1373 XZ_OPT="$XZ_OPT --memlimit-decompress=$NEWLIM"
1374 export XZ_OPT
1375 fi
1376 Custom compressor filter chains
1378 The simplest use for custom filter chains is customizing a LZMA2 pre‐
1379 set. This can be useful, because the presets cover only a subset of
1380 the potentially useful combinations of compression settings.
1381 The CompCPU columns of the tables from the descriptions of the options
1383 -0 ... -9 and --extreme are useful when customizing LZMA2 presets.
1384 Here are the relevant parts collected from those two tables:
1385 Preset CompCPU
1387 -0 0
1388 -1 1
1389 -2 2
1391 -3 3
1392 -4 4
1393 -5 5
1394 -6 6
1395 -5e 7
1396 -6e 8
1397 If you know that a file requires somewhat big dictionary (e.g. 32 MiB)
1399 to compress well, but you want to compress it quicker than xz -8 would
1400 do, a preset with a low CompCPU value (e.g. 1) can be modified to use a
1401 bigger dictionary:
1402 xz --lzma2=preset=1,dict=32MiB foo.tar
1404 With certain files, the above command may be faster than xz -6 while
1406 compressing significantly better. However, it must be emphasized that
1407 only some files benefit from a big dictionary while keeping the CompCPU
1408 value low. The most obvious situation, where a big dictionary can help
1409 a lot, is an archive containing very similar files of at least a few
1410 megabytes each. The dictionary size has to be significantly bigger
1411 than any individual file to allow LZMA2 to take full advantage of the
1412 similarities between consecutive files.
1413 If very high compressor and decompressor memory usage is fine, and the
1415 file being compressed is at least several hundred megabytes, it may be
1416 useful to use an even bigger dictionary than the 64 MiB that xz -9
1417 would use:
1418 xz -vv --lzma2=dict=192MiB big_foo.tar
1420 Using -vv (--verbose --verbose) like in the above example can be useful
1422 to see the memory requirements of the compressor and decompressor.
1423 Remember that using a dictionary bigger than the size of the uncom‐
1424 pressed file is waste of memory, so the above command isn't useful for
1425 small files.
1426 Sometimes the compression time doesn't matter, but the decompressor
1428 memory usage has to be kept low e.g. to make it possible to decompress
1429 the file on an embedded system. The following command uses -6e (-6
1430 --extreme) as a base and sets the dictionary to only 64 KiB. The
1431 resulting file can be decompressed with XZ Embedded (that's why there
1432 is --check=crc32) using about 100 KiB of memory.
1433 xz --check=crc32 --lzma2=preset=6e,dict=64KiB foo
1435 If you want to squeeze out as many bytes as possible, adjusting the
1437 number of literal context bits (lc) and number of position bits (pb)
1438 can sometimes help. Adjusting the number of literal position bits (lp)
1439 might help too, but usually lc and pb are more important. E.g. a
1440 source code archive contains mostly US-ASCII text, so something like
1441 the following might give slightly (like 0.1 %) smaller file than xz -6e
1442 (try also without lc=4):
1443 xz --lzma2=preset=6e,pb=0,lc=4 source_code.tar
1445 Using another filter together with LZMA2 can improve compression with
1447 certain file types. E.g. to compress a x86-32 or x86-64 shared library
1448 using the x86 BCJ filter:
1449 xz --x86 --lzma2 libfoo.so
1451 Note that the order of the filter options is significant. If --x86 is
1453 specified after --lzma2, xz will give an error, because there cannot be
1454 any filter after LZMA2, and also because the x86 BCJ filter cannot be
1455 used as the last filter in the chain.
1456 The Delta filter together with LZMA2 can give good results with bitmap
1458 images. It should usually beat PNG, which has a few more advanced fil‐
1459 ters than simple delta but uses Deflate for the actual compression.
1460 The image has to be saved in uncompressed format, e.g. as uncompressed
1462 TIFF. The distance parameter of the Delta filter is set to match the
1463 number of bytes per pixel in the image. E.g. 24-bit RGB bitmap needs
1464 dist=3, and it is also good to pass pb=0 to LZMA2 to accommodate the
1465 three-byte alignment:
1466 xz --delta=dist=3 --lzma2=pb=0 foo.tiff
1468 If multiple images have been put into a single archive (e.g. .tar), the
1470 Delta filter will work on that too as long as all images have the same
1471 number of bytes per pixel.
1472
1474 xzdec(1), xzdiff(1), xzgrep(1), xzless(1), xzmore(1), gzip(1),
1475 bzip2(1), 7z(1)
1476 XZ Utils: <http://tukaani.org/xz/>
1478 XZ Embedded: <http://tukaani.org/xz/embedded.html>
1479 LZMA SDK: <http://7-zip.org/sdk.html>
1480Tukaani 2015-05-11 XZ(1)