1XZ(1) XZ Utils XZ(1)
2
6 xz, unxz, xzcat, lzma, unlzma, lzcat - Compress or decompress .xz and
7 .lzma files
8
10 xz [option...] [file...]
11
13 unxz is equivalent to xz --decompress.
14 xzcat is equivalent to xz --decompress --stdout.
15 lzma is equivalent to xz --format=lzma.
16 unlzma is equivalent to xz --format=lzma --decompress.
17 lzcat is equivalent to xz --format=lzma --decompress --stdout.
18 When writing scripts that need to decompress files, it is recommended
20 to always use the name xz with appropriate arguments (xz -d or xz -dc)
21 instead of the names unxz and xzcat.
22
24 xz is a general-purpose data compression tool with command line syntax
25 similar to gzip(1) and bzip2(1). The native file format is the .xz
26 format, but the legacy .lzma format used by LZMA Utils and raw com‐
27 pressed streams with no container format headers are also supported.
28 xz compresses or decompresses each file according to the selected oper‐
30 ation mode. If no files are given or file is -, xz reads from standard
31 input and writes the processed data to standard output. xz will refuse
32 (display an error and skip the file) to write compressed data to stan‐
33 dard output if it is a terminal. Similarly, xz will refuse to read
34 compressed data from standard input if it is a terminal.
35 Unless --stdout is specified, files other than - are written to a new
37 file whose name is derived from the source file name:
38 · When compressing, the suffix of the target file format (.xz or
40 .lzma) is appended to the source filename to get the target file‐
41 name.
42 · When decompressing, the .xz or .lzma suffix is removed from the
44 filename to get the target filename. xz also recognizes the suf‐
45 fixes .txz and .tlz, and replaces them with the .tar suffix.
46 If the target file already exists, an error is displayed and the file
48 is skipped.
49 Unless writing to standard output, xz will display a warning and skip
51 the file if any of the following applies:
52 · File is not a regular file. Symbolic links are not followed, and
54 thus they are not considered to be regular files.
55 · File has more than one hard link.
57 · File has setuid, setgid, or sticky bit set.
59 · The operation mode is set to compress and the file already has a
61 suffix of the target file format (.xz or .txz when compressing to
62 the .xz format, and .lzma or .tlz when compressing to the .lzma for‐
63 mat).
64 · The operation mode is set to decompress and the file doesn't have a
66 suffix of any of the supported file formats (.xz, .txz, .lzma, or
67 .tlz).
68 After successfully compressing or decompressing the file, xz copies the
70 owner, group, permissions, access time, and modification time from the
71 source file to the target file. If copying the group fails, the per‐
72 missions are modified so that the target file doesn't become accessible
73 to users who didn't have permission to access the source file. xz
74 doesn't support copying other metadata like access control lists or
75 extended attributes yet.
76 Once the target file has been successfully closed, the source file is
78 removed unless --keep was specified. The source file is never removed
79 if the output is written to standard output.
80 Sending SIGINFO or SIGUSR1 to the xz process makes it print progress
82 information to standard error. This has only limited use since when
83 standard error is a terminal, using --verbose will display an automati‐
84 cally updating progress indicator.
85 Memory usage
87 The memory usage of xz varies from a few hundred kilobytes to several
88 gigabytes depending on the compression settings. The settings used
89 when compressing a file determine the memory requirements of the decom‐
90 pressor. Typically the decompressor needs 5 % to 20 % of the amount of
91 memory that the compressor needed when creating the file. For example,
92 decompressing a file created with xz -9 currently requires 65 MiB of
93 memory. Still, it is possible to have .xz files that require several
94 gigabytes of memory to decompress.
95 Especially users of older systems may find the possibility of very
97 large memory usage annoying. To prevent uncomfortable surprises, xz
98 has a built-in memory usage limiter, which is disabled by default.
99 While some operating systems provide ways to limit the memory usage of
100 processes, relying on it wasn't deemed to be flexible enough (e.g.
101 using ulimit(1) to limit virtual memory tends to cripple mmap(2)).
102 The memory usage limiter can be enabled with the command line option
104 --memlimit=limit. Often it is more convenient to enable the limiter by
105 default by setting the environment variable XZ_DEFAULTS, e.g.
106 XZ_DEFAULTS=--memlimit=150MiB. It is possible to set the limits sepa‐
107 rately for compression and decompression by using --memlimit-com‐
108 press=limit and --memlimit-decompress=limit. Using these two options
109 outside XZ_DEFAULTS is rarely useful because a single run of xz cannot
110 do both compression and decompression and --memlimit=limit (or -M
111 limit) is shorter to type on the command line.
112 If the specified memory usage limit is exceeded when decompressing, xz
114 will display an error and decompressing the file will fail. If the
115 limit is exceeded when compressing, xz will try to scale the settings
116 down so that the limit is no longer exceeded (except when using --for‐
117 mat=raw or --no-adjust). This way the operation won't fail unless the
118 limit is very small. The scaling of the settings is done in steps that
119 don't match the compression level presets, e.g. if the limit is only
120 slightly less than the amount required for xz -9, the settings will be
121 scaled down only a little, not all the way down to xz -8.
122 Concatenation and padding with .xz files
124 It is possible to concatenate .xz files as is. xz will decompress such
125 files as if they were a single .xz file.
126 It is possible to insert padding between the concatenated parts or
128 after the last part. The padding must consist of null bytes and the
129 size of the padding must be a multiple of four bytes. This can be use‐
130 ful e.g. if the .xz file is stored on a medium that measures file sizes
131 in 512-byte blocks.
132 Concatenation and padding are not allowed with .lzma files or raw
134 streams.
135
137 Integer suffixes and special values
138 In most places where an integer argument is expected, an optional suf‐
139 fix is supported to easily indicate large integers. There must be no
140 space between the integer and the suffix.
141 KiB Multiply the integer by 1,024 (2^10). Ki, k, kB, K, and KB are
143 accepted as synonyms for KiB.
144 MiB Multiply the integer by 1,048,576 (2^20). Mi, m, M, and MB are
146 accepted as synonyms for MiB.
147 GiB Multiply the integer by 1,073,741,824 (2^30). Gi, g, G, and GB
149 are accepted as synonyms for GiB.
150 The special value max can be used to indicate the maximum integer value
152 supported by the option.
153 Operation mode
155 If multiple operation mode options are given, the last one takes
156 effect.
157 -z, --compress
159 Compress. This is the default operation mode when no operation
160 mode option is specified and no other operation mode is implied
161 from the command name (for example, unxz implies --decompress).
162 -d, --decompress, --uncompress
164 Decompress.
165 -t, --test
167 Test the integrity of compressed files. This option is equiva‐
168 lent to --decompress --stdout except that the decompressed data
169 is discarded instead of being written to standard output. No
170 files are created or removed.
171 -l, --list
173 Print information about compressed files. No uncompressed out‐
174 put is produced, and no files are created or removed. In list
175 mode, the program cannot read the compressed data from standard
176 input or from other unseekable sources.
177 The default listing shows basic information about files, one
179 file per line. To get more detailed information, use also the
180 --verbose option. For even more information, use --verbose
181 twice, but note that this may be slow, because getting all the
182 extra information requires many seeks. The width of verbose
183 output exceeds 80 characters, so piping the output to e.g.
184 less -S may be convenient if the terminal isn't wide enough.
185 The exact output may vary between xz versions and different
187 locales. For machine-readable output, --robot --list should be
188 used.
189 Operation modifiers
191 -k, --keep
192 Don't delete the input files.
193 -f, --force
195 This option has several effects:
196 · If the target file already exists, delete it before compress‐
198 ing or decompressing.
199 · Compress or decompress even if the input is a symbolic link
201 to a regular file, has more than one hard link, or has the
202 setuid, setgid, or sticky bit set. The setuid, setgid, and
203 sticky bits are not copied to the target file.
204 · When used with --decompress --stdout and xz cannot recognize
206 the type of the source file, copy the source file as is to
207 standard output. This allows xzcat --force to be used like
208 cat(1) for files that have not been compressed with xz. Note
209 that in future, xz might support new compressed file formats,
210 which may make xz decompress more types of files instead of
211 copying them as is to standard output. --format=format can
212 be used to restrict xz to decompress only a single file for‐
213 mat.
214 -c, --stdout, --to-stdout
216 Write the compressed or decompressed data to standard output
217 instead of a file. This implies --keep.
218 --single-stream
220 Decompress only the first .xz stream, and silently ignore possi‐
221 ble remaining input data following the stream. Normally such
222 trailing garbage makes xz display an error.
223 xz never decompresses more than one stream from .lzma files or
225 raw streams, but this option still makes xz ignore the possible
226 trailing data after the .lzma file or raw stream.
227 This option has no effect if the operation mode is not --decom‐
229 press or --test.
230 --no-sparse
232 Disable creation of sparse files. By default, if decompressing
233 into a regular file, xz tries to make the file sparse if the
234 decompressed data contains long sequences of binary zeros. It
235 also works when writing to standard output as long as standard
236 output is connected to a regular file and certain additional
237 conditions are met to make it safe. Creating sparse files may
238 save disk space and speed up the decompression by reducing the
239 amount of disk I/O.
240 -S .suf, --suffix=.suf
242 When compressing, use .suf as the suffix for the target file
243 instead of .xz or .lzma. If not writing to standard output and
244 the source file already has the suffix .suf, a warning is dis‐
245 played and the file is skipped.
246 When decompressing, recognize files with the suffix .suf in
248 addition to files with the .xz, .txz, .lzma, or .tlz suffix. If
249 the source file has the suffix .suf, the suffix is removed to
250 get the target filename.
251 When compressing or decompressing raw streams (--format=raw),
253 the suffix must always be specified unless writing to standard
254 output, because there is no default suffix for raw streams.
255 --files[=file]
257 Read the filenames to process from file; if file is omitted,
258 filenames are read from standard input. Filenames must be ter‐
259 minated with the newline character. A dash (-) is taken as a
260 regular filename; it doesn't mean standard input. If filenames
261 are given also as command line arguments, they are processed
262 before the filenames read from file.
263 --files0[=file]
265 This is identical to --files[=file] except that each filename
266 must be terminated with the null character.
267 Basic file format and compression options
269 -F format, --format=format
270 Specify the file format to compress or decompress:
271 auto This is the default. When compressing, auto is equiva‐
273 lent to xz. When decompressing, the format of the input
274 file is automatically detected. Note that raw streams
275 (created with --format=raw) cannot be auto-detected.
276 xz Compress to the .xz file format, or accept only .xz files
278 when decompressing.
279 lzma, alone
281 Compress to the legacy .lzma file format, or accept only
282 .lzma files when decompressing. The alternative name
283 alone is provided for backwards compatibility with LZMA
284 Utils.
285 raw Compress or uncompress a raw stream (no headers). This
287 is meant for advanced users only. To decode raw streams,
288 you need use --format=raw and explicitly specify the fil‐
289 ter chain, which normally would have been stored in the
290 container headers.
291 -C check, --check=check
293 Specify the type of the integrity check. The check is calcu‐
294 lated from the uncompressed data and stored in the .xz file.
295 This option has an effect only when compressing into the .xz
296 format; the .lzma format doesn't support integrity checks. The
297 integrity check (if any) is verified when the .xz file is decom‐
298 pressed.
299 Supported check types:
301 none Don't calculate an integrity check at all. This is usu‐
303 ally a bad idea. This can be useful when integrity of
304 the data is verified by other means anyway.
305 crc32 Calculate CRC32 using the polynomial from IEEE-802.3
307 (Ethernet).
308 crc64 Calculate CRC64 using the polynomial from ECMA-182. This
310 is the default, since it is slightly better than CRC32 at
311 detecting damaged files and the speed difference is neg‐
312 ligible.
313 sha256 Calculate SHA-256. This is somewhat slower than CRC32
315 and CRC64.
316 Integrity of the .xz headers is always verified with CRC32. It
318 is not possible to change or disable it.
319 --ignore-check
321 Don't verify the integrity check of the compressed data when
322 decompressing. The CRC32 values in the .xz headers will still
323 be verified normally.
324 Do not use this option unless you know what you are doing. Pos‐
326 sible reasons to use this option:
327 · Trying to recover data from a corrupt .xz file.
329 · Speeding up decompression. This matters mostly with SHA-256
331 or with files that have compressed extremely well. It's rec‐
332 ommended to not use this option for this purpose unless the
333 file integrity is verified externally in some other way.
334 -0 ... -9
336 Select a compression preset level. The default is -6. If mul‐
337 tiple preset levels are specified, the last one takes effect.
338 If a custom filter chain was already specified, setting a com‐
339 pression preset level clears the custom filter chain.
340 The differences between the presets are more significant than
342 with gzip(1) and bzip2(1). The selected compression settings
343 determine the memory requirements of the decompressor, thus
344 using a too high preset level might make it painful to decom‐
345 press the file on an old system with little RAM. Specifically,
346 it's not a good idea to blindly use -9 for everything like it
347 often is with gzip(1) and bzip2(1).
348 -0 ... -3
350 These are somewhat fast presets. -0 is sometimes faster
351 than gzip -9 while compressing much better. The higher
352 ones often have speed comparable to bzip2(1) with compa‐
353 rable or better compression ratio, although the results
354 depend a lot on the type of data being compressed.
355 -4 ... -6
357 Good to very good compression while keeping decompressor
358 memory usage reasonable even for old systems. -6 is the
359 default, which is usually a good choice e.g. for dis‐
360 tributing files that need to be decompressible even on
361 systems with only 16 MiB RAM. (-5e or -6e may be worth
362 considering too. See --extreme.)
363 -7 ... -9
365 These are like -6 but with higher compressor and decom‐
366 pressor memory requirements. These are useful only when
367 compressing files bigger than 8 MiB, 16 MiB, and 32 MiB,
368 respectively.
369 On the same hardware, the decompression speed is approximately a
371 constant number of bytes of compressed data per second. In
372 other words, the better the compression, the faster the decom‐
373 pression will usually be. This also means that the amount of
374 uncompressed output produced per second can vary a lot.
375 The following table summarises the features of the presets:
377 Preset DictSize CompCPU CompMem DecMem
379 -0 256 KiB 0 3 MiB 1 MiB
380 -1 1 MiB 1 9 MiB 2 MiB
381 -2 2 MiB 2 17 MiB 3 MiB
382 -3 4 MiB 3 32 MiB 5 MiB
383 -4 4 MiB 4 48 MiB 5 MiB
384 -5 8 MiB 5 94 MiB 9 MiB
385 -6 8 MiB 6 94 MiB 9 MiB
386 -7 16 MiB 6 186 MiB 17 MiB
387 -8 32 MiB 6 370 MiB 33 MiB
388 -9 64 MiB 6 674 MiB 65 MiB
389 Column descriptions:
391 · DictSize is the LZMA2 dictionary size. It is waste of memory
393 to use a dictionary bigger than the size of the uncompressed
394 file. This is why it is good to avoid using the presets -7
395 ... -9 when there's no real need for them. At -6 and lower,
396 the amount of memory wasted is usually low enough to not mat‐
397 ter.
398 · CompCPU is a simplified representation of the LZMA2 settings
400 that affect compression speed. The dictionary size affects
401 speed too, so while CompCPU is the same for levels -6 ... -9,
402 higher levels still tend to be a little slower. To get even
403 slower and thus possibly better compression, see --extreme.
404 · CompMem contains the compressor memory requirements in the
406 single-threaded mode. It may vary slightly between xz ver‐
407 sions. Memory requirements of some of the future multi‐
408 threaded modes may be dramatically higher than that of the
409 single-threaded mode.
410 · DecMem contains the decompressor memory requirements. That
412 is, the compression settings determine the memory require‐
413 ments of the decompressor. The exact decompressor memory
414 usage is slightly more than the LZMA2 dictionary size, but
415 the values in the table have been rounded up to the next full
416 MiB.
417 -e, --extreme
419 Use a slower variant of the selected compression preset level
420 (-0 ... -9) to hopefully get a little bit better compression
421 ratio, but with bad luck this can also make it worse. Decom‐
422 pressor memory usage is not affected, but compressor memory
423 usage increases a little at preset levels -0 ... -3.
424 Since there are two presets with dictionary sizes 4 MiB and
426 8 MiB, the presets -3e and -5e use slightly faster settings
427 (lower CompCPU) than -4e and -6e, respectively. That way no two
428 presets are identical.
429 Preset DictSize CompCPU CompMem DecMem
431 -0e 256 KiB 8 4 MiB 1 MiB
432 -1e 1 MiB 8 13 MiB 2 MiB
433 -2e 2 MiB 8 25 MiB 3 MiB
434 -3e 4 MiB 7 48 MiB 5 MiB
435 -4e 4 MiB 8 48 MiB 5 MiB
436 -5e 8 MiB 7 94 MiB 9 MiB
437 -6e 8 MiB 8 94 MiB 9 MiB
438 -7e 16 MiB 8 186 MiB 17 MiB
439 -8e 32 MiB 8 370 MiB 33 MiB
440 -9e 64 MiB 8 674 MiB 65 MiB
441 For example, there are a total of four presets that use 8 MiB
443 dictionary, whose order from the fastest to the slowest is -5,
444 -6, -5e, and -6e.
445 --fast
447 --best These are somewhat misleading aliases for -0 and -9, respec‐
448 tively. These are provided only for backwards compatibility
449 with LZMA Utils. Avoid using these options.
450 --block-size=size
452 When compressing to the .xz format, split the input data into
453 blocks of size bytes. The blocks are compressed independently
454 from each other, which helps with multi-threading and makes lim‐
455 ited random-access decompression possible. This option is typi‐
456 cally used to override the default block size in multi-threaded
457 mode, but this option can be used in single-threaded mode too.
458 In multi-threaded mode about three times size bytes will be
460 allocated in each thread for buffering input and output. The
461 default size is three times the LZMA2 dictionary size or 1 MiB,
462 whichever is more. Typically a good value is 2-4 times the size
463 of the LZMA2 dictionary or at least 1 MiB. Using size less than
464 the LZMA2 dictionary size is waste of RAM because then the LZMA2
465 dictionary buffer will never get fully used. The sizes of the
466 blocks are stored in the block headers, which a future version
467 of xz will use for multi-threaded decompression.
468 In single-threaded mode no block splitting is done by default.
470 Setting this option doesn't affect memory usage. No size infor‐
471 mation is stored in block headers, thus files created in single-
472 threaded mode won't be identical to files created in multi-
473 threaded mode. The lack of size information also means that a
474 future version of xz won't be able decompress the files in
475 multi-threaded mode.
476 --block-list=sizes
478 When compressing to the .xz format, start a new block after the
479 given intervals of uncompressed data.
480 The uncompressed sizes of the blocks are specified as a comma-
482 separated list. Omitting a size (two or more consecutive com‐
483 mas) is a shorthand to use the size of the previous block.
484 If the input file is bigger than the sum of sizes, the last
486 value in sizes is repeated until the end of the file. A special
487 value of 0 may be used as the last value to indicate that the
488 rest of the file should be encoded as a single block.
489 If one specifies sizes that exceed the encoder's block size
491 (either the default value in threaded mode or the value speci‐
492 fied with --block-size=size), the encoder will create additional
493 blocks while keeping the boundaries specified in sizes. For
494 example, if one specifies --block-size=10MiB
495 --block-list=5MiB,10MiB,8MiB,12MiB,24MiB and the input file is
496 80 MiB, one will get 11 blocks: 5, 10, 8, 10, 2, 10, 10, 4, 10,
497 10, and 1 MiB.
498 In multi-threaded mode the sizes of the blocks are stored in the
500 block headers. This isn't done in single-threaded mode, so the
501 encoded output won't be identical to that of the multi-threaded
502 mode.
503 --flush-timeout=timeout
505 When compressing, if more than timeout milliseconds (a positive
506 integer) has passed since the previous flush and reading more
507 input would block, all the pending input data is flushed from
508 the encoder and made available in the output stream. This can
509 be useful if xz is used to compress data that is streamed over a
510 network. Small timeout values make the data available at the
511 receiving end with a small delay, but large timeout values give
512 better compression ratio.
513 This feature is disabled by default. If this option is speci‐
515 fied more than once, the last one takes effect. The special
516 timeout value of 0 can be used to explicitly disable this fea‐
517 ture.
518 This feature is not available on non-POSIX systems.
520 This feature is still experimental. Currently xz is unsuitable
522 for decompressing the stream in real time due to how xz does
523 buffering.
524 --memlimit-compress=limit
526 Set a memory usage limit for compression. If this option is
527 specified multiple times, the last one takes effect.
528 If the compression settings exceed the limit, xz will adjust the
530 settings downwards so that the limit is no longer exceeded and
531 display a notice that automatic adjustment was done. Such
532 adjustments are not made when compressing with --format=raw or
533 if --no-adjust has been specified. In those cases, an error is
534 displayed and xz will exit with exit status 1.
535 The limit can be specified in multiple ways:
537 · The limit can be an absolute value in bytes. Using an inte‐
539 ger suffix like MiB can be useful. Example: --memlimit-com‐
540 press=80MiB
541 · The limit can be specified as a percentage of total physical
543 memory (RAM). This can be useful especially when setting the
544 XZ_DEFAULTS environment variable in a shell initialization
545 script that is shared between different computers. That way
546 the limit is automatically bigger on systems with more mem‐
547 ory. Example: --memlimit-compress=70%
548 · The limit can be reset back to its default value by setting
550 it to 0. This is currently equivalent to setting the limit
551 to max (no memory usage limit). Once multithreading support
552 has been implemented, there may be a difference between 0 and
553 max for the multithreaded case, so it is recommended to use 0
554 instead of max until the details have been decided.
555 For 32-bit xz there is a special case: if the limit would be
557 over 4020 MiB, the limit is set to 4020 MiB. (The values 0 and
558 max aren't affected by this. A similar feature doesn't exist
559 for decompression.) This can be helpful when a 32-bit exe‐
560 cutable has access to 4 GiB address space while hopefully doing
561 no harm in other situations.
562 See also the section Memory usage.
564 --memlimit-decompress=limit
566 Set a memory usage limit for decompression. This also affects
567 the --list mode. If the operation is not possible without
568 exceeding the limit, xz will display an error and decompressing
569 the file will fail. See --memlimit-compress=limit for possible
570 ways to specify the limit.
571 -M limit, --memlimit=limit, --memory=limit
573 This is equivalent to specifying --memlimit-compress=limit
574 --memlimit-decompress=limit.
575 --no-adjust
577 Display an error and exit if the compression settings exceed the
578 memory usage limit. The default is to adjust the settings down‐
579 wards so that the memory usage limit is not exceeded. Automatic
580 adjusting is always disabled when creating raw streams (--for‐
581 mat=raw).
582 -T threads, --threads=threads
584 Specify the number of worker threads to use. Setting threads to
585 a special value 0 makes xz use as many threads as there are CPU
586 cores on the system. The actual number of threads can be less
587 than threads if the input file is not big enough for threading
588 with the given settings or if using more threads would exceed
589 the memory usage limit.
590 Currently the only threading method is to split the input into
592 blocks and compress them independently from each other. The
593 default block size depends on the compression level and can be
594 overridden with the --block-size=size option.
595 Threaded decompression hasn't been implemented yet. It will
597 only work on files that contain multiple blocks with size infor‐
598 mation in block headers. All files compressed in multi-threaded
599 mode meet this condition, but files compressed in single-
600 threaded mode don't even if --block-size=size is used.
601 Custom compressor filter chains
603 A custom filter chain allows specifying the compression settings in
604 detail instead of relying on the settings associated to the presets.
605 When a custom filter chain is specified, preset options (-0 ... -9 and
606 --extreme) earlier on the command line are forgotten. If a preset
607 option is specified after one or more custom filter chain options, the
608 new preset takes effect and the custom filter chain options specified
609 earlier are forgotten.
610 A filter chain is comparable to piping on the command line. When com‐
612 pressing, the uncompressed input goes to the first filter, whose output
613 goes to the next filter (if any). The output of the last filter gets
614 written to the compressed file. The maximum number of filters in the
615 chain is four, but typically a filter chain has only one or two fil‐
616 ters.
617 Many filters have limitations on where they can be in the filter chain:
619 some filters can work only as the last filter in the chain, some only
620 as a non-last filter, and some work in any position in the chain.
621 Depending on the filter, this limitation is either inherent to the fil‐
622 ter design or exists to prevent security issues.
623 A custom filter chain is specified by using one or more filter options
625 in the order they are wanted in the filter chain. That is, the order
626 of filter options is significant! When decoding raw streams (--for‐
627 mat=raw), the filter chain is specified in the same order as it was
628 specified when compressing.
629 Filters take filter-specific options as a comma-separated list. Extra
631 commas in options are ignored. Every option has a default value, so
632 you need to specify only those you want to change.
633 To see the whole filter chain and options, use xz -vv (that is, use
635 --verbose twice). This works also for viewing the filter chain options
636 used by presets.
637 --lzma1[=options]
639 --lzma2[=options]
640 Add LZMA1 or LZMA2 filter to the filter chain. These filters
641 can be used only as the last filter in the chain.
642 LZMA1 is a legacy filter, which is supported almost solely due
644 to the legacy .lzma file format, which supports only LZMA1.
645 LZMA2 is an updated version of LZMA1 to fix some practical
646 issues of LZMA1. The .xz format uses LZMA2 and doesn't support
647 LZMA1 at all. Compression speed and ratios of LZMA1 and LZMA2
648 are practically the same.
649 LZMA1 and LZMA2 share the same set of options:
651 preset=preset
653 Reset all LZMA1 or LZMA2 options to preset. Preset con‐
654 sist of an integer, which may be followed by single-let‐
655 ter preset modifiers. The integer can be from 0 to 9,
656 matching the command line options -0 ... -9. The only
657 supported modifier is currently e, which matches
658 --extreme. If no preset is specified, the default values
659 of LZMA1 or LZMA2 options are taken from the preset 6.
660 dict=size
662 Dictionary (history buffer) size indicates how many bytes
663 of the recently processed uncompressed data is kept in
664 memory. The algorithm tries to find repeating byte
665 sequences (matches) in the uncompressed data, and replace
666 them with references to the data currently in the dictio‐
667 nary. The bigger the dictionary, the higher is the
668 chance to find a match. Thus, increasing dictionary size
669 usually improves compression ratio, but a dictionary big‐
670 ger than the uncompressed file is waste of memory.
671 Typical dictionary size is from 64 KiB to 64 MiB. The
673 minimum is 4 KiB. The maximum for compression is cur‐
674 rently 1.5 GiB (1536 MiB). The decompressor already sup‐
675 ports dictionaries up to one byte less than 4 GiB, which
676 is the maximum for the LZMA1 and LZMA2 stream formats.
677 Dictionary size and match finder (mf) together determine
679 the memory usage of the LZMA1 or LZMA2 encoder. The same
680 (or bigger) dictionary size is required for decompressing
681 that was used when compressing, thus the memory usage of
682 the decoder is determined by the dictionary size used
683 when compressing. The .xz headers store the dictionary
684 size either as 2^n or 2^n + 2^(n-1), so these sizes are
685 somewhat preferred for compression. Other sizes will get
686 rounded up when stored in the .xz headers.
687 lc=lc Specify the number of literal context bits. The minimum
689 is 0 and the maximum is 4; the default is 3. In addi‐
690 tion, the sum of lc and lp must not exceed 4.
691 All bytes that cannot be encoded as matches are encoded
693 as literals. That is, literals are simply 8-bit bytes
694 that are encoded one at a time.
695 The literal coding makes an assumption that the highest
697 lc bits of the previous uncompressed byte correlate with
698 the next byte. E.g. in typical English text, an upper-
699 case letter is often followed by a lower-case letter, and
700 a lower-case letter is usually followed by another lower-
701 case letter. In the US-ASCII character set, the highest
702 three bits are 010 for upper-case letters and 011 for
703 lower-case letters. When lc is at least 3, the literal
704 coding can take advantage of this property in the uncom‐
705 pressed data.
706 The default value (3) is usually good. If you want maxi‐
708 mum compression, test lc=4. Sometimes it helps a little,
709 and sometimes it makes compression worse. If it makes it
710 worse, test e.g. lc=2 too.
711 lp=lp Specify the number of literal position bits. The minimum
713 is 0 and the maximum is 4; the default is 0.
714 Lp affects what kind of alignment in the uncompressed
716 data is assumed when encoding literals. See pb below for
717 more information about alignment.
718 pb=pb Specify the number of position bits. The minimum is 0
720 and the maximum is 4; the default is 2.
721 Pb affects what kind of alignment in the uncompressed
723 data is assumed in general. The default means four-byte
724 alignment (2^pb=2^2=4), which is often a good choice when
725 there's no better guess.
726 When the aligment is known, setting pb accordingly may
728 reduce the file size a little. E.g. with text files hav‐
729 ing one-byte alignment (US-ASCII, ISO-8859-*, UTF-8),
730 setting pb=0 can improve compression slightly. For
731 UTF-16 text, pb=1 is a good choice. If the alignment is
732 an odd number like 3 bytes, pb=0 might be the best
733 choice.
734 Even though the assumed alignment can be adjusted with pb
736 and lp, LZMA1 and LZMA2 still slightly favor 16-byte
737 alignment. It might be worth taking into account when
738 designing file formats that are likely to be often com‐
739 pressed with LZMA1 or LZMA2.
740 mf=mf Match finder has a major effect on encoder speed, memory
742 usage, and compression ratio. Usually Hash Chain match
743 finders are faster than Binary Tree match finders. The
744 default depends on the preset: 0 uses hc3, 1-3 use hc4,
745 and the rest use bt4.
746 The following match finders are supported. The memory
748 usage formulas below are rough approximations, which are
749 closest to the reality when dict is a power of two.
750 hc3 Hash Chain with 2- and 3-byte hashing
752 Minimum value for nice: 3
753 Memory usage:
754 dict * 7.5 (if dict <= 16 MiB);
755 dict * 5.5 + 64 MiB (if dict > 16 MiB)
756 hc4 Hash Chain with 2-, 3-, and 4-byte hashing
758 Minimum value for nice: 4
759 Memory usage:
760 dict * 7.5 (if dict <= 32 MiB);
761 dict * 6.5 (if dict > 32 MiB)
762 bt2 Binary Tree with 2-byte hashing
764 Minimum value for nice: 2
765 Memory usage: dict * 9.5
766 bt3 Binary Tree with 2- and 3-byte hashing
768 Minimum value for nice: 3
769 Memory usage:
770 dict * 11.5 (if dict <= 16 MiB);
771 dict * 9.5 + 64 MiB (if dict > 16 MiB)
772 bt4 Binary Tree with 2-, 3-, and 4-byte hashing
774 Minimum value for nice: 4
775 Memory usage:
776 dict * 11.5 (if dict <= 32 MiB);
777 dict * 10.5 (if dict > 32 MiB)
778 mode=mode
780 Compression mode specifies the method to analyze the data
781 produced by the match finder. Supported modes are fast
782 and normal. The default is fast for presets 0-3 and nor‐
783 mal for presets 4-9.
784 Usually fast is used with Hash Chain match finders and
786 normal with Binary Tree match finders. This is also what
787 the presets do.
788 nice=nice
790 Specify what is considered to be a nice length for a
791 match. Once a match of at least nice bytes is found, the
792 algorithm stops looking for possibly better matches.
793 Nice can be 2-273 bytes. Higher values tend to give bet‐
795 ter compression ratio at the expense of speed. The
796 default depends on the preset.
797 depth=depth
799 Specify the maximum search depth in the match finder.
800 The default is the special value of 0, which makes the
801 compressor determine a reasonable depth from mf and nice.
802 Reasonable depth for Hash Chains is 4-100 and 16-1000 for
804 Binary Trees. Using very high values for depth can make
805 the encoder extremely slow with some files. Avoid set‐
806 ting the depth over 1000 unless you are prepared to
807 interrupt the compression in case it is taking far too
808 long.
809 When decoding raw streams (--format=raw), LZMA2 needs only the
811 dictionary size. LZMA1 needs also lc, lp, and pb.
812 --x86[=options]
814 --powerpc[=options]
815 --ia64[=options]
816 --arm[=options]
817 --armthumb[=options]
818 --sparc[=options]
819 Add a branch/call/jump (BCJ) filter to the filter chain. These
820 filters can be used only as a non-last filter in the filter
821 chain.
822 A BCJ filter converts relative addresses in the machine code to
824 their absolute counterparts. This doesn't change the size of
825 the data, but it increases redundancy, which can help LZMA2 to
826 produce 0-15 % smaller .xz file. The BCJ filters are always
827 reversible, so using a BCJ filter for wrong type of data doesn't
828 cause any data loss, although it may make the compression ratio
829 slightly worse.
830 It is fine to apply a BCJ filter on a whole executable; there's
832 no need to apply it only on the executable section. Applying a
833 BCJ filter on an archive that contains both executable and non-
834 executable files may or may not give good results, so it gener‐
835 ally isn't good to blindly apply a BCJ filter when compressing
836 binary packages for distribution.
837 These BCJ filters are very fast and use insignificant amount of
839 memory. If a BCJ filter improves compression ratio of a file,
840 it can improve decompression speed at the same time. This is
841 because, on the same hardware, the decompression speed of LZMA2
842 is roughly a fixed number of bytes of compressed data per sec‐
843 ond.
844 These BCJ filters have known problems related to the compression
846 ratio:
847 · Some types of files containing executable code (e.g. object
849 files, static libraries, and Linux kernel modules) have the
850 addresses in the instructions filled with filler values.
851 These BCJ filters will still do the address conversion, which
852 will make the compression worse with these files.
853 · Applying a BCJ filter on an archive containing multiple simi‐
855 lar executables can make the compression ratio worse than not
856 using a BCJ filter. This is because the BCJ filter doesn't
857 detect the boundaries of the executable files, and doesn't
858 reset the address conversion counter for each executable.
859 Both of the above problems will be fixed in the future in a new
861 filter. The old BCJ filters will still be useful in embedded
862 systems, because the decoder of the new filter will be bigger
863 and use more memory.
864 Different instruction sets have different alignment:
866 Filter Alignment Notes
868 x86 1 32-bit or 64-bit x86
869 PowerPC 4 Big endian only
870 ARM 4 Little endian only
871 ARM-Thumb 2 Little endian only
872 IA-64 16 Big or little endian
873 SPARC 4 Big or little endian
874 Since the BCJ-filtered data is usually compressed with LZMA2,
876 the compression ratio may be improved slightly if the LZMA2
877 options are set to match the alignment of the selected BCJ fil‐
878 ter. For example, with the IA-64 filter, it's good to set pb=4
879 with LZMA2 (2^4=16). The x86 filter is an exception; it's usu‐
880 ally good to stick to LZMA2's default four-byte alignment when
881 compressing x86 executables.
882 All BCJ filters support the same options:
884 start=offset
886 Specify the start offset that is used when converting
887 between relative and absolute addresses. The offset must
888 be a multiple of the alignment of the filter (see the ta‐
889 ble above). The default is zero. In practice, the
890 default is good; specifying a custom offset is almost
891 never useful.
892 --delta[=options]
894 Add the Delta filter to the filter chain. The Delta filter can
895 be only used as a non-last filter in the filter chain.
896 Currently only simple byte-wise delta calculation is supported.
898 It can be useful when compressing e.g. uncompressed bitmap
899 images or uncompressed PCM audio. However, special purpose
900 algorithms may give significantly better results than Delta +
901 LZMA2. This is true especially with audio, which compresses
902 faster and better e.g. with flac(1).
903 Supported options:
905 dist=distance
907 Specify the distance of the delta calculation in bytes.
908 distance must be 1-256. The default is 1.
909 For example, with dist=2 and eight-byte input A1 B1 A2 B3
911 A3 B5 A4 B7, the output will be A1 B1 01 02 01 02 01 02.
912 Other options
914 -q, --quiet
915 Suppress warnings and notices. Specify this twice to suppress
916 errors too. This option has no effect on the exit status. That
917 is, even if a warning was suppressed, the exit status to indi‐
918 cate a warning is still used.
919 -v, --verbose
921 Be verbose. If standard error is connected to a terminal, xz
922 will display a progress indicator. Specifying --verbose twice
923 will give even more verbose output.
924 The progress indicator shows the following information:
926 · Completion percentage is shown if the size of the input file
928 is known. That is, the percentage cannot be shown in pipes.
929 · Amount of compressed data produced (compressing) or consumed
931 (decompressing).
932 · Amount of uncompressed data consumed (compressing) or pro‐
934 duced (decompressing).
935 · Compression ratio, which is calculated by dividing the amount
937 of compressed data processed so far by the amount of uncom‐
938 pressed data processed so far.
939 · Compression or decompression speed. This is measured as the
941 amount of uncompressed data consumed (compression) or pro‐
942 duced (decompression) per second. It is shown after a few
943 seconds have passed since xz started processing the file.
944 · Elapsed time in the format M:SS or H:MM:SS.
946 · Estimated remaining time is shown only when the size of the
948 input file is known and a couple of seconds have already
949 passed since xz started processing the file. The time is
950 shown in a less precise format which never has any colons,
951 e.g. 2 min 30 s.
952 When standard error is not a terminal, --verbose will make xz
954 print the filename, compressed size, uncompressed size, compres‐
955 sion ratio, and possibly also the speed and elapsed time on a
956 single line to standard error after compressing or decompressing
957 the file. The speed and elapsed time are included only when the
958 operation took at least a few seconds. If the operation didn't
959 finish, e.g. due to user interruption, also the completion per‐
960 centage is printed if the size of the input file is known.
961 -Q, --no-warn
963 Don't set the exit status to 2 even if a condition worth a warn‐
964 ing was detected. This option doesn't affect the verbosity
965 level, thus both --quiet and --no-warn have to be used to not
966 display warnings and to not alter the exit status.
967 --robot
969 Print messages in a machine-parsable format. This is intended
970 to ease writing frontends that want to use xz instead of
971 liblzma, which may be the case with various scripts. The output
972 with this option enabled is meant to be stable across xz
973 releases. See the section ROBOT MODE for details.
974 --info-memory
976 Display, in human-readable format, how much physical memory
977 (RAM) xz thinks the system has and the memory usage limits for
978 compression and decompression, and exit successfully.
979 -h, --help
981 Display a help message describing the most commonly used
982 options, and exit successfully.
983 -H, --long-help
985 Display a help message describing all features of xz, and exit
986 successfully
987 -V, --version
989 Display the version number of xz and liblzma in human readable
990 format. To get machine-parsable output, specify --robot before
991 --version.
992
994 The robot mode is activated with the --robot option. It makes the out‐
995 put of xz easier to parse by other programs. Currently --robot is sup‐
996 ported only together with --version, --info-memory, and --list. It
997 will be supported for compression and decompression in the future.
998 Version
1000 xz --robot --version will print the version number of xz and liblzma in
1001 the following format:
1002 XZ_VERSION=XYYYZZZS
1004 LIBLZMA_VERSION=XYYYZZZS
1005 X Major version.
1007 YYY Minor version. Even numbers are stable. Odd numbers are alpha
1009 or beta versions.
1010 ZZZ Patch level for stable releases or just a counter for develop‐
1012 ment releases.
1013 S Stability. 0 is alpha, 1 is beta, and 2 is stable. S should be
1015 always 2 when YYY is even.
1016 XYYYZZZS are the same on both lines if xz and liblzma are from the same
1018 XZ Utils release.
1019 Examples: 4.999.9beta is 49990091 and 5.0.0 is 50000002.
1021 Memory limit information
1023 xz --robot --info-memory prints a single line with three tab-separated
1024 columns:
1025 1. Total amount of physical memory (RAM) in bytes
1027 2. Memory usage limit for compression in bytes. A special value of
1029 zero indicates the default setting, which for single-threaded mode
1030 is the same as no limit.
1031 3. Memory usage limit for decompression in bytes. A special value of
1033 zero indicates the default setting, which for single-threaded mode
1034 is the same as no limit.
1035 In the future, the output of xz --robot --info-memory may have more
1037 columns, but never more than a single line.
1038 List mode
1040 xz --robot --list uses tab-separated output. The first column of every
1041 line has a string that indicates the type of the information found on
1042 that line:
1043 name This is always the first line when starting to list a file. The
1045 second column on the line is the filename.
1046 file This line contains overall information about the .xz file. This
1048 line is always printed after the name line.
1049 stream This line type is used only when --verbose was specified. There
1051 are as many stream lines as there are streams in the .xz file.
1052 block This line type is used only when --verbose was specified. There
1054 are as many block lines as there are blocks in the .xz file.
1055 The block lines are shown after all the stream lines; different
1056 line types are not interleaved.
1057 summary
1059 This line type is used only when --verbose was specified twice.
1060 This line is printed after all block lines. Like the file line,
1061 the summary line contains overall information about the .xz
1062 file.
1063 totals This line is always the very last line of the list output. It
1065 shows the total counts and sizes.
1066 The columns of the file lines:
1068 2. Number of streams in the file
1069 3. Total number of blocks in the stream(s)
1070 4. Compressed size of the file
1071 5. Uncompressed size of the file
1072 6. Compression ratio, for example 0.123. If ratio is over
1073 9.999, three dashes (---) are displayed instead of the
1074 ratio.
1075 7. Comma-separated list of integrity check names. The follow‐
1076 ing strings are used for the known check types: None, CRC32,
1077 CRC64, and SHA-256. For unknown check types, Unknown-N is
1078 used, where N is the Check ID as a decimal number (one or
1079 two digits).
1080 8. Total size of stream padding in the file
1081 The columns of the stream lines:
1083 2. Stream number (the first stream is 1)
1084 3. Number of blocks in the stream
1085 4. Compressed start offset
1086 5. Uncompressed start offset
1087 6. Compressed size (does not include stream padding)
1088 7. Uncompressed size
1089 8. Compression ratio
1090 9. Name of the integrity check
1091 10. Size of stream padding
1092 The columns of the block lines:
1094 2. Number of the stream containing this block
1095 3. Block number relative to the beginning of the stream (the
1096 first block is 1)
1097 4. Block number relative to the beginning of the file
1098 5. Compressed start offset relative to the beginning of the
1099 file
1100 6. Uncompressed start offset relative to the beginning of the
1101 file
1102 7. Total compressed size of the block (includes headers)
1103 8. Uncompressed size
1104 9. Compression ratio
1105 10. Name of the integrity check
1106 If --verbose was specified twice, additional columns are included on
1108 the block lines. These are not displayed with a single --verbose,
1109 because getting this information requires many seeks and can thus be
1110 slow:
1111 11. Value of the integrity check in hexadecimal
1112 12. Block header size
1113 13. Block flags: c indicates that compressed size is present,
1114 and u indicates that uncompressed size is present. If the
1115 flag is not set, a dash (-) is shown instead to keep the
1116 string length fixed. New flags may be added to the end of
1117 the string in the future.
1118 14. Size of the actual compressed data in the block (this
1119 excludes the block header, block padding, and check fields)
1120 15. Amount of memory (in bytes) required to decompress this
1121 block with this xz version
1122 16. Filter chain. Note that most of the options used at com‐
1123 pression time cannot be known, because only the options that
1124 are needed for decompression are stored in the .xz headers.
1125 The columns of the summary lines:
1127 2. Amount of memory (in bytes) required to decompress this file
1128 with this xz version
1129 3. yes or no indicating if all block headers have both com‐
1130 pressed size and uncompressed size stored in them
1131 Since xz 5.1.2alpha:
1132 4. Minimum xz version required to decompress the file
1133 The columns of the totals line:
1135 2. Number of streams
1136 3. Number of blocks
1137 4. Compressed size
1138 5. Uncompressed size
1139 6. Average compression ratio
1140 7. Comma-separated list of integrity check names that were
1141 present in the files
1142 8. Stream padding size
1143 9. Number of files. This is here to keep the order of the ear‐
1144 lier columns the same as on file lines.
1145 If --verbose was specified twice, additional columns are included on
1147 the totals line:
1148 10. Maximum amount of memory (in bytes) required to decompress
1149 the files with this xz version
1150 11. yes or no indicating if all block headers have both com‐
1151 pressed size and uncompressed size stored in them
1152 Since xz 5.1.2alpha:
1153 12. Minimum xz version required to decompress the file
1154 Future versions may add new line types and new columns can be added to
1156 the existing line types, but the existing columns won't be changed.
1157
1159 0 All is good.
1160 1 An error occurred.
1162 2 Something worth a warning occurred, but no actual errors
1164 occurred.
1165 Notices (not warnings or errors) printed on standard error don't affect
1167 the exit status.
1168
1170 xz parses space-separated lists of options from the environment vari‐
1171 ables XZ_DEFAULTS and XZ_OPT, in this order, before parsing the options
1172 from the command line. Note that only options are parsed from the
1173 environment variables; all non-options are silently ignored. Parsing
1174 is done with getopt_long(3) which is used also for the command line
1175 arguments.
1176 XZ_DEFAULTS
1178 User-specific or system-wide default options. Typically this is
1179 set in a shell initialization script to enable xz's memory usage
1180 limiter by default. Excluding shell initialization scripts and
1181 similar special cases, scripts must never set or unset
1182 XZ_DEFAULTS.
1183 XZ_OPT This is for passing options to xz when it is not possible to set
1185 the options directly on the xz command line. This is the case
1186 e.g. when xz is run by a script or tool, e.g. GNU tar(1):
1187 XZ_OPT=-2v tar caf foo.tar.xz foo
1189 Scripts may use XZ_OPT e.g. to set script-specific default com‐
1191 pression options. It is still recommended to allow users to
1192 override XZ_OPT if that is reasonable, e.g. in sh(1) scripts one
1193 may use something like this:
1194 XZ_OPT=${XZ_OPT-"-7e"}
1196 export XZ_OPT
1197
1199 The command line syntax of xz is practically a superset of lzma,
1200 unlzma, and lzcat as found from LZMA Utils 4.32.x. In most cases, it
1201 is possible to replace LZMA Utils with XZ Utils without breaking exist‐
1202 ing scripts. There are some incompatibilities though, which may some‐
1203 times cause problems.
1204 Compression preset levels
1206 The numbering of the compression level presets is not identical in xz
1207 and LZMA Utils. The most important difference is how dictionary sizes
1208 are mapped to different presets. Dictionary size is roughly equal to
1209 the decompressor memory usage.
1210 Level xz LZMA Utils
1212 -0 256 KiB N/A
1213 -1 1 MiB 64 KiB
1214 -2 2 MiB 1 MiB
1215 -3 4 MiB 512 KiB
1216 -4 4 MiB 1 MiB
1217 -5 8 MiB 2 MiB
1218 -6 8 MiB 4 MiB
1219 -7 16 MiB 8 MiB
1220 -8 32 MiB 16 MiB
1221 -9 64 MiB 32 MiB
1222 The dictionary size differences affect the compressor memory usage too,
1224 but there are some other differences between LZMA Utils and XZ Utils,
1225 which make the difference even bigger:
1226 Level xz LZMA Utils 4.32.x
1228 -0 3 MiB N/A
1229 -1 9 MiB 2 MiB
1230 -2 17 MiB 12 MiB
1231 -3 32 MiB 12 MiB
1232 -4 48 MiB 16 MiB
1233 -5 94 MiB 26 MiB
1234 -6 94 MiB 45 MiB
1235 -7 186 MiB 83 MiB
1236 -8 370 MiB 159 MiB
1237 -9 674 MiB 311 MiB
1238 The default preset level in LZMA Utils is -7 while in XZ Utils it is
1240 -6, so both use an 8 MiB dictionary by default.
1241 Streamed vs. non-streamed .lzma files
1243 The uncompressed size of the file can be stored in the .lzma header.
1244 LZMA Utils does that when compressing regular files. The alternative
1245 is to mark that uncompressed size is unknown and use end-of-payload
1246 marker to indicate where the decompressor should stop. LZMA Utils uses
1247 this method when uncompressed size isn't known, which is the case for
1248 example in pipes.
1249 xz supports decompressing .lzma files with or without end-of-payload
1251 marker, but all .lzma files created by xz will use end-of-payload
1252 marker and have uncompressed size marked as unknown in the .lzma
1253 header. This may be a problem in some uncommon situations. For exam‐
1254 ple, a .lzma decompressor in an embedded device might work only with
1255 files that have known uncompressed size. If you hit this problem, you
1256 need to use LZMA Utils or LZMA SDK to create .lzma files with known
1257 uncompressed size.
1258 Unsupported .lzma files
1260 The .lzma format allows lc values up to 8, and lp values up to 4. LZMA
1261 Utils can decompress files with any lc and lp, but always creates files
1262 with lc=3 and lp=0. Creating files with other lc and lp is possible
1263 with xz and with LZMA SDK.
1264 The implementation of the LZMA1 filter in liblzma requires that the sum
1266 of lc and lp must not exceed 4. Thus, .lzma files, which exceed this
1267 limitation, cannot be decompressed with xz.
1268 LZMA Utils creates only .lzma files which have a dictionary size of 2^n
1270 (a power of 2) but accepts files with any dictionary size. liblzma
1271 accepts only .lzma files which have a dictionary size of 2^n or 2^n +
1272 2^(n-1). This is to decrease false positives when detecting .lzma
1273 files.
1274 These limitations shouldn't be a problem in practice, since practically
1276 all .lzma files have been compressed with settings that liblzma will
1277 accept.
1278 Trailing garbage
1280 When decompressing, LZMA Utils silently ignore everything after the
1281 first .lzma stream. In most situations, this is a bug. This also
1282 means that LZMA Utils don't support decompressing concatenated .lzma
1283 files.
1284 If there is data left after the first .lzma stream, xz considers the
1286 file to be corrupt unless --single-stream was used. This may break
1287 obscure scripts which have assumed that trailing garbage is ignored.
1288
1290 Compressed output may vary
1291 The exact compressed output produced from the same uncompressed input
1292 file may vary between XZ Utils versions even if compression options are
1293 identical. This is because the encoder can be improved (faster or bet‐
1294 ter compression) without affecting the file format. The output can
1295 vary even between different builds of the same XZ Utils version, if
1296 different build options are used.
1297 The above means that once --rsyncable has been implemented, the result‐
1299 ing files won't necessarily be rsyncable unless both old and new files
1300 have been compressed with the same xz version. This problem can be
1301 fixed if a part of the encoder implementation is frozen to keep rsynca‐
1302 ble output stable across xz versions.
1303 Embedded .xz decompressors
1305 Embedded .xz decompressor implementations like XZ Embedded don't neces‐
1306 sarily support files created with integrity check types other than none
1307 and crc32. Since the default is --check=crc64, you must use
1308 --check=none or --check=crc32 when creating files for embedded systems.
1309 Outside embedded systems, all .xz format decompressors support all the
1311 check types, or at least are able to decompress the file without veri‐
1312 fying the integrity check if the particular check is not supported.
1313 XZ Embedded supports BCJ filters, but only with the default start off‐
1315 set.
1316
1318 Basics
1319 Compress the file foo into foo.xz using the default compression level
1320 (-6), and remove foo if compression is successful:
1321 xz foo
1323 Decompress bar.xz into bar and don't remove bar.xz even if decompres‐
1325 sion is successful:
1326 xz -dk bar.xz
1328 Create baz.tar.xz with the preset -4e (-4 --extreme), which is slower
1330 than e.g. the default -6, but needs less memory for compression and
1331 decompression (48 MiB and 5 MiB, respectively):
1332 tar cf - baz | xz -4e > baz.tar.xz
1334 A mix of compressed and uncompressed files can be decompressed to stan‐
1336 dard output with a single command:
1337 xz -dcf a.txt b.txt.xz c.txt d.txt.lzma > abcd.txt
1339 Parallel compression of many files
1341 On GNU and *BSD, find(1) and xargs(1) can be used to parallelize com‐
1342 pression of many files:
1343 find . -type f \! -name '*.xz' -print0 \
1345 | xargs -0r -P4 -n16 xz -T1
1346 The -P option to xargs(1) sets the number of parallel xz processes.
1348 The best value for the -n option depends on how many files there are to
1349 be compressed. If there are only a couple of files, the value should
1350 probably be 1; with tens of thousands of files, 100 or even more may be
1351 appropriate to reduce the number of xz processes that xargs(1) will
1352 eventually create.
1353 The option -T1 for xz is there to force it to single-threaded mode,
1355 because xargs(1) is used to control the amount of parallelization.
1356 Robot mode
1358 Calculate how many bytes have been saved in total after compressing
1359 multiple files:
1360 xz --robot --list *.xz | awk '/^totals/{print $5-$4}'
1362 A script may want to know that it is using new enough xz. The follow‐
1364 ing sh(1) script checks that the version number of the xz tool is at
1365 least 5.0.0. This method is compatible with old beta versions, which
1366 didn't support the --robot option:
1367 if ! eval "$(xz --robot --version 2> /dev/null)" ||
1369 [ "$XZ_VERSION" -lt 50000002 ]; then
1370 echo "Your xz is too old."
1371 fi
1372 unset XZ_VERSION LIBLZMA_VERSION
1373 Set a memory usage limit for decompression using XZ_OPT, but if a limit
1375 has already been set, don't increase it:
1376 NEWLIM=$((123 << 20)) # 123 MiB
1378 OLDLIM=$(xz --robot --info-memory | cut -f3)
1379 if [ $OLDLIM -eq 0 -o $OLDLIM -gt $NEWLIM ]; then
1380 XZ_OPT="$XZ_OPT --memlimit-decompress=$NEWLIM"
1381 export XZ_OPT
1382 fi
1383 Custom compressor filter chains
1385 The simplest use for custom filter chains is customizing a LZMA2 pre‐
1386 set. This can be useful, because the presets cover only a subset of
1387 the potentially useful combinations of compression settings.
1388 The CompCPU columns of the tables from the descriptions of the options
1390 -0 ... -9 and --extreme are useful when customizing LZMA2 presets.
1391 Here are the relevant parts collected from those two tables:
1392 Preset CompCPU
1394 -0 0
1395 -1 1
1396 -2 2
1397 -3 3
1398 -4 4
1399 -5 5
1400 -6 6
1401 -5e 7
1402 -6e 8
1403 If you know that a file requires somewhat big dictionary (e.g. 32 MiB)
1405 to compress well, but you want to compress it quicker than xz -8 would
1406 do, a preset with a low CompCPU value (e.g. 1) can be modified to use a
1407 bigger dictionary:
1408 xz --lzma2=preset=1,dict=32MiB foo.tar
1410 With certain files, the above command may be faster than xz -6 while
1412 compressing significantly better. However, it must be emphasized that
1413 only some files benefit from a big dictionary while keeping the CompCPU
1414 value low. The most obvious situation, where a big dictionary can help
1415 a lot, is an archive containing very similar files of at least a few
1416 megabytes each. The dictionary size has to be significantly bigger
1417 than any individual file to allow LZMA2 to take full advantage of the
1418 similarities between consecutive files.
1419 If very high compressor and decompressor memory usage is fine, and the
1421 file being compressed is at least several hundred megabytes, it may be
1422 useful to use an even bigger dictionary than the 64 MiB that xz -9
1423 would use:
1424 xz -vv --lzma2=dict=192MiB big_foo.tar
1426 Using -vv (--verbose --verbose) like in the above example can be useful
1428 to see the memory requirements of the compressor and decompressor.
1429 Remember that using a dictionary bigger than the size of the uncom‐
1430 pressed file is waste of memory, so the above command isn't useful for
1431 small files.
1432 Sometimes the compression time doesn't matter, but the decompressor
1434 memory usage has to be kept low e.g. to make it possible to decompress
1435 the file on an embedded system. The following command uses -6e (-6
1436 --extreme) as a base and sets the dictionary to only 64 KiB. The
1437 resulting file can be decompressed with XZ Embedded (that's why there
1438 is --check=crc32) using about 100 KiB of memory.
1439 xz --check=crc32 --lzma2=preset=6e,dict=64KiB foo
1441 If you want to squeeze out as many bytes as possible, adjusting the
1443 number of literal context bits (lc) and number of position bits (pb)
1444 can sometimes help. Adjusting the number of literal position bits (lp)
1445 might help too, but usually lc and pb are more important. E.g. a
1446 source code archive contains mostly US-ASCII text, so something like
1447 the following might give slightly (like 0.1 %) smaller file than xz -6e
1448 (try also without lc=4):
1449 xz --lzma2=preset=6e,pb=0,lc=4 source_code.tar
1451 Using another filter together with LZMA2 can improve compression with
1453 certain file types. E.g. to compress a x86-32 or x86-64 shared library
1454 using the x86 BCJ filter:
1455 xz --x86 --lzma2 libfoo.so
1457 Note that the order of the filter options is significant. If --x86 is
1459 specified after --lzma2, xz will give an error, because there cannot be
1460 any filter after LZMA2, and also because the x86 BCJ filter cannot be
1461 used as the last filter in the chain.
1462 The Delta filter together with LZMA2 can give good results with bitmap
1464 images. It should usually beat PNG, which has a few more advanced fil‐
1465 ters than simple delta but uses Deflate for the actual compression.
1466 The image has to be saved in uncompressed format, e.g. as uncompressed
1468 TIFF. The distance parameter of the Delta filter is set to match the
1469 number of bytes per pixel in the image. E.g. 24-bit RGB bitmap needs
1470 dist=3, and it is also good to pass pb=0 to LZMA2 to accommodate the
1471 three-byte alignment:
1472 xz --delta=dist=3 --lzma2=pb=0 foo.tiff
1474 If multiple images have been put into a single archive (e.g. .tar), the
1476 Delta filter will work on that too as long as all images have the same
1477 number of bytes per pixel.
1478
1480 xzdec(1), xzdiff(1), xzgrep(1), xzless(1), xzmore(1), gzip(1),
1481 bzip2(1), 7z(1)
1482 XZ Utils: <https://tukaani.org/xz/>
1484 XZ Embedded: <https://tukaani.org/xz/embedded.html>
1485 LZMA SDK: <http://7-zip.org/sdk.html>
1486Tukaani 2020-02-01 XZ(1)