1ZSTD(1) User Commands ZSTD(1)
2
6 zstd - zstd, zstdmt, unzstd, zstdcat - Compress or decompress .zst
7 files
8
10 zstd [OPTIONS] [-|INPUT-FILE] [-o OUTPUT-FILE]
11 zstdmt is equivalent to zstd -T0
13 unzstd is equivalent to zstd -d
15 zstdcat is equivalent to zstd -dcf
17
19 zstd is a fast lossless compression algorithm and data compression
20 tool, with command line syntax similar to gzip (1) and xz (1). It is
21 based on the LZ77 family, with further FSE & huff0 entropy stages. zstd
22 offers highly configurable compression speed, with fast modes at > 200
23 MB/s per core, and strong modes nearing lzma compression ratios. It
24 also features a very fast decoder, with speeds > 500 MB/s per core.
25 zstd command line syntax is generally similar to gzip, but features the
27 following differences :
28 • Source files are preserved by default. It´s possible to remove them
30 automatically by using the --rm command.
31 • When compressing a single file, zstd displays progress notifica‐
33 tions and result summary by default. Use -q to turn them off.
34 • zstd does not accept input from console, but it properly accepts
36 stdin when it´s not the console.
37 • zstd displays a short help page when command line is an error. Use
39 -q to turn it off.
40 zstd compresses or decompresses each file according to the selected op‐
44 eration mode. If no files are given or file is -, zstd reads from stan‐
45 dard input and writes the processed data to standard output. zstd will
46 refuse to write compressed data to standard output if it is a terminal
47 : it will display an error message and skip the file. Similarly, zstd
48 will refuse to read compressed data from standard input if it is a ter‐
49 minal.
50 Unless --stdout or -o is specified, files are written to a new file
52 whose name is derived from the source file name:
53 • When compressing, the suffix .zst is appended to the source file‐
55 name to get the target filename.
56 • When decompressing, the .zst suffix is removed from the source
58 filename to get the target filename
59 Concatenation with .zst files
63 It is possible to concatenate .zst files as is. zstd will decompress
64 such files as if they were a single .zst file.
65
67 Integer suffixes and special values
68 In most places where an integer argument is expected, an optional suf‐
69 fix is supported to easily indicate large integers. There must be no
70 space between the integer and the suffix.
71 KiB Multiply the integer by 1,024 (2^10). Ki, K, and KB are accepted
73 as synonyms for KiB.
74 MiB Multiply the integer by 1,048,576 (2^20). Mi, M, and MB are ac‐
76 cepted as synonyms for MiB.
77 Operation mode
79 If multiple operation mode options are given, the last one takes ef‐
80 fect.
81 -z, --compress
83 Compress. This is the default operation mode when no operation
84 mode option is specified and no other operation mode is implied
85 from the command name (for example, unzstd implies --decom‐
86 press).
87 -d, --decompress, --uncompress
89 Decompress.
90 -t, --test
92 Test the integrity of compressed files. This option is equiva‐
93 lent to --decompress --stdout except that the decompressed data
94 is discarded instead of being written to standard output. No
95 files are created or removed.
96 -b# Benchmark file(s) using compression level #
98 --train FILEs
100 Use FILEs as a training set to create a dictionary. The training
101 set should contain a lot of small files (> 100).
102 -l, --list
104 Display information related to a zstd compressed file, such as
105 size, ratio, and checksum. Some of these fields may not be
106 available. This command can be augmented with the -v modifier.
107 Operation modifiers
109 • -#: # compression level [1-19] (default: 3)
110 • --ultra: unlocks high compression levels 20+ (maximum 22), using a
112 lot more memory. Note that decompression will also require more
113 memory when using these levels.
114 • --fast[=#]: switch to ultra-fast compression levels. If =# is not
116 present, it defaults to 1. The higher the value, the faster the
117 compression speed, at the cost of some compression ratio. This set‐
118 ting overwrites compression level if one was set previously. Simi‐
119 larly, if a compression level is set after --fast, it overrides it.
120 • -T#, --threads=#: Compress using # working threads (default: 1). If
122 # is 0, attempt to detect and use the number of physical CPU cores.
123 In all cases, the nb of threads is capped to ZSTDMT_NBWORKERS_MAX,
124 which is either 64 in 32-bit mode, or 256 for 64-bit environments.
125 This modifier does nothing if zstd is compiled without multithread
126 support.
127 • --single-thread: Does not spawn a thread for compression, use a
129 single thread for both I/O and compression. In this mode, compres‐
130 sion is serialized with I/O, which is slightly slower. (This is
131 different from -T1, which spawns 1 compression thread in parallel
132 of I/O). This mode is the only one available when multithread sup‐
133 port is disabled. Single-thread mode features lower memory usage.
134 Final compressed result is slightly different from -T1.
135 • --adapt[=min=#,max=#] : zstd will dynamically adapt compression
137 level to perceived I/O conditions. Compression level adaptation can
138 be observed live by using command -v. Adaptation can be constrained
139 between supplied min and max levels. The feature works when com‐
140 bined with multi-threading and --long mode. It does not work with
141 --single-thread. It sets window size to 8 MB by default (can be
142 changed manually, see wlog). Due to the chaotic nature of dynamic
143 adaptation, compressed result is not reproducible. note : at the
144 time of this writing, --adapt can remain stuck at low speed when
145 combined with multiple worker threads (>=2).
146 • --long[=#]: enables long distance matching with # windowLog, if not
148 # is not present it defaults to 27. This increases the window size
149 (windowLog) and memory usage for both the compressor and decompres‐
150 sor. This setting is designed to improve the compression ratio for
151 files with long matches at a large distance.
152 Note: If windowLog is set to larger than 27, --long=windowLog or
154 --memory=windowSize needs to be passed to the decompressor.
155 • -D DICT: use DICT as Dictionary to compress or decompress FILE(s)
157 • --patch-from FILE: Specify the file to be used as a reference point
159 for zstd´s diff engine. This is effectively dictionary compression
160 with some convenient parameter selection, namely that windowSize >
161 srcSize.
162 Note: cannot use both this and -D together Note: --long mode will
164 be automatically activated if chainLog < fileLog (fileLog being the
165 windowLog required to cover the whole file). You can also manually
166 force it. Node: for all levels, you can use --patch-from in --sin‐
167 gle-thread mode to improve compression ratio at the cost of speed
168 Note: for level 19, you can get increased compression ratio at the
169 cost of speed by specifying --zstd=targetLength= to be something
170 large (i.e 4096), and by setting a large --zstd=chainLog=
171 • --rsyncable : zstd will periodically synchronize the compression
173 state to make the compressed file more rsync-friendly. There is a
174 negligible impact to compression ratio, and the faster compression
175 levels will see a small compression speed hit. This feature does
176 not work with --single-thread. You probably don´t want to use it
177 with long range mode, since it will decrease the effectiveness of
178 the synchronization points, but your milage may vary.
179 • -C, --[no-]check: add integrity check computed from uncompressed
181 data (default: enabled)
182 • --[no-]content-size: enable / disable whether or not the original
184 size of the file is placed in the header of the compressed file.
185 The default option is --content-size (meaning that the original
186 size will be placed in the header).
187 • --no-dictID: do not store dictionary ID within frame header (dic‐
189 tionary compression). The decoder will have to rely on implicit
190 knowledge about which dictionary to use, it won´t be able to check
191 if it´s correct.
192 • -M#, --memory=#: Set a memory usage limit. By default, Zstandard
194 uses 128 MB for decompression as the maximum amount of memory the
195 decompressor is allowed to use, but you can override this manually
196 if need be in either direction (ie. you can increase or decrease
197 it).
198 This is also used during compression when using with --patch-from=.
200 In this case, this parameter overrides that maximum size allowed
201 for a dictionary. (128 MB).
202 • --stream-size=# : Sets the pledged source size of input coming from
204 a stream. This value must be exact, as it will be included in the
205 produced frame header. Incorrect stream sizes will cause an error.
206 This information will be used to better optimize compression param‐
207 eters, resulting in better and potentially faster compression, es‐
208 pecially for smaller source sizes.
209 • --size-hint=#: When handling input from a stream, zstd must guess
211 how large the source size will be when optimizing compression pa‐
212 rameters. If the stream size is relatively small, this guess may be
213 a poor one, resulting in a higher compression ratio than expected.
214 This feature allows for controlling the guess when needed. Exact
215 guesses result in better compression ratios. Overestimates result
216 in slightly degraded compression ratios, while underestimates may
217 result in significant degradation.
218 • -o FILE: save result into FILE
220 • -f, --force: disable input and output checks. Allows overwriting
222 existing files, input from console, output to stdout, operating on
223 links, block devices, etc.
224 • -c, --stdout: force write to standard output, even if it is the
226 console
227 • --[no-]sparse: enable / disable sparse FS support, to make files
229 with many zeroes smaller on disk. Creating sparse files may save
230 disk space and speed up decompression by reducing the amount of
231 disk I/O. default: enabled when output is into a file, and disabled
232 when output is stdout. This setting overrides default and can force
233 sparse mode over stdout.
234 • --rm: remove source file(s) after successful compression or decom‐
236 pression. If used in combination with -o, will trigger a confirma‐
237 tion prompt (which can be silenced with -f), as this is a destruc‐
238 tive operation.
239 • -k, --keep: keep source file(s) after successful compression or de‐
241 compression. This is the default behavior.
242 • -r: operate recursively on directories
244 • --filelist FILE read a list of files to process as content from
246 FILE. Format is compatible with ls output, with one file per line.
247 • --output-dir-flat DIR: resulting files are stored into target DIR
249 directory, instead of same directory as origin file. Be aware that
250 this command can introduce name collision issues, if multiple
251 files, from different directories, end up having the same name.
252 Collision resolution ensures first file with a given name will be
253 present in DIR, while in combination with -f, the last file will be
254 present instead.
255 • --output-dir-mirror DIR: similar to --output-dir-flat, the output
257 files are stored underneath target DIR directory, but this option
258 will replicate input directory hierarchy into output DIR.
259 If input directory contains "..", the files in this directory will
261 be ignored. If input directory is an absolute directory (i.e.
262 "/var/tmp/abc"), it will be stored into the "out‐
263 put-dir/var/tmp/abc". If there are multiple input files or directo‐
264 ries, name collision resolution will follow the same rules as
265 --output-dir-flat.
266 • --format=FORMAT: compress and decompress in other formats. If com‐
268 piled with support, zstd can compress to or decompress from other
269 compression algorithm formats. Possibly available options are zstd,
270 gzip, xz, lzma, and lz4. If no such format is provided, zstd is the
271 default.
272 • -h/-H, --help: display help/long help and exit
274 • -V, --version: display version number and exit. Advanced : -vV also
276 displays supported formats. -vvV also displays POSIX support. -q
277 will only display the version number, suitable for machine reading.
278 • -v, --verbose: verbose mode, display more information
280 • -q, --quiet: suppress warnings, interactivity, and notifications.
282 specify twice to suppress errors too.
283 • --no-progress: do not display the progress bar, but keep all other
285 messages.
286 • --show-default-cparams: Shows the default compression parameters
288 that will be used for a particular src file. If the provided src
289 file is not a regular file (eg. named pipe), the cli will just out‐
290 put the default parameters. That is, the parameters that are used
291 when the src size is unknown.
292 • --: All arguments after -- are treated as files
294
297 Additional options for the pzstd utility
298 -p, --processes
300 number of threads to use for (de)compression (default:4)
301 Restricted usage of Environment Variables
307 Using environment variables to set parameters has security implica‐
308 tions. Therefore, this avenue is intentionally restricted. Only
309 ZSTD_CLEVEL and ZSTD_NBTHREADS are currently supported. They set the
310 compression level and number of threads to use during compression, re‐
311 spectively.
312 ZSTD_CLEVEL can be used to set the level between 1 and 19 (the "normal"
314 range). If the value of ZSTD_CLEVEL is not a valid integer, it will be
315 ignored with a warning message. ZSTD_CLEVEL just replaces the default
316 compression level (3).
317 ZSTD_NBTHREADS can be used to set the number of threads zstd will at‐
319 tempt to use during compression. If the value of ZSTD_NBTHREADS is not
320 a valid unsigned integer, it will be ignored with a warning message.
321 ZSTD_NBTHREADS has a default value of (1), and is capped at ZSTDMT_NB‐
322 WORKERS_MAX==200. zstd must be compiled with multithread support for
323 this to have any effect.
324 They can both be overridden by corresponding command line arguments: -#
326 for compression level and -T# for number of compression threads.
327
329 zstd offers dictionary compression, which greatly improves efficiency
330 on small files and messages. It´s possible to train zstd with a set of
331 samples, the result of which is saved into a file called a dictionary.
332 Then during compression and decompression, reference the same dictio‐
333 nary, using command -D dictionaryFileName. Compression of small files
334 similar to the sample set will be greatly improved.
335 --train FILEs
337 Use FILEs as training set to create a dictionary. The training
338 set should contain a lot of small files (> 100), and weight typ‐
339 ically 100x the target dictionary size (for example, 10 MB for a
340 100 KB dictionary).
341 Supports multithreading if zstd is compiled with threading sup‐
343 port. Additional parameters can be specified with --train-fast‐
344 cover. The legacy dictionary builder can be accessed with
345 --train-legacy. The cover dictionary builder can be accessed
346 with --train-cover. Equivalent to --train-fastcover=d=8,steps=4.
347 -o file
349 Dictionary saved into file (default name: dictionary).
350 --maxdict=#
352 Limit dictionary to specified size (default: 112640).
353 -# Use # compression level during training (optional). Will gener‐
355 ate statistics more tuned for selected compression level, re‐
356 sulting in a small compression ratio improvement for this level.
357 -B# Split input files in blocks of size # (default: no split)
359 --dictID=#
361 A dictionary ID is a locally unique ID that a decoder can use to
362 verify it is using the right dictionary. By default, zstd will
363 create a 4-bytes random number ID. It´s possible to give a pre‐
364 cise number instead. Short numbers have an advantage : an ID <
365 256 will only need 1 byte in the compressed frame header, and an
366 ID < 65536 will only need 2 bytes. This compares favorably to 4
367 bytes default. However, it´s up to the dictionary manager to not
368 assign twice the same ID to 2 different dictionaries.
369 --train-cover[=k#,d=#,steps=#,split=#,shrink[=#]]
371 Select parameters for the default dictionary builder algorithm
372 named cover. If d is not specified, then it tries d = 6 and d =
373 8. If k is not specified, then it tries steps values in the
374 range [50, 2000]. If steps is not specified, then the default
375 value of 40 is used. If split is not specified or split <= 0,
376 then the default value of 100 is used. Requires that d <= k. If
377 shrink flag is not used, then the default value for shrinkDict
378 of 0 is used. If shrink is not specified, then the default value
379 for shrinkDictMaxRegression of 1 is used.
380 Selects segments of size k with highest score to put in the dic‐
382 tionary. The score of a segment is computed by the sum of the
383 frequencies of all the subsegments of size d. Generally d should
384 be in the range [6, 8], occasionally up to 16, but the algorithm
385 will run faster with d <= 8. Good values for k vary widely based
386 on the input data, but a safe range is [2 * d, 2000]. If split
387 is 100, all input samples are used for both training and testing
388 to find optimal d and k to build dictionary. Supports multi‐
389 threading if zstd is compiled with threading support. Having
390 shrink enabled takes a truncated dictionary of minimum size and
391 doubles in size until compression ratio of the truncated dictio‐
392 nary is at most shrinkDictMaxRegression% worse than the compres‐
393 sion ratio of the largest dictionary.
394 Examples:
396 zstd --train-cover FILEs
398 zstd --train-cover=k=50,d=8 FILEs
400 zstd --train-cover=d=8,steps=500 FILEs
402 zstd --train-cover=k=50 FILEs
404 zstd --train-cover=k=50,split=60 FILEs
406 zstd --train-cover=shrink FILEs
408 zstd --train-cover=shrink=2 FILEs
410 --train-fastcover[=k#,d=#,f=#,steps=#,split=#,accel=#]
412 Same as cover but with extra parameters f and accel and differ‐
413 ent default value of split If split is not specified, then it
414 tries split = 75. If f is not specified, then it tries f = 20.
415 Requires that 0 < f < 32. If accel is not specified, then it
416 tries accel = 1. Requires that 0 < accel <= 10. Requires that d
417 = 6 or d = 8.
418 f is log of size of array that keeps track of frequency of sub‐
420 segments of size d. The subsegment is hashed to an index in the
421 range [0,2^f - 1]. It is possible that 2 different subsegments
422 are hashed to the same index, and they are considered as the
423 same subsegment when computing frequency. Using a higher f re‐
424 duces collision but takes longer.
425 Examples:
427 zstd --train-fastcover FILEs
429 zstd --train-fastcover=d=8,f=15,accel=2 FILEs
431 --train-legacy[=selectivity=#]
433 Use legacy dictionary builder algorithm with the given dictio‐
434 nary selectivity (default: 9). The smaller the selectivity
435 value, the denser the dictionary, improving its efficiency but
436 reducing its possible maximum size. --train-legacy=s=# is also
437 accepted.
438 Examples:
440 zstd --train-legacy FILEs
442 zstd --train-legacy=selectivity=8 FILEs
444
446 -b# benchmark file(s) using compression level #
447 -e# benchmark file(s) using multiple compression levels, from -b# to
449 -e# (inclusive)
450 -i# minimum evaluation time, in seconds (default: 3s), benchmark
452 mode only
453 -B#, --block-size=#
455 cut file(s) into independent blocks of size # (default: no
456 block)
457 --priority=rt
459 set process priority to real-time
460 Output Format: CompressionLevel#Filename : IntputSize -> OutputSize
462 (CompressionRatio), CompressionSpeed, DecompressionSpeed
463 Methodology: For both compression and decompression speed, the entire
465 input is compressed/decompressed in-memory to measure speed. A run
466 lasts at least 1 sec, so when files are small, they are compressed/de‐
467 compressed several times per run, in order to improve measurement accu‐
468 racy.
469
471 -B#:
472 Select the size of each compression job. This parameter is only avail‐
473 able when multi-threading is enabled. Each compression job is run in
474 parallel, so this value indirectly impacts the nb of active threads.
475 Default job size varies depending on compression level (generally 4 *
476 windowSize). -B# makes it possible to manually select a custom size.
477 Note that job size must respect a minimum value which is enforced
478 transparently. This minimum is either 512 KB, or overlapSize, whichever
479 is largest. Different job sizes will lead to (slightly) different com‐
480 pressed frames.
481 --zstd[=options]:
483 zstd provides 22 predefined compression levels. The selected or default
484 predefined compression level can be changed with advanced compression
485 options. The options are provided as a comma-separated list. You may
486 specify only the options you want to change and the rest will be taken
487 from the selected or default compression level. The list of available
488 options:
489 strategy=strat, strat=strat
491 Specify a strategy used by a match finder.
492 There are 9 strategies numbered from 1 to 9, from faster to
494 stronger: 1=ZSTD_fast, 2=ZSTD_dfast, 3=ZSTD_greedy, 4=ZSTD_lazy,
495 5=ZSTD_lazy2, 6=ZSTD_btlazy2, 7=ZSTD_btopt, 8=ZSTD_btultra,
496 9=ZSTD_btultra2.
497 windowLog=wlog, wlog=wlog
499 Specify the maximum number of bits for a match distance.
500 The higher number of increases the chance to find a match which
502 usually improves compression ratio. It also increases memory re‐
503 quirements for the compressor and decompressor. The minimum wlog
504 is 10 (1 KiB) and the maximum is 30 (1 GiB) on 32-bit platforms
505 and 31 (2 GiB) on 64-bit platforms.
506 Note: If windowLog is set to larger than 27, --long=windowLog or
508 --memory=windowSize needs to be passed to the decompressor.
509 hashLog=hlog, hlog=hlog
511 Specify the maximum number of bits for a hash table.
512 Bigger hash tables cause less collisions which usually makes
514 compression faster, but requires more memory during compression.
515 The minimum hlog is 6 (64 B) and the maximum is 30 (1 GiB).
517 chainLog=clog, clog=clog
519 Specify the maximum number of bits for a hash chain or a binary
520 tree.
521 Higher numbers of bits increases the chance to find a match
523 which usually improves compression ratio. It also slows down
524 compression speed and increases memory requirements for compres‐
525 sion. This option is ignored for the ZSTD_fast strategy.
526 The minimum clog is 6 (64 B) and the maximum is 29 (524 Mib) on
528 32-bit platforms and 30 (1 Gib) on 64-bit platforms.
529 searchLog=slog, slog=slog
531 Specify the maximum number of searches in a hash chain or a bi‐
532 nary tree using logarithmic scale.
533 More searches increases the chance to find a match which usually
535 increases compression ratio but decreases compression speed.
536 The minimum slog is 1 and the maximum is ´windowLog´ - 1.
538 minMatch=mml, mml=mml
540 Specify the minimum searched length of a match in a hash table.
541 Larger search lengths usually decrease compression ratio but im‐
543 prove decompression speed.
544 The minimum mml is 3 and the maximum is 7.
546 targetLength=tlen, tlen=tlen
548 The impact of this field vary depending on selected strategy.
549 For ZSTD_btopt, ZSTD_btultra and ZSTD_btultra2, it specifies the
551 minimum match length that causes match finder to stop searching.
552 A larger targetLength usually improves compression ratio but de‐
553 creases compression speed. t For ZSTD_fast, it triggers ul‐
554 tra-fast mode when > 0. The value represents the amount of data
555 skipped between match sampling. Impact is reversed : a larger
556 targetLength increases compression speed but decreases compres‐
557 sion ratio.
558 For all other strategies, this field has no impact.
560 The minimum tlen is 0 and the maximum is 128 Kib.
562 overlapLog=ovlog, ovlog=ovlog
564 Determine overlapSize, amount of data reloaded from previous
565 job. This parameter is only available when multithreading is en‐
566 abled. Reloading more data improves compression ratio, but de‐
567 creases speed.
568 The minimum ovlog is 0, and the maximum is 9. 1 means "no over‐
570 lap", hence completely independent jobs. 9 means "full overlap",
571 meaning up to windowSize is reloaded from previous job. Reducing
572 ovlog by 1 reduces the reloaded amount by a factor 2. For exam‐
573 ple, 8 means "windowSize/2", and 6 means "windowSize/8". Value 0
574 is special and means "default" : ovlog is automatically deter‐
575 mined by zstd. In which case, ovlog will range from 6 to 9, de‐
576 pending on selected strat.
577 ldmHashLog=lhlog, lhlog=lhlog
579 Specify the maximum size for a hash table used for long distance
580 matching.
581 This option is ignored unless long distance matching is enabled.
583 Bigger hash tables usually improve compression ratio at the ex‐
585 pense of more memory during compression and a decrease in com‐
586 pression speed.
587 The minimum lhlog is 6 and the maximum is 30 (default: 20).
589 ldmMinMatch=lmml, lmml=lmml
591 Specify the minimum searched length of a match for long distance
592 matching.
593 This option is ignored unless long distance matching is enabled.
595 Larger/very small values usually decrease compression ratio.
597 The minimum lmml is 4 and the maximum is 4096 (default: 64).
599 ldmBucketSizeLog=lblog, lblog=lblog
601 Specify the size of each bucket for the hash table used for long
602 distance matching.
603 This option is ignored unless long distance matching is enabled.
605 Larger bucket sizes improve collision resolution but decrease
607 compression speed.
608 The minimum lblog is 1 and the maximum is 8 (default: 3).
610 ldmHashRateLog=lhrlog, lhrlog=lhrlog
612 Specify the frequency of inserting entries into the long dis‐
613 tance matching hash table.
614 This option is ignored unless long distance matching is enabled.
616 Larger values will improve compression speed. Deviating far from
618 the default value will likely result in a decrease in compres‐
619 sion ratio.
620 The default value is wlog - lhlog.
622 Example
624 The following parameters sets advanced compression options to something
625 similar to predefined level 19 for files bigger than 256 KB:
626 --zstd=wlog=23,clog=23,hlog=22,slog=6,mml=3,tlen=48,strat=6
628
630 Report bugs at: https://github.com/facebook/zstd/issues
631
633 Yann Collet
634zstd 1.5.0 May 2021 ZSTD(1)