1PARALLEL(1) parallel PARALLEL(1)
2
6 parallel - build and execute shell command lines from standard input in
7 parallel
8
10 parallel [options] [command [arguments]] < list_of_arguments
11 parallel [options] [command [arguments]] ( ::: arguments | :::+
13 arguments | :::: argfile(s) | ::::+ argfile(s) ) ...
14 parallel --semaphore [options] command
16 #!/usr/bin/parallel --shebang [options] [command [arguments]]
18 #!/usr/bin/parallel --shebang-wrap [options] [command [arguments]]
20
22 STOP!
23 Read the Reader's guide below if you are new to GNU parallel.
25 GNU parallel is a shell tool for executing jobs in parallel using one
27 or more computers. A job can be a single command or a small script that
28 has to be run for each of the lines in the input. The typical input is
29 a list of files, a list of hosts, a list of users, a list of URLs, or a
30 list of tables. A job can also be a command that reads from a pipe. GNU
31 parallel can then split the input into blocks and pipe a block into
32 each command in parallel.
33 If you use xargs and tee today you will find GNU parallel very easy to
35 use as GNU parallel is written to have the same options as xargs. If
36 you write loops in shell, you will find GNU parallel may be able to
37 replace most of the loops and make them run faster by running several
38 jobs in parallel.
39 GNU parallel makes sure output from the commands is the same output as
41 you would get had you run the commands sequentially. This makes it
42 possible to use output from GNU parallel as input for other programs.
43 For each line of input GNU parallel will execute command with the line
45 as arguments. If no command is given, the line of input is executed.
46 Several lines will be run in parallel. GNU parallel can often be used
47 as a substitute for xargs or cat | bash.
48 Reader's guide
50 Start by watching the intro videos for a quick introduction:
51 http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
52 Then look at the EXAMPLEs after the list of OPTIONS (Use
54 LESS=+/EXAMPLE\: man parallel). That will give you an idea of what GNU
55 parallel is capable of.
56 Then spend an hour walking through the tutorial (man
58 parallel_tutorial). Your command line will love you for it.
59 Finally you may want to look at the rest of this manual if you have
61 special needs not already covered.
62 If you want to know the design decisions behind GNU parallel, try: man
64 parallel_design. This is also a good intro if you intend to change GNU
65 parallel.
66
68 command
69 Command to execute. If command or the following arguments contain
70 replacement strings (such as {}) every instance will be substituted
71 with the input.
72 If command is given, GNU parallel solve the same tasks as xargs. If
74 command is not given GNU parallel will behave similar to cat | sh.
75 The command must be an executable, a script, a composed command, an
77 alias, or a function.
78 Bash functions: export -f the function first or use env_parallel.
80 Bash, Csh, or Tcsh aliases: Use env_parallel.
82 Zsh, Fish, Ksh, and Pdksh functions and aliases: Use env_parallel.
84 {} Input line. This replacement string will be replaced by a full line
86 read from the input source. The input source is normally stdin
87 (standard input), but can also be given with -a, :::, or ::::.
88 The replacement string {} can be changed with -I.
90 If the command line contains no replacement strings then {} will be
92 appended to the command line.
93 Replacement strings are normally quoted, so special characters are
95 not parsed by the shell. The exception is if the command starts
96 with a replacement string; then the string is not quoted.
97 {.} Input line without extension. This replacement string will be
99 replaced by the input with the extension removed. If the input line
100 contains . after the last / the last . till the end of the string
101 will be removed and {.} will be replaced with the remaining. E.g.
102 foo.jpg becomes foo, subdir/foo.jpg becomes subdir/foo,
103 sub.dir/foo.jpg becomes sub.dir/foo, sub.dir/bar remains
104 sub.dir/bar. If the input line does not contain . it will remain
105 unchanged.
106 The replacement string {.} can be changed with --er.
108 To understand replacement strings see {}.
110 {/} Basename of input line. This replacement string will be replaced by
112 the input with the directory part removed.
113 The replacement string {/} can be changed with --basenamereplace.
115 To understand replacement strings see {}.
117 {//}
119 Dirname of input line. This replacement string will be replaced by
120 the dir of the input line. See dirname(1).
121 The replacement string {//} can be changed with --dirnamereplace.
123 To understand replacement strings see {}.
125 {/.}
127 Basename of input line without extension. This replacement string
128 will be replaced by the input with the directory and extension part
129 removed. It is a combination of {/} and {.}.
130 The replacement string {/.} can be changed with
132 --basenameextensionreplace.
133 To understand replacement strings see {}.
135 {#} Sequence number of the job to run. This replacement string will be
137 replaced by the sequence number of the job being run. It contains
138 the same number as $PARALLEL_SEQ.
139 The replacement string {#} can be changed with --seqreplace.
141 To understand replacement strings see {}.
143 {%} Job slot number. This replacement string will be replaced by the
145 job's slot number between 1 and number of jobs to run in parallel.
146 There will never be 2 jobs running at the same time with the same
147 job slot number.
148 The replacement string {%} can be changed with --slotreplace.
150 To understand replacement strings see {}.
152 {n} Argument from input source n or the n'th argument. This positional
154 replacement string will be replaced by the input from input source
155 n (when used with -a or ::::) or with the n'th argument (when used
156 with -N). If n is negative it refers to the n'th last argument.
157 To understand replacement strings see {}.
159 {n.}
161 Argument from input source n or the n'th argument without
162 extension. It is a combination of {n} and {.}.
163 This positional replacement string will be replaced by the input
165 from input source n (when used with -a or ::::) or with the n'th
166 argument (when used with -N). The input will have the extension
167 removed.
168 To understand positional replacement strings see {n}.
170 {n/}
172 Basename of argument from input source n or the n'th argument. It
173 is a combination of {n} and {/}.
174 This positional replacement string will be replaced by the input
176 from input source n (when used with -a or ::::) or with the n'th
177 argument (when used with -N). The input will have the directory (if
178 any) removed.
179 To understand positional replacement strings see {n}.
181 {n//}
183 Dirname of argument from input source n or the n'th argument. It
184 is a combination of {n} and {//}.
185 This positional replacement string will be replaced by the dir of
187 the input from input source n (when used with -a or ::::) or with
188 the n'th argument (when used with -N). See dirname(1).
189 To understand positional replacement strings see {n}.
191 {n/.}
193 Basename of argument from input source n or the n'th argument
194 without extension. It is a combination of {n}, {/}, and {.}.
195 This positional replacement string will be replaced by the input
197 from input source n (when used with -a or ::::) or with the n'th
198 argument (when used with -N). The input will have the directory (if
199 any) and extension removed.
200 To understand positional replacement strings see {n}.
202 {=perl expression=}
204 Replace with calculated perl expression. $_ will contain the same
205 as {}. After evaluating perl expression $_ will be used as the
206 value. It is recommended to only change $_ but you have full access
207 to all of GNU parallel's internal functions and data structures. A
208 few convenience functions and data structures have been made:
209 Q(string) shell quote a string
211 pQ(string) perl quote a string
213 total_jobs() number of jobs in total
215 slot() slot number of job
217 seq() sequence number of job
219 @arg the arguments
221 Example:
223 seq 10 | parallel echo {} + 1 is {= '$_++' =}
225 parallel csh -c {= '$_="mkdir ".Q($_)' =} ::: '12" dir'
226 seq 50 | parallel echo job {#} of {= '$_=total_jobs()' =}
227 See also: --rpl --parens
229 {=n perl expression=}
231 Positional equivalent to {=perl expression=}. To understand
232 positional replacement strings see {n}.
233 See also: {=perl expression=} {n}.
235 ::: arguments
237 Use arguments from the command line as input source instead of
238 stdin (standard input). Unlike other options for GNU parallel :::
239 is placed after the command and before the arguments.
240 The following are equivalent:
242 (echo file1; echo file2) | parallel gzip
244 parallel gzip ::: file1 file2
245 parallel gzip {} ::: file1 file2
246 parallel --arg-sep ,, gzip {} ,, file1 file2
247 parallel --arg-sep ,, gzip ,, file1 file2
248 parallel ::: "gzip file1" "gzip file2"
249 To avoid treating ::: as special use --arg-sep to set the argument
251 separator to something else. See also --arg-sep.
252 If multiple ::: are given, each group will be treated as an input
254 source, and all combinations of input sources will be generated.
255 E.g. ::: 1 2 ::: a b c will result in the combinations (1,a) (1,b)
256 (1,c) (2,a) (2,b) (2,c). This is useful for replacing nested for-
257 loops.
258 ::: and :::: can be mixed. So these are equivalent:
260 parallel echo {1} {2} {3} ::: 6 7 ::: 4 5 ::: 1 2 3
262 parallel echo {1} {2} {3} :::: <(seq 6 7) <(seq 4 5) \
263 :::: <(seq 1 3)
264 parallel -a <(seq 6 7) echo {1} {2} {3} :::: <(seq 4 5) \
265 :::: <(seq 1 3)
266 parallel -a <(seq 6 7) -a <(seq 4 5) echo {1} {2} {3} \
267 ::: 1 2 3
268 seq 6 7 | parallel -a - -a <(seq 4 5) echo {1} {2} {3} \
269 ::: 1 2 3
270 seq 4 5 | parallel echo {1} {2} {3} :::: <(seq 6 7) - \
271 ::: 1 2 3
272 :::+ arguments
274 Like ::: but linked like --link to the previous input source.
275 Contrary to --link, values do not wrap: The shortest input source
277 determines the length.
278 Example:
280 parallel echo ::: a b c :::+ 1 2 3 ::: X Y :::+ 11 22
282 :::: argfiles
284 Another way to write -a argfile1 -a argfile2 ...
285 ::: and :::: can be mixed.
287 See -a, ::: and --link.
289 ::::+ argfiles
291 Like :::: but linked like --link to the previous input source.
292 Contrary to --link, values do not wrap: The shortest input source
294 determines the length.
295 --null
297 -0 Use NUL as delimiter. Normally input lines will end in \n
298 (newline). If they end in \0 (NUL), then use this option. It is
299 useful for processing arguments that may contain \n (newline).
300 --arg-file input-file
302 -a input-file
303 Use input-file as input source. If you use this option, stdin
304 (standard input) is given to the first process run. Otherwise,
305 stdin (standard input) is redirected from /dev/null.
306 If multiple -a are given, each input-file will be treated as an
308 input source, and all combinations of input sources will be
309 generated. E.g. The file foo contains 1 2, the file bar contains a
310 b c. -a foo -a bar will result in the combinations (1,a) (1,b)
311 (1,c) (2,a) (2,b) (2,c). This is useful for replacing nested for-
312 loops.
313 See also --link and {n}.
315 --arg-file-sep sep-str
317 Use sep-str instead of :::: as separator string between command and
318 argument files. Useful if :::: is used for something else by the
319 command.
320 See also: ::::.
322 --arg-sep sep-str
324 Use sep-str instead of ::: as separator string. Useful if ::: is
325 used for something else by the command.
326 Also useful if you command uses ::: but you still want to read
328 arguments from stdin (standard input): Simply change --arg-sep to a
329 string that is not in the command line.
330 See also: :::.
332 --bar
334 Show progress as a progress bar. In the bar is shown: % of jobs
335 completed, estimated seconds left, and number of jobs started.
336 It is compatible with zenity:
338 seq 1000 | parallel -j30 --bar '(echo {};sleep 0.1)' \
340 2> >(zenity --progress --auto-kill) | wc
341 --basefile file
343 --bf file
344 file will be transferred to each sshlogin before a job is started.
345 It will be removed if --cleanup is active. The file may be a script
346 to run or some common base data needed for the job. Multiple --bf
347 can be specified to transfer more basefiles. The file will be
348 transferred the same way as --transferfile.
349 --basenamereplace replace-str
351 --bnr replace-str
352 Use the replacement string replace-str instead of {/} for basename
353 of input line.
354 --basenameextensionreplace replace-str
356 --bner replace-str
357 Use the replacement string replace-str instead of {/.} for basename
358 of input line without extension.
359 --bg
361 Run command in background thus GNU parallel will not wait for
362 completion of the command before exiting. This is the default if
363 --semaphore is set.
364 See also: --fg, man sem.
366 Implies --semaphore.
368 --bibtex
370 --citation
371 Print the citation notice and BibTeX entry for GNU parallel,
372 silence citation notice for all future runs, and exit. It will not
373 run any commands.
374 If it is impossible for you to run --citation you can instead use
376 --will-cite, which will run commands, but which will only silence
377 the citation notice for this single run.
378 If you use --will-cite in scripts to be run by others you are
380 making it harder for others to see the citation notice. The
381 development of GNU parallel is indirectly financed through
382 citations, so if your users do not know they should cite then you
383 are making it harder to finance development. However, if you pay
384 10000 EUR, you have done your part to finance future development
385 and should feel free to use --will-cite in scripts.
386 --block size
388 --block-size size
389 Size of block in bytes to read at a time. The size can be postfixed
390 with K, M, G, T, P, E, k, m, g, t, p, or e which would multiply the
391 size with 1024, 1048576, 1073741824, 1099511627776,
392 1125899906842624, 1152921504606846976, 1000, 1000000, 1000000000,
393 1000000000000, 1000000000000000, or 1000000000000000000
394 respectively.
395 GNU parallel tries to meet the block size but can be off by the
397 length of one record. For performance reasons size should be bigger
398 than a two records. GNU parallel will warn you and automatically
399 increase the size if you choose a size that is too small.
400 If you use -N, --block-size should be bigger than N+1 records.
402 size defaults to 1M.
404 When using --pipepart a negative block size is not interpreted as a
406 blocksize but as the number of blocks each jobslot should have. So
407 this will run 10*5 = 50 jobs in total:
408 parallel --pipepart -a myfile --block -10 -j5 wc
410 This is an efficient alternative to --round-robin because data is
412 never read by GNU parallel, but you can still have very few
413 jobslots process a large amount of data.
414 See --pipe and --pipepart for use of this.
416 --cat
418 Create a temporary file with content. Normally --pipe/--pipepart
419 will give data to the program on stdin (standard input). With --cat
420 GNU parallel will create a temporary file with the name in {}, so
421 you can do: parallel --pipe --cat wc {}.
422 Implies --pipe unless --pipepart is used.
424 See also --fifo.
426 --cleanup
428 Remove transferred files. --cleanup will remove the transferred
429 files on the remote computer after processing is done.
430 find log -name '*gz' | parallel \
432 --sshlogin server.example.com --transferfile {} \
433 --return {.}.bz2 --cleanup "zcat {} | bzip -9 >{.}.bz2"
434 With --transferfile {} the file transferred to the remote computer
436 will be removed on the remote computer. Directories created will
437 not be removed - even if they are empty.
438 With --return the file transferred from the remote computer will be
440 removed on the remote computer. Directories created will not be
441 removed - even if they are empty.
442 --cleanup is ignored when not used with --transferfile or --return.
444 --colsep regexp
446 -C regexp
447 Column separator. The input will be treated as a table with regexp
448 separating the columns. The n'th column can be accessed using {n}
449 or {n.}. E.g. {3} is the 3rd column.
450 If there are more input sources, each input source will be
452 separated, but the columns from each input source will be linked
453 (see --link).
454 parallel --colsep '-' echo {4} {3} {2} {1} \
456 ::: A-B C-D ::: e-f g-h
457 --colsep implies --trim rl, which can be overridden with --trim n.
459 regexp is a Perl Regular Expression:
461 http://perldoc.perl.org/perlre.html
462 --compress
464 Compress temporary files. If the output is big and very
465 compressible this will take up less disk space in $TMPDIR and
466 possibly be faster due to less disk I/O.
467 GNU parallel will try pzstd, lbzip2, pbzip2, zstd, pigz, lz4, lzop,
469 plzip, lzip, lrz, gzip, pxz, lzma, bzip2, xz, clzip, in that order,
470 and use the first available.
471 --compress-program prg
473 --decompress-program prg
474 Use prg for (de)compressing temporary files. It is assumed that prg
475 -dc will decompress stdin (standard input) to stdout (standard
476 output) unless --decompress-program is given.
477 --delimiter delim
479 -d delim
480 Input items are terminated by delim. Quotes and backslash are not
481 special; every character in the input is taken literally. Disables
482 the end-of-file string, which is treated like any other argument.
483 The specified delimiter may be characters, C-style character
484 escapes such as \n, or octal or hexadecimal escape codes. Octal
485 and hexadecimal escape codes are understood as for the printf
486 command. Multibyte characters are not supported.
487 --dirnamereplace replace-str
489 --dnr replace-str
490 Use the replacement string replace-str instead of {//} for dirname
491 of input line.
492 -E eof-str
494 Set the end of file string to eof-str. If the end of file string
495 occurs as a line of input, the rest of the input is not read. If
496 neither -E nor -e is used, no end of file string is used.
497 --delay mytime
499 Delay starting next job by mytime. GNU parallel will pause mytime
500 after starting each job. mytime is normally in seconds, but can be
501 floats postfixed with s, m, h, or d which would multiply the float
502 by 1, 60, 3600, or 86400. Thus these are equivalent: --delay 100000
503 and --delay 1d3.5h16.6m4s.
504 --dry-run
506 Print the job to run on stdout (standard output), but do not run
507 the job. Use -v -v to include the wrapping that GNU Parallel
508 generates (for remote jobs, --tmux, --nice, --pipe, --pipepart,
509 --fifo and --cat). Do not count on this literaly, though, as the
510 job may be scheduled on another computer or the local computer if :
511 is in the list.
512 --eof[=eof-str]
514 -e[eof-str]
515 This option is a synonym for the -E option. Use -E instead,
516 because it is POSIX compliant for xargs while this option is not.
517 If eof-str is omitted, there is no end of file string. If neither
518 -E nor -e is used, no end of file string is used.
519 --embed (alpha testing)
521 Embed GNU parallel in a shell script. If you need to distribute
522 your script to someone who does not want to install GNU parallel
523 you can embed GNU parallel in your own shell script:
524 parallel --embed > new_script
526 After which you add your code at the end of new_script. This is
528 tested on ash, bash, dash, ksh, sh, and zsh.
529 --env var
531 Copy environment variable var. This will copy var to the
532 environment that the command is run in. This is especially useful
533 for remote execution.
534 In Bash var can also be a Bash function - just remember to export
536 -f the function, see command.
537 The variable '_' is special. It will copy all exported environment
539 variables except for the ones mentioned in
540 ~/.parallel/ignored_vars.
541 To copy the full environment (both exported and not exported
543 variables, arrays, and functions) use env_parallel.
544 See also: --record-env.
546 --eta
548 Show the estimated number of seconds before finishing. This forces
549 GNU parallel to read all jobs before starting to find the number of
550 jobs. GNU parallel normally only reads the next job to run.
551 The estimate is based on the runtime of finished jobs, so the first
553 estimate will only be shown when the first job has finished.
554 Implies --progress.
556 See also: --bar, --progress.
558 --fg
560 Run command in foreground.
561 With --tmux and --tmuxpane GNU parallel will start tmux in the
563 foreground.
564 With --semaphore GNU parallel will run the command in the
566 foreground (opposite --bg), and wait for completion of the command
567 before exiting.
568 See also --bg, man sem.
570 --fifo
572 Create a temporary fifo with content. Normally --pipe and
573 --pipepart will give data to the program on stdin (standard input).
574 With --fifo GNU parallel will create a temporary fifo with the name
575 in {}, so you can do: parallel --pipe --fifo wc {}.
576 Beware: If data is not read from the fifo, the job will block
578 forever.
579 Implies --pipe unless --pipepart is used.
581 See also --cat.
583 --filter-hosts
585 Remove down hosts. For each remote host: check that login through
586 ssh works. If not: do not use this host.
587 For performance reasons, this check is performed only at the start
589 and every time --sshloginfile is changed. If an host goes down
590 after the first check, it will go undetected until --sshloginfile
591 is changed; --retries can be used to mitigate this.
592 Currently you can not put --filter-hosts in a profile, $PARALLEL,
594 /etc/parallel/config or similar. This is because GNU parallel uses
595 GNU parallel to compute this, so you will get an infinite loop.
596 This will likely be fixed in a later release.
597 --gnu
599 Behave like GNU parallel. This option historically took precedence
600 over --tollef. The --tollef option is now retired, and therefore
601 may not be used. --gnu is kept for compatibility.
602 --group
604 Group output. Output from each job is grouped together and is only
605 printed when the command is finished. stdout (standard output)
606 first followed by stderr (standard error).
607 This takes in the order of 0.5ms per job and depends on the speed
609 of your disk for larger output. It can be disabled with -u, but
610 this means output from different commands can get mixed.
611 --group is the default. Can be reversed with -u.
613 See also: --line-buffer --ungroup
615 --help
617 -h Print a summary of the options to GNU parallel and exit.
618 --halt-on-error val
620 --halt val
621 When should GNU parallel terminate? In some situations it makes no
622 sense to run all jobs. GNU parallel should simply give up as soon
623 as a condition is met.
624 val defaults to never, which runs all jobs no matter what.
626 val can also take on the form of when,why.
628 when can be 'now' which means kill all running jobs and halt
630 immediately, or it can be 'soon' which means wait for all running
631 jobs to complete, but start no new jobs.
632 why can be 'fail=X', 'fail=Y%', 'success=X', 'success=Y%',
634 'done=X', or 'done=Y%' where X is the number of jobs that has to
635 fail, succeed, or be done before halting, and Y is the percentage
636 of jobs that has to fail, succeed, or be done before halting.
637 Example:
639 --halt now,fail=1 exit when the first job fails. Kill running
641 jobs.
642 --halt soon,fail=3 exit when 3 jobs fail, but wait for running
644 jobs to complete.
645 --halt soon,fail=3% exit when 3% of the jobs have failed, but
647 wait for running jobs to complete.
648 --halt now,success=1 exit when a job succeeds. Kill running jobs.
650 --halt soon,success=3 exit when 3 jobs succeeds, but wait for
652 running jobs to complete.
653 --halt now,success=3% exit when 3% of the jobs have succeeded.
655 Kill running jobs.
656 --halt now,done=1 exit when one of the jobs finishes. Kill
658 running jobs.
659 --halt soon,done=3 exit when 3 jobs finishes, but wait for
661 running jobs to complete.
662 --halt now,done=3% exit when 3% of the jobs have finished. Kill
664 running jobs.
665 For backwards compability these also work:
667 0 never
669 1 soon,fail=1
671 2 now,fail=1
673 -1 soon,success=1
675 -2 now,success=1
677 1-99% soon,fail=1-99%
679 --header regexp
681 Use regexp as header. For normal usage the matched header
682 (typically the first line: --header '.*\n') will be split using
683 --colsep (which will default to '\t') and column names can be used
684 as replacement variables: {column name}, {column name/}, {column
685 name//}, {column name/.}, {column name.}, {=column name perl
686 expression =}, ..
687 For --pipe the matched header will be prepended to each output.
689 --header : is an alias for --header '.*\n'.
691 If regexp is a number, it is a fixed number of lines.
693 --hostgroups
695 --hgrp
696 Enable hostgroups on arguments. If an argument contains '@' the
697 string after '@' will be removed and treated as a list of
698 hostgroups on which this job is allowed to run. If there is no
699 --sshlogin with a corresponding group, the job will run on any
700 hostgroup.
701 Example:
703 parallel --hostgroups \
705 --sshlogin @grp1/myserver1 -S @grp1+grp2/myserver2 \
706 --sshlogin @grp3/myserver3 \
707 echo ::: my_grp1_arg@grp1 arg_for_grp2@grp2 third@grp1+grp3
708 my_grp1_arg may be run on either myserver1 or myserver2, third may
710 be run on either myserver1 or myserver3, but arg_for_grp2 will only
711 be run on myserver2.
712 See also: --sshlogin.
714 -I replace-str
716 Use the replacement string replace-str instead of {}.
717 --replace[=replace-str]
719 -i[replace-str]
720 This option is a synonym for -Ireplace-str if replace-str is
721 specified, and for -I {} otherwise. This option is deprecated; use
722 -I instead.
723 --joblog logfile
725 Logfile for executed jobs. Save a list of the executed jobs to
726 logfile in the following TAB separated format: sequence number,
727 sshlogin, start time as seconds since epoch, run time in seconds,
728 bytes in files transferred, bytes in files returned, exit status,
729 signal, and command run.
730 For --pipe bytes transferred and bytes returned are number of input
732 and output of bytes.
733 If logfile is prepended with '+' log lines will be appended to the
735 logfile.
736 To convert the times into ISO-8601 strict do:
738 cat logfile | perl -a -F"\t" -ne \
740 'chomp($F[2]=`date -d \@$F[2] +%FT%T`); print join("\t",@F)'
741 If the host is long, you can use column -t to pretty print it:
743 cat joblog | column -t
745 See also --resume --resume-failed.
747 --jobs N
749 -j N
750 --max-procs N
751 -P N
752 Number of jobslots on each machine. Run up to N jobs in parallel.
753 0 means as many as possible. Default is 100% which will run one job
754 per CPU core on each machine.
755 If --semaphore is set, the default is 1 thus making a mutex.
757 --jobs +N
759 -j +N
760 --max-procs +N
761 -P +N
762 Add N to the number of CPU cores. Run this many jobs in parallel.
763 See also --use-cpus-instead-of-cores.
764 --jobs -N
766 -j -N
767 --max-procs -N
768 -P -N
769 Subtract N from the number of CPU cores. Run this many jobs in
770 parallel. If the evaluated number is less than 1 then 1 will be
771 used. See also --use-cpus-instead-of-cores.
772 --jobs N%
774 -j N%
775 --max-procs N%
776 -P N%
777 Multiply N% with the number of CPU cores. Run this many jobs in
778 parallel. See also --use-cpus-instead-of-cores.
779 --jobs procfile
781 -j procfile
782 --max-procs procfile
783 -P procfile
784 Read parameter from file. Use the content of procfile as parameter
785 for -j. E.g. procfile could contain the string 100% or +2 or 10. If
786 procfile is changed when a job completes, procfile is read again
787 and the new number of jobs is computed. If the number is lower than
788 before, running jobs will be allowed to finish but new jobs will
789 not be started until the wanted number of jobs has been reached.
790 This makes it possible to change the number of simultaneous running
791 jobs while GNU parallel is running.
792 --keep-order
794 -k Keep sequence of output same as the order of input. Normally the
795 output of a job will be printed as soon as the job completes. Try
796 this to see the difference:
797 parallel -j4 sleep {}\; echo {} ::: 2 1 4 3
799 parallel -j4 -k sleep {}\; echo {} ::: 2 1 4 3
800 If used with --onall or --nonall the output will grouped by
802 sshlogin in sorted order.
803 If used with --pipe --roundrobin and the same input, the jobslots
805 will get the same blocks in the same order in every run.
806 -L max-lines
808 When used with --pipe: Read records of max-lines.
809 When used otherwise: Use at most max-lines nonblank input lines per
811 command line. Trailing blanks cause an input line to be logically
812 continued on the next input line.
813 -L 0 means read one line, but insert 0 arguments on the command
815 line.
816 Implies -X unless -m, --xargs, or --pipe is set.
818 --max-lines[=max-lines]
820 -l[max-lines]
821 When used with --pipe: Read records of max-lines.
822 When used otherwise: Synonym for the -L option. Unlike -L, the
824 max-lines argument is optional. If max-lines is not specified, it
825 defaults to one. The -l option is deprecated since the POSIX
826 standard specifies -L instead.
827 -l 0 is an alias for -l 1.
829 Implies -X unless -m, --xargs, or --pipe is set.
831 --limit "command args"
833 Dynamic job limit. Before starting a new job run command with args.
834 The exit value of command determines what GNU parallel will do:
835 0 Below limit. Start another job.
837 1 Over limit. Start no jobs.
839 2 Way over limit. Kill the youngest job.
841 You can use any shell command. There are 3 predefined commands:
843 "io n" Limit for I/O. The amount of disk I/O will be computed as
845 a value 0-100, where 0 is no I/O and 100 is at least one
846 disk is 100% saturated.
847 "load n" Similar to --load.
849 "mem n" Similar to --memfree.
851 --line-buffer
853 --lb
854 Buffer output on line basis. --group will keep the output together
855 for a whole job. --ungroup allows output to mixup with half a line
856 coming from one job and half a line coming from another job.
857 --line-buffer fits between these two: GNU parallel will print a
858 full line, but will allow for mixing lines of different jobs.
859 --line-buffer takes more CPU power than both --group and --ungroup,
861 but can be much faster than --group if the CPU is not the limiting
862 factor.
863 Normally --line-buffer does not buffer on disk, and can thus
865 process an infinite amount of data, but it will buffer on disk when
866 combined with: --keep-order, --results, --compress, and --files.
867 This will make it as slow as --group and will limit output to the
868 available disk space.
869 With --keep-order --line-buffer will output lines from the first
871 job while it is running, then lines from the second job while that
872 is running. It will buffer full lines, but jobs will not mix.
873 Compare:
874 parallel -j0 'echo {};sleep {};echo {}' ::: 1 3 2 4
876 parallel -j0 --lb 'echo {};sleep {};echo {}' ::: 1 3 2 4
877 parallel -j0 -k --lb 'echo {};sleep {};echo {}' ::: 1 3 2 4
878 See also: --group --ungroup
880 --xapply
882 --link
883 Link input sources. Read multiple input sources like xapply. If
884 multiple input sources are given, one argument will be read from
885 each of the input sources. The arguments can be accessed in the
886 command as {1} .. {n}, so {1} will be a line from the first input
887 source, and {6} will refer to the line with the same line number
888 from the 6th input source.
889 Compare these two:
891 parallel echo {1} {2} ::: 1 2 3 ::: a b c
893 parallel --link echo {1} {2} ::: 1 2 3 ::: a b c
894 Arguments will be recycled if one input source has more arguments
896 than the others:
897 parallel --link echo {1} {2} {3} \
899 ::: 1 2 ::: I II III ::: a b c d e f g
900 See also --header, :::+, ::::+.
902 --load max-load
904 Do not start new jobs on a given computer unless the number of
905 running processes on the computer is less than max-load. max-load
906 uses the same syntax as --jobs, so 100% for one per CPU is a valid
907 setting. Only difference is 0 which is interpreted as 0.01.
908 --controlmaster
910 -M Use ssh's ControlMaster to make ssh connections faster. Useful if
911 jobs run remote and are very fast to run. This is disabled for
912 sshlogins that specify their own ssh command.
913 --xargs
915 Multiple arguments. Insert as many arguments as the command line
916 length permits.
917 If {} is not used the arguments will be appended to the line. If
919 {} is used multiple times each {} will be replaced with all the
920 arguments.
921 Support for --xargs with --sshlogin is limited and may fail.
923 See also -X for context replace. If in doubt use -X as that will
925 most likely do what is needed.
926 -m Multiple arguments. Insert as many arguments as the command line
928 length permits. If multiple jobs are being run in parallel:
929 distribute the arguments evenly among the jobs. Use -j1 or --xargs
930 to avoid this.
931 If {} is not used the arguments will be appended to the line. If
933 {} is used multiple times each {} will be replaced with all the
934 arguments.
935 Support for -m with --sshlogin is limited and may fail.
937 See also -X for context replace. If in doubt use -X as that will
939 most likely do what is needed.
940 --memfree size
942 Minimum memory free when starting another job. The size can be
943 postfixed with K, M, G, T, P, k, m, g, t, or p which would multiply
944 the size with 1024, 1048576, 1073741824, 1099511627776,
945 1125899906842624, 1000, 1000000, 1000000000, 1000000000000, or
946 1000000000000000, respectively.
947 If the jobs take up very different amount of RAM, GNU parallel will
949 only start as many as there is memory for. If less than size bytes
950 are free, no more jobs will be started. If less than 50% size bytes
951 are free, the youngest job will be killed, and put back on the
952 queue to be run later.
953 --retries must be set to determine how many times GNU parallel
955 should retry a given job.
956 --minversion version
958 Print the version GNU parallel and exit. If the current version of
959 GNU parallel is less than version the exit code is 255. Otherwise
960 it is 0.
961 This is useful for scripts that depend on features only available
963 from a certain version of GNU parallel.
964 --nonall
966 --onall with no arguments. Run the command on all computers given
967 with --sshlogin but take no arguments. GNU parallel will log into
968 --jobs number of computers in parallel and run the job on the
969 computer. -j adjusts how many computers to log into in parallel.
970 This is useful for running the same command (e.g. uptime) on a list
972 of servers.
973 --onall
975 Run all the jobs on all computers given with --sshlogin. GNU
976 parallel will log into --jobs number of computers in parallel and
977 run one job at a time on the computer. The order of the jobs will
978 not be changed, but some computers may finish before others.
979 When using --group the output will be grouped by each server, so
981 all the output from one server will be grouped together.
982 --joblog will contain an entry for each job on each server, so
984 there will be several job sequence 1.
985 --output-as-files
987 --outputasfiles
988 --files
989 Instead of printing the output to stdout (standard output) the
990 output of each job is saved in a file and the filename is then
991 printed.
992 See also: --results
994 --pipe
996 --spreadstdin
997 Spread input to jobs on stdin (standard input). Read a block of
998 data from stdin (standard input) and give one block of data as
999 input to one job.
1000 The block size is determined by --block. The strings --recstart and
1002 --recend tell GNU parallel how a record starts and/or ends. The
1003 block read will have the final partial record removed before the
1004 block is passed on to the job. The partial record will be prepended
1005 to next block.
1006 If --recstart is given this will be used to split at record start.
1008 If --recend is given this will be used to split at record end.
1010 If both --recstart and --recend are given both will have to match
1012 to find a split position.
1013 If neither --recstart nor --recend are given --recend defaults to
1015 '\n'. To have no record separator use --recend "".
1016 --files is often used with --pipe.
1018 --pipe maxes out at around 1 GB/s input, and 100 MB/s output. If
1020 performance is important use --pipepart.
1021 See also: --recstart, --recend, --fifo, --cat, --pipepart, --files.
1023 --pipepart
1025 Pipe parts of a physical file. --pipepart works similar to --pipe,
1026 but is much faster.
1027 --pipepart has a few limitations:
1029 · The file must be a normal file or a block device (technically it
1031 must be seekable) and must be given using -a or ::::. The file
1032 cannot be a pipe or a fifo as they are not seekable.
1033 If using a block device with lot of NUL bytes, remember to set
1035 --recend ''.
1036 · Record counting (-N) and line counting (-L/-l) do not work.
1038 --plain
1040 Ignore any --profile, $PARALLEL, and ~/.parallel/config to get full
1041 control on the command line (used by GNU parallel internally when
1042 called with --sshlogin).
1043 --plus
1045 Activate additional replacement strings: {+/} {+.} {+..} {+...}
1046 {..} {...} {/..} {/...} {##}. The idea being that '{+foo}' matches
1047 the opposite of '{foo}' and {} = {+/}/{/} = {.}.{+.} =
1048 {+/}/{/.}.{+.} = {..}.{+..} = {+/}/{/..}.{+..} = {...}.{+...} =
1049 {+/}/{/...}.{+...}
1050 {##} is the number of jobs to be run. It is incompatible with
1052 -X/-m/--xargs.
1053 {choose_k} is inspired by n choose k: Given a list of n elements,
1055 choose k. k is the number of input sources and n is the number of
1056 arguments in an input source. The content of the input sources
1057 must be the same and the arguments must be unique.
1058 The following dynamic replacement strings are also activated. They
1060 are inspired by bash's parameter expansion:
1061 {:-str} str if the value is empty
1063 {:num} remove the first num characters
1064 {:num1:num2} characters from num1 to num2
1065 {#str} remove prefix str
1066 {%str} remove postfix str
1067 {/str1/str2} replace str1 with str2
1068 {^str} uppercase str if found at the start
1069 {^^str} uppercase str
1070 {,str} lowercase str if found at the start
1071 {,,str} lowercase str
1072 --progress
1074 Show progress of computations. List the computers involved in the
1075 task with number of CPU cores detected and the max number of jobs
1076 to run. After that show progress for each computer: number of
1077 running jobs, number of completed jobs, and percentage of all jobs
1078 done by this computer. The percentage will only be available after
1079 all jobs have been scheduled as GNU parallel only read the next job
1080 when ready to schedule it - this is to avoid wasting time and
1081 memory by reading everything at startup.
1082 By sending GNU parallel SIGUSR2 you can toggle turning on/off
1084 --progress on a running GNU parallel process.
1085 See also --eta and --bar.
1087 --max-args=max-args
1089 -n max-args
1090 Use at most max-args arguments per command line. Fewer than max-
1091 args arguments will be used if the size (see the -s option) is
1092 exceeded, unless the -x option is given, in which case GNU parallel
1093 will exit.
1094 -n 0 means read one argument, but insert 0 arguments on the command
1096 line.
1097 Implies -X unless -m is set.
1099 --max-replace-args=max-args
1101 -N max-args
1102 Use at most max-args arguments per command line. Like -n but also
1103 makes replacement strings {1} .. {max-args} that represents
1104 argument 1 .. max-args. If too few args the {n} will be empty.
1105 -N 0 means read one argument, but insert 0 arguments on the command
1107 line.
1108 This will set the owner of the homedir to the user:
1110 tr ':' '\n' < /etc/passwd | parallel -N7 chown {1} {6}
1112 Implies -X unless -m or --pipe is set.
1114 When used with --pipe -N is the number of records to read. This is
1116 somewhat slower than --block.
1117 --max-line-length-allowed
1119 Print the maximal number of characters allowed on the command line
1120 and exit (used by GNU parallel itself to determine the line length
1121 on remote computers).
1122 --number-of-cpus
1124 Print the number of physical CPUs and exit (used by GNU parallel
1125 itself to determine the number of physical CPUs on remote
1126 computers).
1127 --number-of-cores
1129 Print the number of CPU cores and exit (used by GNU parallel itself
1130 to determine the number of CPU cores on remote computers).
1131 --no-keep-order
1133 Overrides an earlier --keep-order (e.g. if set in
1134 ~/.parallel/config).
1135 --nice niceness
1137 Run the command at this niceness. For simple commands you can just
1138 add nice in front of the command. But if the command consists of
1139 more sub commands (Like: ls|wc) then prepending nice will not
1140 always work. --nice will make sure all sub commands are niced -
1141 even on remote servers.
1142 --interactive
1144 -p Prompt the user about whether to run each command line and read a
1145 line from the terminal. Only run the command line if the response
1146 starts with 'y' or 'Y'. Implies -t.
1147 --parens parensstring
1149 Define start and end parenthesis for {= perl expression =}. The
1150 left and the right parenthesis can be multiple characters and are
1151 assumed to be the same length. The default is {==} giving {= as the
1152 start parenthesis and =} as the end parenthesis.
1153 Another useful setting is ,,,, which would make both parenthesis
1155 ,,:
1156 parallel --parens ,,,, echo foo is ,,s/I/O/g,, ::: FII
1158 See also: --rpl {= perl expression =}
1160 --profile profilename
1162 -J profilename
1163 Use profile profilename for options. This is useful if you want to
1164 have multiple profiles. You could have one profile for running jobs
1165 in parallel on the local computer and a different profile for
1166 running jobs on remote computers. See the section PROFILE FILES for
1167 examples.
1168 profilename corresponds to the file ~/.parallel/profilename.
1170 You can give multiple profiles by repeating --profile. If parts of
1172 the profiles conflict, the later ones will be used.
1173 Default: config
1175 --quote
1177 -q Quote command. This will quote the command line so special
1178 characters are not interpreted by the shell. See the section
1179 QUOTING. Most people will never need this. Quoting is disabled by
1180 default.
1181 --no-run-if-empty
1183 -r If the stdin (standard input) only contains whitespace, do not run
1184 the command.
1185 If used with --pipe this is slow.
1187 --noswap
1189 Do not start new jobs on a given computer if there is both swap-in
1190 and swap-out activity.
1191 The swap activity is only sampled every 10 seconds as the sampling
1193 takes 1 second to do.
1194 Swap activity is computed as (swap-in)*(swap-out) which in practice
1196 is a good value: swapping out is not a problem, swapping in is not
1197 a problem, but both swapping in and out usually indicates a
1198 problem.
1199 --memfree may give better results, so try using that first.
1201 --record-env
1203 Record current environment variables in ~/.parallel/ignored_vars.
1204 This is useful before using --env _.
1205 See also --env.
1207 --recstart startstring
1209 --recend endstring
1210 If --recstart is given startstring will be used to split at record
1211 start.
1212 If --recend is given endstring will be used to split at record end.
1214 If both --recstart and --recend are given the combined string
1216 endstringstartstring will have to match to find a split position.
1217 This is useful if either startstring or endstring match in the
1218 middle of a record.
1219 If neither --recstart nor --recend are given then --recend defaults
1221 to '\n'. To have no record separator use --recend "".
1222 --recstart and --recend are used with --pipe.
1224 Use --regexp to interpret --recstart and --recend as regular
1226 expressions. This is slow, however.
1227 --regexp
1229 Use --regexp to interpret --recstart and --recend as regular
1230 expressions. This is slow, however.
1231 --remove-rec-sep
1233 --removerecsep
1234 --rrs
1235 Remove the text matched by --recstart and --recend before piping it
1236 to the command.
1237 Only used with --pipe.
1239 --results name
1241 --res name
1242 Save the output into files.
1243 Simple string output dir
1245 If name does not contain replacement strings and does not end in
1247 .csv/.tsv, the output will be stored in a directory tree rooted at
1248 name. Within this directory tree, each command will result in
1249 three files: name/<ARGS>/stdout and name/<ARGS>/stderr,
1250 name/<ARGS>/seq, where <ARGS> is a sequence of directories
1251 representing the header of the input source (if using --header :)
1252 or the number of the input source and corresponding values.
1253 E.g:
1255 parallel --header : --results foo echo {a} {b} \
1257 ::: a I II ::: b III IIII
1258 will generate the files:
1260 foo/a/II/b/III/seq
1262 foo/a/II/b/III/stderr
1263 foo/a/II/b/III/stdout
1264 foo/a/II/b/IIII/seq
1265 foo/a/II/b/IIII/stderr
1266 foo/a/II/b/IIII/stdout
1267 foo/a/I/b/III/seq
1268 foo/a/I/b/III/stderr
1269 foo/a/I/b/III/stdout
1270 foo/a/I/b/IIII/seq
1271 foo/a/I/b/IIII/stderr
1272 foo/a/I/b/IIII/stdout
1273 and
1275 parallel --results foo echo {1} {2} ::: I II ::: III IIII
1277 will generate the files:
1279 foo/1/II/2/III/seq
1281 foo/1/II/2/III/stderr
1282 foo/1/II/2/III/stdout
1283 foo/1/II/2/IIII/seq
1284 foo/1/II/2/IIII/stderr
1285 foo/1/II/2/IIII/stdout
1286 foo/1/I/2/III/seq
1287 foo/1/I/2/III/stderr
1288 foo/1/I/2/III/stdout
1289 foo/1/I/2/IIII/seq
1290 foo/1/I/2/IIII/stderr
1291 foo/1/I/2/IIII/stdout
1292 CSV file output
1294 If name ends in .csv/.tsv the output will be a CSV-file named name.
1296 .csv gives a comma separated value file. .tsv gives a TAB separated
1298 value file.
1299 -.csv/-.tsv are special: It will give the file on stdout (standard
1301 output).
1302 Replacement string output file
1304 If name contains a replacement string and the replaced result does
1306 not end in /, then the standard output will be stored in a file
1307 named by this result. Standard error will be stored in the same
1308 file name with '.err' added, and the sequence number will be stored
1309 in the same file name with '.seq' added.
1310 E.g.
1312 parallel --results my_{} echo ::: foo bar baz
1314 will generate the files:
1316 my_bar
1318 my_bar.err
1319 my_bar.seq
1320 my_baz
1321 my_baz.err
1322 my_baz.seq
1323 my_foo
1324 my_foo.err
1325 my_foo.seq
1326 Replacement string output dir
1328 If name contains a replacement string and the replaced result ends
1330 in /, then output files will be stored in the resulting dir.
1331 E.g.
1333 parallel --results my_{}/ echo ::: foo bar baz
1335 will generate the files:
1337 my_bar/seq
1339 my_bar/stderr
1340 my_bar/stdout
1341 my_baz/seq
1342 my_baz/stderr
1343 my_baz/stdout
1344 my_foo/seq
1345 my_foo/stderr
1346 my_foo/stdout
1347 See also --files, --tag, --header, --joblog.
1349 --resume
1351 Resumes from the last unfinished job. By reading --joblog or the
1352 --results dir GNU parallel will figure out the last unfinished job
1353 and continue from there. As GNU parallel only looks at the sequence
1354 numbers in --joblog then the input, the command, and --joblog all
1355 have to remain unchanged; otherwise GNU parallel may run wrong
1356 commands.
1357 See also --joblog, --results, --resume-failed, --retries.
1359 --resume-failed
1361 Retry all failed and resume from the last unfinished job. By
1362 reading --joblog GNU parallel will figure out the failed jobs and
1363 run those again. After that it will resume last unfinished job and
1364 continue from there. As GNU parallel only looks at the sequence
1365 numbers in --joblog then the input, the command, and --joblog all
1366 have to remain unchanged; otherwise GNU parallel may run wrong
1367 commands.
1368 See also --joblog, --resume, --retry-failed, --retries.
1370 --retry-failed
1372 Retry all failed jobs in joblog. By reading --joblog GNU parallel
1373 will figure out the failed jobs and run those again.
1374 --retry-failed ignores the command and arguments on the command
1376 line: It only looks at the joblog.
1377 Differences between --resume, --resume-failed, --retry-failed
1379 In this example exit {= $_%=2 =} will cause every other job to
1381 fail.
1382 timeout -k 1 4 parallel --joblog log -j10 \
1384 'sleep {}; exit {= $_%=2 =}' ::: {10..1}
1385 4 jobs completed. 2 failed:
1387 Seq [...] Exitval Signal Command
1389 10 [...] 1 0 sleep 1; exit 1
1390 9 [...] 0 0 sleep 2; exit 0
1391 8 [...] 1 0 sleep 3; exit 1
1392 7 [...] 0 0 sleep 4; exit 0
1393 --resume does not care about the Exitval, but only looks at Seq. If
1395 the Seq is run, it will not be run again. So if needed, you can
1396 change the command for the seqs not run yet:
1397 parallel --resume --joblog log -j10 \
1399 'sleep .{}; exit {= $_%=2 =}' ::: {10..1}
1400 Seq [...] Exitval Signal Command
1402 [... as above ...]
1403 1 [...] 0 0 sleep .10; exit 0
1404 6 [...] 1 0 sleep .5; exit 1
1405 5 [...] 0 0 sleep .6; exit 0
1406 4 [...] 1 0 sleep .7; exit 1
1407 3 [...] 0 0 sleep .8; exit 0
1408 2 [...] 1 0 sleep .9; exit 1
1409 --resume-failed cares about the Exitval, but also only looks at Seq
1411 to figure out which commands to run. Again this means you can
1412 change the command, but not the arguments. It will run the failed
1413 seqs and the seqs not yet run:
1414 parallel --resume-failed --joblog log -j10 \
1416 'echo {};sleep .{}; exit {= $_%=3 =}' ::: {10..1}
1417 Seq [...] Exitval Signal Command
1419 [... as above ...]
1420 10 [...] 1 0 echo 1;sleep .1; exit 1
1421 8 [...] 0 0 echo 3;sleep .3; exit 0
1422 6 [...] 2 0 echo 5;sleep .5; exit 2
1423 4 [...] 1 0 echo 7;sleep .7; exit 1
1424 2 [...] 0 0 echo 9;sleep .9; exit 0
1425 --retry-failed cares about the Exitval, but takes the command from
1427 the joblog. It ignores any arguments or commands given on the
1428 command line:
1429 parallel --retry-failed --joblog log -j10 this part is ignored
1431 Seq [...] Exitval Signal Command
1433 [... as above ...]
1434 10 [...] 1 0 echo 1;sleep .1; exit 1
1435 6 [...] 2 0 echo 5;sleep .5; exit 2
1436 4 [...] 1 0 echo 7;sleep .7; exit 1
1437 See also --joblog, --resume, --resume-failed, --retries.
1439 --retries n
1441 If a job fails, retry it on another computer on which it has not
1442 failed. Do this n times. If there are fewer than n computers in
1443 --sshlogin GNU parallel will re-use all the computers. This is
1444 useful if some jobs fail for no apparent reason (such as network
1445 failure).
1446 --return filename
1448 Transfer files from remote computers. --return is used with
1449 --sshlogin when the arguments are files on the remote computers.
1450 When processing is done the file filename will be transferred from
1451 the remote computer using rsync and will be put relative to the
1452 default login dir. E.g.
1453 echo foo/bar.txt | parallel --return {.}.out \
1455 --sshlogin server.example.com touch {.}.out
1456 This will transfer the file $HOME/foo/bar.out from the computer
1458 server.example.com to the file foo/bar.out after running touch
1459 foo/bar.out on server.example.com.
1460 parallel -S server --trc out/./{}.out touch {}.out ::: in/file
1462 This will transfer the file in/file.out from the computer
1464 server.example.com to the files out/in/file.out after running touch
1465 in/file.out on server.
1466 echo /tmp/foo/bar.txt | parallel --return {.}.out \
1468 --sshlogin server.example.com touch {.}.out
1469 This will transfer the file /tmp/foo/bar.out from the computer
1471 server.example.com to the file /tmp/foo/bar.out after running touch
1472 /tmp/foo/bar.out on server.example.com.
1473 Multiple files can be transferred by repeating the option multiple
1475 times:
1476 echo /tmp/foo/bar.txt | parallel \
1478 --sshlogin server.example.com \
1479 --return {.}.out --return {.}.out2 touch {.}.out {.}.out2
1480 --return is often used with --transferfile and --cleanup.
1482 --return is ignored when used with --sshlogin : or when not used
1484 with --sshlogin.
1485 --round-robin
1487 --round
1488 Normally --pipe will give a single block to each instance of the
1489 command. With --round-robin all blocks will at random be written to
1490 commands already running. This is useful if the command takes a
1491 long time to initialize.
1492 --keep-order will not work with --round-robin as it is impossible
1494 to track which input block corresponds to which output.
1495 --round-robin implies --pipe, except if --pipepart is given.
1497 --rpl 'tag perl expression'
1499 Use tag as a replacement string for perl expression. This makes it
1500 possible to define your own replacement strings. GNU parallel's 7
1501 replacement strings are implemented as:
1502 --rpl '{} '
1504 --rpl '{#} 1 $_=$job->seq()'
1505 --rpl '{%} 1 $_=$job->slot()'
1506 --rpl '{/} s:.*/::'
1507 --rpl '{//} $Global::use{"File::Basename"} ||=
1508 eval "use File::Basename; 1;"; $_ = dirname($_);'
1509 --rpl '{/.} s:.*/::; s:\.[^/.]+$::;'
1510 --rpl '{.} s:\.[^/.]+$::'
1511 The --plus replacement strings are implemented as:
1513 --rpl '{+/} s:/[^/]*$::'
1515 --rpl '{+.} s:.*\.::'
1516 --rpl '{+..} s:.*\.([^.]*\.):$1:'
1517 --rpl '{+...} s:.*\.([^.]*\.[^.]*\.):$1:'
1518 --rpl '{..} s:\.[^/.]+$::; s:\.[^/.]+$::'
1519 --rpl '{...} s:\.[^/.]+$::; s:\.[^/.]+$::; s:\.[^/.]+$::'
1520 --rpl '{/..} s:.*/::; s:\.[^/.]+$::; s:\.[^/.]+$::'
1521 --rpl '{/...} s:.*/::;s:\.[^/.]+$::;s:\.[^/.]+$::;s:\.[^/.]+$::'
1522 --rpl '{##} $_=total_jobs()'
1523 --rpl '{:-(.+?)} $_ ||= $$1'
1524 --rpl '{:(\d+?)} substr($_,0,$$1) = ""'
1525 --rpl '{:(\d+?):(\d+?)} $_ = substr($_,$$1,$$2);'
1526 --rpl '{#([^#].*?)} s/^$$1//;'
1527 --rpl '{%(.+?)} s/$$1$//;'
1528 --rpl '{/(.+?)/(.*?)} s/$$1/$$2/;'
1529 --rpl '{^(.+?)} s/^($$1)/uc($1)/e;'
1530 --rpl '{^^(.+?)} s/($$1)/uc($1)/eg;'
1531 --rpl '{,(.+?)} s/^($$1)/lc($1)/e;'
1532 --rpl '{,,(.+?)} s/($$1)/lc($1)/eg;'
1533 If the user defined replacement string starts with '{' it can also
1535 be used as a positional replacement string (like {2.}).
1536 It is recommended to only change $_ but you have full access to all
1538 of GNU parallel's internal functions and data structures.
1539 Here are a few examples:
1541 Is the job sequence even or odd?
1543 --rpl '{odd} $_ = seq() % 2 ? "odd" : "even"'
1544 Pad job sequence with leading zeros to get equal width
1545 --rpl '{0#} $f=1+int("".(log(total_jobs())/log(10)));
1546 $_=sprintf("%0${f}d",seq())'
1547 Job sequence counting from 0
1548 --rpl '{#0} $_ = seq() - 1'
1549 Job slot counting from 2
1550 --rpl '{%1} $_ = slot() + 1'
1551 Remove all extensions
1552 --rpl '{:} s:(\.[^/]+)*$::'
1553 You can have dynamic replacement strings by including parenthesis
1555 in the replacement string and adding a regular expression between
1556 the parenthesis. The matching string will be inserted as $$1:
1557 parallel --rpl '{%(.*?)} s/$$1//' echo {%.tar.gz} ::: my.tar.gz
1559 parallel --rpl '{:%(.+?)} s:$$1(\.[^/]+)*$::' \
1560 echo {:%_file} ::: my_file.tar.gz
1561 parallel -n3 --rpl '{/:%(.*?)} s:.*/(.*)$$1(\.[^/]+)*$:$1:' \
1562 echo job {#}: {2} {2.} {3/:%_1} ::: a/b.c c/d.e f/g_1.h.i
1563 You can even use multiple matches:
1565 parallel --rpl '{/(.+?)/(.*?)} s/$$1/$$2/;'
1567 echo {/replacethis/withthis} {/b/C} ::: a_replacethis_b
1568 parallel --rpl '{(.*?)/(.*?)} $_="$$2$_$$1"' \
1570 echo {swap/these} ::: -middle-
1571 See also: {= perl expression =} --parens
1573 --rsync-opts options
1575 Options to pass on to rsync. Setting --rsync-opts takes precedence
1576 over setting the environment variable $PARALLEL_RSYNC_OPTS.
1577 --max-chars=max-chars
1579 -s max-chars
1580 Use at most max-chars characters per command line, including the
1581 command and initial-arguments and the terminating nulls at the ends
1582 of the argument strings. The largest allowed value is system-
1583 dependent, and is calculated as the argument length limit for exec,
1584 less the size of your environment. The default value is the
1585 maximum.
1586 Implies -X unless -m is set.
1588 --show-limits
1590 Display the limits on the command-line length which are imposed by
1591 the operating system and the -s option. Pipe the input from
1592 /dev/null (and perhaps specify --no-run-if-empty) if you don't want
1593 GNU parallel to do anything.
1594 --semaphore
1596 Work as a counting semaphore. --semaphore will cause GNU parallel
1597 to start command in the background. When the number of jobs given
1598 by --jobs is reached, GNU parallel will wait for one of these to
1599 complete before starting another command.
1600 --semaphore implies --bg unless --fg is specified.
1602 --semaphore implies --semaphorename `tty` unless --semaphorename is
1604 specified.
1605 Used with --fg, --wait, and --semaphorename.
1607 The command sem is an alias for parallel --semaphore.
1609 See also man sem.
1611 --semaphorename name
1613 --id name
1614 Use name as the name of the semaphore. Default is the name of the
1615 controlling tty (output from tty).
1616 The default normally works as expected when used interactively, but
1618 when used in a script name should be set. $$ or my_task_name are
1619 often a good value.
1620 The semaphore is stored in ~/.parallel/semaphores/
1622 Implies --semaphore.
1624 See also man sem.
1626 --semaphoretimeout secs
1628 --st secs
1629 If secs > 0: If the semaphore is not released within secs seconds,
1630 take it anyway.
1631 If secs < 0: If the semaphore is not released within secs seconds,
1633 exit.
1634 Implies --semaphore.
1636 See also man sem.
1638 --seqreplace replace-str
1640 Use the replacement string replace-str instead of {#} for job
1641 sequence number.
1642 --shebang
1644 --hashbang
1645 GNU parallel can be called as a shebang (#!) command as the first
1646 line of a script. The content of the file will be treated as
1647 inputsource.
1648 Like this:
1650 #!/usr/bin/parallel --shebang -r wget
1652 https://ftpmirror.gnu.org/parallel/parallel-20120822.tar.bz2
1654 https://ftpmirror.gnu.org/parallel/parallel-20130822.tar.bz2
1655 https://ftpmirror.gnu.org/parallel/parallel-20140822.tar.bz2
1656 --shebang must be set as the first option.
1658 On FreeBSD env is needed:
1660 #!/usr/bin/env -S parallel --shebang -r wget
1662 https://ftpmirror.gnu.org/parallel/parallel-20120822.tar.bz2
1664 https://ftpmirror.gnu.org/parallel/parallel-20130822.tar.bz2
1665 https://ftpmirror.gnu.org/parallel/parallel-20140822.tar.bz2
1666 There are many limitations of shebang (#!) depending on your
1668 operating system. See details on
1669 http://www.in-ulm.de/~mascheck/various/shebang/
1670 --shebang-wrap
1672 GNU parallel can parallelize scripts by wrapping the shebang line.
1673 If the program can be run like this:
1674 cat arguments | parallel the_program
1676 then the script can be changed to:
1678 #!/usr/bin/parallel --shebang-wrap /original/parser --options
1680 E.g.
1682 #!/usr/bin/parallel --shebang-wrap /usr/bin/python
1684 If the program can be run like this:
1686 cat data | parallel --pipe the_program
1688 then the script can be changed to:
1690 #!/usr/bin/parallel --shebang-wrap --pipe /orig/parser --opts
1692 E.g.
1694 #!/usr/bin/parallel --shebang-wrap --pipe /usr/bin/perl -w
1696 --shebang-wrap must be set as the first option.
1698 --shellquote
1700 Does not run the command but quotes it. Useful for making quoted
1701 composed commands for GNU parallel.
1702 --shuf
1704 Shuffle jobs. When having multiple input sources it is hard to
1705 randomize jobs. --shuf will generate all jobs, and shuffle them
1706 before running them. This is useful to get a quick preview of the
1707 results before running the full batch.
1708 --skip-first-line
1710 Do not use the first line of input (used by GNU parallel itself
1711 when called with --shebang).
1712 --sql DBURL (obsolete)
1714 Use --sqlmaster instead.
1715 --sqlmaster DBURL
1717 Submit jobs via SQL server. DBURL must point to a table, which will
1718 contain the same information as --joblog, the values from the input
1719 sources (stored in columns V1 .. Vn), and the output (stored in
1720 columns Stdout and Stderr).
1721 If DBURL is prepended with '+' GNU parallel assumes the table is
1723 already made with the correct columns and appends the jobs to it.
1724 If DBURL is not prepended with '+' the table will be dropped and
1726 created with the correct amount of V-columns unless
1727 --sqlmaster does not run any jobs, but it creates the values for
1729 the jobs to be run. One or more --sqlworker must be run to actually
1730 execute the jobs.
1731 If --wait is set, GNU parallel will wait for the jobs to complete.
1733 The format of a DBURL is:
1735 [sql:]vendor://[[user][:pwd]@][host][:port]/[db]/table
1737 E.g.
1739 sql:mysql://hr:hr@localhost:3306/hrdb/jobs
1741 mysql://scott:tiger@my.example.com/pardb/paralleljobs
1742 sql:oracle://scott:tiger@ora.example.com/xe/parjob
1743 postgresql://scott:tiger@pg.example.com/pgdb/parjob
1744 pg:///parjob
1745 sqlite3:///pardb/parjob
1746 It can also be an alias from ~/.sql/aliases:
1748 :myalias mysql:///mydb/paralleljobs
1750 --sqlandworker DBURL
1752 Shorthand for: --sqlmaster DBURL --sqlworker DBURL.
1753 --sqlworker DBURL
1755 Execute jobs via SQL server. Read the input sources variables from
1756 the table pointed to by DBURL. The command on the command line
1757 should be the same as given by --sqlmaster.
1758 If you have more than one --sqlworker jobs may be run more than
1760 once.
1761 If --sqlworker runs on the local machine, the hostname in the SQL
1763 table will not be ':' but instead the hostname of the machine.
1764 --ssh sshcommand
1766 GNU parallel defaults to using ssh for remote access. This can be
1767 overridden with --ssh. It can also be set on a per server basis
1768 (see --sshlogin).
1769 --sshdelay secs
1771 Delay starting next ssh by secs seconds. GNU parallel will pause
1772 secs seconds after starting each ssh. secs can be less than 1
1773 seconds.
1774 -S
1776 [@hostgroups/][ncores/]sshlogin[,[@hostgroups/][ncores/]sshlogin[,...]]
1777 -S @hostgroup
1778 --sshlogin
1779 [@hostgroups/][ncores/]sshlogin[,[@hostgroups/][ncores/]sshlogin[,...]]
1780 --sshlogin @hostgroup
1781 Distribute jobs to remote computers. The jobs will be run on a list
1782 of remote computers.
1783 If hostgroups is given, the sshlogin will be added to that
1785 hostgroup. Multiple hostgroups are separated by '+'. The sshlogin
1786 will always be added to a hostgroup named the same as sshlogin.
1787 If only the @hostgroup is given, only the sshlogins in that
1789 hostgroup will be used. Multiple @hostgroup can be given.
1790 GNU parallel will determine the number of CPU cores on the remote
1792 computers and run the number of jobs as specified by -j. If the
1793 number ncores is given GNU parallel will use this number for number
1794 of CPU cores on the host. Normally ncores will not be needed.
1795 An sshlogin is of the form:
1797 [sshcommand [options]] [username@]hostname
1799 The sshlogin must not require a password (ssh-agent, ssh-copy-id,
1801 and sshpass may help with that).
1802 The sshlogin ':' is special, it means 'no ssh' and will therefore
1804 run on the local computer.
1805 The sshlogin '..' is special, it read sshlogins from
1807 ~/.parallel/sshloginfile or $XDG_CONFIG_HOME/parallel/sshloginfile
1808 The sshlogin '-' is special, too, it read sshlogins from stdin
1810 (standard input).
1811 To specify more sshlogins separate the sshlogins by comma, newline
1813 (in the same string), or repeat the options multiple times.
1814 For examples: see --sshloginfile.
1816 The remote host must have GNU parallel installed.
1818 --sshlogin is known to cause problems with -m and -X.
1820 --sshlogin is often used with --transferfile, --return, --cleanup,
1822 and --trc.
1823 --sshloginfile filename
1825 --slf filename
1826 File with sshlogins. The file consists of sshlogins on separate
1827 lines. Empty lines and lines starting with '#' are ignored.
1828 Example:
1829 server.example.com
1831 username@server2.example.com
1832 8/my-8-core-server.example.com
1833 2/my_other_username@my-dualcore.example.net
1834 # This server has SSH running on port 2222
1835 ssh -p 2222 server.example.net
1836 4/ssh -p 2222 quadserver.example.net
1837 # Use a different ssh program
1838 myssh -p 2222 -l myusername hexacpu.example.net
1839 # Use a different ssh program with default number of cores
1840 //usr/local/bin/myssh -p 2222 -l myusername hexacpu
1841 # Use a different ssh program with 6 cores
1842 6//usr/local/bin/myssh -p 2222 -l myusername hexacpu
1843 # Assume 16 cores on the local computer
1844 16/:
1845 # Put server1 in hostgroup1
1846 @hostgroup1/server1
1847 # Put myusername@server2 in hostgroup1+hostgroup2
1848 @hostgroup1+hostgroup2/myusername@server2
1849 # Force 4 cores and put 'ssh -p 2222 server3' in hostgroup1
1850 @hostgroup1/4/ssh -p 2222 server3
1851 When using a different ssh program the last argument must be the
1853 hostname.
1854 Multiple --sshloginfile are allowed.
1856 GNU parallel will first look for the file in current dir; if that
1858 fails it look for the file in ~/.parallel.
1859 The sshloginfile '..' is special, it read sshlogins from
1861 ~/.parallel/sshloginfile
1862 The sshloginfile '.' is special, it read sshlogins from
1864 /etc/parallel/sshloginfile
1865 The sshloginfile '-' is special, too, it read sshlogins from stdin
1867 (standard input).
1868 If the sshloginfile is changed it will be re-read when a job
1870 finishes though at most once per second. This makes it possible to
1871 add and remove hosts while running.
1872 This can be used to have a daemon that updates the sshloginfile to
1874 only contain servers that are up:
1875 cp original.slf tmp2.slf
1877 while [ 1 ] ; do
1878 nice parallel --nonall -j0 -k --slf original.slf \
1879 --tag echo | perl 's/\t$//' > tmp.slf
1880 if diff tmp.slf tmp2.slf; then
1881 mv tmp.slf tmp2.slf
1882 fi
1883 sleep 10
1884 done &
1885 parallel --slf tmp2.slf ...
1886 --slotreplace replace-str
1888 Use the replacement string replace-str instead of {%} for job slot
1889 number.
1890 --silent
1892 Silent. The job to be run will not be printed. This is the
1893 default. Can be reversed with -v.
1894 --tty
1896 Open terminal tty. If GNU parallel is used for starting an
1897 interactive program then this option may be needed. It will start
1898 only one job at a time (i.e. -j1), not buffer the output (i.e. -u),
1899 and it will open a tty for the job. When the job is done, the next
1900 job will get the tty.
1901 You can of course override -j1 and -u.
1903 --tag
1905 Tag lines with arguments. Each output line will be prepended with
1906 the arguments and TAB (\t). When combined with --onall or --nonall
1907 the lines will be prepended with the sshlogin instead.
1908 --tag is ignored when using -u.
1910 --tagstring str
1912 Tag lines with a string. Each output line will be prepended with
1913 str and TAB (\t). str can contain replacement strings such as {}.
1914 --tagstring is ignored when using -u, --onall, and --nonall.
1916 --tee
1918 Pipe all data to all jobs. Used with --pipe/--pipepart and :::.
1919 seq 1000 | parallel --pipe --tee -v wc {} ::: -w -l -c
1921 How many numbers in 1..1000 contain 0..9, and how many bytes do
1923 they fill:
1924 seq 1000 | parallel --pipe --tee --tag \
1926 'grep {1} | wc {2}' ::: {0..9} ::: -l -c
1927 How many words contain a..z and how many bytes do they fill?
1929 parallel -a /usr/share/dict/words --pipepart --tee --tag \
1931 'grep {1} | wc {2}' ::: {a..z} ::: -l -c
1932 --termseq sequence
1934 Termination sequence. When a job is killed due to --timeout,
1935 --memfree, --halt, or abnormal termination of GNU parallel,
1936 sequence determines how the job is killed. The default is:
1937 TERM,200,TERM,100,TERM,50,KILL,25
1939 which sends a TERM signal, waits 200 ms, sends another TERM signal,
1941 waits 100 ms, sends another TERM signal, waits 50 ms, sends a KILL
1942 signal, waits 25 ms, and exits. GNU parallel detects if a process
1943 dies before the waiting time is up.
1944 --tmpdir dirname
1946 Directory for temporary files. GNU parallel normally buffers output
1947 into temporary files in /tmp. By setting --tmpdir you can use a
1948 different dir for the files. Setting --tmpdir is equivalent to
1949 setting $TMPDIR.
1950 --tmux (Long beta testing)
1952 Use tmux for output. Start a tmux session and run each job in a
1953 window in that session. No other output will be produced.
1954 --tmuxpane (Long beta testing)
1956 Use tmux for output but put output into panes in the first window.
1957 Useful if you want to monitor the progress of less than 100
1958 concurrent jobs.
1959 --timeout duration
1961 Time out for command. If the command runs for longer than duration
1962 seconds it will get killed as per --termseq.
1963 If duration is followed by a % then the timeout will dynamically be
1965 computed as a percentage of the median average runtime of
1966 successful jobs. Only values > 100% will make sense.
1967 duration is normally in seconds, but can be floats postfixed with
1969 s, m, h, or d which would multiply the float by 1, 60, 3600, or
1970 86400. Thus these are equivalent: --timeout 100000 and --timeout
1971 1d3.5h16.6m4s.
1972 --verbose
1974 -t Print the job to be run on stderr (standard error).
1975 See also -v, -p.
1977 --transfer
1979 Transfer files to remote computers. Shorthand for: --transferfile
1980 {}.
1981 --transferfile filename
1983 --tf filename
1984 --transferfile is used with --sshlogin to transfer files to the
1985 remote computers. The files will be transferred using rsync and
1986 will be put relative to the default work dir. If the path contains
1987 /./ the remaining path will be relative to the work dir. E.g.
1988 echo foo/bar.txt | parallel --transferfile {} \
1990 --sshlogin server.example.com wc
1991 This will transfer the file foo/bar.txt to the computer
1993 server.example.com to the file $HOME/foo/bar.txt before running wc
1994 foo/bar.txt on server.example.com.
1995 echo /tmp/foo/bar.txt | parallel --transferfile {} \
1997 --sshlogin server.example.com wc
1998 This will transfer the file /tmp/foo/bar.txt to the computer
2000 server.example.com to the file /tmp/foo/bar.txt before running wc
2001 /tmp/foo/bar.txt on server.example.com.
2002 echo /tmp/./foo/bar.txt | parallel --transferfile {} \
2004 --sshlogin server.example.com wc {= s:.*/./:./: =}
2005 This will transfer the file /tmp/foo/bar.txt to the computer
2007 server.example.com to the file foo/bar.txt before running wc
2008 ./foo/bar.txt on server.example.com.
2009 --transferfile is often used with --return and --cleanup. A
2011 shorthand for --transferfile {} is --transfer.
2012 --transferfile is ignored when used with --sshlogin : or when not
2014 used with --sshlogin.
2015 --trc filename
2017 Transfer, Return, Cleanup. Shorthand for:
2018 --transferfile {} --return filename --cleanup
2020 --trim <n|l|r|lr|rl>
2022 Trim white space in input.
2023 n No trim. Input is not modified. This is the default.
2025 l Left trim. Remove white space from start of input. E.g. " a bc
2027 " -> "a bc ".
2028 r Right trim. Remove white space from end of input. E.g. " a bc "
2030 -> " a bc".
2031 lr
2033 rl Both trim. Remove white space from both start and end of input.
2034 E.g. " a bc " -> "a bc". This is the default if --colsep is
2035 used.
2036 --ungroup
2038 -u Ungroup output. Output is printed as soon as possible and by
2039 passes GNU parallel internal processing. This may cause output from
2040 different commands to be mixed thus should only be used if you do
2041 not care about the output. Compare these:
2042 seq 4 | parallel -j0 \
2044 'sleep {};echo -n start{};sleep {};echo {}end'
2045 seq 4 | parallel -u -j0 \
2046 'sleep {};echo -n start{};sleep {};echo {}end'
2047 It also disables --tag. GNU parallel outputs faster with -u.
2049 Compare the speeds of these:
2050 parallel seq ::: 300000000 >/dev/null
2052 parallel -u seq ::: 300000000 >/dev/null
2053 parallel --line-buffer seq ::: 300000000 >/dev/null
2054 Can be reversed with --group.
2056 See also: --line-buffer --group
2058 --extensionreplace replace-str
2060 --er replace-str
2061 Use the replacement string replace-str instead of {.} for input
2062 line without extension.
2063 --use-cpus-instead-of-cores
2065 Count the number of physical CPUs instead of CPU cores. When
2066 computing how many jobs to run simultaneously relative to the
2067 number of CPU cores you can ask GNU parallel to instead look at the
2068 number of physical CPUs. This will make sense for computers that
2069 have hyperthreading as two jobs running on one CPU with
2070 hyperthreading will run slower than two jobs running on two
2071 physical CPUs. Some multi-core CPUs can run faster if only one
2072 thread is running per physical CPU. Most users will not need this
2073 option.
2074 -v Verbose. Print the job to be run on stdout (standard output). Can
2076 be reversed with --silent. See also -t.
2077 Use -v -v to print the wrapping ssh command when running remotely.
2079 --version
2081 -V Print the version GNU parallel and exit.
2082 --workdir mydir
2084 --wd mydir
2085 Files transferred using --transferfile and --return will be
2086 relative to mydir on remote computers, and the command will be
2087 executed in the dir mydir.
2088 The special mydir value ... will create working dirs under
2090 ~/.parallel/tmp/ on the remote computers. If --cleanup is given
2091 these dirs will be removed.
2092 The special mydir value . uses the current working dir. If the
2094 current working dir is beneath your home dir, the value . is
2095 treated as the relative path to your home dir. This means that if
2096 your home dir is different on remote computers (e.g. if your login
2097 is different) the relative path will still be relative to your home
2098 dir.
2099 To see the difference try:
2101 parallel -S server pwd ::: ""
2103 parallel --wd . -S server pwd ::: ""
2104 parallel --wd ... -S server pwd ::: ""
2105 mydir can contain GNU parallel's replacement strings.
2107 --wait
2109 Wait for all commands to complete.
2110 Used with --semaphore or --sqlmaster.
2112 See also man sem.
2114 -X Multiple arguments with context replace. Insert as many arguments
2116 as the command line length permits. If multiple jobs are being run
2117 in parallel: distribute the arguments evenly among the jobs. Use
2118 -j1 to avoid this.
2119 If {} is not used the arguments will be appended to the line. If
2121 {} is used as part of a word (like pic{}.jpg) then the whole word
2122 will be repeated. If {} is used multiple times each {} will be
2123 replaced with the arguments.
2124 Normally -X will do the right thing, whereas -m can give unexpected
2126 results if {} is used as part of a word.
2127 Support for -X with --sshlogin is limited and may fail.
2129 See also -m.
2131 --exit
2133 -x Exit if the size (see the -s option) is exceeded.
2134
2136 GNU parallel can work similar to xargs -n1.
2137 To compress all html files using gzip run:
2139 find . -name '*.html' | parallel gzip --best
2141 If the file names may contain a newline use -0. Substitute FOO BAR with
2143 FUBAR in all files in this dir and subdirs:
2144 find . -type f -print0 | parallel -q0 perl -i -pe 's/FOO BAR/FUBAR/g'
2146 Note -q is needed because of the space in 'FOO BAR'.
2148
2150 GNU parallel can take the arguments from command line instead of stdin
2151 (standard input). To compress all html files in the current dir using
2152 gzip run:
2153 parallel gzip --best ::: *.html
2155 To convert *.wav to *.mp3 using LAME running one process per CPU core
2157 run:
2158 parallel lame {} -o {.}.mp3 ::: *.wav
2160
2162 When moving a lot of files like this: mv *.log destdir you will
2163 sometimes get the error:
2164 bash: /bin/mv: Argument list too long
2166 because there are too many files. You can instead do:
2168 ls | grep -E '\.log$' | parallel mv {} destdir
2170 This will run mv for each file. It can be done faster if mv gets as
2172 many arguments that will fit on the line:
2173 ls | grep -E '\.log$' | parallel -m mv {} destdir
2175 In many shells you can also use printf:
2177 printf '%s\0' *.log | parallel -0 -m mv {} destdir
2179
2181 To remove the files pict0000.jpg .. pict9999.jpg you could do:
2182 seq -w 0 9999 | parallel rm pict{}.jpg
2184 You could also do:
2186 seq -w 0 9999 | perl -pe 's/(.*)/pict$1.jpg/' | parallel -m rm
2188 The first will run rm 10000 times, while the last will only run rm as
2190 many times needed to keep the command line length short enough to avoid
2191 Argument list too long (it typically runs 1-2 times).
2192 You could also run:
2194 seq -w 0 9999 | parallel -X rm pict{}.jpg
2196 This will also only run rm as many times needed to keep the command
2198 line length short enough.
2199
2201 If ImageMagick is installed this will generate a thumbnail of a jpg
2202 file:
2203 convert -geometry 120 foo.jpg thumb_foo.jpg
2205 This will run with number-of-cpu-cores jobs in parallel for all jpg
2207 files in a directory:
2208 ls *.jpg | parallel convert -geometry 120 {} thumb_{}
2210 To do it recursively use find:
2212 find . -name '*.jpg' | parallel convert -geometry 120 {} {}_thumb.jpg
2214 Notice how the argument has to start with {} as {} will include path
2216 (e.g. running convert -geometry 120 ./foo/bar.jpg thumb_./foo/bar.jpg
2217 would clearly be wrong). The command will generate files like
2218 ./foo/bar.jpg_thumb.jpg.
2219 Use {.} to avoid the extra .jpg in the file name. This command will
2221 make files like ./foo/bar_thumb.jpg:
2222 find . -name '*.jpg' | parallel convert -geometry 120 {} {.}_thumb.jpg
2224
2226 This will generate an uncompressed version of .gz-files next to the
2227 .gz-file:
2228 parallel zcat {} ">"{.} ::: *.gz
2230 Quoting of > is necessary to postpone the redirection. Another solution
2232 is to quote the whole command:
2233 parallel "zcat {} >{.}" ::: *.gz
2235 Other special shell characters (such as * ; $ > < | >> <<) also need to
2237 be put in quotes, as they may otherwise be interpreted by the shell and
2238 not given to GNU parallel.
2239
2241 A job can consist of several commands. This will print the number of
2242 files in each directory:
2243 ls | parallel 'echo -n {}" "; ls {}|wc -l'
2245 To put the output in a file called <name>.dir:
2247 ls | parallel '(echo -n {}" "; ls {}|wc -l) >{}.dir'
2249 Even small shell scripts can be run by GNU parallel:
2251 find . | parallel 'a={}; name=${a##*/};' \
2253 'upper=$(echo "$name" | tr "[:lower:]" "[:upper:]");'\
2254 'echo "$name - $upper"'
2255 ls | parallel 'mv {} "$(echo {} | tr "[:upper:]" "[:lower:]")"'
2257 Given a list of URLs, list all URLs that fail to download. Print the
2259 line number and the URL.
2260 cat urlfile | parallel "wget {} 2>/dev/null || grep -n {} urlfile"
2262 Create a mirror directory with the same filenames except all files and
2264 symlinks are empty files.
2265 cp -rs /the/source/dir mirror_dir
2267 find mirror_dir -type l | parallel -m rm {} '&&' touch {}
2268 Find the files in a list that do not exist
2270 cat file_list | parallel 'if [ ! -e {} ] ; then echo {}; fi'
2272
2274 You have a dir with files named as 24 hours in 5 minute intervals:
2275 00:00, 00:05, 00:10 .. 23:55. You want to find the files missing:
2276 parallel [ -f {1}:{2} ] "||" echo {1}:{2} does not exist \
2278 ::: {00..23} ::: {00..55..5}
2279
2281 If the composed command is longer than a line, it becomes hard to read.
2282 In Bash you can use functions. Just remember to export -f the function.
2283 doit() {
2285 echo Doing it for $1
2286 sleep 2
2287 echo Done with $1
2288 }
2289 export -f doit
2290 parallel doit ::: 1 2 3
2291 doubleit() {
2293 echo Doing it for $1 $2
2294 sleep 2
2295 echo Done with $1 $2
2296 }
2297 export -f doubleit
2298 parallel doubleit ::: 1 2 3 ::: a b
2299 To do this on remote servers you need to transfer the function using
2301 --env:
2302 parallel --env doit -S server doit ::: 1 2 3
2304 parallel --env doubleit -S server doubleit ::: 1 2 3 ::: a b
2305 If your environment (aliases, variables, and functions) is small you
2307 can copy the full environment without having to export -f anything. See
2308 env_parallel.
2309
2311 To test a program with different parameters:
2312 tester() {
2314 if (eval "$@") >&/dev/null; then
2315 perl -e 'printf "\033[30;102m[ OK ]\033[0m @ARGV\n"' "$@"
2316 else
2317 perl -e 'printf "\033[30;101m[FAIL]\033[0m @ARGV\n"' "$@"
2318 fi
2319 }
2320 export -f tester
2321 parallel tester my_program ::: arg1 arg2
2322 parallel tester exit ::: 1 0 2 0
2323 If my_program fails a red FAIL will be printed followed by the failing
2325 command; otherwise a green OK will be printed followed by the command.
2326
2328 Log rotation renames a logfile to an extension with a higher number:
2329 log.1 becomes log.2, log.2 becomes log.3, and so on. The oldest log is
2330 removed. To avoid overwriting files the process starts backwards from
2331 the high number to the low number. This will keep 10 old versions of
2332 the log:
2333 seq 9 -1 1 | parallel -j1 mv log.{} log.'{= $_++ =}'
2335 mv log log.1
2336
2338 When processing files removing the file extension using {.} is often
2339 useful.
2340 Create a directory for each zip-file and unzip it in that dir:
2342 parallel 'mkdir {.}; cd {.}; unzip ../{}' ::: *.zip
2344 Recompress all .gz files in current directory using bzip2 running 1 job
2346 per CPU core in parallel:
2347 parallel "zcat {} | bzip2 >{.}.bz2 && rm {}" ::: *.gz
2349 Convert all WAV files to MP3 using LAME:
2351 find sounddir -type f -name '*.wav' | parallel lame {} -o {.}.mp3
2353 Put all converted in the same directory:
2355 find sounddir -type f -name '*.wav' | \
2357 parallel lame {} -o mydir/{/.}.mp3
2358
2360 If you have directory with tar.gz files and want these extracted in the
2361 corresponding dir (e.g foo.tar.gz will be extracted in the dir foo) you
2362 can do:
2363 parallel --plus 'mkdir {..}; tar -C {..} -xf {}' ::: *.tar.gz
2365 If you want to remove a different ending, you can use {%string}:
2367 parallel --plus echo {%_demo} ::: mycode_demo keep_demo_here
2369 You can also remove a starting string with {#string}
2371 parallel --plus echo {#demo_} ::: demo_mycode keep_demo_here
2373 To remove a string anywhere you can use regular expressions with
2375 {/regexp/replacement} and leave the replacement empty:
2376 parallel --plus echo {/demo_/} ::: demo_mycode remove_demo_here
2378
2380 Let us assume a website stores images like:
2381 http://www.example.com/path/to/YYYYMMDD_##.jpg
2383 where YYYYMMDD is the date and ## is the number 01-24. This will
2385 download images for the past 30 days:
2386 getit() {
2388 date=$(date -d "today -$1 days" +%Y%m%d)
2389 num=$2
2390 echo wget http://www.example.com/path/to/${date}_${num}.jpg
2391 }
2392 export -f getit
2393 parallel getit ::: $(seq 30) ::: $(seq -w 24)
2395 $(date -d "today -$1 days" +%Y%m%d) will give the dates in YYYYMMDD
2397 with $1 days subtracted.
2398
2400 find . | parallel cp {} '../destdir/{= $a=int(10000*rand); $_=pQ($_);
2401 $_=`date -r "$_" +%FT%T"$a"`; chomp; =}'
2402 {= and =} mark a perl expression. pQ quotes the string. date +%FT%T is
2404 the date in ISO8601 with time.
2405
2407 The : in a digital clock blinks. To make every other line have a ':'
2408 and the rest a ' ' a perl expression is used to look at the 3rd input
2409 source. If the value modudo 2 is 1: Use ":" otherwise use " ":
2410 parallel -k echo {1}'{=3 $_=$_%2?":":" "=}'{2}{3} \
2412 ::: {0..12} ::: {0..5} ::: {0..9}
2413
2415 This:
2416 parallel --header : echo x{X}y{Y}z{Z} \> x{X}y{Y}z{Z} \
2418 ::: X {1..5} ::: Y {01..10} ::: Z {1..5}
2419 will generate the files x1y01z1 .. x5y10z5. If you want to aggregate
2421 the output grouping on x and z you can do this:
2422 parallel eval 'cat {=s/y01/y*/=} > {=s/y01//=}' ::: *y01*
2424 For all values of x and z it runs commands like:
2426 cat x1y*z1 > x1z1
2428 So you end up with x1z1 .. x5z5 each containing the content of all
2430 values of y.
2431
2433 This script below will crawl and mirror a URL in parallel. It
2434 downloads first pages that are 1 click down, then 2 clicks down, then
2435 3; instead of the normal depth first, where the first link link on each
2436 page is fetched first.
2437 Run like this:
2439 PARALLEL=-j100 ./parallel-crawl http://gatt.org.yeslab.org/
2441 Remove the wget part if you only want a web crawler.
2443 It works by fetching a page from a list of URLs and looking for links
2445 in that page that are within the same starting URL and that have not
2446 already been seen. These links are added to a new queue. When all the
2447 pages from the list is done, the new queue is moved to the list of URLs
2448 and the process is started over until no unseen links are found.
2449 #!/bin/bash
2451 # E.g. http://gatt.org.yeslab.org/
2453 URL=$1
2454 # Stay inside the start dir
2455 BASEURL=$(echo $URL | perl -pe 's:#.*::; s:(//.*/)[^/]*:$1:')
2456 URLLIST=$(mktemp urllist.XXXX)
2457 URLLIST2=$(mktemp urllist.XXXX)
2458 SEEN=$(mktemp seen.XXXX)
2459 # Spider to get the URLs
2461 echo $URL >$URLLIST
2462 cp $URLLIST $SEEN
2463 while [ -s $URLLIST ] ; do
2465 cat $URLLIST |
2466 parallel lynx -listonly -image_links -dump {} \; \
2467 wget -qm -l1 -Q1 {} \; echo Spidered: {} \>\&2 |
2468 perl -ne 's/#.*//; s/\s+\d+.\s(\S+)$/$1/ and
2469 do { $seen{$1}++ or print }' |
2470 grep -F $BASEURL |
2471 grep -v -x -F -f $SEEN | tee -a $SEEN > $URLLIST2
2472 mv $URLLIST2 $URLLIST
2473 done
2474 rm -f $URLLIST $URLLIST2 $SEEN
2476
2478 If the files to be processed are in a tar file then unpacking one file
2479 and processing it immediately may be faster than first unpacking all
2480 files.
2481 tar xvf foo.tgz | perl -ne 'print $l;$l=$_;END{print $l}' | \
2483 parallel echo
2484 The Perl one-liner is needed to make sure the file is complete before
2486 handing it to GNU parallel.
2487
2489 for-loops like this:
2490 (for x in `cat list` ; do
2492 do_something $x
2493 done) | process_output
2494 and while-read-loops like this:
2496 cat list | (while read x ; do
2498 do_something $x
2499 done) | process_output
2500 can be written like this:
2502 cat list | parallel do_something | process_output
2504 For example: Find which host name in a list has IP address 1.2.3 4:
2506 cat hosts.txt | parallel -P 100 host | grep 1.2.3.4
2508 If the processing requires more steps the for-loop like this:
2510 (for x in `cat list` ; do
2512 no_extension=${x%.*};
2513 do_step1 $x scale $no_extension.jpg
2514 do_step2 <$x $no_extension
2515 done) | process_output
2516 and while-loops like this:
2518 cat list | (while read x ; do
2520 no_extension=${x%.*};
2521 do_step1 $x scale $no_extension.jpg
2522 do_step2 <$x $no_extension
2523 done) | process_output
2524 can be written like this:
2526 cat list | parallel "do_step1 {} scale {.}.jpg ; do_step2 <{} {.}" |\
2528 process_output
2529 If the body of the loop is bigger, it improves readability to use a
2531 function:
2532 (for x in `cat list` ; do
2534 do_something $x
2535 [... 100 lines that do something with $x ...]
2536 done) | process_output
2537 cat list | (while read x ; do
2539 do_something $x
2540 [... 100 lines that do something with $x ...]
2541 done) | process_output
2542 can both be rewritten as:
2544 doit() {
2546 x=$1
2547 do_something $x
2548 [... 100 lines that do something with $x ...]
2549 }
2550 export -f doit
2551 cat list | parallel doit
2552
2554 Nested for-loops like this:
2555 (for x in `cat xlist` ; do
2557 for y in `cat ylist` ; do
2558 do_something $x $y
2559 done
2560 done) | process_output
2561 can be written like this:
2563 parallel do_something {1} {2} :::: xlist ylist | process_output
2565 Nested for-loops like this:
2567 (for colour in red green blue ; do
2569 for size in S M L XL XXL ; do
2570 echo $colour $size
2571 done
2572 done) | sort
2573 can be written like this:
2575 parallel echo {1} {2} ::: red green blue ::: S M L XL XXL | sort
2577
2579 diff is good for finding differences in text files. diff | wc -l gives
2580 an indication of the size of the difference. To find the differences
2581 between all files in the current dir do:
2582 parallel --tag 'diff {1} {2} | wc -l' ::: * ::: * | sort -nk3
2584 This way it is possible to see if some files are closer to other files.
2586
2588 When doing multiple nested for-loops it can be easier to keep track of
2589 the loop variable if is is named instead of just having a number. Use
2590 --header : to let the first argument be an named alias for the
2591 positional replacement string:
2592 parallel --header : echo {colour} {size} \
2594 ::: colour red green blue ::: size S M L XL XXL
2595 This also works if the input file is a file with columns:
2597 cat addressbook.tsv | \
2599 parallel --colsep '\t' --header : echo {Name} {E-mail address}
2600
2602 GNU parallel makes all combinations when given two lists.
2603 To make all combinations in a single list with unique values, you
2605 repeat the list and use replacement string with a Perl expression that
2606 skips the job if the value from input source 1 is greater than or equal
2607 to the value from input source 2:
2608 parallel echo {= 'if($arg[1] ge $arg[2]) { skip() }' =} ::: A B C D ::: A B C D
2610 Or more generally:
2612 parallel echo \
2614 '{= for $t (2..$#arg){ if($arg[$t-1] ge $arg[$t]) { skip() } } =}' \
2615 ::: A B C D ::: A B C D ::: A B C D
2616
2618 Assume you have input like:
2619 aardvark
2621 babble
2622 cab
2623 dab
2624 each
2625 and want to run combinations like:
2627 aardvark babble
2629 babble cab
2630 cab dab
2631 dab each
2632 If the input is in the file in.txt:
2634 parallel echo {1} - {2} ::::+ <(head -n -1 in.txt) <(tail -n +2 in.txt)
2636 If the input is in the array $a here are two solutions:
2638 seq $((${#a[@]}-1)) | env_parallel --env a echo '${a[{=$_--=}]} - ${a[{}]}'
2640 parallel echo {1} - {2} ::: "${a[@]::${#a[@]}-1}" :::+ "${a[@]:1}"
2641
2643 Using --results the results are saved in /tmp/diffcount*.
2644 parallel --results /tmp/diffcount "diff -U 0 {1} {2} | \
2646 tail -n +3 |grep -v '^@'|wc -l" ::: * ::: *
2647 To see the difference between file A and file B look at the file
2649 '/tmp/diffcount/1/A/2/B'.
2650
2652 Starting a job on the local machine takes around 10 ms. This can be a
2653 big overhead if the job takes very few ms to run. Often you can group
2654 small jobs together using -X which will make the overhead less
2655 significant. Compare the speed of these:
2656 seq -w 0 9999 | parallel touch pict{}.jpg
2658 seq -w 0 9999 | parallel -X touch pict{}.jpg
2659 If your program cannot take multiple arguments, then you can use GNU
2661 parallel to spawn multiple GNU parallels:
2662 seq -w 0 999999 | parallel -j10 --pipe parallel -j0 touch pict{}.jpg
2664 If -j0 normally spawns 252 jobs, then the above will try to spawn 2520
2666 jobs. On a normal GNU/Linux system you can spawn 32000 jobs using this
2667 technique with no problems. To raise the 32000 jobs limit raise
2668 /proc/sys/kernel/pid_max to 4194303.
2669
2671 When using shell variables you need to quote them correctly as they may
2672 otherwise be interpreted by the shell.
2673 Notice the difference between:
2675 ARR=("My brother's 12\" records are worth <\$\$\$>"'!' Foo Bar)
2677 parallel echo ::: ${ARR[@]} # This is probably not what you want
2678 and:
2680 ARR=("My brother's 12\" records are worth <\$\$\$>"'!' Foo Bar)
2682 parallel echo ::: "${ARR[@]}"
2683 When using variables in the actual command that contains special
2685 characters (e.g. space) you can quote them using '"$VAR"' or using "'s
2686 and -q:
2687 VAR="My brother's 12\" records are worth <\$\$\$>"
2689 parallel -q echo "$VAR" ::: '!'
2690 export VAR
2691 parallel echo '"$VAR"' ::: '!'
2692 If $VAR does not contain ' then "'$VAR'" will also work (and does not
2694 need export):
2695 VAR="My 12\" records are worth <\$\$\$>"
2697 parallel echo "'$VAR'" ::: '!'
2698 If you use them in a function you just quote as you normally would do:
2700 VAR="My brother's 12\" records are worth <\$\$\$>"
2702 export VAR
2703 myfunc() { echo "$VAR" "$1"; }
2704 export -f myfunc
2705 parallel myfunc ::: '!'
2706
2708 When running jobs that output data, you often do not want the output of
2709 multiple jobs to run together. GNU parallel defaults to grouping the
2710 output of each job, so the output is printed when the job finishes. If
2711 you want full lines to be printed while the job is running you can use
2712 --line-buffer. If you want output to be printed as soon as possible you
2713 can use -u.
2714 Compare the output of:
2716 parallel wget --limit-rate=100k \
2718 https://ftpmirror.gnu.org/parallel/parallel-20{}0822.tar.bz2 \
2719 ::: {12..16}
2720 parallel --line-buffer wget --limit-rate=100k \
2721 https://ftpmirror.gnu.org/parallel/parallel-20{}0822.tar.bz2 \
2722 ::: {12..16}
2723 parallel -u wget --limit-rate=100k \
2724 https://ftpmirror.gnu.org/parallel/parallel-20{}0822.tar.bz2 \
2725 ::: {12..16}
2726
2728 GNU parallel groups the output lines, but it can be hard to see where
2729 the different jobs begin. --tag prepends the argument to make that more
2730 visible:
2731 parallel --tag wget --limit-rate=100k \
2733 https://ftpmirror.gnu.org/parallel/parallel-20{}0822.tar.bz2 \
2734 ::: {12..16}
2735 --tag works with --line-buffer but not with -u:
2737 parallel --tag --line-buffer wget --limit-rate=100k \
2739 https://ftpmirror.gnu.org/parallel/parallel-20{}0822.tar.bz2 \
2740 ::: {12..16}
2741 Check the uptime of the servers in ~/.parallel/sshloginfile:
2743 parallel --tag -S .. --nonall uptime
2745
2747 Give each job a new color. Most terminals support ANSI colors with the
2748 escape code "\033[30;3Xm" where 0 <= X <= 7:
2749 parallel --tagstring '\033[30;3{=$_=++$::color%8=}m' seq {} ::: {1..10}
2751 parallel --rpl '{color} $_="\033[30;3".(++$::color%8)."m"' \
2752 --tagstring {color} seq {} ::: {1..10}
2753 To get rid of the initial \t (which comes from --tagstring):
2755 ... | perl -pe 's/\t//'
2757
2759 Normally the output of a job will be printed as soon as it completes.
2760 Sometimes you want the order of the output to remain the same as the
2761 order of the input. This is often important, if the output is used as
2762 input for another system. -k will make sure the order of output will be
2763 in the same order as input even if later jobs end before earlier jobs.
2764 Append a string to every line in a text file:
2766 cat textfile | parallel -k echo {} append_string
2768 If you remove -k some of the lines may come out in the wrong order.
2770 Another example is traceroute:
2772 parallel traceroute ::: qubes-os.org debian.org freenetproject.org
2774 will give traceroute of qubes-os.org, debian.org and
2776 freenetproject.org, but it will be sorted according to which job
2777 completed first.
2778 To keep the order the same as input run:
2780 parallel -k traceroute ::: qubes-os.org debian.org freenetproject.org
2782 This will make sure the traceroute to qubes-os.org will be printed
2784 first.
2785 A bit more complex example is downloading a huge file in chunks in
2787 parallel: Some internet connections will deliver more data if you
2788 download files in parallel. For downloading files in parallel see:
2789 "EXAMPLE: Download 10 images for each of the past 30 days". But if you
2790 are downloading a big file you can download the file in chunks in
2791 parallel.
2792 To download byte 10000000-19999999 you can use curl:
2794 curl -r 10000000-19999999 http://example.com/the/big/file >file.part
2796 To download a 1 GB file we need 100 10MB chunks downloaded and combined
2798 in the correct order.
2799 seq 0 99 | parallel -k curl -r \
2801 {}0000000-{}9999999 http://example.com/the/big/file > file
2802
2804 grep -r greps recursively through directories. On multicore CPUs GNU
2805 parallel can often speed this up.
2806 find . -type f | parallel -k -j150% -n 1000 -m grep -H -n STRING {}
2808 This will run 1.5 job per core, and give 1000 arguments to grep.
2810
2812 The simplest solution to grep a big file for a lot of regexps is:
2813 grep -f regexps.txt bigfile
2815 Or if the regexps are fixed strings:
2817 grep -F -f regexps.txt bigfile
2819 There are 3 limiting factors: CPU, RAM, and disk I/O.
2821 RAM is easy to measure: If the grep process takes up most of your free
2823 memory (e.g. when running top), then RAM is a limiting factor.
2824 CPU is also easy to measure: If the grep takes >90% CPU in top, then
2826 the CPU is a limiting factor, and parallelization will speed this up.
2827 It is harder to see if disk I/O is the limiting factor, and depending
2829 on the disk system it may be faster or slower to parallelize. The only
2830 way to know for certain is to test and measure.
2831 Limiting factor: RAM
2833 The normal grep -f regexs.txt bigfile works no matter the size of
2834 bigfile, but if regexps.txt is so big it cannot fit into memory, then
2835 you need to split this.
2836 grep -F takes around 100 bytes of RAM and grep takes about 500 bytes of
2838 RAM per 1 byte of regexp. So if regexps.txt is 1% of your RAM, then it
2839 may be too big.
2840 If you can convert your regexps into fixed strings do that. E.g. if the
2842 lines you are looking for in bigfile all looks like:
2843 ID1 foo bar baz Identifier1 quux
2845 fubar ID2 foo bar baz Identifier2
2846 then your regexps.txt can be converted from:
2848 ID1.*Identifier1
2850 ID2.*Identifier2
2851 into:
2853 ID1 foo bar baz Identifier1
2855 ID2 foo bar baz Identifier2
2856 This way you can use grep -F which takes around 80% less memory and is
2858 much faster.
2859 If it still does not fit in memory you can do this:
2861 parallel --pipepart -a regexps.txt --block 1M grep -Ff - -n bigfile |
2863 sort -un | perl -pe 's/^\d+://'
2864 The 1M should be your free memory divided by the number of cores and
2866 divided by 200 for grep -F and by 1000 for normal grep. On GNU/Linux
2867 you can do:
2868 free=$(awk '/^((Swap)?Cached|MemFree|Buffers):/ { sum += $2 }
2870 END { print sum }' /proc/meminfo)
2871 percpu=$((free / 200 / $(parallel --number-of-cores)))k
2872 parallel --pipepart -a regexps.txt --block $percpu --compress \
2874 grep -F -f - -n bigfile |
2875 sort -un | perl -pe 's/^\d+://'
2876 If you can live with duplicated lines and wrong order, it is faster to
2878 do:
2879 parallel --pipepart -a regexps.txt --block $percpu --compress \
2881 grep -F -f - bigfile
2882 Limiting factor: CPU
2884 If the CPU is the limiting factor parallelization should be done on the
2885 regexps:
2886 cat regexp.txt | parallel --pipe -L1000 --round-robin --compress \
2888 grep -f - -n bigfile |
2889 sort -un | perl -pe 's/^\d+://'
2890 The command will start one grep per CPU and read bigfile one time per
2892 CPU, but as that is done in parallel, all reads except the first will
2893 be cached in RAM. Depending on the size of regexp.txt it may be faster
2894 to use --block 10m instead of -L1000.
2895 Some storage systems perform better when reading multiple chunks in
2897 parallel. This is true for some RAID systems and for some network file
2898 systems. To parallelize the reading of bigfile:
2899 parallel --pipepart --block 100M -a bigfile -k --compress \
2901 grep -f regexp.txt
2902 This will split bigfile into 100MB chunks and run grep on each of these
2904 chunks. To parallelize both reading of bigfile and regexp.txt combine
2905 the two using --fifo:
2906 parallel --pipepart --block 100M -a bigfile --fifo cat regexp.txt \
2908 \| parallel --pipe -L1000 --round-robin grep -f - {}
2909 If a line matches multiple regexps, the line may be duplicated.
2911 Bigger problem
2913 If the problem is too big to be solved by this, you are probably ready
2914 for Lucene.
2915
2917 To run commands on a remote computer SSH needs to be set up and you
2918 must be able to login without entering a password (The commands ssh-
2919 copy-id, ssh-agent, and sshpass may help you do that).
2920 If you need to login to a whole cluster, you typically do not want to
2922 accept the host key for every host. You want to accept them the first
2923 time and be warned if they are ever changed. To do that:
2924 # Add the servers to the sshloginfile
2926 (echo servera; echo serverb) > .parallel/my_cluster
2927 # Make sure .ssh/config exist
2928 touch .ssh/config
2929 cp .ssh/config .ssh/config.backup
2930 # Disable StrictHostKeyChecking temporarily
2931 (echo 'Host *'; echo StrictHostKeyChecking no) >> .ssh/config
2932 parallel --slf my_cluster --nonall true
2933 # Remove the disabling of StrictHostKeyChecking
2934 mv .ssh/config.backup .ssh/config
2935 The servers in .parallel/my_cluster are now added in .ssh/known_hosts.
2937 To run echo on server.example.com:
2939 seq 10 | parallel --sshlogin server.example.com echo
2941 To run commands on more than one remote computer run:
2943 seq 10 | parallel --sshlogin server.example.com,server2.example.net echo
2945 Or:
2947 seq 10 | parallel --sshlogin server.example.com \
2949 --sshlogin server2.example.net echo
2950 If the login username is foo on server2.example.net use:
2952 seq 10 | parallel --sshlogin server.example.com \
2954 --sshlogin foo@server2.example.net echo
2955 If your list of hosts is server1-88.example.net with login foo:
2957 seq 10 | parallel -Sfoo@server{1..88}.example.net echo
2959 To distribute the commands to a list of computers, make a file
2961 mycomputers with all the computers:
2962 server.example.com
2964 foo@server2.example.com
2965 server3.example.com
2966 Then run:
2968 seq 10 | parallel --sshloginfile mycomputers echo
2970 To include the local computer add the special sshlogin ':' to the list:
2972 server.example.com
2974 foo@server2.example.com
2975 server3.example.com
2976 :
2977 GNU parallel will try to determine the number of CPU cores on each of
2979 the remote computers, and run one job per CPU core - even if the remote
2980 computers do not have the same number of CPU cores.
2981 If the number of CPU cores on the remote computers is not identified
2983 correctly the number of CPU cores can be added in front. Here the
2984 computer has 8 CPU cores.
2985 seq 10 | parallel --sshlogin 8/server.example.com echo
2987
2989 To recompress gzipped files with bzip2 using a remote computer run:
2990 find logs/ -name '*.gz' | \
2992 parallel --sshlogin server.example.com \
2993 --transfer "zcat {} | bzip2 -9 >{.}.bz2"
2994 This will list the .gz-files in the logs directory and all directories
2996 below. Then it will transfer the files to server.example.com to the
2997 corresponding directory in $HOME/logs. On server.example.com the file
2998 will be recompressed using zcat and bzip2 resulting in the
2999 corresponding file with .gz replaced with .bz2.
3000 If you want the resulting bz2-file to be transferred back to the local
3002 computer add --return {.}.bz2:
3003 find logs/ -name '*.gz' | \
3005 parallel --sshlogin server.example.com \
3006 --transfer --return {.}.bz2 "zcat {} | bzip2 -9 >{.}.bz2"
3007 After the recompressing is done the .bz2-file is transferred back to
3009 the local computer and put next to the original .gz-file.
3010 If you want to delete the transferred files on the remote computer add
3012 --cleanup. This will remove both the file transferred to the remote
3013 computer and the files transferred from the remote computer:
3014 find logs/ -name '*.gz' | \
3016 parallel --sshlogin server.example.com \
3017 --transfer --return {.}.bz2 --cleanup "zcat {} | bzip2 -9 >{.}.bz2"
3018 If you want run on several computers add the computers to --sshlogin
3020 either using ',' or multiple --sshlogin:
3021 find logs/ -name '*.gz' | \
3023 parallel --sshlogin server.example.com,server2.example.com \
3024 --sshlogin server3.example.com \
3025 --transfer --return {.}.bz2 --cleanup "zcat {} | bzip2 -9 >{.}.bz2"
3026 You can add the local computer using --sshlogin :. This will disable
3028 the removing and transferring for the local computer only:
3029 find logs/ -name '*.gz' | \
3031 parallel --sshlogin server.example.com,server2.example.com \
3032 --sshlogin server3.example.com \
3033 --sshlogin : \
3034 --transfer --return {.}.bz2 --cleanup "zcat {} | bzip2 -9 >{.}.bz2"
3035 Often --transfer, --return and --cleanup are used together. They can be
3037 shortened to --trc:
3038 find logs/ -name '*.gz' | \
3040 parallel --sshlogin server.example.com,server2.example.com \
3041 --sshlogin server3.example.com \
3042 --sshlogin : \
3043 --trc {.}.bz2 "zcat {} | bzip2 -9 >{.}.bz2"
3044 With the file mycomputers containing the list of computers it becomes:
3046 find logs/ -name '*.gz' | parallel --sshloginfile mycomputers \
3048 --trc {.}.bz2 "zcat {} | bzip2 -9 >{.}.bz2"
3049 If the file ~/.parallel/sshloginfile contains the list of computers the
3051 special short hand -S .. can be used:
3052 find logs/ -name '*.gz' | parallel -S .. \
3054 --trc {.}.bz2 "zcat {} | bzip2 -9 >{.}.bz2"
3055
3057 Convert *.mp3 to *.ogg running one process per CPU core on local
3058 computer and server2:
3059 parallel --trc {.}.ogg -S server2,: \
3061 'mpg321 -w - {} | oggenc -q0 - -o {.}.ogg' ::: *.mp3
3062
3064 To run the command uptime on remote computers you can do:
3065 parallel --tag --nonall -S server1,server2 uptime
3067 --nonall reads no arguments. If you have a list of jobs you want to run
3069 on each computer you can do:
3070 parallel --tag --onall -S server1,server2 echo ::: 1 2 3
3072 Remove --tag if you do not want the sshlogin added before the output.
3074 If you have a lot of hosts use '-j0' to access more hosts in parallel.
3076
3078 If the workers are behind a NAT wall, you need some trickery to get to
3079 them.
3080 If you can ssh to a jumphost, and reach the workers from there, then
3082 the obvious solution would be this, but it does not work:
3083 parallel --ssh 'ssh jumphost ssh' -S host1 echo ::: DOES NOT WORK
3085 It does not work because the command is dequoted by ssh twice where as
3087 GNU parallel only expects it to be dequoted once.
3088 So instead put this in ~/.ssh/config:
3090 Host host1 host2 host3
3092 ProxyCommand ssh jumphost.domain nc -w 1 %h 22
3093 It requires nc(netcat) to be installed on jumphost. With this you can
3095 simply:
3096 parallel -S host1,host2,host3 echo ::: This does work
3098 No jumphost, but port forwards
3100 If there is no jumphost but each server has port 22 forwarded from the
3101 firewall (e.g. the firewall's port 22001 = port 22 on host1, 22002 =
3102 host2, 22003 = host3) then you can use ~/.ssh/config:
3103 Host host1.v
3105 Port 22001
3106 Host host2.v
3107 Port 22002
3108 Host host3.v
3109 Port 22003
3110 Host *.v
3111 Hostname firewall
3112 And then use host{1..3}.v as normal hosts:
3114 parallel -S host1.v,host2.v,host3.v echo ::: a b c
3116 No jumphost, no port forwards
3118 If ports cannot be forwarded, you need some sort of VPN to traverse the
3119 NAT-wall. TOR is one options for that, as it is very easy to get
3120 working.
3121 You need to install TOR and setup a hidden service. In torrc put:
3123 HiddenServiceDir /var/lib/tor/hidden_service/
3125 HiddenServicePort 22 127.0.0.1:22
3126 Then start TOR: /etc/init.d/tor restart
3128 The TOR hostname is now in /var/lib/tor/hidden_service/hostname and is
3130 something similar to izjafdceobowklhz.onion. Now you simply prepend
3131 torsocks to ssh:
3132 parallel --ssh 'torsocks ssh' -S izjafdceobowklhz.onion \
3134 -S zfcdaeiojoklbwhz.onion,auclucjzobowklhi.onion echo ::: a b c
3135 If not all hosts are accessible through TOR:
3137 parallel -S 'torsocks ssh izjafdceobowklhz.onion,host2,host3' \
3139 echo ::: a b c
3140 See more ssh tricks on
3142 https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Proxies_and_Jump_Hosts
3143
3145 rsync is a great tool, but sometimes it will not fill up the available
3146 bandwidth. This is often a problem when copying several big files over
3147 high speed connections.
3148 The following will start one rsync per big file in src-dir to dest-dir
3150 on the server fooserver:
3151 cd src-dir; find . -type f -size +100000 | \
3153 parallel -v ssh fooserver mkdir -p /dest-dir/{//}\; \
3154 rsync -s -Havessh {} fooserver:/dest-dir/{}
3155 The dirs created may end up with wrong permissions and smaller files
3157 are not being transferred. To fix those run rsync a final time:
3158 rsync -Havessh src-dir/ fooserver:/dest-dir/
3160 If you are unable to push data, but need to pull them and the files are
3162 called digits.png (e.g. 000000.png) you might be able to do:
3163 seq -w 0 99 | parallel rsync -Havessh fooserver:src/*{}.png destdir/
3165
3167 Copy files like foo.es.ext to foo.ext:
3168 ls *.es.* | perl -pe 'print; s/\.es//' | parallel -N2 cp {1} {2}
3170 The perl command spits out 2 lines for each input. GNU parallel takes 2
3172 inputs (using -N2) and replaces {1} and {2} with the inputs.
3173 Count in binary:
3175 parallel -k echo ::: 0 1 ::: 0 1 ::: 0 1 ::: 0 1 ::: 0 1 ::: 0 1
3177 Print the number on the opposing sides of a six sided die:
3179 parallel --link -a <(seq 6) -a <(seq 6 -1 1) echo
3181 parallel --link echo :::: <(seq 6) <(seq 6 -1 1)
3182 Convert files from all subdirs to PNG-files with consecutive numbers
3184 (useful for making input PNG's for ffmpeg):
3185 parallel --link -a <(find . -type f | sort) \
3187 -a <(seq $(find . -type f|wc -l)) convert {1} {2}.png
3188 Alternative version:
3190 find . -type f | sort | parallel convert {} {#}.png
3192
3194 Content of table_file.tsv:
3195 foo<TAB>bar
3197 baz <TAB> quux
3198 To run:
3200 cmd -o bar -i foo
3202 cmd -o quux -i baz
3203 you can run:
3205 parallel -a table_file.tsv --colsep '\t' cmd -o {2} -i {1}
3207 Note: The default for GNU parallel is to remove the spaces around the
3209 columns. To keep the spaces:
3210 parallel -a table_file.tsv --trim n --colsep '\t' cmd -o {2} -i {1}
3212
3214 GNU parallel can output to a database table and a CSV-file:
3215 DBURL=csv:///%2Ftmp%2Fmy.csv
3217 DBTABLEURL=$DBURL/mytable
3218 parallel --sqlandworker $DBTABLEURL seq ::: {1..10}
3219 It is rather slow and takes up a lot of CPU time because GNU parallel
3221 parses the whole CSV file for each update.
3222 A better approach is to use an SQLite-base and then convert that to
3224 CSV:
3225 DBURL=sqlite3:///%2Ftmp%2Fmy.sqlite
3227 DBTABLEURL=$DBURL/mytable
3228 parallel --sqlandworker $DBTABLEURL seq ::: {1..10}
3229 sql $DBURL '.headers on' '.mode csv' 'SELECT * FROM mytable;'
3230 This takes around a second per job.
3232 If you have access to a real database system, such as PostgreSQL, it is
3234 even faster:
3235 DBURL=pg://user:pass@host/mydb
3237 DBTABLEURL=$DBURL/mytable
3238 parallel --sqlandworker $DBTABLEURL seq ::: {1..10}
3239 sql $DBURL "COPY (SELECT * FROM mytable) TO stdout DELIMITER ',' CSV HEADER;"
3240 Or MySQL:
3242 DBURL=mysql://user:pass@host/mydb
3244 DBTABLEURL=$DBURL/mytable
3245 parallel --sqlandworker $DBTABLEURL seq ::: {1..10}
3246 sql -p -B $DBURL "SELECT * FROM mytable;" > mytable.tsv
3247 perl -pe 's/"/""/g; s/\t/","/g; s/^/"/; s/$/"/; s/\\\\/\\/g;
3248 s/\\t/\t/g; s/\\n/\n/g;' mytable.tsv
3249
3251 If you have no need for the advanced job distribution control that a
3252 database provides, but you simply want output into a CSV file that you
3253 can read into R or LibreCalc, then you can use --results:
3254 parallel --results my.csv seq ::: 10 20 30
3256 R
3257 > mydf <- read.csv("my.csv");
3258 > print(mydf[2,])
3259 > write(as.character(mydf[2,c("Stdout")]),'')
3260
3262 The show Aflyttet on Radio 24syv publishes an RSS feed with their audio
3263 podcasts on: http://arkiv.radio24syv.dk/audiopodcast/channel/4466232
3264 Using xpath you can extract the URLs for 2016 and download them using
3266 GNU parallel:
3267 wget -O - http://arkiv.radio24syv.dk/audiopodcast/channel/4466232 |
3269 xpath -e "//ancestor::pubDate[contains(text(),'2016')]/../enclosure/@url" |
3270 parallel -u wget '{= s/ url="//; s/"//; =}'
3271
3273 If you want to run the same command with the same arguments 10 times in
3274 parallel you can do:
3275 seq 10 | parallel -n0 my_command my_args
3277
3279 GNU parallel can work similar to cat | sh.
3280 A resource inexpensive job is a job that takes very little CPU, disk
3282 I/O and network I/O. Ping is an example of a resource inexpensive job.
3283 wget is too - if the webpages are small.
3284 The content of the file jobs_to_run:
3286 ping -c 1 10.0.0.1
3288 wget http://example.com/status.cgi?ip=10.0.0.1
3289 ping -c 1 10.0.0.2
3290 wget http://example.com/status.cgi?ip=10.0.0.2
3291 ...
3292 ping -c 1 10.0.0.255
3293 wget http://example.com/status.cgi?ip=10.0.0.255
3294 To run 100 processes simultaneously do:
3296 parallel -j 100 < jobs_to_run
3298 As there is not a command the jobs will be evaluated by the shell.
3300
3302 To process a big file or some output you can use --pipe to split up the
3303 data into blocks and pipe the blocks into the processing program.
3304 If the program is gzip -9 you can do:
3306 cat bigfile | parallel --pipe --recend '' -k gzip -9 > bigfile.gz
3308 This will split bigfile into blocks of 1 MB and pass that to gzip -9 in
3310 parallel. One gzip will be run per CPU core. The output of gzip -9 will
3311 be kept in order and saved to bigfile.gz
3312 gzip works fine if the output is appended, but some processing does not
3314 work like that - for example sorting. For this GNU parallel can put the
3315 output of each command into a file. This will sort a big file in
3316 parallel:
3317 cat bigfile | parallel --pipe --files sort |\
3319 parallel -Xj1 sort -m {} ';' rm {} >bigfile.sort
3320 Here bigfile is split into blocks of around 1MB, each block ending in
3322 '\n' (which is the default for --recend). Each block is passed to sort
3323 and the output from sort is saved into files. These files are passed to
3324 the second parallel that runs sort -m on the files before it removes
3325 the files. The output is saved to bigfile.sort.
3326 GNU parallel's --pipe maxes out at around 100 MB/s because every byte
3328 has to be copied through GNU parallel. But if bigfile is a real
3329 (seekable) file GNU parallel can by-pass the copying and send the parts
3330 directly to the program:
3331 parallel --pipepart --block 100m -a bigfile --files sort |\
3333 parallel -Xj1 sort -m {} ';' rm {} >bigfile.sort
3334
3336 When processing with --pipe you may have lines grouped by a value. Here
3337 is my.csv:
3338 Transaction Customer Item
3340 1 a 53
3341 2 b 65
3342 3 b 82
3343 4 c 96
3344 5 c 67
3345 6 c 13
3346 7 d 90
3347 8 d 43
3348 9 d 91
3349 10 d 84
3350 11 e 72
3351 12 e 102
3352 13 e 63
3353 14 e 56
3354 15 e 74
3355 Let us assume you want GNU parallel to process each customer. In other
3357 words: You want all the transactions for a single customer to be
3358 treated as a single record.
3359 To do this we preprocess the data with a program that inserts a record
3361 separator before each customer (column 2 = $F[1]). Here we first make a
3362 50 character random string, which we then use as the separator:
3363 sep=`perl -e 'print map { ("a".."z","A".."Z")[rand(52)] } (1..50);'`
3365 cat my.csv | perl -ape '$F[1] ne $l and print "'$sep'"; $l = $F[1]' |
3366 parallel --recend $sep --rrs --pipe -N1 wc
3367 If your program can process multiple customers replace -N1 with a
3369 reasonable --blocksize.
3370
3372 If you need to run a massive amount of jobs in parallel, then you will
3373 likely hit the filehandle limit which is often around 250 jobs. If you
3374 are super user you can raise the limit in /etc/security/limits.conf but
3375 you can also use this workaround. The filehandle limit is per process.
3376 That means that if you just spawn more GNU parallels then each of them
3377 can run 250 jobs. This will spawn up to 2500 jobs:
3378 cat myinput |\
3380 parallel --pipe -N 50 --round-robin -j50 parallel -j50 your_prg
3381 This will spawn up to 62500 jobs (use with caution - you need 64 GB RAM
3383 to do this, and you may need to increase /proc/sys/kernel/pid_max):
3384 cat myinput |\
3386 parallel --pipe -N 250 --round-robin -j250 parallel -j250 your_prg
3387
3389 The command sem is an alias for parallel --semaphore.
3390 A counting semaphore will allow a given number of jobs to be started in
3392 the background. When the number of jobs are running in the background,
3393 GNU sem will wait for one of these to complete before starting another
3394 command. sem --wait will wait for all jobs to complete.
3395 Run 10 jobs concurrently in the background:
3397 for i in *.log ; do
3399 echo $i
3400 sem -j10 gzip $i ";" echo done
3401 done
3402 sem --wait
3403 A mutex is a counting semaphore allowing only one job to run. This will
3405 edit the file myfile and prepends the file with lines with the numbers
3406 1 to 3.
3407 seq 3 | parallel sem sed -i -e '1i{}' myfile
3409 As myfile can be very big it is important only one process edits the
3411 file at the same time.
3412 Name the semaphore to have multiple different semaphores active at the
3414 same time:
3415 seq 3 | parallel sem --id mymutex sed -i -e 'i{}' myfile
3417
3419 Assume a script is called from cron or from a web service, but only one
3420 instance can be run at a time. With sem and --shebang-wrap the script
3421 can be made to wait for other instances to finish. Here in bash:
3422 #!/usr/bin/sem --shebang-wrap -u --id $0 --fg /bin/bash
3424 echo This will run
3426 sleep 5
3427 echo exclusively
3428 Here perl:
3430 #!/usr/bin/sem --shebang-wrap -u --id $0 --fg /usr/bin/perl
3432 print "This will run ";
3434 sleep 5;
3435 print "exclusively\n";
3436 Here python:
3438 #!/usr/local/bin/sem --shebang-wrap -u --id $0 --fg /usr/bin/python
3440 import time
3442 print "This will run ";
3443 time.sleep(5)
3444 print "exclusively";
3445
3447 You can use GNU parallel to start interactive programs like emacs or
3448 vi:
3449 cat filelist | parallel --tty -X emacs
3451 cat filelist | parallel --tty -X vi
3452 If there are more files than will fit on a single command line, the
3454 editor will be started again with the remaining files.
3455
3457 sudo requires a password to run a command as root. It caches the
3458 access, so you only need to enter the password again if you have not
3459 used sudo for a while.
3460 The command:
3462 parallel sudo echo ::: This is a bad idea
3464 is no good, as you would be prompted for the sudo password for each of
3466 the jobs. You can either do:
3467 sudo echo This
3469 parallel sudo echo ::: is a good idea
3470 or:
3472 sudo parallel echo ::: This is a good idea
3474 This way you only have to enter the sudo password once.
3476
3478 GNU parallel can work as a simple job queue system or batch manager.
3479 The idea is to put the jobs into a file and have GNU parallel read from
3480 that continuously. As GNU parallel will stop at end of file we use tail
3481 to continue reading:
3482 true >jobqueue; tail -n+0 -f jobqueue | parallel
3484 To submit your jobs to the queue:
3486 echo my_command my_arg >> jobqueue
3488 You can of course use -S to distribute the jobs to remote computers:
3490 true >jobqueue; tail -n+0 -f jobqueue | parallel -S ..
3492 If you keep this running for a long time, jobqueue will grow. A way of
3494 removing the jobs already run is by making GNU parallel stop when it
3495 hits a special value and then restart. To use --eof to make GNU
3496 parallel exit, tail also needs to be forced to exit:
3497 true >jobqueue;
3499 while true; do
3500 tail -n+0 -f jobqueue |
3501 (parallel -E StOpHeRe -S ..; echo GNU Parallel is now done;
3502 perl -e 'while(<>){/StOpHeRe/ and last};print <>' jobqueue > j2;
3503 (seq 1000 >> jobqueue &);
3504 echo Done appending dummy data forcing tail to exit)
3505 echo tail exited;
3506 mv j2 jobqueue
3507 done
3508 In some cases you can run on more CPUs and computers during the night:
3510 # Day time
3512 echo 50% > jobfile
3513 cp day_server_list ~/.parallel/sshloginfile
3514 # Night time
3515 echo 100% > jobfile
3516 cp night_server_list ~/.parallel/sshloginfile
3517 tail -n+0 -f jobqueue | parallel --jobs jobfile -S ..
3518 GNU Parallel discovers if jobfile or ~/.parallel/sshloginfile changes.
3520 There is a a small issue when using GNU parallel as queue system/batch
3522 manager: You have to submit JobSlot number of jobs before they will
3523 start, and after that you can submit one at a time, and job will start
3524 immediately if free slots are available. Output from the running or
3525 completed jobs are held back and will only be printed when JobSlots
3526 more jobs has been started (unless you use --ungroup or --line-buffer,
3527 in which case the output from the jobs are printed immediately). E.g.
3528 if you have 10 jobslots then the output from the first completed job
3529 will only be printed when job 11 has started, and the output of second
3530 completed job will only be printed when job 12 has started.
3531
3533 If you have a dir in which users drop files that needs to be processed
3534 you can do this on GNU/Linux (If you know what inotifywait is called on
3535 other platforms file a bug report):
3536 inotifywait -qmre MOVED_TO -e CLOSE_WRITE --format %w%f my_dir |\
3538 parallel -u echo
3539 This will run the command echo on each file put into my_dir or subdirs
3541 of my_dir.
3542 You can of course use -S to distribute the jobs to remote computers:
3544 inotifywait -qmre MOVED_TO -e CLOSE_WRITE --format %w%f my_dir |\
3546 parallel -S .. -u echo
3547 If the files to be processed are in a tar file then unpacking one file
3549 and processing it immediately may be faster than first unpacking all
3550 files. Set up the dir processor as above and unpack into the dir.
3551 Using GNU Parallel as dir processor has the same limitations as using
3553 GNU Parallel as queue system/batch manager.
3554
3556 If you have downloaded source and tried compiling it, you may have
3557 seen:
3558 $ ./configure
3560 [...]
3561 checking for something.h... no
3562 configure: error: "libsomething not found"
3563 Often it is not obvious which package you should install to get that
3565 file. Debian has `apt-file` to search for a file. `tracefile` from
3566 https://gitlab.com/ole.tange/tangetools can tell which files a program
3567 tried to access. In this case we are interested in one of the last
3568 files:
3569 $ tracefile -un ./configure | tail | parallel -j0 apt-file search
3571
3573 GNU parallel is very liberal in quoting. You only need to quote
3574 characters that have special meaning in shell:
3575 ( ) $ ` ' " < > ; | \
3577 and depending on context these needs to be quoted, too:
3579 ~ & # ! ? space * {
3581 Therefore most people will never need more quoting than putting '\' in
3583 front of the special characters.
3584 Often you can simply put \' around every ':
3586 perl -ne '/^\S+\s+\S+$/ and print $ARGV,"\n"' file
3588 can be quoted:
3590 parallel perl -ne \''/^\S+\s+\S+$/ and print $ARGV,"\n"'\' ::: file
3592 However, when you want to use a shell variable you need to quote the
3594 $-sign. Here is an example using $PARALLEL_SEQ. This variable is set by
3595 GNU parallel itself, so the evaluation of the $ must be done by the sub
3596 shell started by GNU parallel:
3597 seq 10 | parallel -N2 echo seq:\$PARALLEL_SEQ arg1:{1} arg2:{2}
3599 If the variable is set before GNU parallel starts you can do this:
3601 VAR=this_is_set_before_starting
3603 echo test | parallel echo {} $VAR
3604 Prints: test this_is_set_before_starting
3606 It is a little more tricky if the variable contains more than one space
3608 in a row:
3609 VAR="two spaces between each word"
3611 echo test | parallel echo {} \'"$VAR"\'
3612 Prints: test two spaces between each word
3614 If the variable should not be evaluated by the shell starting GNU
3616 parallel but be evaluated by the sub shell started by GNU parallel,
3617 then you need to quote it:
3618 echo test | parallel VAR=this_is_set_after_starting \; echo {} \$VAR
3620 Prints: test this_is_set_after_starting
3622 It is a little more tricky if the variable contains space:
3624 echo test |\
3626 parallel VAR='"two spaces between each word"' echo {} \'"$VAR"\'
3627 Prints: test two spaces between each word
3629 $$ is the shell variable containing the process id of the shell. This
3631 will print the process id of the shell running GNU parallel:
3632 seq 10 | parallel echo $$
3634 And this will print the process ids of the sub shells started by GNU
3636 parallel.
3637 seq 10 | parallel echo \$\$
3639 If the special characters should not be evaluated by the sub shell then
3641 you need to protect it against evaluation from both the shell starting
3642 GNU parallel and the sub shell:
3643 echo test | parallel echo {} \\\$VAR
3645 Prints: test $VAR
3647 GNU parallel can protect against evaluation by the sub shell by using
3649 -q:
3650 echo test | parallel -q echo {} \$VAR
3652 Prints: test $VAR
3654 This is particularly useful if you have lots of quoting. If you want to
3656 run a perl script like this:
3657 perl -ne '/^\S+\s+\S+$/ and print $ARGV,"\n"' file
3659 It needs to be quoted like one of these:
3661 ls | parallel perl -ne '/^\\S+\\s+\\S+\$/\ and\ print\ \$ARGV,\"\\n\"'
3663 ls | parallel perl -ne \''/^\S+\s+\S+$/ and print $ARGV,"\n"'\'
3664 Notice how spaces, \'s, "'s, and $'s need to be quoted. GNU parallel
3666 can do the quoting by using option -q:
3667 ls | parallel -q perl -ne '/^\S+\s+\S+$/ and print $ARGV,"\n"'
3669 However, this means you cannot make the sub shell interpret special
3671 characters. For example because of -q this WILL NOT WORK:
3672 ls *.gz | parallel -q "zcat {} >{.}"
3674 ls *.gz | parallel -q "zcat {} | bzip2 >{.}.bz2"
3675 because > and | need to be interpreted by the sub shell.
3677 If you get errors like:
3679 sh: -c: line 0: syntax error near unexpected token
3681 sh: Syntax error: Unterminated quoted string
3682 sh: -c: line 0: unexpected EOF while looking for matching `''
3683 sh: -c: line 1: syntax error: unexpected end of file
3684 then you might try using -q.
3686 If you are using bash process substitution like <(cat foo) then you may
3688 try -q and prepending command with bash -c:
3689 ls | parallel -q bash -c 'wc -c <(echo {})'
3691 Or for substituting output:
3693 ls | parallel -q bash -c \
3695 'tar c {} | tee >(gzip >{}.tar.gz) | bzip2 >{}.tar.bz2'
3696 Conclusion: To avoid dealing with the quoting problems it may be easier
3698 just to write a small script or a function (remember to export -f the
3699 function) and have GNU parallel call that.
3700
3702 If you want a list of the jobs currently running you can run:
3703 killall -USR1 parallel
3705 GNU parallel will then print the currently running jobs on stderr
3707 (standard error).
3708
3710 If you regret starting a lot of jobs you can simply break GNU parallel,
3711 but if you want to make sure you do not have half-completed jobs you
3712 should send the signal SIGTERM to GNU parallel:
3713 killall -TERM parallel
3715 This will tell GNU parallel to not start any new jobs, but wait until
3717 the currently running jobs are finished before exiting.
3718
3720 $PARALLEL_HOME
3721 Dir where GNU parallel stores config files, semaphores, and
3722 caches information between invocations. Default:
3723 $HOME/.parallel.
3724 $PARALLEL_PID
3726 The environment variable $PARALLEL_PID is set by GNU parallel
3727 and is visible to the jobs started from GNU parallel. This
3728 makes it possible for the jobs to communicate directly to GNU
3729 parallel. Remember to quote the $, so it gets evaluated by
3730 the correct shell.
3731 Example: If each of the jobs tests a solution and one of jobs
3733 finds the solution the job can tell GNU parallel not to start
3734 more jobs by: kill -TERM $PARALLEL_PID. This only works on the
3735 local computer.
3736 $PARALLEL_RSYNC_OPTS
3738 Options to pass on to rsync. Defaults to: -rlDzR.
3739 $PARALLEL_SHELL
3741 Use this shell for the commands run by GNU Parallel:
3742 · $PARALLEL_SHELL. If undefined use:
3744 · The shell that started GNU Parallel. If that cannot be
3746 determined:
3747 · $SHELL. If undefined use:
3749 · /bin/sh
3751 $PARALLEL_SSH
3753 GNU parallel defaults to using ssh for remote access. This can
3754 be overridden with $PARALLEL_SSH, which again can be
3755 overridden with --ssh. It can also be set on a per server
3756 basis (see --sshlogin).
3757 $PARALLEL_SEQ
3759 $PARALLEL_SEQ will be set to the sequence number of the job
3760 running. Remember to quote the $, so it gets evaluated by the
3761 correct shell.
3762 Example:
3764 seq 10 | parallel -N2 \
3766 echo seq:'$'PARALLEL_SEQ arg1:{1} arg2:{2}
3767 $PARALLEL_TMUX
3769 Path to tmux. If unset the tmux in $PATH is used.
3770 $TMPDIR Directory for temporary files. See: --tmpdir.
3772 $PARALLEL
3774 The environment variable $PARALLEL will be used as default
3775 options for GNU parallel. If the variable contains special
3776 shell characters (e.g. $, *, or space) then these need to be
3777 to be escaped with \.
3778 Example:
3780 cat list | parallel -j1 -k -v ls
3782 cat list | parallel -j1 -k -v -S"myssh user@server" ls
3783 can be written as:
3785 cat list | PARALLEL="-kvj1" parallel ls
3787 cat list | PARALLEL='-kvj1 -S myssh\ user@server' \
3788 parallel echo
3789 Notice the \ in the middle is needed because 'myssh' and
3791 'user@server' must be one argument.
3792
3794 The global configuration file /etc/parallel/config, followed by user
3795 configuration file ~/.parallel/config (formerly known as .parallelrc)
3796 will be read in turn if they exist. Lines starting with '#' will be
3797 ignored. The format can follow that of the environment variable
3798 $PARALLEL, but it is often easier to simply put each option on its own
3799 line.
3800 Options on the command line take precedence, followed by the
3802 environment variable $PARALLEL, user configuration file
3803 ~/.parallel/config, and finally the global configuration file
3804 /etc/parallel/config.
3805 Note that no file that is read for options, nor the environment
3807 variable $PARALLEL, may contain retired options such as --tollef.
3808
3810 If --profile set, GNU parallel will read the profile from that file
3811 rather than the global or user configuration files. You can have
3812 multiple --profiles.
3813 Example: Profile for running a command on every sshlogin in
3815 ~/.ssh/sshlogins and prepend the output with the sshlogin:
3816 echo --tag -S .. --nonall > ~/.parallel/n
3818 parallel -Jn uptime
3819 Example: Profile for running every command with -j-1 and nice
3821 echo -j-1 nice > ~/.parallel/nice_profile
3823 parallel -J nice_profile bzip2 -9 ::: *
3824 Example: Profile for running a perl script before every command:
3826 echo "perl -e '\$a=\$\$; print \$a,\" \",'\$PARALLEL_SEQ',\" \";';" \
3828 > ~/.parallel/pre_perl
3829 parallel -J pre_perl echo ::: *
3830 Note how the $ and " need to be quoted using \.
3832 Example: Profile for running distributed jobs with nice on the remote
3834 computers:
3835 echo -S .. nice > ~/.parallel/dist
3837 parallel -J dist --trc {.}.bz2 bzip2 -9 ::: *
3838
3840 Exit status depends on --halt-on-error if one of these is used:
3841 success=X, success=Y%, fail=Y%.
3842 0 All jobs ran without error. If success=X is used: X jobs ran
3844 without error. If success=Y% is used: Y% of the jobs ran without
3845 error.
3846 1-100 Some of the jobs failed. The exit status gives the number of
3848 failed jobs. If Y% is used the exit status is the percentage of
3849 jobs that failed.
3850 101 More than 100 jobs failed.
3852 255 Other error.
3854 -1 (In joblog and SQL table)
3856 Killed by Ctrl-C, timeout, not enough memory or similar.
3857 -2 (In joblog and SQL table)
3859 skip() was called in {= =}.
3860 -1000 (In SQL table)
3862 Job is ready to run (set by --sqlmaster).
3863 -1220 (In SQL table)
3865 Job is taken by worker (set by --sqlworker).
3866 If fail=1 is used, the exit status will be the exit status of the
3868 failing job.
3869
3871 See: man parallel_alternatives
3872
3874 Quoting of newline
3875 Because of the way newline is quoted this will not work:
3876 echo 1,2,3 | parallel -vkd, "echo 'a{}b'"
3878 However, these will all work:
3880 echo 1,2,3 | parallel -vkd, echo a{}b
3882 echo 1,2,3 | parallel -vkd, "echo 'a'{}'b'"
3883 echo 1,2,3 | parallel -vkd, "echo 'a'"{}"'b'"
3884 Speed
3886 Startup
3887 GNU parallel is slow at starting up - around 250 ms the first time and
3889 150 ms after that.
3890 Job startup
3892 Starting a job on the local machine takes around 10 ms. This can be a
3894 big overhead if the job takes very few ms to run. Often you can group
3895 small jobs together using -X which will make the overhead less
3896 significant. Or you can run multiple GNU parallels as described in
3897 EXAMPLE: Speeding up fast jobs.
3898 SSH
3900 When using multiple computers GNU parallel opens ssh connections to
3902 them to figure out how many connections can be used reliably
3903 simultaneously (Namely SSHD's MaxStartups). This test is done for each
3904 host in serial, so if your --sshloginfile contains many hosts it may be
3905 slow.
3906 If your jobs are short you may see that there are fewer jobs running on
3908 the remote systems than expected. This is due to time spent logging in
3909 and out. -M may help here.
3910 Disk access
3912 A single disk can normally read data faster if it reads one file at a
3914 time instead of reading a lot of files in parallel, as this will avoid
3915 disk seeks. However, newer disk systems with multiple drives can read
3916 faster if reading from multiple files in parallel.
3917 If the jobs are of the form read-all-compute-all-write-all, so
3919 everything is read before anything is written, it may be faster to
3920 force only one disk access at the time:
3921 sem --id diskio cat file | compute | sem --id diskio cat > file
3923 If the jobs are of the form read-compute-write, so writing starts
3925 before all reading is done, it may be faster to force only one reader
3926 and writer at the time:
3927 sem --id read cat file | compute | sem --id write cat > file
3929 If the jobs are of the form read-compute-read-compute, it may be faster
3931 to run more jobs in parallel than the system has CPUs, as some of the
3932 jobs will be stuck waiting for disk access.
3933 --nice limits command length
3935 The current implementation of --nice is too pessimistic in the max
3936 allowed command length. It only uses a little more than half of what it
3937 could. This affects -X and -m. If this becomes a real problem for you,
3938 file a bug-report.
3939 Aliases and functions do not work
3941 If you get:
3942 Can't exec "command": No such file or directory
3944 or:
3946 open3: exec of by command failed
3948 or:
3950 /bin/bash: command: command not found
3952 it may be because command is not known, but it could also be because
3954 command is an alias or a function. If it is a function you need to
3955 export -f the function first or use env_parallel. An alias will only
3956 work if you use env_parallel.
3957 Database with MySQL fails randomly
3959 The --sql* options may fail randomly with MySQL. This problem does not
3960 exist with PostgreSQL.
3961
3963 Report bugs to <bug-parallel@gnu.org> or
3964 https://savannah.gnu.org/bugs/?func=additem&group=parallel
3965 See a perfect bug report on
3967 https://lists.gnu.org/archive/html/bug-parallel/2015-01/msg00000.html
3968 Your bug report should always include:
3970 · The error message you get (if any). If the error message is not from
3972 GNU parallel you need to show why you think GNU parallel caused
3973 these.
3974 · The complete output of parallel --version. If you are not running the
3976 latest released version (see http://ftp.gnu.org/gnu/parallel/) you
3977 should specify why you believe the problem is not fixed in that
3978 version.
3979 · A minimal, complete, and verifiable example (See description on
3981 http://stackoverflow.com/help/mcve).
3982 It should be a complete example that others can run that shows the
3984 problem including all files needed to run the example. This should
3985 preferably be small and simple, so try to remove as many options as
3986 possible. A combination of yes, seq, cat, echo, and sleep can
3987 reproduce most errors. If your example requires large files, see if
3988 you can make them by something like seq 1000000 > file or yes | head
3989 -n 10000000 > file.
3990 If your example requires remote execution, see if you can use
3992 localhost - maybe using another login.
3993 If you have access to a different system, test if the MCVE shows the
3995 problem on that system.
3996 · The output of your example. If your problem is not easily reproduced
3998 by others, the output might help them figure out the problem.
3999 · Whether you have watched the intro videos
4001 (http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1), walked
4002 through the tutorial (man parallel_tutorial), and read the EXAMPLE
4003 section in the man page (man parallel - search for EXAMPLE:).
4004 If you suspect the error is dependent on your environment or
4006 distribution, please see if you can reproduce the error on one of these
4007 VirtualBox images:
4008 http://sourceforge.net/projects/virtualboximage/files/
4009 http://www.osboxes.org/virtualbox-images/
4010 Specifying the name of your distribution is not enough as you may have
4012 installed software that is not in the VirtualBox images.
4013 If you cannot reproduce the error on any of the VirtualBox images
4015 above, see if you can build a VirtualBox image on which you can
4016 reproduce the error. If not you should assume the debugging will be
4017 done through you. That will put more burden on you and it is extra
4018 important you give any information that help. In general the problem
4019 will be fixed faster and with less work for you if you can reproduce
4020 the error on a VirtualBox.
4021
4023 When using GNU parallel for a publication please cite:
4024 O. Tange (2011): GNU Parallel - The Command-Line Power Tool, ;login:
4026 The USENIX Magazine, February 2011:42-47.
4027 This helps funding further development; and it won't cost you a cent.
4029 If you pay 10000 EUR you should feel free to use GNU Parallel without
4030 citing.
4031 Copyright (C) 2007-10-18 Ole Tange, http://ole.tange.dk
4033 Copyright (C) 2008,2009,2010 Ole Tange, http://ole.tange.dk
4035 Copyright (C) 2010,2011,2012,2013,2014,2015,2016,2017,2018 Ole Tange,
4037 http://ole.tange.dk and Free Software Foundation, Inc.
4038 Parts of the manual concerning xargs compatibility is inspired by the
4040 manual of xargs from GNU findutils 4.4.2.
4041
4043 Copyright (C)
4044 2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017,2018 Free
4045 Software Foundation, Inc.
4046 This program is free software; you can redistribute it and/or modify it
4048 under the terms of the GNU General Public License as published by the
4049 Free Software Foundation; either version 3 of the License, or at your
4050 option any later version.
4051 This program is distributed in the hope that it will be useful, but
4053 WITHOUT ANY WARRANTY; without even the implied warranty of
4054 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
4055 General Public License for more details.
4056 You should have received a copy of the GNU General Public License along
4058 with this program. If not, see <http://www.gnu.org/licenses/>.
4059 Documentation license I
4061 Permission is granted to copy, distribute and/or modify this
4062 documentation under the terms of the GNU Free Documentation License,
4063 Version 1.3 or any later version published by the Free Software
4064 Foundation; with no Invariant Sections, with no Front-Cover Texts, and
4065 with no Back-Cover Texts. A copy of the license is included in the
4066 file fdl.txt.
4067 Documentation license II
4069 You are free:
4070 to Share to copy, distribute and transmit the work
4072 to Remix to adapt the work
4074 Under the following conditions:
4076 Attribution
4078 You must attribute the work in the manner specified by the
4079 author or licensor (but not in any way that suggests that they
4080 endorse you or your use of the work).
4081 Share Alike
4083 If you alter, transform, or build upon this work, you may
4084 distribute the resulting work only under the same, similar or
4085 a compatible license.
4086 With the understanding that:
4088 Waiver Any of the above conditions can be waived if you get
4090 permission from the copyright holder.
4091 Public Domain
4093 Where the work or any of its elements is in the public domain
4094 under applicable law, that status is in no way affected by the
4095 license.
4096 Other Rights
4098 In no way are any of the following rights affected by the
4099 license:
4100 · Your fair dealing or fair use rights, or other applicable
4102 copyright exceptions and limitations;
4103 · The author's moral rights;
4105 · Rights other persons may have either in the work itself or
4107 in how the work is used, such as publicity or privacy
4108 rights.
4109 Notice For any reuse or distribution, you must make clear to others
4111 the license terms of this work.
4112 A copy of the full license is included in the file as cc-by-sa.txt.
4114
4116 GNU parallel uses Perl, and the Perl modules Getopt::Long, IPC::Open3,
4117 Symbol, IO::File, POSIX, and File::Temp. For remote usage it also uses
4118 rsync with ssh.
4119
4121 ssh(1), ssh-agent(1), sshpass(1), ssh-copy-id(1), rsync(1), find(1),
4122 xargs(1), dirname(1), make(1), pexec(1), ppss(1), xjobs(1), prll(1),
4123 dxargs(1), mdm(1)
412420180222 2018-03-19 PARALLEL(1)