- Notifications
You must be signed in to change notification settings - Fork10
Manual
- Command line options
- Commands from arguments
- Commands from stdin
- Command and initial arguments on command line
- Reading multiple inputs
- Parallelism
- Dry run
- Debug logging
- Error handling
- Timeout
- Path cache
- Progress bar
- Regular Expression
- Shell Commands
- Bash Function
$ rust-parallel --helpExecute commands in parallelBy Aaron Riekenberg <aaron.riekenberg@gmail.com>https://github.com/aaronriekenberg/rust-parallelhttps://crates.io/crates/rust-parallelUsage: rust-parallel [OPTIONS] [COMMAND_AND_INITIAL_ARGUMENTS]...Arguments: [COMMAND_AND_INITIAL_ARGUMENTS]... Optional command and initial arguments. If this contains 1 or more ::: delimiters the cartesian product of arguments from all groups are run.Options: -d, --discard-output <DISCARD_OUTPUT> Discard output for commands Possible values: - stdout: Redirect stdout for commands to /dev/null - stderr: Redirect stderr for commands to /dev/null - all: Redirect stdout and stderr for commands to /dev/null -i, --input-file <INPUT_FILE> Input file or - for stdin. Defaults to stdin if no inputs are specified -j, --jobs <JOBS> Maximum number of commands to run in parallel, defauts to num cpus [default: 8] -0, --null-separator Use null separator for reading input files instead of newline -p, --progress-bar Display progress bar -r, --regex <REGEX> Apply regex pattern to inputs -s, --shell Use shell mode for running commands. Each command line is passed to "<shell-path> <shell-argument>" as a single argument. -t, --timeout-seconds <TIMEOUT_SECONDS> Timeout seconds for running commands. Defaults to infinite timeout if not specified --channel-capacity <CHANNEL_CAPACITY> Input and output channel capacity, defaults to num cpus * 2 [default: 16] --disable-path-cache Disable command path cache --dry-run Dry run mode Do not actually run commands just log. --exit-on-error Exit on error mode Exit immediately when a command fails. --no-run-if-empty Do not run commands for empty buffered input lines --shell-path <SHELL_PATH> Path to shell to use for shell mode [default: /bin/bash] --shell-argument <SHELL_ARGUMENT> Argument to shell for shell mode [default: -c] -h, --help Print help (see a summary with '-h') -V, --version Print versionThe::: separator can be used to run theCartesian Product of command line arguments. This is similar to the::: behavior in GNU Parallel.
$ rust-parallel echo ::: A B ::: C D ::: E F GA C EA C FA D GA D FA D EA C GB C EB C FB D EB C GB D FB D G$ rust-parallel echo hello ::: larry curly moehello moehello curlyhello larry# run gzip -k on all *.html files in current directory$ rust-parallel gzip -k ::: *.htmlWhen using commands from arguments, numbered variables{0},{1}, etc are automatically available based on the number of arguments.{0} will be replaced by the entire input line, and other groups match individual argument groups. This is useful for building more complex command lines. For example:
$ rust-parallel echo group0={0} group1={1} group2={2} group3={3} group2again={2} ::: A B ::: C D ::: E F Ggroup0=A C E group1=A group2=C group3=E group2again=Cgroup0=A C G group1=A group2=C group3=G group2again=Cgroup0=A C F group1=A group2=C group3=F group2again=Cgroup0=A D G group1=A group2=D group3=G group2again=Dgroup0=B C F group1=B group2=C group3=F group2again=Cgroup0=A D F group1=A group2=D group3=F group2again=Dgroup0=A D E group1=A group2=D group3=E group2again=Dgroup0=B C E group1=B group2=C group3=E group2again=Cgroup0=B D E group1=B group2=D group3=E group2again=Dgroup0=B C G group1=B group2=C group3=G group2again=Cgroup0=B D F group1=B group2=D group3=F group2again=Dgroup0=B D G group1=B group2=D group3=G group2again=DInternally these variables are implemented using an auto-generatedregular expression. If a regular expression is manually specified this will override the auto-generated one.
Run complete commands from stdin.
$ cat >./test <<EOLecho hiecho thereecho howecho areecho youEOL$ cat test | rust-parallelhiareyoutherehowHeremd5 -s will be prepended to each input line to form a command likemd5 -s aal
$ head -100 /usr/share/dict/words | rust-parallel md5 -s | head -10MD5 ("Aani") = e9b22dd6213c3d29648e8ad7a8642f2fMD5 ("aardvark") = 88571e5d5e13a4a60f82cea7802f6255MD5 ("aa") = 4124bc0a9335c27f086f24ba207a4912MD5 ("aalii") = 0a1ea2a8d75d02ae052f8222e36927a5MD5 ("a") = 0cc175b9c0f1b6a831c399e269772661MD5 ("A") = 7fc56270e7a70fa81a5935b72eacbe29MD5 ("aam") = 35c2d90f7c06b623fe763d0a4e5b7ed9MD5 ("aal") = ff45e881572ca2c987460932660d320cMD5 ("Aaronic") = 0390cf1718c4f2d76f770c7c35b40c50MD5 ("aardwolf") = 66a4a1a2b442e8d218e8e99100069877By defaultrust-parallel reads input from stdin only. The-i option can be used 1 or more times to override this behavior.-i - means read from stdin,-i ./test means read from the file./test:
$ cat >./test <<EOLfoobarbazEOL$ head -5 /usr/share/dict/words | rust-parallel -i - -i ./test echoAaaaalbaraaliibazfooaBy default the number of parallel jobs to run simulatenously is the number of cpus detected at run time.
This can be override with the-j/--jobs option.
With-j5 all echo commands below run in parallel.
With-j1 all jobs run sequentially.
$ rust-parallel -j5 echo ::: hi there how are youhowhiyouthereare$ rust-parallel -j1 echo ::: hi there how are youhitherehowareyouUse option--dry-run for dry run mode.
In this mode the commands that would be run are ouput as info level logs.
No commands are actually run - this is useful for testing before running a job.
$ rust-parallel --dry-run echo ::: hi there how are you2024-05-04T16:08:20.294681Z INFO rust_parallel::command: cmd="/bin/echo",args=["hi"],line=command_line_args:12024-05-04T16:08:20.294705Z INFO rust_parallel::command: cmd="/bin/echo",args=["there"],line=command_line_args:22024-05-04T16:08:20.294713Z INFO rust_parallel::command: cmd="/bin/echo",args=["how"],line=command_line_args:32024-05-04T16:08:20.294721Z INFO rust_parallel::command: cmd="/bin/echo",args=["are"],line=command_line_args:42024-05-04T16:08:20.294728Z INFO rust_parallel::command: cmd="/bin/echo",args=["you"],line=command_line_args:5Set environment variableRUST_LOG=debug to see debug output.
This logs structured information about command line arguments and commands being run.
Recommend enabling debug logging for all examples to understand what is happening in more detail.
$ RUST_LOG=debug rust-parallel echo ::: hi there how are you | grep command_line_args | head -12024-05-04T16:08:20.297176Z DEBUG try_main: rust_parallel::command_line_args: command_line_args = CommandLineArgs { discard_output: None, input_file: [], jobs: 8, null_separator: false, progress_bar: false, regex: None, shell: false, timeout_seconds: None, channel_capacity: 16, disable_path_cache: false, dry_run: false, exit_on_error: false, no_run_if_empty: false, shell_path: "/bin/bash", shell_argument: "-c", command_and_initial_arguments: ["echo", ":::", "hi", "there", "how", "are", "you"] }$ RUST_LOG=debug rust-parallel echo ::: hi there how are you | grep command_line_args:12024-05-04T16:08:20.302757Z DEBUG Command::run{cmd="/bin/echo" args=["hi"] line=command_line_args:1}: rust_parallel::command: begin run2024-05-04T16:08:20.302961Z DEBUG Command::run{cmd="/bin/echo" args=["hi"] line=command_line_args:1 child_pid=40443}: rust_parallel::command: spawned child process, awaiting completion2024-05-04T16:08:20.303670Z DEBUG Command::run{cmd="/bin/echo" args=["hi"] line=command_line_args:1 child_pid=40443}: rust_parallel::command: command exit status = exit status: 02024-05-04T16:08:20.303681Z DEBUG Command::run{cmd="/bin/echo" args=["hi"] line=command_line_args:1 child_pid=40443}: rust_parallel::command: end runThe following are considered command failures and error will be logged:
- Spawn error
- Timeout
- I/O error
- Command exits with non-0 status
By default rust-parallel runs all commands even if failures occur.
When rust-parallel terminates, if any command failed it logs failure metrics and exits with status 1.
Here we try to usecat to show non-existing filesA,B, andC, so each command exits with status 1:
$ rust-parallel cat ::: A B Ccat: A: No such file or directory2024-05-04T16:08:20.308001Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["A"],line=command_line_args:1 exit_status=1cat: C: No such file or directory2024-05-04T16:08:20.308071Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["C"],line=command_line_args:3 exit_status=1cat: B: No such file or directory2024-05-04T16:08:20.308132Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["B"],line=command_line_args:2 exit_status=12024-05-04T16:08:20.308165Z ERROR rust_parallel: fatal error in main: command failures: commands_run=3 total_failures=3 spawn_errors=0 timeouts=0 io_errors=0 exit_status_errors=3$ echo $?1The--exit-on-error option can be used to exit after one command fails.
rust-parallel waits for in-progress commands to finish before exiting and then exits with status 1.
$ head -100 /usr/share/dict/words | rust-parallel --exit-on-error catcat: aa: No such file or directory2024-05-04T16:08:20.311987Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["aa"],line=stdin:3 exit_status=1cat: A: No such file or directory2024-05-04T16:08:20.312064Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["A"],line=stdin:1 exit_status=1cat: a: No such file or directory2024-05-04T16:08:20.312329Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["a"],line=stdin:2 exit_status=1cat: Aani: No such file or directory2024-05-04T16:08:20.312412Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["Aani"],line=stdin:7 exit_status=1cat: aam: No such file or directory2024-05-04T16:08:20.312521Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["aam"],line=stdin:6 exit_status=1cat: aardvark: No such file or directory2024-05-04T16:08:20.312615Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["aardvark"],line=stdin:8 exit_status=1cat: aal: No such file or directory2024-05-04T16:08:20.312681Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["aal"],line=stdin:4 exit_status=1cat: aalii: No such file or directory2024-05-04T16:08:20.312736Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["aalii"],line=stdin:5 exit_status=1cat: aardwolf: No such file or directory2024-05-04T16:08:20.312777Z ERROR rust_parallel::output::task: command failed: cmd="/bin/cat",args=["aardwolf"],line=stdin:9 exit_status=12024-05-04T16:08:20.313071Z ERROR rust_parallel: fatal error in main: command failures: commands_run=9 total_failures=9 spawn_errors=0 timeouts=0 io_errors=0 exit_status_errors=9$ echo $?1The-t/--timeout-seconds option can be used to specify a command timeout in seconds. If any command times out this is considered a command failure (seeerror handling).
$ rust-parallel -t 0.5 sleep ::: 0 3 52024-05-04T16:08:20.818579Z ERROR rust_parallel::command: child process error command: cmd="/bin/sleep",args=["5"],line=command_line_args:3 error: timeout: deadline has elapsed2024-05-04T16:08:20.818583Z ERROR rust_parallel::command: child process error command: cmd="/bin/sleep",args=["3"],line=command_line_args:2 error: timeout: deadline has elapsed2024-05-04T16:08:20.819071Z ERROR rust_parallel: fatal error in main: command failures: commands_run=3 total_failures=2 spawn_errors=0 timeouts=2 io_errors=0 exit_status_errors=0$ echo $?1By default as commands are run the full paths are resolved usingwhich. Resolved paths are stored in a cache to prevent duplicate resolutions. This is generallygood for performance.
The path cache can be disabled using the--disable-path-cache option.
The-p/--progress-bar option can be used to enable a graphical progress bar.
This is best used for commands which are running for at least a few seconds, and which do not produce output to stdout or stderr. In the below commands-d all is used to discard all output from commands run.
Progress styles can be chosen with thePROGRESS_STYLE environment variable. IfPROGRESS_STYLE is not set it defaults tolight_bg.
The following progress styles are available:
PROGRESS_STYLE=light_bggood for light terminal background with colors, spinner, and steady tick enabled:
PROGRESS_STYLE=dark_bggood for dark terminal background with colors, spinner, and steady tick enabled:
PROGRESS_STYLE=simplegood for simple or non-ansi terminals/jobs with colors, spinner, and steady tick disabled:
Regular expressions can be specified by the-r or--regex command line argument.
Named or numbered capture groups are expanded with data values from the current input before the command is executed.
In these examples using command line arguments{url} and{filename} are named capture groups.{0} is a numbered capture group.
$ rust-parallel -r '(?P<url>.*),(?P<filename>.*)' echo got url={url} filename={filename} ::: URL1,filename1 URL2,filename2got url=URL1 filename=filename1got url=URL2 filename=filename2$ rust-parallel -r '(?P<url>.*) (?P<filename>.*)' echo got url={url} filename={filename} full input={0} ::: URL1 URL2 ::: filename1 filename2got url=URL2 filename=filename1 full input=URL2 filename1got url=URL1 filename=filename2 full input=URL1 filename2got url=URL1 filename=filename1 full input=URL1 filename1got url=URL2 filename=filename2 full input=URL2 filename2In the next example input file arguments{0}{1}{2}{3} are numbered capture groups, and the input is a csv file:
$ cat >./test <<EOLfoo,bar,bazfoo2,bar2,baz2foo3,bar3,baz3EOL$ cat test | rust-parallel -r '(.*),(.*),(.*)' echo got arg1={1} arg2={2} arg3={3} full input={0}got arg1=foo arg2=bar arg3=baz full input=foo,bar,bazgot arg1=foo2 arg2=bar2 arg3=baz2 full input=foo2,bar2,baz2got arg1=foo3 arg2=bar3 arg3=baz3 full input=foo3,bar3,baz3All occurrences of capture groups are replaced as exact strings. Surrounding characters have no effect on this.
This means capture groups can be nested with other{ or} characters such as when building json:
$ cat >./test <<EOL1,2,34,5,67,8,9EOL$ cat test | rust-parallel -r '(.*),(.*),(.*)' echo '{"one":{1},"two":{2},"nested_object":{"three":{3}}}'{"one":7,"two":8,"nested_object":{"three":9}}{"one":1,"two":2,"nested_object":{"three":3}}{"one":4,"two":5,"nested_object":{"three":6}}Shell commands can be written using-s shell mode.
Multiline commands can be written using;.
Environment variables,$ characters, nested commands and much more are possible:
$ rust-parallel -s -r '(?P<arg1>.*) (?P<arg2>.*)' 'FOO={arg1}; BAR={arg2}; echo "FOO = ${FOO}, BAR = ${BAR}, shell pid = $$, date = $(date)"' ::: A B ::: CAT DOGFOO = B, BAR = CAT, shell pid = 40492, date = Sat May 4 11:08:20 CDT 2024FOO = B, BAR = DOG, shell pid = 40493, date = Sat May 4 11:08:20 CDT 2024FOO = A, BAR = DOG, shell pid = 40491, date = Sat May 4 11:08:20 CDT 2024FOO = A, BAR = CAT, shell pid = 40490, date = Sat May 4 11:08:20 CDT 2024-s shell mode can be used to invoke an arbitrary bash function.
Similar to normal commands bash functions can be called using stdin, input files, or from command line arguments.
Define a bash fuctionlogargs that logs all arguments and make visible withexport -f:
$ logargs() { echo "logargs got $@"}$ export -f logargs$ rust-parallel -s logargs ::: A B C ::: D E Flogargs got A Dlogargs got A Elogargs got B Flogargs got B Dlogargs got B Elogargs got A Flogargs got C Elogargs got C Dlogargs got C F$ cat >./test <<EOLlogargs hello alicelogargs hello boblogargs hello charlieEOL$ cat test | rust-parallel -slogargs got hello alicelogargs got hello charlielogargs got hello bob$ cat >./test <<EOLalicebobcharlieEOL$ cat test | rust-parallel -s logargs hellologargs got hello alicelogargs got hello boblogargs got hello charlie