| xargs | |
|---|---|
| Developers | Variousopen-source andcommercial developers |
| Operating system | Unix,Unix-like,Plan 9,IBM i |
| Platform | Cross-platform |
| Type | Command |
xargs (short for "extended arguments")[1] is acommand onUnix and mostUnix-likeoperating systems used to build and execute commands fromstandard input. It converts input from standard input into arguments to a command.
Some commands such asgrep andawk can take input either as command-line arguments or from the standard input. However, others such ascp andecho can only take input as arguments, which is whyxargs is necessary.
A port of an older version of GNUxargs is available forMicrosoft Windows as part of theUnxUtils collection ofnativeWin32ports of common GNU Unix-like utilities.[2] A ground-up rewrite namedwargs is part of the open-source TextTools[3] project. Thexargs command has also been ported to theIBM i operating system.[4]
One use case of thexargs command is to remove a list of files using therm command.POSIX systems have anARG_MAX for the maximum total length of the command line,[5][6] so the command may fail with an error message of "Argument list too long" (meaning that the exec system call's limit on the length of a command line was exceeded):rm/path/* orrm$(find/path-typef). (The latter invocation is incorrect, as it may expandglobs in the output.)
This can be rewritten using thexargs command to break the list of arguments into sublists small enough to be acceptable:
$find/path-typef-print|xargsrm
In the above example, thefind utility feeds the input ofxargs with a long list of file names.xargs then splits this list into sublists and callsrm once for every sublist.
Some implementations ofxargs can also be used to parallelize operations with the-P maxprocs argument to specify how many parallel processes should be used to execute the commands over the input argument lists. However, the output streams may not be synchronized. This can be overcome by using an--output file argument where possible, and then combining the results after processing. The following example queues 24 processes and waits on each to finish before launching another.
$find/path-name'*.foo'|xargs-P24-I'{}'/cpu/bound/process'{}'-o'{}'.out
xargs often covers the same functionality as thecommand substitution feature of manyshells, denoted by thebackquote notation (`...` or$(...)).xargs is also a good companion for commands that output long lists of files such asfind,locate andgrep, but only if one uses-0 (or equivalently--null), sincexargs without-0 deals badly with file names containing'," and space.GNU Parallel is a similar tool that offers better compatibility withfind,locate andgrep when file names may contain',", and space (newline still requires-0).
-I option: single argumentThexargs command offers options to insert the listed arguments at some position other than the end of the command line. The-I option toxargs takes a string that will be replaced with the supplied input before the command is executed. A common choice is%.
$mkdir~/backups$find/path-typef-name'*~'-print0|xargs-0-I%cp-a%~/backups
The string to replace may appear multiple times in the command part. Using-I at all limits the number of lines used each time to one.
Another way to achieve a similar effect is to use a shell as the launched command, and deal with the complexity in that shell, for example:
$mkdir~/backups$find/path-typef-name'*~'-print0|xargs-0sh-c'for filename; do cp -a "$filename" ~/backups; done'sh
The wordsh at the end of the line is for thePOSIX shellsh -c to fill in for$0, the "executable name" part of the positional parameters (argv). If it weren't present, the name of the first matched file would be instead assigned to$0 and the file wouldn't be copied to~/backups. One can also use any other word to fill in that blank,my-xargs-script for example.
Sincecp accepts multiple files at once, one can also simply do the following:
$find/path-typef-name'*~'-print0|xargs-0sh-c'if [ $# -gt 0 ]; then cp -a "$@" ~/backup; fi'sh
This script runscp with all the files given to it when there are any arguments passed. Doing so is more efficient since only one invocation ofcp is done for each invocation ofsh.
Many Unix utilities are line-oriented. These may work withxargs as long as the lines do not contain',", or a space. Some of the Unix utilities can useNUL as record separator (e.g.Perl (requires-0 and\0 instead of\n),locate (requires using-0),find (requires using-print0),grep (requires-z or-Z),sort (requires using-z)). Using-0 forxargs deals with the problem, but many Unix utilities cannot use NUL as separator (e.g.head,tail,ls,echo,sed,tar -v,wc,which).
But often people forget this and assumexargs is also line-oriented, which isnot the case (per defaultxargs separates on newlinesand blanks within lines, substrings with blanks must be single- or double-quoted).
The separator problem is illustrated here:
# Make some targets to practice ontouchimportant_filetouch'not important_file'mkdir-p'12" records'find.-namenot\*|tail-1|xargsrmfind\!-name.-typed|tail-1|xargsrmdir
Running the above will causeimportant_file to be removed but will remove neither the directory called12" records, nor the file callednot important_file.
The proper fix is to use the GNU-specific-print0 option, buttail (and other tools) do not support NUL-terminated strings:
# use the same preparation commands as abovefind.-namenot\*-print0|xargs-0rmfind\!-name.-typed-print0|xargs-0rmdir
When using the-print0 option, entries are separated by a null character instead of an end-of-line. This is equivalent to the more verbose command:find.-namenot\*|tr\\n\\0|xargs-0rm or shorter, by switchingxargs to (non-POSIX)line-oriented mode with the-d (delimiter) option:find.-namenot\*|xargs-d'\n'rm
but in general using-0 with-print0 should be preferred, since newlines in filenames are still a problem.
GNUparallel is an alternative toxargs that is designed to have the same options, but is line-oriented. Thus, using GNU Parallel instead, the above would work as expected.[7]
For Unix environments wherexargs does not support the-0 nor the-d option (e.g. Solaris, AIX), the POSIX standard states that one can simply backslash-escape every character:find.-namenot\*|sed's/\(.\)/\\\1/g'|xargsrm.[8] Alternatively, one can avoid using xargs at all, either by using GNU parallel or using the-exec ... + functionality offind.
One might be dealing with commands that can only accept one or maybe two arguments at a time. For example, thediff command operates on two files at a time. The-n option toxargs specifies how many arguments at a time to supply to the given command. The command will be invoked repeatedly until all input is exhausted. Note that on the last invocation one might get fewer than the desired number of arguments if there is insufficient input. Usexargs to break up the input into two arguments per line:
$echo{0..9}|xargs-n20 12 34 56 78 9
In addition to running based on a specified number of arguments at a time, one can also invoke a command for each line of input with the-L 1 option. One can use an arbitrary number of lines at a time, but one is most common. Here is how one mightdiff every git commit against its parent.[9]
$gitlog--format="%H %P"|xargs-L1gitdiff
The argument separator processing ofxargs is not the only problem with using thexargs program in its default mode. Most Unix tools which are often used to manipulate filenames (for examplesed,basename,sort, etc.) are text processing tools. However, Unix path names are not really text. Consider a path name /aaa/bbb/ccc. The /aaa directory and its bbb subdirectory can in general be created by different users with different environments. That means these users could have a different locale setup, and that means that aaa and bbb do not even necessarily have to have the same character encoding. For example, aaa could be in UTF-8 and bbb in Shift JIS. As a result, an absolute path name in a Unix system may not be correctly processable as text under a single character encoding. Tools which rely on their input being text may fail on such strings.
One workaround for this problem is to run such tools in the C locale, which essentially processes the bytes of the input as-is. However, this will change the behavior of the tools in ways the user may not expect (for example, some of the user's expectations about case-folding behavior may not be met).
xargs – Shell and Utilities Reference,The Single UNIX Specification, Version 5 fromThe Open Groupxargs: construct argument lists and invoke utility – Shell and Utilities Reference,The Single UNIX Specification, Version 5 fromThe Open Groupxargs(1) – GNUFindutils referencexargs(1): construct argument list(s) and execute utility – FreeBSD General CommandsManualxargs(1): construct argument list(s) and execute utility – NetBSD General CommandsManualxargs(1): construct argument list(s) and execute utility – OpenBSD General CommandsManualxargs(1): construct argument lists and invoke utility – Solaris 11.4 User Commands ReferenceManual