All of the standard Linux utilities have three data streams associated with them. They are the standard input stream (stdin), the standard output stream (stdout), and the standard error stream (stderr).
These streams work with text. We send input (stdin) to a command using text, and the response (stdout) is written to the terminal window as text. Error messages are also written to the terminal window as text (stderr).
One of the great features of Linux and Unix-like operating systems is the ability to pipe the stdout output from one command into the stdin input of a second command. The first command doesn’t care that its output is not going to a terminal window, and the second command doesn’t care that its input isn’t coming from a keyboard.
Although all of the Linux commands have the three standard streams, not all of them accept another command’s stdout as input to their stdin. That means you can’t pipe input to them.
xargs is a command for building execution pipelines using the standard data streams. By using xargs we can make commands such as echo, rm, and mkdir accept standard input as arguments.
The xargs Command
xargs will accept piped input. It can also accept input from a file. xargs uses that input as parameters for the commands we’ve told it to work with. If we do not tell xargs to work with a specific command it will default to use echo.
We can use that to demonstrate how xargs will always generate a single line of output, even from multi-line input.
If we use the -1 (list one file per line) option with ls, we get a single column of filenames.
This lists the shell script files in the current directory.
We get a single column as expected. If we pipe it through xargs what do we get?
The output is written to the terminal window, as one long stream of text.
It’s this capability that let’s xargs feed parameters into other commands.
Using xargs With wc
We can use xargs to easily have wc count the words, characters, and lines in multiple files.
This is what happens:
ls lists the *. page files and passes the list to xargs. xargs passes the filenames to wc. wc treats the filenames as if it had received them as command line parameters.
The statistics for each file are displayed together with an overall total.
Using xargs With Confirmation
We can use the -p (interactive) option to have xargs prompt us for confirmation that we are happy for it to proceed.
If we pass a string of filenames to touch, through xargs, touch will create the files for us.
The command that is going to be executed is displayed and xargs waits for us to respond by typing “y” or “Y”, or “n” or “N”, and pressing Enter.
If you just press Enter, it is treated as “n”. The command is only executed if you type “y” or “Y”.
We pressed “y” and pressed Enter. We can use ls to check that the files have been created.
Using xargs With Multiple Commands
We can use multiple commands with xargs by using the -I (initial arguments) option.
This option defines a “replace-string.” Wherever the token for the replace-string appears in the command line, the values that were supplied to xargs are inserted.
Let’s use the tree command to look at the subdirectories from the current directory. The -d (directory) option causes tree to ignore files and only report on directories.
There is a single subdirectory called “images.”
In a file called “directories.txt”, we have the names of some directories that we wish to have created. We can look at its contents using cat.
We’re going to use this as the input data for xargs. The command we’re going to is this:
This breaks down like this:
cat directories. txt |: This pushes the contents of the directrories. txt file (all the new directory names) into xargs. xargs -I %: This defines a “replace-string” with the token “%”. sh -c: This starts a new subshell. The -c (command) tells the shell to read commands from the command line. ‘echo %; mkdir %’: each of the “%” tokens will be replaced by the directory names that are passed by xargs. The echo command will print the directory name; the mkdir command will create the directory.
The directories are listed one by one.
We can use tree once more to verify the directories have been created.
Copying Files To Multiple Locations
We can use xargs to allow us to copy files to multiple locations with a single command.
We are going to pipe the names of two directories into xargs as the input parameters. We’ll tell xargs to only pass one of these parameters at a time to the command it is working with.
In this case, the command is cp. So the effect is to call cp twice, each time with one of the two directories as a command-line parameter. The xargs parameter that allows this to happen is the -n (max number) option. We’re going to set this to be one.
We’re also using the -v (verbose) option with cp so that it reports what is happening.
The files are copied to the two directories, one directory at a time. cp reports each file copy action so that we can see them taking place.
Deleting Files in Nested Directories
If filenames have spaces and strange characters in them—such as newline characters— xargs will not be able to interpret them correctly. We can overcome that problem by using the -0 (null terminator) option. This tells xargs to use the null character as the final delimiter for filenames.
We’re going to use find in this example. find has its own option for dealing with whitespace and strange characters in filenames. It is the -print0 (full name, null character) option.
This breaks down like this:
find . -name “. png”: find is going to search from the current directory “. ” for objects with names that match “. png” that are files (type -f). -print0: names will be terminated by a null character, and spaces and strange characters will be catered for. xargs -0: xargs is also going to consider filenames to be null-terminated, and spaces and strange characters will not cause problems. rm -v -rf “{}”: rm is going to be verbose and report what is happening (-v). It is going to be recursive (-r) and look through nested subdirectories, and will remove files without prompting (-f). The “{}” is replaced by each filename.
All subdirectories are searched, and the files that match the search pattern are deleted.
Removing Nested Directories
Let’s say we want to remove a set of nested subdirectories. tree will let us see them.
This command will use find to search recursively within the current directory. The search target is a directory called “level_one”. The directory names are passed through xargs to rm.
The only significant changes between this command and the previous command are, the search term is the name of the topmost directory, and -type d tells find to look for directories, not files.
The name of each directory is printed as it is removed. We can check with tree :
All of the nested subdirectories are deleted.
Deleting All Files, Except for One File Type
We can use find, xargs and rm to delete all files apart from one type we want to retain. It’s slightly counterintuitive, but we provide the name of the filetype we wish to keep, not the name of the ones we want to delete.
The -not option tells find to return the names of the files that don’t match the search pattern. We’re using the -I (initial arguments) option with xargs once more. This time the replace-string token we’re defining is “{}”. This will behave exactly the same as the replace-string token we generated previously, which happened to be a “%”.
We can check with ls. The only files left in the directory are the ones that matched the “*.sh” search pattern.
Creating an Archive File With Xargs
We can use find to search for files and pass them through xargs to tar, to create an archive file.
We’re going to search in the current directory. The search pattern is “*.page” so we’re going to be looking for “.page” files.
The files are listed as expected, as the archive file is created.
The Data Mediator
Sometimes you need a little scaffolding when you’re stacking things together. xargs bridges the gap between commands that can pump out information and commands that aren’t built to take it in.
Both xargs and find have a huge number of options. You’re encouraged to check out their man pages to learn more.
