Streams Join Two Points
As soon as you start to learn about Linux and Unix-like operating systems, you’ll come across the terms stdin, stdout, and stederr. These are three standard streams that are established when a Linux command is executed. In computing, a stream is something that can transfer data. In the case of these streams, that data is text.
Data streams, like water streams, have two ends. They have a source and an outflow. Whichever Linux command you’re using provides one end of each stream. The other end is determined by the shell that launched the command. That end will be connected to the terminal window, connected to a pipe, or redirected to a file or other command, according to the command line that launched the command.
The Linux Standard Streams
In Linux, stdin is the standard input stream. This accepts text as its input. Text output from the command to the shell is delivered via the stdout (standard out) stream. Error messages from the command are sent through the stderr (standard error) stream.
So you can see that there are two output streams, stdout and stderr, and one input stream, stdin. Because error messages and normal output each have their own conduit to carry them to the terminal window, they can be handled independently of one another.
Streams Are Handled Like Files
Streams in Linux—like almost everything else—are treated as though they were files. You can read text from a file, and you can write text into a file. Both of these actions involve a stream of data. So the concept of handling a stream of data as a file isn’t that much of a stretch.
Each file associated with a process is allocated a unique number to identify it. This is known as the file descriptor. Whenever an action is required to be performed on a file, the file descriptor is used to identify the file.
These values are always used for stdin, stdout, and stderr:
0: stdin 1: stdout 2: stderr
Reacting to Pipes and Redirects
To ease someone’s introduction to a subject, a common technique is to teach a simplified version of the topic. For example, with grammar, we are told that the rule is “I before E, except after C.” But actually, there are more exceptions to this rule than there are cases that obey it.
In a similar vein, when talking about stdin, stdout, and stderr it is convenient to trot out the accepted axiom that a process neither knows nor cares where its three standard streams are terminated. Should a process care whether its output is going to the terminal or being redirected into a file? Can it even tell if its input is coming from the keyboard or is being piped into it from another process?
Actually, a process does know—or at least it can find out, should it choose to check—and it can change its behavior accordingly if the software author decided to add that functionality.
We can see this change in behavior very easily. Try these two commands:
The ls command behaves differently if its output (stdout) is being piped into another command. It is ls that switches to a single column output, it isn’t a conversion performed by cat. And ls does the same thing if its output is being redirected:
Redirecting stdout and stderr
There’s an advantage to having error messages delivered by a dedicated stream. It means we can redirect a command’s output (stdout) to a file and still see any error messages (stderr) in the terminal window. You can react to the errors if you need to, as they occur. It also stops the error messages from contaminating the file that stdout has been redirected into.
Type the following text into an editor and save it to a file called error.sh.
Make the script executable with this command:
The first line of the script echoes text to the terminal window, via the stdout stream. The second line tries to access a file that doesn’t exist. This will generate an error message that is delivered via stderr.
Run the script with this command:
We can see that both streams of output, stdout and stderr, have been displayed in the terminal windows.
Let’s try to redirect the output to a file:
The error message that is delivered via stderr is still sent to the terminal window. We can check the contents of the file to see whether the stdout output went to the file.
The output from stdin was redirected to the file as expected.
The > redirection symbol works with stdout by default. You can use one of the numeric file descriptors to indicate which standard output stream you wish to redirect.
To explicitly redirect stdout, use this redirection instruction:
To explicitly redirect stderr, use this redirection instruction:
Let’s try to our test again, and this time we’ll use 2>:
The error message is redirected and the stdout echo message is sent to the terminal window:
Let’s see what is in the capture.txt file.
The stderr message is in capture.txt as expected.
Redirecting Both stdout and stderr
Surely, if we can redirect either stdout or stderr to a file independently of one another, we ought to be able to redirect them both at the same time, to two different files?
Yes, we can. This command will direct stdout to a file called capture.txt and stderr to a file called error.txt.
Because both streams of output–standard output and standard error—are redirected to files, there is no visible output in the terminal window. We are returned to the command line prompt as though nothing has occurred.
Let’s check the contents of each file:
Redirecting stdout and stderr to the Same File
That’s neat, we’ve got each of the standard output streams going to its own dedicated file. The only other combination we can do is to send both stdout and stderr to the same file.
We can achieve this with the following command:
Let’s break that down.
. /error. sh: Launches the error. sh script file.
capture. txt: Redirects the stdout stream to the capture. txt file. is shorthand for 1>. 2>&1: This uses the &> redirect instruction. This instruction allows you to tell the shell to make one stream got to the same destination as another stream. In this case, we’re saying “redirect stream 2, stderr, to the same destination that stream 1, stdout, is being redirected to. ”
There is no visible output. That’s encouraging.
Let’s check the capture.txt file and see what’s in it.
Both the stdout and stderr streams have been redirected to a single destination file.
To have the output of a stream redirected and silently thrown away, direct the output to /dev/null.
Detecting Redirection Within a Script
We discussed how a command can detect if any of the streams are being redirected, and can choose to alter its behavior accordingly. Can we accomplish this in our own scripts? Yes, we can. And it is a very easy technique to understand and employ.
Type the following text into an editor and save it as input.sh.
Use the following command to make it executable:
The clever part is the test within the square brackets. The -t (terminal) option returns true (0) if the file associated with the file descriptor terminates in the terminal window. We’ve used the file descriptor 0 as the argument to the test, which represents stdin.
If stdin is connected to a terminal window the test will prove true. If stdin is connected to a file or a pipe, the test will fail.
We can use any convenient text file to generate input to the script. Here we’re using one called dummy.txt.
The output shows that the script recognizes that the input isn’t coming from a keyboard, it is coming from a file. If you chose to, you could vary your script’s behavior accordingly.
That was with a file redirection, let’s try it with a pipe.
The script recognizes that its input is being piped into it. Or more precisely, it recognizes once more that the stdin stream is not connected to a terminal window.
Let’s run the script with neither pipes nor redirects.
The stdin stream is connected to the terminal window, and the script reports this accordingly.
To check the same thing with the output stream, we need a new script. Type the following into an editor and save it as output.sh.
Use the following command to make it executable:
The only significant change to this script is in the test in the square brackets. We’re using the digit 1 to represent the file descriptor for stdout.
Let’s try it out. We’ll pipe the output through cat.
The script recognizes that its output is no going directly to a terminal window.
We can also test the script by redirecting the output to a file.
There is no output to the terminal window, we are silently returned to the command prompt. As we’d expect.
We can look inside the capture.txt file to see what was captured. Use the following command to do so.
Again, the simple test in our script detects that the stdout stream is not being sent directly to a terminal window.
If we run the script without any pipes or redirections, it should detect that stdout is being delivered directly to the terminal window.
And that’s exactly what we see.
Streams Of Consciousness
Knowing how to tell if your scripts are connected to the terminal window, or a pipe, or are being redirected, allows you to adjust their behavior accordingly.
Logging and diagnostic output can be more or less detailed, depending on whether it is going to the screen or to a file. Error messages can be logged to a different file than the normal program output.
As is usually the case, more knowledge brings more options.
