A tale of broken pipes

Written by Erik Mogensen updated: Monday January 02 2017 13:11

At Escenic we use a lot of bash as our glue language; it's widely available and pretty compatible across unixes. It also has an immense array of features, some of which are turned off by default, for good reason. One of these is pipefail which might catch you off guard if you (like us) weren't aware of some of the problems of pipes.

What is a pipe?

A pipe is simply the act of sending the output of one command into another. cat is a simple command, which outputs the contents of a file, while tr is a command that can make simple transformations. Both of these can operate on the streams called "standard input" and "standard output".

A simple example:

cat myfile.txt | tr LEET 1337

The cat program reads myfile.txt and prints it out to the pipe; while tr reads from the pipe and does some leet-speak transformations. Nothing too complicated. Where trouble lies is when errors occur. The most obvious error is that the file myfile.txt does not exist. But the pipe hides this from us:

cat my-non-existent-file.txt | tr LEET 1337; echo rc=$?
rc=0

The return code of the pipe itself is the return code of the last command in the pipe. So if the tr command is successful, so is the whole pipe. To know how the pipe worked, you need to inspect the shell builtin PIPESTATUS which happens to be an array, each representing the return code of each command in the pipeline.

cat my-non-existent-file.txt | tr LEET 1337; echo rc=$? pipestatus=${PIPESTATUS[@]}
cat: my-non-existent-file.txt: File not found 
rc=0 pipestatus=1 0

So we can now determine if any command in the pipeline failed. But it could be tedious to have to check this all the time, so bash has this mechanism of "bubbling up" any errors to the pipeline. This is called pipefail:

set -o pipefail
cat my-non-existent-file.txt | tr LEET 1337; echo rc=$? pipestatus=${PIPESTATUS[@]}
cat: my-non-existent-file.txt: File not found 
rc=1 pipestatus=1 0

The pipe now fails (rc=1) if any command in the pipe fails. One might wonder why this isn't on by default. When you first discover it, you want to add it to all your commands so that you are alerted of failures early. But if you do start pipefailing you should be aware of some of its pitfalls.

When pipefail fails

A command fails when the command exits with a nonzero return code. cat can obviously fail, because the file it's reading might not be there, or while reading the file, someone could unplug the disk from where it was reading the file, or maybe the file system was corrupt and the data is simply unavailable. But disregarding these types of failures, consider the following command:

find / -type -f | grep "myfile.txt" | head -n 1

This series of commands looks for a file called myfile.txt on my file system in a rather roundabout way. The return code will be 0 because head -n 1 can probably never fail, but the PIPESTATUS will reveal that both find and grep failed with the return code 141.

141??

In bash(1) (the man-page), we can see that an exit code for a process is its return code, or 128+n if it is terminated with a signal. 141 is 13 more than 128, and indeed, according to signal(7), Signal 13 is SIGPIPE:

SIGPIPE      13       Term    Broken pipe: write to pipe with no readers

This means that in any pipeline of commands, the downstream of a pipe can close the pipe, and cause upstream ripple effects. They will try to write to the pipe and will abruptly be killed with SIG_PIPE.

This means that if pipefail is on, the return code of this crude locate(1) replacement would be nondeterministic:

find / -type -f | grep "myfile.txt" | head -n 1

It would return 141 in most cases, and sometimes (if the file was in the last few kilobytes of find's output) it would return 0. It would depend on a range of factors such as the buffer sizes of each command, the buffer sizes of the shell's pipes, and even the order of the results. Pretty nondeterministic, and my guess is that this is the main reason why pipefail is off by default.

Safe commands

Something as simple as tr LEET 1337 simply cannot fail, right? Or echo. It is safe to assume that these commands won't fail, right?

As it turns out, even these "safe" commands can fail under certain circumstances, and inspecting the PIPESTATUS or turning on pipefail will at some point expose you to these circumstances.

Let's start with echo:

echo "$variable" | head -n 1; echo pipestatus=${PIPESTATUS[@]}
the first line
0 0

Imagine $variable is a multi-line variable and you'd like to see the first line of that variable. If you inspect PIPESTATUS you will see that both echo and head completed successfully.

Sometimes, this happens:

echo "$variable" | head -n 1; echo pipestatus=${PIPESTATUS[@]}
the first line
141 0

This only happens if $variable is so big that the head command completes before echo has had the chance to print out what it's supposed to. On a reasonably modern Linux, that's many tens of kilobytes, and you probably won't run into that unless you're misusing bash variables. But this is to highlight the problem with using pipefail and inspecting PIPESTATUS.

Conclusion

Pipe return codes can be a very useful tool to see the results of a pipe, and if each command completed their job. But beware of commands that close their input early: head is an obvious one, but grep -q will close its input after the first occurrence. These exits will (nondeterministically) cause otherwise well behaved commands like echo and cat to exit with SIGPIPE, causing the PIPESTATUS to indicate failures. Happy bashing!