Chapter 22. Process Substitution

Piping the stdout of a command into the stdin of another is a powerful technique. But, what if you need to pipe the stdout of multiple commands? This is where process substitution comes in.

Process substitution feeds the output of a process (or processes) into the stdin of another process.

bash$ echo >(true) /dev/fd/63 bash$ echo <(true) /dev/fd/63

Bash creates a pipe with two file descriptors, --fIn and fOut--. The stdin of true connects to fOut (dup2(fOut, 0)), then Bash passes a /dev/fd/fIn argument to echo. On systems lacking /dev/fd/<n> files, Bash may use temporary files. (Thanks, S.C.)

Process substitution can compare the output of two different commands, or even the output of different options to the same command.

bash$ comm <(ls -l) <(ls -al)
total 12
-rw-rw-r--    1 bozo bozo       78 Mar 10 12:58 File0
-rw-rw-r--    1 bozo bozo       42 Mar 10 12:58 File2
-rw-rw-r--    1 bozo bozo      103 Mar 10 12:58 t2.sh
        total 20
        drwxrwxrwx    2 bozo bozo     4096 Mar 10 18:10 .
        drwx------   72 bozo bozo     4096 Mar 10 17:58 ..
        -rw-rw-r--    1 bozo bozo       78 Mar 10 12:58 File0
        -rw-rw-r--    1 bozo bozo       42 Mar 10 12:58 File2
        -rw-rw-r--    1 bozo bozo      103 Mar 10 12:58 t2.sh

Using process substitution to compare the contents of two directories (to see which filenames are in one, but not the other):

diff <(ls $first_directory) <(ls $second_directory)

Some other usages and uses of process substitution:

read -a list < <( od -Ad -w24 -t u2 /dev/urandom ) # Read a list of random numbers from /dev/urandom, #+ process with "od" #+ and feed into stdin of "read" . . . # From "insertion-sort.bash" example script. # Courtesy of JuanJo Ciarlante.

cat <(ls -l) # Same as ls -l | cat sort -k 9 <(ls -l /bin) <(ls -l /usr/bin) <(ls -l /usr/X11R6/bin) # Lists all the files in the 3 main 'bin' directories, and sorts by filename. # Note that three (count 'em) distinct commands are fed to 'sort'. diff <(command1) <(command2) # Gives difference in command output. tar cf >(bzip2 -c > file.tar.bz2) $directory_name # Calls "tar cf /dev/fd/?? $directory_name", and "bzip2 -c > file.tar.bz2". # # Because of the /dev/fd/<n> system feature, # the pipe between both commands does not need to be named. # # This can be emulated. # bzip2 -c < pipe > file.tar.bz2& tar cf pipe $directory_name rm pipe # or exec 3>&1 tar cf /dev/fd/4 $directory_name 4>&1 >&3 3>&- | bzip2 -c > file.tar.bz2 3>&- exec 3>&- # Thanks, Stéphane Chazelas

A reader sent in the following interesting example of process substitution.

# Script fragment taken from SuSE distribution: # --------------------------------------------------------------# while read des what mask iface; do # Some commands ... done < <(route -n) # ^ ^ First < is redirection, second is process substitution. # To test it, let's make it do something. while read des what mask iface; do echo $des $what $mask $iface done < <(route -n) # Output: # Kernel IP routing table # Destination Gateway Genmask Flags Metric Ref Use Iface # 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo # --------------------------------------------------------------# # As Stéphane Chazelas points out, #+ an easier-to-understand equivalent is: route -n | while read des what mask iface; do # Variables set from output of pipe. echo $des $what $mask $iface done # This yields the same output as above. # However, as Ulrich Gayer points out . . . #+ this simplified equivalent uses a subshell for the while loop, #+ and therefore the variables disappear when the pipe terminates. # --------------------------------------------------------------# # However, Filip Moritz comments that there is a subtle difference #+ between the above two examples, as the following shows. ( route -n | while read x; do ((y++)); done echo $y # $y is still unset while read x; do ((y++)); done < <(route -n) echo $y # $y has the number of lines of output of route -n ) More generally spoken ( : | x=x # seems to start a subshell like : | ( x=x ) # while x=x < <(:) # does not ) # This is useful, when parsing csv and the like. # That is, in effect, what the original SuSE code fragment does.