Objectives

By the end of this chapter, you should be able to:

• Explain what the head, tail, sort, uniq, wc and grep commands do
• Define what piping is
• Understand use cases for piping
• Use piping to better work in Terminal

Piping

At the end of the last chapter, we saw how we could use redirection to combine a couple of commands into one. In that case, in a single line we were able to both sort a text file and output the contents to a new file.

But what if we want to chain even more commands together? This is where piping comes into play. Before we learn about piping, though, let's learn/review a couple other terminal commands.

head - display the first lines of a file (using the -n flag we can specify the number of lines)

tail - display the last lines of a file (using the -n flag we can specify the number of lines)

sort - sort lines of a text file

uniq - removes duplicated lines (your data must be sorted for this to work)

wc - word, line, character and byte count

Now let's create two files - first.txt which contains the text

First
Second
Third

And second.txt which contains:

Fourth
Fifth
Sixth

If we want to concatenate (join) these two files together, we use the cat command:

cat first.txt second.txt (make sure the files and commands are separated by a space).

But what happens if we want to concatenate these two files and then find the word count? What if we want to concatenate and then sort it? This is where we need piping! You can think of a pipe as a connection between the output of one command into the input of another command. So once we concatenate two files we then want to send (or pipe) the result of that to another command. We can even combine this with redirection!

To pipe a command we use the | character. So if we want to pipe cat into sort it would look like this:

cat first.txt second.txt | sort or cat first.txt second.txt | sort | head -n 2

Take a look at this command and try to figure out what it's doing. You'll find a step-by-step answer below.

cat first.txt second.txt | sort | tail -n 3 | head -n 1

1. Concatenate the two files first.txt and second.txt
2. Sort the results
3. Find the last 3 lines
4. Find the first line of those last 3 lines

This is how we can find the third from last line in a file (without knowing how many lines the file has).

grep

Let's examine another useful command called grep which is extremely powerful for finding text. On its own it is helpful, but it is quite useful when piped with cat. Let's try a simple example with cat first.txt | grep First - what do you see?

You should see the word First output to the terminal. This is because grep searched the file for the text First and found a match!

Notice that if grep doesn't find a match, it won't output anything. If it finds multiple matches, it will print them all. Try out these commands and see what grep returns to you!

cat first.txt second.txt | grep Nope
cat first.txt second.txt | grep th

Let's look at another example. Imagine you have a file called petnames.txt, and inside you have the following list of names:

Lassie
Moxie
Whiskey
Fido
Lassie
Moxie

We see here there are quite a few duplicates, so let's try to use the uniq command to remove these duplicate names. The problem is, when we run uniq petnames.txt we get the following

Lassie
Moxie
Whiskey
Fido
Lassie
Moxie

If we look back at our definition of how the uniq command works, we see that our data must be sorted! So how can we first use sort on petnames.txt then attach the uniq command? Piping to the rescue!

sort petnames.txt | uniq gives us

Fido
Lassie
Moxie
Whiskey

This looks great! But this text is just being output to the terminal, what if we want to output this text to a new file called petnames_sorted.txt? We can combine piping with redirection and use sort petnames.txt | uniq > petnames_sorted.txt. Now if we cat petnames_sorted.txt we should see our four unique sorted names!