Looping in Bash: An Introduction

When I first learned about Linux in the 90’s, I read that it was possible to even write your own commands to use at the command line. Later I learned about bash scripting, and it wasn’t long before I needed to learn how to loop in bash. Looping in bash is one of the fundamental building blocks of bash programming. It isn’t hard to do at all and is worth learning. The main reason to learn looping in bash is to handle doing the same thing over and over again. They’re easy to do even at the command line. Please follow along as we look a couple of basic examples, and how you can expand on them.

Getting Started

Simply put, a loop says “As long as a variable or condition is true, do this. When the variable or condition is not true, stop.” So you have to have a variable, and you have to have something to do while the variable is true.

I’ve generated a list of random names at http://listofrandomnames.com and put them in a file called names.txt. Using a simple loop, we’re going to do some things with those names.

The first part of a loop is defining what to use as data. In this case it’s in names.txt so let’s get that file into a variable.

names=$(cat names.txt)

Using $(command) tells bash to execute that as a command. Now, the contents of names.txt are contained in the variable called names.

Next, we’re doing to define a loop using the variable $names. You’ll notice that we did not use $ when defining the variable, only when calling it. This is called a ‘for’ loop:

for name in $names

Now we’ve started the loop. We’ve defined a new variable here: “name”. It means “For a piece of data in this variable.” We’re going to use $name in the next part:

do echo $name
done

We have to say ‘done’ or we haven’t completed the loop. Lets look at it all together:

#!/bin/bash
names=$(cat names.txt)
for name in $names
do echo $name
done

We started with #!/bin/bash, which tells the shell to execute the script using /bin/bash, but that’s only if we want to use it as a script inside a file. For something this small (and not something I’ll re-use a lot) I just write it on the command line like this:

# names=$(cat names.txt); for name in $names; do echo $name; done

The # signifies the terminal prompt; don’t type that part. The colon separates the elements of the script as if they were on separate lines even though they are not. How does it look when we run it?

# names=$(cat names.txt); for name in $names; do echo $name; done
Tawna
Adam
Lawanda
Klara
Grazyna

Well that doesn’t do much. We could have just done `cat list` and got the same result. Let’s do something more interesting to the names. Looping in bash is a lot more useful than this.

The Main Reason for Looping in Bash

Now to the fun part. Just because we can, we’ll use ‘wc -m’ to find out how many characters are in each name. Notice that we’re using backticks to tell echo to pipe (transfer the output of) the command ‘echo $name’ into the input of ‘wc -m’ which causes ‘wc’ to count the number of letters in the output of ‘echo’. More on pipes later.

# names=$(cat names.txt); for name in $names; do echo $(echo $name | wc -m) $name; done
6 Tawna
5 Adam
8 Lawanda
6 Klara
8 Grazyna

Now that’s more interesting. How about we put them in order, using ‘sort’?

# names=$(cat names.txt); for name in $names; do echo $(echo $name | wc -m) $name; done | sort
5 Adam
6 Klara
6 Tawna
8 Grazyna
8 Lawanda

So now we have the basics of sorting using a loop. Let’s go a bit farther. Let’s sort them, count how many have the same number of letters in them, and sort by how many letters each name has. We’ll use ‘uniq -c’ to find out how many of them are unique, and then sort again. This is a very handy way to create and sort lists.

# names=$(cat 50names.txt); for name in $names; do echo $(echo $name | wc -m) $name; done | sort | awk '{ print "names have " $1 " letters" }' | uniq -c | sort -n
      2 names have 4 letters
      3 names have 10 letters
      3 names have 9 letters
      6 names have 5 letters
     10 names have 7 letters
     12 names have 8 letters
     14 names have 6 letters

We can also express that script as follows, and put it into the following format in a file called ‘namelength.sh’

#!/bin/bash
names=$(cat 50names.txt)
for name in $names
    do 
       echo $(echo $name | wc -m) $name
    done | sort | awk '{ print "names have " $1 " letters" }' | uniq -c | sort -n

You’re looking at the exact same thing. You’d run this by simply typing

bash namelength.sh

and pressing enter. And the reason we have to have that big long line at the end is because we’re using pipe “|” to connect the output of the loop to the input of Sort, then awk, then uniq, then sort again. Each one modifies the input and produces it as output.

While /bin/true; do echo I Love You; done

Bash poetry notwithstanding, you might recall at the beginning that we said that a loop would run as long as the variable was true. In the above case, as long as ‘cat names.txt’ had content that hadn’t been processed, it returns the value ‘true’ to bash. But when it’s empty, it returns ‘false’ and the loop stops. But what if you don’t want the loop to stop?

Let’s use a real world example. In my line of work, I find myself transferring files from flaky connections at times. Often I don’t have access to the status of the server that’s sending the files (often via FTP) so there’s no decent way to check progress of the files. So I’ll use /bin/true as my variable, count the size of the files transferred so far, and then ‘sleep’ to tell it to pause so it doesn’t run out of control. Using ‘sleep’ is important because if you do not, the process will never pause. Depending on what you’re doing, you can exhaust the resources of the machine you’re working on. Trust me.

while /bin/true; du -m --max-depth=1 /path/to/flaky/transfer/files; sleep 30; done

This says “as long as /bin/true is true” (it always is) check the disk usage of 1 level of directories in /path/to/flaky/transfer/files, wait 30 seconds, and then check to see if /bin/true is still true. This way I can see the numbers get bigger every time the ‘du’ command runs. Lather, rinse repeat. To end the loop, just use ‘ctrl+c’ and that will bring you back to a command prompt.

More on variables

For a final exercise we’ll take a look at our first example.

# names=$(cat names.txt); for name in $names; do echo $(echo $name | wc -m) $name; done
6 Tawna
5 Adam
8 Lawanda
6 Klara
8 Grazyna

See how we defined a variable and then used it only once? Defining a variable is great, but if you’re only using it once, why bother making it a variable? It can be expressed more directly this way:

# for name in $(cat names.txt); do echo $(echo $name | wc -m) $name; done
6 Tawna
5 Adam
8 Lawanda
6 Klara
8 Grazyna

You can see how it simplifies the arrangement. When looping in bash right at the command line, I usually do it just like that. If it gets more complicated, I’ll write a shell script in a file and run it as shown earlier.

Another nice thing about looping in bash right at the command prompt is that if you want to see different results, you can just use the up arrow, modify something, and hit enter and get results immediately. For many of the shell scripts I write, I will start them out this way, or figure out a function that way. It makes for quick debugging and learning how to use the commands you need.

I hope you’ve found this article on looping in bash helpful. If you find inaccuracies, errors, or other issues please leave a comment below.