Shell Scripting

Table of Contents

1. Some Useful Commands
1.1. find, -prune and -print
2. Understand the Examples from the Lecture
3. Assessment
4. Last Words

Note

For this lab, you should do your work on Gallardo instead of Client1, as it contains some required software components that are not installed on Client1.

Before you proceed to go on with this lab, please make sure you have done the previous lab that covered the use of the Vim editor. Also, make sure you have read the handouts that were provided for the accompanying lecture. If you have questions as to how they work, please ask a demonstrator, in order to clear up any confusion.

This lab will be fairly simple, but there will be some essential commands that we shall need to cover. After that has been completed, you will work through the examples given in the lecture to appreciate how a pipeline is built up, then we’ll let you develop your own simple script.

You will likely find this lab very foundational, but also enormously empowering, once you begin to appreciate everything that scripting could do for you.

1. Some Useful Commands

There are many commands that you might find useful for scripting, but if we were to give them to you all at once, you would be overwhelmed. So we’ll give you some commands that are used very commonly, along with a brief description. You can refer to their manual pages to understand them better. Many of these commands are demonstrated in the lecture handouts.

cat

Concatenate files and print on the standard output. Commonly this is just used to output a (single) file to stdout, and thus introduce it into a pipeline, although that is commonly unnecessary.

echo

Display a line of text. Rather like a simple ‘print’ command.

printf

Format and print data. Much like C’s printf(3) function.

head

Output only the first n lines of the input.

tail

Output only the last n lines of the input.

tr

Transliterate (eg. change to upper-case) or delete characters.

cut

Remove sections from each line of input. Not very sophisticated; sed offers more power.

sort

Sort lines of text, possibly as numbers.

uniq

Remove duplicate lines from sorted input.

grep

Print lines from input that match a pattern.

wc

Print the number of bytes, words, and lines in input.

tee

Read from standard input and write to standard output and files. If you think of the plumbing analogy, this provides a ‘T’ junction. It can be very useful when inspecting what is going through a particular part of a pipeline.

ps

Report process status. Using ps -eo pid,command can be useful in scripts, as using a custom output format can be easier to parse. Supports a huge number of other options as well, most of which are not useful for scripting purposes.

kill

Send a signal to a process. Signals are not necessarily fatal, but are a primitive form of inter-process communication.

xargs

Build and execute command lines from standard input. See the last slide of the lecture notes for an example. Very useful in conjunction with find.

find

Search for files (or directories, etc.) in a directory hierarchy.

find needs plenty of examples to show how best to use it. The manual page for find(1) should contain a section showing various examples. We shall revisit find later in this section.

mktemp

Make temporary file name (unique).

du

Estimate file space usage. Note that this program can be particularly annoying at times.

Here is an example you can put into a file called big5, that shows the largest five entries in the current directory. It’s useful for figuring out where all the disk space is being used up.

$ du -k -d1 | grep -v '^[0-9]*[[:space:]]*\.$' | sort -rn | head -5
Your output will look different, you might not have any.
42080   ./Lectures
7020    ./Labbook

We should mention that there are GUI tools that are much more enligtening about where disk-space is being used, such as the Disk Usage Analyzer application in Ubuntu.

basename

Strip directory, and optionally a named suffix, from filenames.

$ basename "/path/to/foo.txt"
foo.txt
dirname

Strip non-directory suffix from file name.

$ dirname "/path/to/foo.txt"
/path/to
date

Print or set the system date and time.

$ date "+Today: %y/%m/%d"
Today: 07/04/18
sleep

Delay for a specified amount of time, in seconds. Some systems may have usleep available, for sub-second intervals.

getopt

Parse command options (enhanced).

An example was provided with the lecture notes.

1.1. find, -prune and -print

find can become a very confusing command to understand if you don’t specify -print as the operation to perform upon match. Note that if you do not specify an operation, such as -print or -exec then you will get something that appears to be the same as -print, but it is not. You will notice a (very confusing) difference if you, for example, use the -prune option to omit directories from your search. Here is an example, which took me quite a while to figure out, of how to use -prune correctly. The reason it took me a long time figuring out was because I was previously under the misapprehension that find behaves as if -print was the default operation.

Let’s assume that our files may have spaces between them, which could prove problematic as spaces seperate arguments in the Unix command-line. find and xargs have the ability to seperate arguments using an ASCII NUL character, so we shall use that as well.

In this example, we want to remove certain files from a Subversion working directory, but we never want to change anything inside any directory called .svn because that is private to Subversion. This is perhaps one of the most common use-cases for wanting to use -prune, and it will be useful to you later in your later studies, which is why I’m showing you this now. The way to remove a file under a Subversion repository is with the command svn remove filename ....

The particular files we wish to remove all start with ._, which were put there by my Mac and we accidentally imported them. We could do a similar thing with, for example, LaTeX temporary files.

This is an example, we don’t expect you to run this command.
$ find . -name .svn -prune -or -name ._\* -print0 \
>  | xargs -0 svn remove --

We’ve also added -- to the end of the svn remove command, as this should cause svn remove to not treat as an option any filename which might otherwise be seen as an option. Note that this is unnecessary in this case as we know all results are going to start with ._ and never with - or --, but in principle this is a very good thing to do.