Table of Contents
For this lab, you should do your work on Gallardo instead of Client1, as it contains some required software components that are not installed on Client1.
Before you proceed to go on with this lab, please make sure you have done the previous lab that covered the use of the Vim editor. Also, make sure you have read the handouts that were provided for the accompanying lecture. If you have questions as to how they work, please ask a demonstrator, in order to clear up any confusion.
This lab will be fairly simple, but there will be some essential commands that we shall need to cover. After that has been completed, you will work through the examples given in the lecture to appreciate how a pipeline is built up, then we’ll let you develop your own simple script.
You will likely find this lab very foundational, but also enormously empowering, once you begin to appreciate everything that scripting could do for you.
There are many commands that you might find useful for scripting, but if we were to give them to you all at once, you would be overwhelmed. So we’ll give you some commands that are used very commonly, along with a brief description. You can refer to their manual pages to understand them better. Many of these commands are demonstrated in the lecture handouts.
Concatenate files and print on the standard output. Commonly this is just used to output a (single) file to stdout, and thus introduce it into a pipeline, although that is commonly unnecessary.
Display a line of text. Rather like a simple ‘print’ command.
Format and print data. Much like C’s printf(3) function.
Output only the first
n lines of the input.
Output only the last
n lines of the input.
Transliterate (eg. change to upper-case) or delete characters.
Remove sections from each line of input. Not very sophisticated; sed offers more power.
Sort lines of text, possibly as numbers.
Remove duplicate lines from sorted input.
Print lines from input that match a pattern.
Print the number of bytes, words, and lines in input.
Read from standard input and write to standard output and files. If you think of the plumbing analogy, this provides a ‘T’ junction. It can be very useful when inspecting what is going through a particular part of a pipeline.
Report process status. Using ps -eo pid,command can be useful in scripts, as using a custom output format can be easier to parse. Supports a huge number of other options as well, most of which are not useful for scripting purposes.
Send a signal to a process. Signals are not necessarily fatal, but are a primitive form of inter-process communication.
Build and execute command lines from standard input. See the last slide of the lecture notes for an example. Very useful in conjunction with find.
Search for files (or directories, etc.) in a directory hierarchy.
find needs plenty of examples to show how best to use it. The manual page for find(1) should contain a section showing various examples. We shall revisit find later in this section.
Make temporary file name (unique).
Estimate file space usage. Note that this program can be particularly annoying at times.
Here is an example you can put into a file called
big5, that shows the largest five
entries in the current directory. It’s useful for figuring
out where all the disk space is being used up.
du -k -d1 | grep -v '^[0-9]*[[:space:]]*\.$' | sort -rn | head -5Your output will look different, you might not have any. 42080 ./Lectures 7020 ./Labbook …
We should mention that there are GUI tools that are much more enligtening about where disk-space is being used, such as the Disk Usage Analyzer application in Ubuntu.
Strip directory, and optionally a named suffix, from filenames.
Strip non-directory suffix from file name.
Print or set the system date and time.
date "+Today: %y/%m/%d"Today: 07/04/18
Delay for a specified amount of time, in seconds. Some systems may have usleep available, for sub-second intervals.
Parse command options (enhanced).
An example was provided with the lecture notes.
find can become a very confusing
command to understand if you don’t specify
you will get something that appears to be the same as
-prune option to omit directories from
your search. Here is an example, which took me quite a
while to figure out, of how to use
correctly. The reason it took me a long time figuring out
was because I was previously under the misapprehension
that find behaves as if
Let’s assume that our files may have spaces between them, which could prove problematic as spaces seperate arguments in the Unix command-line. find and xargs have the ability to seperate arguments using an ASCII NUL character, so we shall use that as well.
In this example, we want to remove certain files from a
Subversion working directory, but we never
want to change anything inside any directory called
.svn because that is private to
Subversion. This is perhaps one of the most common use-cases for
wanting to use
-prune, and it will be useful to
you later in your later studies, which is why I’m showing you
this now. The way to remove a file under a Subversion repository
is with the command svn remove
The particular files we wish to remove all start with
._, which were put there by my Mac and
we accidentally imported them. We could do a similar thing
with, for example, LaTeX temporary files.
This is an example, we don’t expect you to run this command.
find . -name .svn -prune -or -name ._\* -print0 \
| xargs -0 svn remove --
We’ve also added
-- to the end of the
svn remove command, as this should cause
svn remove to not treat as an option any
filename which might otherwise be seen as an option. Note that
this is unnecessary in this case as we know all results are
going to start with
._ and never with
--, but in principle
this is a very good thing to do.