Scheduled Tasks and Log Management

Table of Contents

1. Cron
1.1. Assessment
2. Syslog
2.1. Assessment
3. Rotating Logs
3.1. Assessment
4. Filtering Logs
4.1. Assessment
5. Regular Expressions
5.1. Assessment
6. Final Words

Today we learn about how to schedule programs to run in the background at a certain time. We also learn about logging on a UNIX-like system, how to manage the system logs, and cover a couple of very useful logging related tools for rotating and reporting on the various log entries. We’ll also learn about simple regular expressions, which gives us a powerful tool for matching text against a pattern.


In this lab, we’ll just be doing all our work on your Server1 unless specified otherwise.

1. Cron

Being able to run jobs at scheduled time, and in the background, is a great assistance to a system administrator (and to the user as well). Cron is able to assist you to run jobs at scheduled times. Here is just a sample of the types of things you might use cron to do:

  • Fetch any mail from your ISP every five minutes. You could then automatically have it sorted and processed.

  • Download package updates at night, when traffic costs may be lower, or the network less busy.

  • Check the system periodically, and page the administrator when something is wrong.

  • Initiate system maintainance tasks, such as rotating logs or performing backups.

  • Run commands once only, at a certain time (using at).

  • Run commands when the system is not heavily loaded (using batch)

To use Cron, you need to know how to use crontab, which is the command for editing your own Cron Table (crontab). You also need to know the format of a crontab file. See crontab(1) for information about the command, and crontab(5) for more information on the format of a crontab.

For most system administration tasks, we don’t use personal crontabs. Instead, we use some standard directories into which we can place jobs. These directories are /etc/cron.d/, cron.hourly/, cron.daily/, cron.weekly/ and cron.monthly/. With the exception of cron.d/, these contain executable scripts, which are run every day, week and month. cron.d/ stores normal crontab entries, including a time and specification. This modularity helps immensely with package management, due to its drop-in nature. You do not use crontab to edit these files, just a regular text-editor.

There is also the file /etc/crontab which is the crontab for the system (ie. the root user). The main items it contains are entries to start all the daily, weekly and monthly jobs. You do not generally need to edit /etc/crontab; use the directories instead. Shown below is what you would see in /etc/crontab. Note that it differs from a normal users crontab, in that the user field does not appear in a users crontab.

# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.


# m h dom mon dow user command
Note, I've edited this for clarity.
17 *	* * *	root    run-parts --report /etc/cron.hourly
25 6	* * *	root	run-parts --report /etc/cron.daily
47 6	* * 7	root	run-parts --report /etc/cron.weekly
52 6	1 * *	root	run-parts --report /etc/cron.monthly

run-parts is a helper program that runs all the executable scripts in a directory, such as cron.daily/.

The scripts in cron.when/ etc. are run with the privileges of the root user, and run one after another (sequentially, based on its filename). The files in cron.d/, which can be seen of as fragments of /etc/crontab, have a user field in them, commonly root, but can easily be changed.

Each user can have a crontab file under /var/spool/cron/crontabs/[48]. This directory will most likely be empty on your system, because users don’t most often will not have created any crontab entries. A user modifies or creates a crontab using the crontab -e command. This will start the editor specified by the VISUAL or EDITOR environment variable, or the powerful-but-not-very-newbie-friendly vi[49]. The format for the file is fairly simple, but easy to forget, so you might like to put a comment at the start of the file that reminds you of the format.

Here is an example of the crontab you would be editing using crontab -e. Remember, as this is a users crontab, it does not allow you to specify a user field. Consider carefully the difference between each time specification. Note that you are not required to make any modifications at this stage.

* 8 * * *     command  Will run every minute from 8:00am to 8:59am
0 8 * * *     command  Will run at 8:00am
0 */8 * * *   command  Will run every 8th hour
* */8 * * *   command  Will run every minute of every 8th hour

All entries can be thought of as being checked every minute. The * is a wildcard which always matches. */2 means every 2 hours (when in the hourly column). Likewise, 9,17,21 matches 9am, 5pm, and 9pm in 24-hour format. If you want something to run once in a particular hour(s), then the minute field must be absolutely specified. Consider the difference beween the first two entries above.

1.1. Assessment


When will the following crontab entry run?

* * 13 jan * command


When will the following crontab entry run? Notice that both day of week and day of month are specified; check crontab(5) for how this edge-case is treated on your system.

30 4 1,15 * 5 command


Imagine you have a virtualisation host with a lot of identically configured virtual machines (guests). You notice that you get massive load spikes on a regular basis, and upon investigation during one of these load spikes, you determine that they are caused by cron jobs all starting at the same time (and thus every machine simultaneously wants to wake up and run something, causing the host to become very busy, even though each virtual machine has very little to do). How could you improve this situation? (You do not need to implement anything for this question, just brainstorm some solutions.)

[48] This can be disabled or allowed only to particular users.

[49] vi is a common Unix default, but other systems can differ. Debian-based systems in particular, use its “alternatives” subsystem to run whatever is configured as sensible-editor, which is nano by default.