3. Rotating Logs

It’s a fact of life, logs grow. Just like your lawn, the logs need processed every so often to keep them manageable, and to help you find any snakes in the grass. What you do with your old logs will reflect on your local policies, and possibly even laws eg. logs may need to be kept for several years, or on the other hand, logs perhaps may not be kept for longer than a few years due to privacy laws; you should seek advice from your lawyer or industry.

In Linux distributions, logs are usually rotated using a tool called logrotate. Have a look at the default Ubuntu configuration of logrotate by looking in the file /etc/logrotate.conf. Note that it includes all the files in /etc/logrotate.d/, to enable package-maintainers to easily install rules for having logrotate rotate its logs, simply by dropping a file into the logrotate.d/ directory. Here is what the /etc/logrotate.conf file looks like. The file has been edited to save space.

Specifying global defaults
weekly               Rotate logs weekly
rotate 4             Keep 4 weeks worth of backlogs
create               Create new (empty) log files after rotating old ones
#compress            Uncomment to compress rotated files
A directory so packages can install log rotation policies.
include /etc/logrotate.d/
/var/log/wtmp {      Stores login history.
    missingok        Don’t complain if wtmp file is missing.
    monthly
    create 0664 root utmp   perm user group
    rotate 1
}

The above is a simple example; we could do a lot more. We’ve selected the condition to rotate based on time (weekly and monthly), but we could also choose to rotate based on file size. You can also specify commands to be run before and after rotating the log file. Here is a fictitious entry which shows some of the other useful entries. This could be a file in /etc/logrotate.d/ Remember that the following is an example, so I don’t expect you to input it.

You can specify multiple files
/var/log/foo/access /var/log/foo/errors {
  size=100k           Rotate when it reaches a certain size
  sharedscripts       Run post and prerotate only once for all files in this set
  postrotate          You can run a list of commands after rotation
    killall -HUP food food would be the daemon for the foo service
  endscript           End the list using endscript
}

logrotate gets run by cron, using /etc/cron.daily/logrotate. The scripts in cron.daily get run early each morning, typically about 6am. You can alter this by editing /etc/crontab.

You can force logrotate to rotate (ie. ignoring the selection tags such as weekly or size) the files by using the -f argument to logrotate. Have a look, using ls -l, in the /var/log directory, and then run the command logrotate -f /etc/logrotate.conf. Have a look to see what changed. Repeat a few times, and describe what happens.

Although Ubuntu doesn’t compress them by default, logrotate is able to compress logs using gzip, and thus get an extension of .gz. Compressed logs can be viewed using the zless or zcat program, and grepped using zgrep (we’ll cover grep later in the lab.

3.1. Assessment

3.1.1.

On paper, write logrotate entry that rotates the file /var/log/auth.log on a weekly basis. This file is where you would find any log messages relating to failed login attempts etc. Any further rotation options are up to you. Be prepared to rationalise why you chose particular options, particularly the number of logs to keep.

3.1.2.

This question is designed to help you understand the concept of log rotation. Before we continue, we must first disable (temporarily comment-out) the notifempty directive in /etc/logrotate.d/rsyslog. This directive tells rsyslogd not to bother rotating a file if that file is empty. Normally, that is useful, but it does get in the way of understanding the concept we are trying to illustrate.

After disabling notifempty and restarting rsyslogd, cd into /var/log. From there, run the following sequence of commands three or so times, until you see a pattern emerge. If you don’t see a pattern after doing this, go on with the next question, and ask a demonstrator to explain.

$ ls auth.log* | while read file; do
> echo -en "$file\t"
> (zcat "$file" 2>/dev/null || cat "$file"; echo) \
>   | head -1 | cut -b-60
> done
# logrotate -f /etc/logrotate.conf

This somewhat lengthy command, besides being another example of how useful scripting can be, will print out all the files matching auth.log*, with their name, and the first 60 characters of the first line. Pay attention to the association between the file name and the contents. The output from a single run will look something like this:

auth.log        May 11 12:25:36 client1 sudo: mal : TTY=unknown ; PWD=/
auth.log.1      Mar 30 08:09:01 client1 CRON[3513]: pam_unix(cron:session): 
auth.log.2.gz   Mar 26 08:09:02 client1 CRON[13480]: pam_unix(cron:session):
auth.log.3.gz   Mar 15 22:00:39 client1 gnome-keyring-daemon[1504]: removing
auth.log.4.gz   Jan 12 15:49:48 client1 gdm-session-worker[1341]: pam_unix(g

Run the lengthy command three or so times (you may like to put the commands into a script) and describe what happens to the files in /var/log/ when they are rotated, based on the pattern you find from the outputs.

3.1.3.

What would be a good thing to do with archived logs? What factors might you need to take into consideration?