Site Feed

Monday, September 15, 2008

Backup Logs with a Shell Script



Here is a very simple way to backup and compress subdirectories. Normally you would want to copy them to a remote location in case something happens to the original drive, but in this case the concern is that disk space is filling up.

Suppose you have a program that is generating log files every day, let's say they are key performance indicators (kpi) for some system that you are testing. The log files are pure text, but after a while, they take up a lot of space. One way to compress text is to use gzip, but since tar has an option (-z) to tar and gzip all at once, it may be one of the easiest solutions.


#!/bin/sh

# compress logs in /opt/multicast_kpi/record
# by ScottM, 20080613

# Set variables
TODAY="`date +%m%d%Y`"
ARCHIVE_DIR=/opt/multicast_kpi/record

# move to working directory
cd $ARCHIVE_DIR

# For each subdirectory, make a .tgz file of it's contents, and then delete the originals.
for dir in `ls -1 $ARCHIVE_DIR | grep -v ".tgz" | grep -v "${TODAY}"`
do tar -C ${ARCHIVE_DIR} -czvf ${dir}.tgz ${dir}
# Note that the archive directory is hard-coded here:
# We don't want to use rm -rf just anywhere...
# Even then, we still verify that the directory exists before running rm
[ "${dir}" != "" ] && [ -d ${ARCHIVE_DIR}/${dir} ] && rm -rf /opt/multicast_kpi/record/${dir}
done

# go back to previous directory
cd -

The results will be:

Original
ls -1 /opt/multicast_kpi/record/
20080101
20080102

Zipped
ls -1 /opt/multicast_kpi/record/
20080101.tgz
20080102.tgz

Then the program can continue to create subdirectories of any name, and place log files there. Our script does not care what name the subdirectory has. Of course, as some point the .tgz files may fill up the disk, so that is another problem you will have to deal with.

For more fun, take a look at the logrotate command, which can be used to rotate, compress, and mail system logs.

No comments: