If you have to manage many ubuntu server, legal log archiving and log consolidation could be an issue. This article describes a simple solution to address this problem.
Logrotate is a default package in all ubuntu distribution and should be already installed in any Ubuntu system.
S3cmd is a simple utility that you can install with:
sudo apt-get install s3cmd
After s3cmd package installation you must configure it running:
sudo s3cmd –configure
You will be asked for the two keys - copy and paste them from your Amazon account page. Be careful when copying them! They are case sensitive and must be entered accurately or you'll keep getting errors about invalid signatures or similar.
This should create the file /root/.s3cfg with your s3 credential and paramethers. Please have a look to it and see if all parameters match your desiderata.
Now you can create a bucket to store your centralized log archive on s3:
sudo s3cmd mb s3://yourname-log-archive
This is where your logs will be archived.
It is a good idea to customize bucket properties using AWS console. You could setup auto-expire facility to limit the age of logs archived (5 years could be enough for many regulatory constraints). You can enable logging (on log archive) and refine access policy for the bucket.
Now you need to install a couple of scripts in all server you want to attach to the centralized archive
First download last script version from googlecode:
svn co http://eautils.googlecode.com/svn/trunk/s3logarchiver s3logarchiver
than run install scripts:
cd s3logarchiver; sudo sh install.sh s3://yourname-log-archive yourtag
yourtag is just a name used to tag logs, if omitted the hostname will be used.
the install script does:
- install s3uploadfile in /usr/local/bin,
- add s3logrotate to the server rc*.d scripts. And in /etc/init.d/ This file allow you to send logs to s3 before rebooting. It just force for two time logrotate script on service shutdown.
- install s3logrotated in /usr/local/bin/ . This file is used as a logrotate callback to send logs to s3 when logrotate occurs.
- Creates the file /etc/ s3logarchiver.conf containing the bucket you will use to store logs
Modify the relevant logrotate scripts to use s3logrotated callback.
When configuring logrotate: don’t use dateext, do use compress, do use delaycompress, do use sharedscripts.
Logrotate should be configured to rotate logs on a daily base. You can still save logs on local server, maintain 5 days localy could be a good choice (remember that a copy of your log will be daily sored on your s3 bucket.
Here’s what I use to patch default apache2 and syslog logrotate configuration on Ubuntu 12.04 LTS:
Change defaults in logrotate.conf
sudo sed -i /etc/logrotate.conf -e 's/^dateext$/#dateext/;s/^weekly$/daily/;s/^rotate 4$/rotate 15/;s/^#compress$/compress\ndelaycompress\nsharedscripts/;s/^[ \t]*monthly$/\tdaily/;'
Patch apache logrotate configuration
sudo sed -i /etc/logrotate.d/apache2 -e 's/^[ \t]*weekly$/\tdaily/;s/^\([ \t]*postrotate$\)/\1 \n\t\t\/usr\/local\/bin\/s3logrotated ".2.gz" "\$@"/'
Patch syslog logrotate configuration
sudo sed -i /etc/logrotate.d/rsyslog -e ' s/^[ \t]*weekly$/\tdaily/;s/^[ \t]*rotate 4$/\trotate 5/;s/^[ \t]*rotate 7$/\trotate 5/;s/^\([ \t]*endscript$\)/\t\t\/usr\/local\/bin\/s3logrotated ".2.gz" "\$@"\n\1 /'
Please note that this could change in other Ubuntu releases. Spaces in sed commands are relevant.
Note that the ".2.gz" is because of compress and delaycompress - it is the first compressed log file. This is why you need /etc/init.d/s3logrotateto run ”logrotate -f” twice on server shutdown.
This article is inspired on a great post of Benjie Gillam.