Recently, a project that I undertook was to have rotating backups of our Chef servers. We have two in our lab, one on premise in our data center, and three in different VPC's in AWS. None of which were being backed up when I joined the engineering team.The main reason that this came about was that one of the servers in our lab was shutdown which caused some automatic build jobs to fail, and for some reason could not be connected to after powering it back on in AWS. Luckily I was able to recover it by taking snapshots of the drives then mount them to a new server in a different account (many more steps were taken to restore the full server).
With Chef Manage, there is a great chef-server-ctl backup command that creates a tar file of the database and files to restore to a new server. After the mishap with one of our servers, I worked up a script to create backups on the servers, and to keep only seven of them. Because I didn't want to run this as a power user (such as root), I added a line into the sudoers file to allow the user chefBackup to run the backup command:
chefBackup ALL= NOPASSWD: /usr/bin/chef-server-ctl backup --yes
With a simple script, to create the backup file
#!/bin/bash
#--Back up server
sudo /usr/bin/chef-server-ctl backup --yes
#--Remove old files
#--Crontab in Root
and then run with cron
0 2 * * * /home/chefServer/chefBackup.sh > /home/chefBackup/chefBackup.log 2>&1
To rotate the files, another cron under the root account of the server, it gets the list of files and deletes anything that is older than a certain amount of time. In this case, 7 days
0 3 * * * ls -1 /var/opt/chef-backup/*.tgz | sort -r | tail -n +6 | xargs rm > /dev/null 2>&1
After the script is creating the backup files, a simple S3 bucket with a expiration policy and user who has access was created. Then the AWS cli is installed on each server to copy the file, and the script was modified to push the backup file. Then modifying the script to add the aws s3 cp command of the latest file.
#!/bin/bash
HOSTNAME=($hostname)
#--Back up server
sudo /usr/bin/chef-server-ctl backup --yes
BACKUP="`ls -dtr1 /var/opt/chef-backup/* | tail -1`"
/usr/local/bin/aws s3 cp $BACKUP s3://chef-backup-bucket/$HOSTNAME$BACKUP
#--Remove old files
#--Crontab in Root
The process can be simplified to run under one account, or give the backup user sudo access to the system. I felt like this is a good option for now, or at least until there is another solution in place to run the backups. If you want the CloudFormation template to create the bucket yourself, it can be found here as a gist on GitHub.
Hope this helps!
