Automatic Backup

From DreamHost
Revision as of 05:36, 20 September 2007 by Mousee (Talk | contribs)

Jump to: navigation, search

Note: Automated domain snapshots and Automated MySQL snapshots are now done by Dreamhost; however, the Dreamhost snapshots do not provide downloadable or transportable archive files as the method described below does. Also, the Dreamhost snapshot frequencies are relatively short-term (days or weeks) and subject to change. For MySQL snapshots, also see the MySQL page, Restore Database button, on the Control Panel for more info.


DreamHost customer matttail has come up with a system for automatically (or automagically as I like to say) backing-up domains and databases. By using the Crontab command to run a series of scripts, the process creates tar file archives. For the purposes of the article, assume the domain that needs backing-up is domain.com - so anywhere you see that you'll need to switch it out with your own domain to be backed up.

This script creates a tar file of the domain.com directory in your FTP space. Bold text must be substituted for the domain that requires the backup.

This script does not backup Maildir or other directories under /home/username/, so you may want to use a similar script for them.

When I say to 'give the command' you need to be logged into SSH and type (or paste) the commands there, then press enter. Sometimes if your server is busy, or the command requires a bit of processing it may take a bit till your get an other prompt. Just be patient. Even though you don't have a little time glass, it's working.

There is also the question of which machine user you should use to make the backups. As long as all the domains are owned by users under the same Dream Host plan it doesn't matter. Each user in your plan has read access to the other's files. So just pick one user to pull in all the backups.

Domain Backup

First we need to create a few directories. Give these commands from your home directory (that's the directory you're in when you first log in via SSH.

mkdir backups
mkdir backups/archives

This is the script to create the tar ball of your data. Copy and paste the 3 lines into your favorite text editor. I recommend using notepad. Update the bold parts, then Save As domain.sh, select the Save as Type: All files, and the encoding to be ANSI. Upload this file to the backups directory created above and chmod to 755. Simply repeat this process for each domain you want to backup.

#!/bin/bash
suffix=$(date +%y%m%d)
cd /home/username/.snapshot/nightly.0/
tar -cf /home/username/backups/archives/domain.$suffix.tar domain.com/ 

It is easy to set things up so that user 1 can backup the files of user 2 and user 3 if you have several domains with different 'owning' users. Replace the first username with the user who is creating the backups, and the second username with the one who owns the domain. Make sure that all the tar files go into the same directory, and that the directory only contains the tar files. Later in the process, the contents of the directory is deleted.

Line by line explanation

For those of you interested, I'll provide a line by line explanation of each script here. If you're in a hurry or simply not interested, just skip over this part. If you're wanting to figure out what's going on, read away.

  1. Defines a Bash script. Basically this line says, "I'm a script, use bash to execute me"
  2. Sets up the date structure to append to the file name. If you want to change how your files are dated, give the command "man date" and it will show you different time stamps you can make. If you want to see the date in your timezone you should use the date command with the TZ variable. Eg. suffix=$(TZ='Europe/Budapest' date "+%y-%m-%d_%H-%M")
  3. Changes the current working directory to provide relative paths so tar can be happy.
  4. Creates the tar file. the flags "-cf" tell the program tar to Create Forcefully an archive. type the command "man tar" for more information about this. The first path /home/...$suffix.tar tell it where to create the archive, and the second path tells it what to archive.

Backup ALL Domains

If you need backup ALL domains. List this in array

domains=( "domain1.com" "domain2.com" "domain3.com" )

and edit your script backup ALL in array:

#!/bin/bash
suffix=$(date +%y%m%d)
domains=( "domain1.com" "domain2.com" "domain3.com" )
cd /home/username/.snapshot/nightly.0/
for domain in ${domains[@]}
do
    tar -cf /home/username/backups/archives/${domain}.$suffix.tar ${domain}/     
done

Front Page Extensions Backup

DreamHost makes backups of your site that has FrontPage Server Extensions installed, but you can't access them. When FrontPage Server Extensisons are enabled, the .snapshot backup directory is no longer updated. Besides that, it's always a good idea to keep your own backups. You should make sure that this backup is working properly by comparing the files backed up to your current one.

Copy and paste the lines into your favorite text editor. I recommend using notepad. Update the bold parts, then Save As fp.sh, select the Save as Type: All files, and the encoding to be ANSI. Upload this file to the backups directory created above and chmod to 755. Simply repeat this process for each domain you want to backup.

#!/bin/bash
suffix=$(date +%y%m%d)
mkdir /home/username/FPbackup
cd /home/username/FPbackup
wget --mirror --non-verbose --domains=www.FPdomain.com www.FPdomain.com
tar -cf /home/username/backups/archives/domain.$suffix.tar www.FPdomain.com/
rm -r /home/username/FPbackup

Line by line explanation

  1. Defines a Bash script. Basically this line says, "I'm a script, use bash to execute me"
  2. Sets up the date structure to append to the file name. If you want to change how your files are dated, give the command "man date" and it will show you different time stamps you can make.
  3. makes a temporary directory to store the files in
  4. changes the working directory into our newly created temp direcotry.
  5. Using the wget utility in mirror mode basically pulls down the index file of the site, then gets all of the images and other stuff it calls in, then follows the links and pulls those pages and images. It should keep directory structure as well. So as long as everypage on your site is apart of a your site tree it will get backed up. "-domains=" Tells it to stick with this domain and not follow links to other sites that we don't want backed up. "--non-verbose" is rather noisy. It writes a single line for every file it downloads, and that will get E-mailed to you by cron. Not the best, hmm? you can change that to "--quiet" which will make wget shut up entirely. The only problem, and the reason I'm using --non-verbose, is that quiet turns off errors too.
  6. Creates the tar file. The flags "-cf" tell the program tar to 'Create' an archive in the specified 'File'. Type the command "man tar" for more information about this. The first path /home/...$suffix.tar tell it where to create the archive, and the second path tells it what to archive.
  7. Finally we remove the temporary directory

Mysql Database Backup

This script creates a backup of one or several MySQL database(s), and places them in the same directory as the tar files. As before, Copy and paste the lines into your favorite text editor. I recommend using notepad. Update the bold parts, then Save As mysql.sh, select the Save as Type: All files, and the encoding to be ANSI. Upload this file to the backups directory created above and chmod to 755. As you can see there are entries to backup two databases here. To add in more databases just copy that line and full in the proper information for each database.

#!/bin/bash
cd /home/username/backups/
mkdir mysql 
suffix=$(date +%y%m%d)
mysqldump --opt -uUser -ppass -h mysqlA.domain.com db_nameA > mysql/db_nameA.$suffix.sql
mysqldump --opt -uUser -ppass -h mysqlB.domain.com db_nameB > mysql/db_nameB.$suffix.sql
tar -cf archives/mysql_backup.$suffix.tar mysql/*
rm -r mysql/

Line by line explanation

  1. Defines a Bash script (again)
  2. Changes the current working directory. It's necessary to do this so we can give tar relative paths so it doesn't complain.
  3. Creates a temporary mysql directory for the MySQL backup(s).
  4. Sets up the date structure to append to the file name.
  5. Performs the MySQL dump. More of these can be added for other databases.
  6. Creates the tar file of the database(s) dumped.
  7. Removes the temporary mysql directory.

If you would like to email the tar file backup to yourself add this line before the last line:

mutt you@domain.com -a /home/username/backups/archives/mysql_backup.$suffix.tar -s "MySQL Backup"

To compress the tar file (especially to reduce email attachment size), substitute the following command for the tar line above:

tar zcf archives/mysql_backup.$suffix.tar.gz mysql/*

To uncompress a compressed tar archive:

tar zxf filename.tar.gz

Test your scripts

now it's time to test your two scripts and make sure that you've done everything correctly. Give these commands:

cd backups/
./domain.sh
./mysql.sh

Look at what out-put you get, if any.

troubleshooting

  • If you get something like
"/bin/sh: /home/myusername/backups/mysql.sh: /bin/bash
: bad interpreter: No such file or directory"

then something didn't go right when you saved the script. Convert the script into a unix format like this:

dos2unix domain.sh

just change out domain.sh for each of the scripts that you've uploaded. Now try testing your script again.

  • If you get a 'permission denied' you probably forgot to chmod the file.
  • If you get 'no such file or directory:...' or 'cp: can not stat' you have a path wrong and need to check that you're using the correct user names and folder names. Also remember that cAsE maTTeRs.
  • If you can't figure out what's wrong feel free to post in the forum for extra help.
  • if you're getting "tar - removing leading / from member names" make sure that you're using the domain backup script as it is now. I had changed this script to try and shorten it, which apparently produced this error for people. Silly me not verifying the changes I make. If you care to know, tar complains because it's expecting to be run by hand, working with relative paths. When the script gives full paths starting with "/" it fusses, so that's why each script change the working directory so we can give relative paths to tar and make it happy.

Note - To evade the error need to use "-P" (like tar cfzpP sample.tar.gz sample/*)
-P, --absolute-names
don’t strip leading ‘/’s from file names

Editing the Crontab

Next, we need to edit the Crontab so that the backup scripts are run on a regular basis. Give the command:

crontab -e

You'll need to add a line for each script that you have, like this:

MAILTO="you@yourdomain.com"
27 22 * * 3 /home/username/backups/domain.sh
27 23 * * 3 /home/username/backups/mysql.sh

Press Ctrl O to save the file, and then Ctrl X to exit the editor. You should receive a message stating that the Crontab was installed successfully. If you get an error about bad time, you either set the time wrong or your command wrapped onto two lines. A command in cron must always appear on one line, or else Cron will think it's two commands.

Line by line explanation

  1. Indicates where to send error messages. Set it to MAILTO="NULL" if you don't wish to receive any.
  2. Indicates when to run the domain.sh script (in this case, 10:27pm on Wednesdays).
  3. Indicates when to run the mysql.sh script (11:27pm on Wednesdays).

If you want to change the times when your scripts are run, it's easy. The Crontab article will help you change it to run at just the right times for you.

At this point you should have some scripts being run on a weekly basis, creating tar archives of your Domain(s) and Database(s).

Automatic FTP downloading of archive

The idea now is to get these backups that your Dreamhost server is creating for you onto your computer, without you having to do a thing. It wouldn't quite be an automagic system if this part wasn't working as well.

Windows

Copy and paste the following lines into your favorite text editor. Save As downloads.bat, select the Save as Type: TXT, and the encoding at it's default setting ANSI.

ftp  -i -s:ftp.txt
pause

This is a simple batch file that tells Windows to open up the command line FTP program in non-interactive mode and to access the file ftp.txt (that we're about to create) for instructions on what to do. The second line, pause tells it to leave the box open after the downloads have completed (you can view transfer speed and check successful download). If you don't want that, just remove the line.

Copy and paste the lines into your favorite text editor. Update the bold parts, then Save As ftp.txt, select the Save as Type:TXT, and the encoding at it's default setting.

open domain.com
username
password
cd backups/
binary
mget archives/
mdelete archives/
quit

Move both files to the folder you wish you archives to be downloaded to.

Line by line explanation

This is also fairly simple...

  1. Log into FTP
  2. Give the server your username
  3. Give the server you password
  4. Move into the directory that holds our backups
  5. Download the backups
  6. Delete the archives, so we don't download them again next week.
  7. log out of FTP.

In the Windows Control Panel, double-click the Scheduled Tasks icon. Follow the wizard for creating a new task and schedule download.bat to run about 20 minutes after the shell scripts have been executed. If everything is working properly, you'll now have tar archives created of your Dream Host data automagically downloaded to your computer every week.


Linux

Start by following the SSH article to password-less login. Next copy and paste this bash script, updating the parts in bold.

#!/bin/bash
cd /home/username/backups/archives
sftp -bftp username'@domain.com

here is the batch file called 'ftp' above:

cd backups/
get archives/*
rm archives/*
bye

Finally add a Crontab entry to run your download script. My Entry looks like this:

* 04 * * 5 /home/username/backups/ftp.sh > /home/username/backups/ftp.log 2>&1

I'm specifically routing any output (including errors) to a log file in this case because I don't have a mail system installed where such things would normally be received.

Variations

User Gordaen from the Dream Host Forums has written this script which backups up both site files and database in one script. Take note that the script creates a tar archive for each mysql database. You could switch this out for the method above if you'd prefer to have separate files for each database rather than one big one.

#!/bin/bash
suffix=$(date +%y%m%d)
nice -19 tar -c /home/username/.snapshot/nightly.0/domain.com/ | gzip -c > archives/domain.com-$suffix.tar.gz
mysqldump --opt -uusername -ppassword -h mysql.DOMAIN.com DBNAME | gzip -c > archives/domainDB-$suffix.sql.gz

Another minor variation of the tar command above is:

nice -19 tar -czf archives/domain.com-$suffix.tar.gz /home/username/.snapshot/nightly.0/domain.com/

Encrypted Backups (via GPG)

Feel safe emailing your entire site/database backup? You shouldn't. Here's an overview of how to add encryption to the backup scripts above. Specifically, this section details the use of The Gnu Privacy Guard (GPG) to create encrypted files without including yet another password in your script files.

GPG uses the mathematical magic of public-key cryptography, which is beyond the scope of this section. The important part is that you create a key pair (consisting of a secret key and a public key, which are mathematically related). The public key file can be stored on your DreamHost server (and distributed to the world). Anything encrypted by GPG with this public key can only be decrypted with the associated private key (which you keep safe on your home machine, or even totally offline).

Here's a good article on setting up your keys. Following those instructions, you should end up with:

  1. secring.gpg and pubring.gpg (and other) files on your safe and secure home machine
  2. pubring.gpg file imported to your DreamHost $HOME/.gnupg directory
  3. You've set the key's trust to ultimate (so the encryption command will not require any user input)

Then, add a line like this to your script to create the encrypted file:

gpg --armor --recipient keyname --output outputfile.asc --encrypt backupfile
  • armor means to use ASCII armor, which is good for emailing the file
  • keyname is the way GPG recognizes the recipient key (the email address or ID of the key)
  • outputfile specifies where the file is saved (the .asc extension just indicates ASCII content)

Run the command yourself, then cat outfile.asc to see what it looks like. Change your backup script to email this encrypted file.

To decrypt, you must be on the machine with your secring.gpg file. In addition, this secret key should be protected by a passphrase (specified when you created the key pair). So, you need to know your passphrase and have your secret key file to decrypt (two-factor authentication).

gpg --decrypt encrypted.asc --output decrypted

Assuming your secret key is on the local machine, GPG will ask for your passphrase. If correct, the file is decrypted.

In summary, this allows you to encrypt your backup files before they leave the comfy confines of your DreamHost account. Anyone who gets their hands on the encrypted backup cannot decrypt it without both your secret key file and its passphrase.

See also