Perform Incremental Backups With Rsync

Backing up your information is very important to you, or at least it should be. One of my college professors once told me, “It is not a question of if your hard drive will crash; it is when your hard drive will crash.”

Rsync is a tool that allows you to keep two distinct directories in sync, either locally (two locations on the same machine, including attached devices) or remotely (local computer to server or even server to backup server). It is free and open source software and runs on Linux and most other Unix-like operating systems, including FreeBSD and Mac OS X.

What you need?

1) A local computer (or virtual machine) running Linux (Debian/Ubuntu/CentOS preferred)
It doesn’t matter if the local Linux distribution is different from the Linux distribution on your server. Since both systems run Linux we can proceed. 
2) A hard disk formatted as ext3 or ext4 with enough space to backup your files and databases from your server. This disk should be mounted to your local Linux system via /etc/fstab

Steps:

All commands below should be run via SSH as root

Step 1) First verify that rsync is installed on your local Linux system. If it is running Debian or Ubuntu, type:

If it is running CentOS, type:

Step 2) Now that we have rsync installed, we must create the SSH authentication keys, so that we can connect to the remote server via SSH without asking for password.

On your local Linux system run:

[bash]ssh-keygen -t rsa[/bash]

when asked “Enter file in which to save the key”
type carefully:

and press Enter. You will see a prompt: “Enter passphrase”
Don’t type anything, just press Enter

The next question is “Enter same passphrase again:”
Again don’t type anything and just press Enter.

Now we have created the authorization key and we need to send it to the remote server.

Step 3) In the following command replace 101.102.103.104 with the IP address of the remote server.
On your local Linux system run:

you will be asked for the root password on your remote server, type it and press Enter.

Step 4) Connect to your remote server and type, as root, the following commands:

Step 5) Back to your local Linux system, now test the passwordless SSH connection by using this command:

(again replace 101.102.103.104 with the IP address of your server)

You should be connected to your server without asking a root password! If it still asks for password, remember to add PubkeyAcceptedKeyTypes=+ssh-dss to end of /etc/ssh/sshd_config file.

Step 6) Now it’s time to set a rsync job using this passwordless SSH connection. Back to your local Linux system, run this command:

In the above example, replace:
101.102.103.104 with the IP address of your remote server
/media/localbackupdisk/ with the real path of the hard disk where you want to save the backups. If you do not want to sync owner (chown), just simply use rsync -rltzuv.

Now your local computer should connect to the remote server without asking for password and start retrieving a backup of the entire home directory, to your local disk. Depending on the size of your files on the server, this first backup will need some time to complete. But the next backup tasks will need much less time, as they will be retrieving only new or changed files.

Step 7) Set a cron job, running automatically the above command, every day at 5 am:

paste this command in a separate line:

Again, replace:
101.102.103.104 with the IP address of your remote server
/media/localbackupdisk/ with the real path of the hard disk where you want to save the backups

Save the file by pressing:
Ctrl+O
Enter
Ctrl+X

If you see the error “nano: command not found”, type:

[bash]apt-get install nano[/bash]

or:

[bash]yum install nano[/bash]

After saving the cron file, type this command to reload the cron service and read the new cron job:
On CentOS

On Debian or Ubuntu:

That was all! Repeat step 2 – 7 for additional server.

Some notes on rsync commands

1. Only include some file types

• Include pdf files and all directory containing them, exclude all others:

  [bash]rsync -a –include=’*.pdf’ –include=’*/’ –exclude=’*’ ~/LaTeX/ ~/Output/[/bash]

2. Exclude some file types

• E.g. exclude .mp4 files:

  [bash]rsync -uptorgvlHaz –exclude=’*.mp4′ -e "ssh -p 22 -i /root/.ssh/id_myserver" /home/localuser/public_html [email protected]:/home/remotedir[/bash]

Additional Tip: Script to backup database:

[bash]#!/bin/sh

FOLDER_STORE_DB="/home/database"
BK_FILE_NAME_PATTERN="*.gz"

MYSQL="$(which mysql)"
MYSQLDUMP="$(which mysqldump)"

MYSQL_USER="root"
MYSQL_PASSWORD="PASSWORD"
MYSQL_HOST="localhost"
IGNORED_DBS="test information_schema mysql yourDB"

## Step 1: Backup databases
DBS="$($MYSQL -u $MYSQL_USER -h $MYSQL_HOST -p$MYSQL_PASSWORD -Bse ‘show databases’)"
for db in $DBS
do
skipdb=-1
if [ "$IGNORED_DBS" != "" ]; then
for i in $IGNORED_DBS
do
[ "$db" == "$i" ] && skipdb=1 || :
done
fi

if [ "$skipdb" == "-1" ] ; then
FILE="$FOLDER_STORE_DB/SERVER.$db.$(date +%Y%m_%d).sql.gz"
$MYSQLDUMP -u $MYSQL_USER -h $MYSQL_HOST -p$MYSQL_PASSWORD $db | gzip -9 > $FILE
fi
done

## Step 2: Remove all backups older than 2 days
find $FOLDER_STORE_DB -type f -name $BK_FILE_NAME_PATTERN -mtime +5 |xargs rm
[/bash]

Running RSync On Windows

• Install cwRsync – Rsync for Windows
• On remote Linux server, create SSH key

  ssh-keygen -t rsa

• On Windows server, generate RSA public-private key using PuTTYgen (id_rsa_windows2008.ppk and id_rsa_windows2008.pub). Remember to generate keys without passphrases.
• Transfer the public key (id_rsa_windows2008.pub) from Windows server to backup Linux server.
• On Linux server, accept the public key as in the above “Step 4)”
• Create a kind of following batch script in the Windows box:

  [bash]SETLOCAL
  SET CWRSYNCHOME=C:\Program Files (x86)\cwRsync
  SET CWOLDPATH=%PATH%
  SET CYGWIN=nontsec
  SET HOME=%HOMEDRIVE%%HOMEPATH%
  SET PATH=%CWRSYNCHOME%\bin;%PATH%
  SET DESTINATION_USER=[user name on your backup server]
  SET DESTINATION_HOST=[IP address of your backup server]
  SET DESTINATION_ROOT_FOLDER=/home/%DESTINATION_USER%/BACKUPS/Windows_Server
  SET DESTINATION_FULL=%DESTINATION_USER%@%DESTINATION_HOST%:%DESTINATION_ROOT_FOLDER%
  SET EXCLUDE_COMMANDS=–exclude="**/My Documents/$RECYCLE.BIN/**" –exclude="**/Downloads/**" –exclude="**/$RECYCLE.BIN/**"

  CALL :s_rsync pdrive
  CALL :s_rsync Finance
  CALL :s_rsync Users

  GOTO s_end

  :s_rsync

  echo %DATE% > d:\%1\.sync
  SET LOGFILE="%CWRSYNCHOME%/logs/%1.txt"
  echo Starting backup of %1 > %LOGFILE%
  echo %DATE% >> %LOGFILE%
  echo %TIME% >> %LOGFILE%
  echo Logs >> %LOGFILE%
  rsync -a -v -v -v –delete –timeout=120 %EXCLUDE_COMMANDS% –chmod u+rwx -e "ssh -i c:\.ssh\id_rsa" "/cygdrive/d/%1" %DESTINATION_FULL% >> %LOGFILE%
  GOTO :eof

  :s_end
  ENDLOCAL
  ECHO "Done"[/bash]