Practical UNIX style backups using an ArchLinux PogoPlug

If there was ever a reason to get a $15 pogoplug hacked to run Arch Linux, this article may sway you.

I'm going to show you how I turn an ordinary $15 linux gadget into a useful "rsync" backup client.

The tasks which I will share with my readers is the typical things I would do administering *NIX based servers. We set up redundancy and failover using some simple and tried and true methods. There really is no rocket science involved. After doing it a few times, it becomes second nature. And because of the simplicity, it becomes apparent why I love small gadgets running Linux.

The articles and methods are pretty much simplified but they illustrate the simple and powerful nature of the powerful command line.

Today, I decided to turn one of my Pogos into a remote robo-copying slave. Its only job is to do remote backups of my GIT server. Then I realize, it should be running independently as a working droid (thinking Star Wars) to find,scan my network and backup any *NIX computers running in my household. I have a few spare older 250-320GB drives that needed to put to good use, so I pared them to a Pogo.

Normally, you would initiate the copy and backup from your desktop/laptop to the PogoPlug running some form of Linux/NAS. Here, I do the reverse. I have my pogoplug go out and do all the work.

So if I turn on my NetBSD G4 mac from 5 years ago, or come home with my Thinkpad, or turn on my iMac, it would automatically back them up without my intervention. I wouldn't have to think about it. If I was working on some code on my Thinkpad. I could go out to the back-yard to my patio and from my Macbook, I can pull source code that was already synced 5 minutes earlier off the Thinkpad.

Think of it as a reverse time machine. The concept is not new. We have dedicated backup servers that do nothing else but do remote backups off-site. Here, I am using a low powered PogoPlug running ArchLinux.

In fact, it took me 30 minutes to implement it. This is the power of *NIX. So I will share it with my readers today. Everything is meant to be done on the Pogo itself.

First, I logged into my Pogo and copy my SSH keys from my Pogo to all my target computers. So if I change my passwords, it would still authenticate against it. As I write this, I am thinking R2-D2 talking to the main-frame of the Death Star. If you don't know what SSH keys are, take a detour and google it before proceeding further. In short, keys allow machines to talk to one another without using passwords.

In arch, it is pretty easy.

ssh-copy-id username@remote_server

It will simply copy your keys over to the remote machine.

Next, I wrote a small bash script that pings the remote machine. If the machine pings, it means it is online and it will then attempt to rsync with it. Rsync is the tried and true industry standard for remote file synchronization/file copies.

Feel free to use this bash script. Simply, change the variable of the "remote_machine" to the IP or hostname of the computer you want it to ping and rsync. Since my network employs Avahi/Bonjour, it pretty much works by hostname. EG. My Thinkpad is accessible via ThinkpadT420.local
In this example, my remoteserver.local is my intended target.

I do a simple IF THEN conditional check in my bash script. If the machine doesn't ping, alert us with a message. Otherwise, proceed to rsync.

 #!/bin/bash  
 remote_machine="remoteserver.local"  
 PINGCOUNT=2  
 PING=$(ping -c $PINGCOUNT $remote_machine | grep received | cut -d ',' -f2 | cut -d ' ' -f2)  
 if [ $PING -eq 0 ]; then  
           echo "Something wrong! server: $remote_machine down"  
 else  
           echo "All good: $remote_machine"  
           echo "We will rsync now"  
           rsync -au --progress --stats root@remoteserver.local:/var/www/ /media/passport/rsync/vps/www/  
 fi  

My rsync command is pretty straightforward here. Archive and update. I added progress and stats for my own reference.

rsync -au --progress --stats root@remoteserver.local:/var/www/ /media/passport/rsync/vps/www/ 

Basically, my rsync logs into the remote server, copying the /var/ww/ into my destination of my 1TB Western Digital USB drive labelled, passport. The copies go into a folder, /media/passport/rsync/vps/www


After writing it, I tested it. I named my script vps_backup.sh

I then change the permissions to executable and did a dry run.

As you can see, the files and folders populate from my remote machine. The below screenshot shows two web directories. One from my iMac and another from a remote VPS.

Lastly, I would use a cron job to run every few minutes. You can even do cron job schedules in Webmin. I would recommend installing webmin so you can have a web based administratie interface of your Pogo. It simplify things quite a bit.

When you log into webmin, head over to System . Scheduled Cron Jobs.

Cron Jobs under webmin is pretty much a point and click.

I specified the user running as "root" to have full system rights and pointed to the script I wrote earlier in /root/vps_backup.sh. For this example, I specified a daily midnight schedule but I could change to hourly or every 5-10 minutes.

And there you have it. A simple (well, at least for me) way to do interval backups from a Pogo.

Legacy Windows Rsync Backup to FreeNAS

I have some old Windows servers (10 years and counting) and I have been using rsync to back them up to my FreeNAS box. It has been working great for me.

First of all, I do have my Windows servers backup in virtualized format. However, those are only one-time snapshops that I run once in a while. These are classic ASP IIS web servers that I can easily put up on a new VM. However, many of these legacy servers generate gigabytes of data a day in their repositories. Running VM conversion daily is not ideal.

My solution was to use some sort of rsync solution just for the data repos. I've tried some applications that didn't work too well with Samba shares and these old servers have slow I/O. Copying files to external sata or usb drive was not ideal. We've moved on from Windows to Linux and do not have any Windows file servers of capacity to provide network backups.  Hence, I decided to use Delta Copy with FreeNAS. So here is a little write up on how to set it up. I have 4 Windows 2000 servers backing up daily with this method.

First, download Delta Copy and install it. It is open-source and pretty much free. It is basically a wrapper for cygwin's rsync. When you install it, it will ask you to install the Server services which allows you to run it as a Rsync server on Windows. You don't need to do this. Instead, you will be just using the Delat Copy Client application. But before we do that, we will need to configure our Rsync service for our Windows Clients on FreeNAS.

In FreeNAS, go under Services , Select Rsync >  Rsync Modules > Add Rsync Module.

Then fill out the form; giving the module a name and set the path. In my example, I simply called it WIN and linked it to a user called backupuser.

This process is much easier than trying to configure the daemon rsyncd.conf file by hand.

Now, on the Windows Client, start the DeltaCopy Client. You will create a new Profile.

You will need to enter the IP of the Rsync server (FreeNAS) and specify the module name which will be called "Virtual Directory Name."  When you pull the select menu, the list of Rsync Modules you created earlier in FreeNAS will populate.

You can set authentication. On the server, you can restrict by IP and do other things to lock down your rsync. 

Next, you will add folders (and/or files) you want to synchronize.

Once the paths are set up, you can run a sync by right clicking the profile name.

Here, I made a test sync to a home folder of a virtualized windows box. As you can see, I mounted the rsync volume on my mac to see the progress. The rsync worked beautifully. DeltaCopy did what it was told.

Once you get everything working. The next thing to do is set schedules. If you done tasks schedules in Windows before, it is pretty straightforward. DeltaCopy has a link in the application to directly create a new task for you. I set my backups to run nightly and it has been working great.

There you have it. Windows rsync to FreeNAS using DeltaCopy.
The nice thing about FreeNAS is you don't have to modify /etc/rsyncd.conf files. Everything can be done in the web admin.

Practical iTunes backup with Rsync

If you are a mac user with a large iTunes collection, backups can be a chore. Some people use Time Machine, carbon-cloner/super-duper, or the traditional method of simple drag-n-drop. Well, I suggest you embrace the built-in *NIX way of doing things and just use rsync. 

It is natively built in the operating system and many of the cloner type apps are just pretty GUI skins on top of Rsync. Time machine is notorious at making duplicates when you don't need to. Who needs to revert back 4-5 revisions of the same file?

Since I started iTunes Match (review coming), I've been updating my music collection and re-tagging my mp3s quite a bit. For example, I would update artist and genre one week. Another week, I would swap out my album art for higher resolution ones that will look nice on the retina iPad. Time Machine will store the 3 different iterations in my example.

Also, rsync is more intuitive than simply drag-n-dropping files. I have over 20,000 files in a music folder. I may be updating or changing 3-4 files at any given time. I may not know the location of the files but I've made updates within iTunes. Instead of looking for the files or copying the entire folder, rsync will simply match up what needs to be synchronized. This is a real time saver.

I won't go into the details of Rsync. You can google and read the man page.  Here is a very simple usage example with options explained:

 rsync -auvp --progress /Users/username/Music/ /Volumes/MUSIC_BK/Music/  
 -a archive  
 -u update only (don't update newer files)  
 -v verbose  
 -p preserve permissions  
 --progress = show progress  

Here, I am simply copying updates from my Music folder to a USB drive. In fact, you can simply write a shell script and leave it on the USB/Firewire drive. When I need to synch, I simply drag the script over to a Terminal window to start my rsync session.

Rsync is also good in multi-platform environments and cloud. I always rsync files to my DropBox or my PogoPlug. When I have a bunch of shell scripts or excel files, I use rsync to synchronize files between OSX and Linux.