26 Nov, 2008
In this article I'm going to explain how I manage and back up files. This article will serve as a guide for me and a handy place to keep all the commands I use. Hopefully it will be of some use to you too.
Hard drives always fail eventually. A recent study (http://usenix.org/events/fast07/tech/schroeder.html) of large scale IT infrastructure showed that in the field on average about 2-4% of drives are replaced every year but sometimes the figure is as much as 13%. There is always a risk of a disk failing and if you don't have backups you are just playing the odds.
Your computer could be lost or stolen - particularly relevant if your main computer is a laptop
You could get a virus or damage your files in some way
You could delete or accidentally overwrite important files or want to go back to a previous version
I have a MacBook Air which only has an 80Gb hard drive (but frankly even a 300Gb drive wouldn't be large enough to store all my files which is why I use the stratey I do).
I have three folders I store files in:
Files which are large and which I don't mind losing. These therefore don't form part of the Sync process described below.
Files which are backed up to DVD but which I still want on my computer. Folders in this directory are named according to the DVDs on which the files are stored eg Photos 7, Files 2 etc. Generally my most recent music or photos are stored in the DVD folder because I still want access to them but don't need them syncronised any more.
This is the equivalent of a Mac OS X Documents folder or Windows My Documents folder and contains all the files I am currently working on which still require backing up. This directory is regularly syncronised with a MyBook networked storage device on my LAN.
Within Sync are the following directories:
This folder contains files which I no longer need to work on all the time but which, when grouped together, are large enough to fill a DVD and can therefore be burned. Ideally this directory should always be empty because I should always burn DVDs as soon as I can.
This is where I dump new Music, pictures from my digital or any other files which I need to sort through into the proper directory structure. Once the files are sorted they can be move to To Burn where they wait until there are enough of them to be worth burning a DVD for. As an example if I have 1.6Gb of photos I don't look at any more, they can go into To Burn until I have enough to fill a Photos 8 DVD, then I'll burn them.
Contains the photos I need to sort
Contains the music I need to sort
Although I try to organise files properly, a lot of the time I just end up with lots of junk files I might access again but which aren't worth trying to organise properly. The Junk folder contains lots of sub-folders numbered from 1 onwards. Junk files go in one of these sub-folders and when the latest one gets quite large or when there is some sort of conceptual change in the contents of a numbered sub-folder I just create the next folder and start putting junk files in there instead.
These contain personal files which need backing up
These contain work files which need backing up
The backup DVDs are copied to the MyBook when they are made. The Sync directory is syncronised with the MyBook every week. Any files which need to be version controlled are stored in a mercurial repository. Any particularly sensitive data is also copied onto my 1Gb memory stick. This means that at the very least there are two copies of all my data.
Burn any DVDs roughly 4.37Gb in size in the To Burn folder
While that's happening check the MyBook to see if you can move any files into the same place they will need to be after rsync is run to syncronise the burned folders. This reduces the amount of files which will need to be transferred later on
Now run rsync on the To Burn folder to copy its contents to the MyBook Sync/To Burn folder
Here's an appropriate command to see what needs sending (notice the -n switch):
/opt/local/bin/rsync -aHxv -n --delete --progress --numeric-ids -e "ssh -c arcfour -o Compression=no -x" "/Users/james/Documents/To Burn/" "root@192.168.1.10:/shares/internal/PUBLIC/Linux/MyBook/To\ Burn/"
Notice some important things:
I'm using the MacPorts version of rsync (although I think the ordinary version should work fine too)
I put all the paths in quotes because they contain spaces but for the Linux version of the path I have to escape the space characters otherwise rsync on the MyBook thinks that I want to syn with /shares/internal/PUBLIC/Linux/MyBook/To not /shares/internal/PUBLIC/Linux/MyBook/To Burn/
Both directories end with a / character, otherwise you end up copying a directory into the other directory (this is unlike the cp command.
Now you can actually perform the sync if you are happy with the changes rsync will make. Notice that I'm using the the -z option instead of -n because I want the changes compressed - I sync over wifi so the connection speed is the limiting factor rather than the CPU compress/decompress time:
/opt/local/bin/rsync -aHxv -z --delete --progress --numeric-ids -e "ssh -c arcfour -o Compression=no -x" "/Users/james/Documents/To Burn/" "root@192.168.1.10:/shares/internal/PUBLIC/Linux/MyBook/To\ Burn/"
Move the contents of each burned DVD from the To Burn folder to the DVD folder on the MacBook Air
Move the contents each DVD in the Sync/To Burn folder on the MyBook to the appropriate permanent location so it is no longer synced
Under no account do I use samba or SSH to transfer files between the MacBook Air and the MyBook because I find that if I do, rsync will sometimes think they are old versions of the files and overwrite them anyway. By using rsync I can be sure they are copied correctly, efficiently and that rsync won't need to overwrite them next time it is run.
One a week I sync the whole Sync directory to back up all my files. Here's how:
If you know you've moved some files around, move them on the MyBook too so that rsync does less copying and deleting
Run the rsync command to see what will be changed:
/opt/local/bin/rsync -aHxv -n --delete --progress --numeric-ids -e "ssh -c arcfour -o Compression=no -x" "/Users/james/Documents/" "root@192.168.1.10:/shares/internal/PUBLIC/Linux/MyBook/
Once you are happy, run this to perform the sync:
/opt/local/bin/rsync -aHxv -z --delete --progress --numeric-ids -e "ssh -c arcfour -o Compression=no -x" "/Users/james/Documents/" "root@192.168.1.10:/shares/internal/PUBLIC/Linux/MyBook/"
Delete any files in Temp you don't want or any files in DVD you are happy to recover from DVD if you decide you need them again.
That's the jist of it. At a later date I plan to come back to this article to describe how I manage my music and photos.
Copyright James Gardner 1996-2020 All Rights Reserved. Admin.