Notes On rsync

The first step is to set up an SSH key pair so that back can log into main. This may be a security problem in general, but later once things are working, we’ll restrict what this key pair can do so that it will only apply to back performing this particular backup task.

On back do the following entering a name (I used main-puller_rsa) and just hitting enter at the password prompt:

Now you have to put the public key on the target machine. There are ways to do this using cut and paste, but this should be unambiguous:

This appended your newly created key (the public part) to the authorized keys file.

Test to make sure that the key pair works. You need to specify the key you want to use explicitly like so:

Cool. Now we have a way to make easy SSH connections. Next is making that connection a lot less easy for everything but the back up mission.

The hardest thing about this process is knowing exactly what command you want the main host to run. In this example, it’s some kind of rsync but that’s not good enough. We need to know exactly what the command is. The way to find this out is to create a small script that simply dumps out the exact command as submitted by the SSH client (on back in this case). The way we intercept our client’s command is by using the command= key directive. By putting this phrase followed by the command you want to run when that key makes a connection, you can control what the key pair does. It turns out that it doesn’t matter what the client asked for. If a client makes a connection using a key pair with a command= directive, that command will be run. To answer our question, we create this temporary script and make it the command that is run:

Notice that all this script does is take the variable $SSH_ORIGINAL_COMMAND and append it to a file called sshcmdlog. It Then runs the submitted command.

Here’s a test command:

Seems that it ran that normally on main from back. But over on main here is the setup and the result:

Notice that we were able to capture the exact command sent by the client as the server saw it. In this case, the command was simple and I could have just guessed that directly, but you’ll see that with rsync commands it can get tricky and it’s best not to spend all day guessing what it’s receiving.

Now I run a test of the real thing on back:

Check what turned up in sshcmdlog:

Now you can see why this is so hard to get right by guessing. I didn’t submit this command in this form, but the server interpreted as such. This is how you have to set the key on the SSH server machine so it will only honor these jobs.

This is the final form of the command directive in the SSH key in the authorized_keys file on main:

Now this unencrypted key pair really can’t do anything but make a backup. But just to limit mischief, we can further restrict the operation of that key to specific hosts. This means that if somehow credentials are stolen, a backup won’t be made to some unknown IP number. Here’s how to implement that:

Note that our host here is back but it can be an IP number or even wild card values (so I’ve heard).

Now that you have set things up so your back up server can directly make a legitimate rsync backup job using SSH, you can now automate that in a cron job. Just run crontab -e and put the rsync command in a cron job (in my example situation, this is done on back, the requesting client). This will do my example every night at one in the morning:

(see cron help)

I’ve had the very odd situation where there is a key installed and you don’t want to use it but it wants to insist. Here’s the answer.

Using ssh to transfer a filesystem:

(see tar help)

Here’s a way to use lftp to get something from lftp with it prompting the user for a password.

FTP:

HTTP:

lftp can also do a mirror. How about some Gene Ontologies? Here’s a little mirroring script showing a way it can be done.

Want to have a topology nightmare? Unison might be just what you need. Unison allows changes on either of two file systems which can then be reconciled. Imagine that you have a laptop and a desktop, L and D respectively. You synchronize your file systems somehow so they are the same. Then you delete L:temp and add L:new. Over on D you create D:different_new and delete D:bad. When you run Unison, in theory, it will delete D:temp and L:bad while adding D:new and L:different_new. That’s already a wee bit confusing for me, but now imagine that you get a second laptop, L2 and you sync to that from L. Ok. Now what if you try syncing L2 to D? You’ve got a graph loop. You need to keep a star topology. Fun! Enjoy!

You want snapshots? You’re jealous of those fruity computer people going on about "Time Machine"? Well that functionality has been around for a long time in the free software world. Check out rdiff-backup which can take bandwidth efficient snapshots of the differences in a file system allowing you to reconstruct its state at any other point a snapshot was made. If you run this in a cron job every night you will be able to go back to a file system state from any day. This is very handy for certain kinds of backups. The downside is that if you deal with large temporary files (video editing, let’s say), rdiff-backup won’t really delete those but just make a note of the fact that they were deleted. This can be circumvented a bit, but generally rdiff-backup takes up more room than a straight mirror.