Chris' SSH Notes

When people think of SSH they think of "logging in" but SSH is actually simpler than that. It merely creates a secure tunnel to a host and then runs a command. In the normal case as just shown, most people most of the time don’t specify a special command. In that case SSH uses the default command of the user’s shell.

It can be very useful to not run an interactive shell as the command SSH executes. Here I connect — but don’t "log in" — to the host xed.ch, run the date program there, and return control to the original local host.

Running specific commands is extremely useful for dealing with multiple hosts. For example, if I have a file called workstations which contains comments (starting with #) and hostnames; I can check the date on all my workstations with this command.

I have written a pretty complete utility called nodessh for dealing with cluster nodes which can simplify most cases of running things like this. I note it to remind myself. Email me if you’re interested.

If you just need a couple of commands, you can pipe them into the ssh command and the default shell will run them. Here’s one I just did to cure dozens of NFS stale file handles on cluster nodes after updating the NFS server.

This basically has the same effect as the following, but connecting to a remote host first.

Sometimes complete wankers write software and they often like to make graphical installers or monitors or some other nonsense for software that has no business being graphical. This means a simple installation or management issue turns into a GUI nightmare. It is well known that the -X option of SSH will tunnel remote X clients to your local server. The -Y is just like -X but better (use it). This allows you to log in somewhere and run that machine’s xclock on your display. Slowly. Great.

But what if you need to run something as sudo? This then gets to be a right pain. The following has worked for me.

You should make sure that your sudoers file contains

Maybe even this.

Either way, double check the DISPLAY variable at each step even if sudo doesn’t preserve it automatically.

SSH is a public key cryptographic system. This means that it does not use a symmetrical "shared secret", i.e. the password I encode with may not be the password you decode with. Instead it generates a key pair, one private and one public. The private key is very important and must be very closely guarded. To let someone have access to it is to improve their odds of breaking into what it protects substantially. The public key is not sensitive information and that can be freely distributed without concern.

Let’s say you have a host called house and another called office. You want to work from home which entails logging into office from house. On house generate a key pair. Create a file on office called ~/.ssh/authorized_keys which contains the public key from house. This file can contain several public keys, one per line. It’s fine to edit it manually by cutting and pasting or any way you can.

Note the permissions section below to sort out any permissions issues.

Assuming your key is named housekey, and the contents of housekey.pub are in office:~/.ssh/authorized_keys, you should be able to log into office from house like this:

Generally just use the command ssh-keygen. It will prompt for the name which should include the full path or else it will use $PWD. This name will be the name of the private key. The public key will also be generated and be the same name with a .pub added to it. It will then ask you for the passphrase.

Here’s a good command line way to generate keys with minimal questions:

If you don’t provide the comment (-C) it defaults to $USER@$HOSTNAME.

Sometimes I create a key pair and come back to it much later. I think I know the passphrase I’ve encrypted the private key with, but I’d like to check privately before getting too excited. The correct way to do that is with -y.

If you need to change the passphrase the private key is encrypted with you only need to worry about the private key. Do this.

It will prompt you for the old and new passphrase.

It is weird that there is a minimum of 5 characters to this password. It’s weird because 5 is ok, 4 is insufficient, but 0 is ok. Hmm.

If you set up an account for someone and give them access to it by placing their public key as described below, you can run into an odd problem where the new account holder can log in but can not change their own password. This is because they never knew it and the passwd program requires it. This is a reminder for the administrator making the account to leave a copy of the password in a file the new user can access. This will allow them to change it. I sometimes make a strong password and then know that it stays that way if the account holder never uses (i.e. complains about) password authentication.

If you ever work with a cluster, it is common that each node on the cluster will have a common shared file server where a user’s home directory is stored. This means that a file changed on node n12 will also be changed on node n17. Often it is useful to set up a key pair with no password so that you can easily control things on any node of the cluster. Since each node is probably on a local network (i.e. not available to the general internet) the security issues are less problematic. Also, if any node is compromised, with a shared file system all nodes are then compromised regardless of what key pairs are lying around.

To set this up is relatively easy since your home and office hosts as used in the previous examples are basically the same machine as far as setup goes. Perhaps the easiest way is to just run:

And then accept defaults and when it asks for a password, just hit enter (it will ask for confirmation - hit enter again). Then it will create the key pair in ~/.ssh. You can ls -l ~/.ssh to have a look. Now that you have a key pair you can use, you have to tell your account (which is replicated on all the nodes) that you will accept this key pair for log in attempts. A good way to do that would be:

If you had an authorized_keys file you have now just appended the contents of your new public key to it (this example assumes that the new public key was ~/.ssh/id_rsa.pub, but it could be ~/.ssh/id_dsa.pub or some other name you gave it).

Note the permissions section below to sort out any permissions issues.

Once the keys are set up and have the correct permissions, you should be able to log into another machine without a password. Just

If node22 has the same home directory source, it should find your authorized_keys file there (even though you set it up "here") and it should log in nicely. If there are problems, try using the -v option. This always is a helpful first step to diagnosing SSH key problems.

Often there are problems with permissions. Permissions are important because you don’t want all this security only to have some one else log into the system and just read your ssh set up and learn what strategies would best be used to attack your account. This is why permissions can be fussy.

Note that the authorized_keys file should have no permissions for group or other or it will not work. Do this to make sure:

You also have to have private keys be private for SSH to take them seriously:

The public keys can be any way you like; they’re public after all:

I also think it’s a good idea to do the following:

This limits everybody but you and the SSH server from seeing how your security is configured.

If your private key is encrypted with a passphrase (a very good idea), here is how you change it:

Here is how to limit what a key can do by origin and by function:

Just manually edit your authorized_keys file and include the keys you want to authorize with any restriction prefixes.

Note that the command= directive won’t block a different command, but rather it will just run the command specified in the authorized_keys file. So if you have this key in host alpha:.ssh/authorized_keys:

And do this from beta:

You’ll get the results of cal. In other words, instead of simply not working, it works differently than you explicitly commanded it to.

See also ForceCommand for how to configure this on the sshd side with no need for keys to be set up.

How do you know what command is really being run remotely? Have SSH monitor it and report it back to you. To do this use a program like this:

Make sure it’s 755 so that it can be run as a command.

This is useful if you’re trying to set up a key that only allows a specific rsync job. Using test_ssh_cmd, you can run your rsync command and see that it really runs on the sshd server like this:

Then you can clip that out and put it verbatim into a command="" specifier and it should work. To otherwise just guess this can be difficult.

Private keys that are used interactively should always be protected with a passphrase. However, it’s reasonable to decrypt the passphrase and leave it available in memory for the duration of a session you might want to use it. This allows you to use the key without entering a passphrase every time. This is accomplished with a key agent. This is a special program that SSH knows how to consult with to see if any of the encrypted keys it needs to use have been decrypted recently and loaded in memory. If so, it just uses it again.

Often your window manager will politely fire up ssh-agent automatically so that it is a parent process. To see if you have a key agent up and running, try this.

If you get some number, that probably means yes.

To add your keys use the ssh-add command like this.

From then on you shouldn’t have to worry about decrypting that key!

I don’t know the details, but it looks like by default Ubuntu (and probably other distributions' display managers) uses a Gnome key manager that seems pretty easy to use. Basically when you need a private key decrypted, a window pops up and offers you the chance to load it into its manager. Then subsequent usage of this key will use it from the key manager’s copy and not bug you for the decryption passphrase.

Sometimes you don’t have a GUI and a fancy integrated keyring manager won’t be possible. If you’re trying to get a key agent working with text console logins you have to resort to the standard key agent. The classic key agent is called ssh-agent. It’s kind of an unusual program. It basically needs to be the top level parent process of all the processes that might be running that would require its services. This can be a pain if you’re already running a shell that requires it and the agent wasn’t set up to be a parent process. So you have to plan in advance. This is done by running it and then running your shell as a sub process.

Note that for many window managers, the entire window manager process kicks off wrapped in an SSH agent. For example, here’s what my processes look like using the Awesome window manager.

This leaves terminals and other subprocesses set up with this kind of thing.

The most basic way to have a process (and its children) know about the key-agent is to make that process a subprocess of the agent. Here are some likely examples of how you’d use it in a simple way:

Here’s an example I use:

This alias is useful if you often use a computer without graphics, but when you do want graphics, this starts it up and makes the entire X Windows session a sub process of the ssh-agent. This means that whatever xterm you might have open, all of it is in touch with the key-agent. (Note that this alias also cleanly shuts down the tty console you were using which I think is a nice security feature so that people can’t Ctrl-Alt-F1 into the shell you started X from when you’ve locked the screen.)

Basically the agent works by exporting some important ssh-agent environment variables to the subprocess. You can also just have ssh-agent generate the shell commands to establish the current shell as an environment that knows about the agent. To do this, run (only one of these):

or

After this is done the environment variables should be in place for ssh clients to know how to use the agent.

These are untested, but seem reasonable for getting a key agent environment set up automatically from a .bashrc (in Bash):

Having this in bash allows the new shells to find the agent. To start the agent the first time, use something like this:

You can also install the program "keychain" which manages a lot of this mess. Apparently, you run the program:

And it creates files in a directory .keychain:

Now you just have to source one of the files in ~/.keychain:

or

Now when you log in from somewhere else, you can use the same key agent by sourcing this keychain script.

To stop a keychain spawned agent (and get rid of the .keychain scripts):

Having an agent keep your keys in memory is cool, but sometimes you need something fancier. You’re on hostA and you need to log in to hostC from hostB. This could happen, for example, if you’re at home on hostA and hostB is the secure gateway that lets you in through the firewall to hostC. I also use this setup to have a system administration machine (hostB) at work that controls various machines (hostC) from home (hostA).

The first thing to check is that hostB allows agent forwarding.

If not, reset this to yes and restart your ssh server. Once this is enabled, hostB pretends to have an agent running even when it doesn’t. When you ask the agent for keys, it basically forwards this request over the already established connection to hostA. Apparently if hostB is messy with untrusted activity, it might be best not to do this since an attacker can hijack all your loaded keys if there is a compromise. Otherwise I think it’s pretty safe.

Also your client on hostA needs to be cool about giving the agent information to your connections that ask for it. You need to set this on hostA:

Note that you can do this per host so that only certain hosts get the forwarded keys stored in the agent. Shown above is how to do it for all hosts. You can also do it on the command line using:

To see if things have a shot at working you can use this command to see what keys are loaded on hostA:

Use -L to see the full keys. If hostA shows you some keys, then you know your agent is working at least. Next run that same command on hostB. You should see the exact same keys available. If in doubt, you can delete a key from the list back on hostA and check to see if that did the same thing on hostB. Use this to remove a key from the agent’s control:

Once you have keys loaded in an agent, real or forwarded from somewhere else, you should be able to proceed as if it were a simple A to B connection.

Now for a very unusual but very irritating problem. Let’s say that for some crazy reason, you want to have a home directory that is not writable. This means that the normal place (~/.ssh/known_hosts) can’t accumulate host keys.

There are some things that can be tried. If you are the admin for a bunch of machines (maybe a compute cluster) you can think about adding all the nodes' host keys to /etc/ssh/ssh_known_hosts which contains a system wide definition of known host keys. This relieves users of having to verify them.

Another possibility that might be even easier is this option:

That’s a lot to type and I’ve had trouble with it believing my intentions sometimes.

Another trick is to collect the host keys in a non-typical place and put them in the right place some other way (elsewhere maybe). To do that use this option:

Sometimes you’re trying to sort things out with the known host keys and you find something like this:

This is a hashed known host key. The idea here is that if an attacker compromises an account, the first thing they will attempt is to look at your known hosts file for a record of exactly where else you like to log in. Then they’ll try those credentials that got them into this account on those accounts. This is how SSH viruses propagate. By hashing the host key, you have enough information for verifying a host, but not enough to propagate an attack to other machines SSH is known to have contacted. That said, this is a huge pain to deal with in many cases. The way to turn off (or on) hashed host keys is to set the option:

To fix this permanently, add this option to /etc/ssh/ssh_config. It’s off by default (if not configured) though configured on by default in some distributions.

Also take a look at the -F option of ssh-keygen to deal with unruly hashed keys. If the idea of hashed host keys sounds great to you you can convert a plain host key file with ssh-keygen -H filename.

Sometimes you need to see an old fashioned MD5 fingerprint of a key. For example, here I’m verifying that my key was installed in Github properly. This is how I got the file to produce the classic fingerprint.

Ever seen something like this and wondered what it was?

This is a visual representation of the host key’s fingerprint (also shown in the example). Since humans are pretty terrible at comparing such long strings of numbers (confession - I have only ever looked at the first few and last few characters when checking such things), this rendering is hoped to be better for humans to more easily notice inconsistencies from one occasion to the next. Here’s an early paper describing the concept. Here’s a paper which authoritatively talks about how it’s implemented in OpenSSH.

One thing I found interesting is that the "S" is for "start" and it is always the same (which seems dumb), the middle. The pattern is made of a random walk, a move for each of 64 bit pairs of a 128 bit MD5 hash. Cells visited once get a . while multiple visits get the following: 2="o", 3="+", 4="=", 5="*", 6="B", 7="O", 8="X", 9="@", 10="%", 11="&", 12="#", 13="/", 14="^". "E" is the end of the walk.

If you’re very serious about this, see the file sshkey.c in the source code. Search for the phrase "human brian". A copy of this might be here.

To just see the ascii art for an arbitrary key, use -l -v.

If you like the ASCII art stuff, you might also like bubblebabble. Take a look at one of your keys with this.

Let’s say you have a computer that’s behind some kind of firewall. Perhaps it requires a VPN connection to access and you don’t have an invasive client because you only have access to a system with user SSH accounts (like an ISP), i.e. you can’t change the network. If you can log in to the remote machine some other way initially to set things up (either by being on site or using the VPN from a suitable client machine) you can set up a tunnel which will allow you to later reach back to that hidden machine. Here’s an example.

Imagine that we have a machine called hidingbox that is on a local network. It may be using NAT or it might just have a serious firewall. We want to connect to it from a machine called openbox. First log in to the hidingbox, either by using a VPN or a bastion host. Or visit the hidden machine in person.

Note that 6166 is a port I completely made up. Use whatever is convenient.

You might need to run screen or nohup to put this open connection in the background (or leave it just sitting open on the terminal). Or consider the ssh -N option. Essentially, this connection needs to be connected. Also consider settings like this if needed.

Or even an aggressive dedicated solution to keep the tunnel open. You can also add a command like sleep 50000 to make sure you don’t leave an open usable shell back to your real computer lying around on the remote hidden machine.

Once that is done, you can go to your openbox and log in to the hidingbox like this.

Note that by logging into the tunnel on openbox, I am connected, over the internet, to hidingbox which really doesn’t even have a real IP number.

Imagine you’re away from home and you want to work on a computer that is on but usually behind a router with a local IP address. Normally while still at home you’d go to (local IP) 192.168.1.1 and configure the router to forward connections to this target machine. Note that the normal way people do this is to leave the router configured to accept outside configuration connections to the router directly. But if you didn’t do this (which is reasonable from a security standpoint, especially because there is a good work around) and you have any machine you can SSH into inside the network (with forwarding to its port 22 already established in the router) you can tunnel in and make the changes using the router’s web interface as if you were inside the network.

Another use is if some service insists that you originate your web session from the last IP address, your home, but you’re on vacation. Maybe it can be used to bypass annoying regional restrictions.

Start by making a tunnel to the IP address that is SSH accessible from the outside. It’s the normal way you’d SSH to it with the normal port you’ve exposed but with a -D option — this setting is dynamic application-level port forwarding.

Now on your "away" machine you can open a browser, go to the Network Settings, and choose Socks Proxy to 127.0.0.1 port 9000. Now requests from this away browser will go through your home machine’s network. This allows you to put in something like 192.168.1.1 in the away browser outside your internal network and see the configuration settings as if you were on the home network. Note you can check this is working by something like an IP location service.

Some possible problems can be that the SSH server needs permissions for some things. Make sure your sshd_config where you log in to has these settings.

You may find that the tunnel connection gets intermittent errors like this.

I looked in sudo tail /var/log/auth.log and found a bunch of connection attempts on IPv6 — so it makes sense to disable that (try setting IPv6 to "ignore" in sudo nmtui). There may be nothing that can really be done about this if you don’t want to humor IPv6. For example, I got calls to an IPv6 number resolving to 1e100.net (n.b. a googol) which is a vehicle for Google’s spycraft.