Contents

General SSH Functionality

SSH is one of the most important tools for serious networked computing. It stands for "Secure SHell" and is a secure upgrade from the ancient unprotected rsh (Remote SHell). While it is mostly used by people to "log in" to another machine, what is really happening is that the other machine is running its shell with output going to the SSH process which then forwards it somewhere else. Most people don’t realize that the core element of SSH is not the shell, but rather this remote forwarding. SSH runs a text shell (Bash usually) by default, but it can actually run anything.

Here I ping a remote host, xed.ch and the time is about 24ms. But I then use ssh to run the same command on that host and the time for this machine to basically ping itself is 300 times faster.

$ H=xed.ch
$ ping -c1 $H | grep icmp_
64 bytes from xed.ch (111.222.222.111): icmp_seq=1 ttl=47 time=23.7 ms
$ ssh $H ping -c1 $H | grep icmp_
64 bytes from xed.ch (111.222.222.111): icmp_req=1 ttl=64 time=0.079 ms

Even though I’m sitting at the same computer and pinging the same computer, the SSH version of the command simulates me actually being present at the remote computer with input and output tunneled over a secure connection.

To "log in" to remote machines the normal command is something like this:

$ ssh remote.example.com

If the login name on the remote system is different (you’re logged in as cxedwards here but need to log in as chris there) you can specify the correct login name to use like this:

$ ssh -l chris remote.example.com

A common short cut for this option is the following format:

$ ssh chris@remote.example.com

Commands

When people think of SSH they think of "logging in" but SSH is actually simpler than that. It merely creates a secure tunnel to a host and then runs a command. In the normal case as just shown, most people most of the time don’t specify a special command. In that case SSH uses the default command of the user’s shell.

getent passwd $USER | cut -d: -f7

It can be very useful to not run an interactive shell as the command SSH executes. Here I connect — but don’t "log in" — to the host xed.ch, run the date program there, and return control to the original local host.

ssh xed.ch date

Multiple Commands

Running specific commands is extremely useful for dealing with multiple hosts. For example, if I have a file called workstations which contains comments (starting with #) and hostnames; I can check the date on all my workstations with this command.

for W in $(grep -v '^#' ~/workstations); do echo $W; ssh $W date; done

I have written a pretty complete utility called nodessh for dealing with cluster nodes which can simplify most cases of running things like this. I note it to remind myself. Email me if you’re interested.

If you just need a couple of commands, you can pipe them into the ssh command and the default shell will run them. Here’s one I just did to cure dozens of NFS stale file handles on cluster nodes after updating the NFS server.

ssh xed.ch "umount -l /cfs; mount /cfs"
echo "umount -l /cfs; mount /cfs" | ssh xed.ch

This basically has the same effect as the following, but connecting to a remote host first.

echo "umount -l /cfs; mount /cfs" | bash

Graphics Forwarding - X11

Sometimes complete wankers write software and they often like to make graphical installers or monitors or some other nonsense for software that has no business being graphical. This means a simple installation or management issue turns into a GUI nightmare. It is well known that the -X option of SSH will tunnel remote X clients to your local server. The -Y is just like -X but better (use it). This allows you to log in somewhere and run that machine’s xclock on your display. Slowly. Great.

But what if you need to run something as sudo? This then gets to be a right pain. The following has worked for me.

ssh -Y user@host
sudo su
xauth merge /home/user/.Xauthority
bozo_config # The stupid program you need to run remotely as UID=0

You should make sure that your sudoers file contains

Defaults    env_keep += "DISPLAY"

Maybe even this.

Defaults    env_keep += "DISPLAY XAUTHORIZATION XAUTHORITY"

Either way, double check the DISPLAY variable at each step even if sudo doesn’t preserve it automatically.

SSH Configuration Files

I don’t know how I’ve been using SSH for nearly two decades without using personal configuration files. They are very easy to use and very helpful eliminating almost all command line annoyances in one organized place. Here is a typical configuration file that I use.

/home/$USER/.ssh/config
# Xed's Client User SSH Configuration
# ===================================
# Must have: chmod 0600 ~/.ssh/config

# IPv4 only.
AddressFamily=inet

# Also blowfish is ok too.
Cipher=3des

# No obsolete insecure protocols.
Protocol=2

# == dusty3
Host 172.19.199.199 dusty3 pharm-dusty3
    # Can list as many aliases as you like.
    # Put IP number as "HostName" to skip look up. Right?
    HostName= 172.19.199.199
    # Very handy for awkward $USER transitions.
    User=admin-dusty
    Port 22
    # Run remote graphical clients?
    ForwardX11=yes
    ForwardX11Trusted=yes
    # Trade off CPU vs. network speed.
    Compression=no
    # Show some textual artwork of the host key.
    VisualHostKey=yes

# == Work Station One
Host ws1* 137.111.123.111
    # (Includes: ws1, ws1-alab, ws1-alab.example.edu)
    HostName=137.111.123.111
    User=xed
    Port 22
    ForwardX11=yes
    ForwardX11Trusted=yes
    Compression=no
    VisualHostKey=yes
    IdentityFile=~/.ssh/id_rsa
    GSSAPIAuthentication=no

# == Other Work Stations
Match host ws?
    Hostname=%h-alab.example.edu
    User=xed
    VisualHostKey=no
    Port 22
    PreferredAuthentications=publickey
    IdentityFile=~/.ssh/id_rsa
    GSSAPIAuthentication=no

# == Beeeater
Host 192.168.0.9 b be bee beeater
    IdentityFile=~/.ssh/be-rsa
    HostName=192.168.0.9

I’m not 100% sure about turning off the GSSAPIAuthentication but it does seem to keep it from performing a futile check which may save time.

Keys

If you are interested in how public key cryptography works, you should watch this extremely well made video which explains in simple clear terms exactly how asymmetric cryptography works.

General Strategy

SSH is a public key cryptographic system. This means that it does not use a symmetrical "shared secret", i.e. the password I encode with may not be the password you decode with. Instead it generates a key pair, one private and one public. The private key is very important and must be very closely guarded. To let someone have access to it is to improve their odds of breaking into what it protects substantially. The public key is not sensitive information and that can be freely distributed without concern.

Let’s say you have a host called house and another called office. You want to work from home which entails logging into office from house. On house generate a key pair. Create a file on office called ~/.ssh/authorized_keys which contains the public key from house. This file can contain several public keys, one per line. It’s fine to edit it manually by cutting and pasting or any way you can.

Note the permissions section below to sort out any permissions issues.

Assuming your key is named housekey, and the contents of housekey.pub are in office:~/.ssh/authorized_keys, you should be able to log into office from house like this:

$ ssh -i ~/.ssh/housekey office

Key Generation

Generally just use the command ssh-keygen. It will prompt for the name which should include the full path or else it will use $PWD. This name will be the name of the private key. The public key will also be generated and be the same name with a .pub added to it. It will then ask you for the passphrase.

Here’s a good command line way to generate keys with minimal questions:

$ ssh-keygen -C "Xed General Purpose" -f ~/.ssh/xedgen

If you don’t provide the comment (-C) it defaults to $USER@$HOSTNAME.

Checking Keys

Sometimes I create a key pair and come back to it much later. I think I know the passphrase I’ve encrypted the private key with, but I’d like to check privately before getting too excited. The correct way to do that is with -y.

:->[raven][~/.ssh]$ ssh-keygen -y -f fancykey_rsa
Enter passphrase:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDE/W/3W5.....full public key...etc
:->[raven][~/.ssh]$    # <- Note the happy exit code. Unhappy and no public key = bad pass.

Changing A Key’s Password

If you need to change the passphrase the private key is encrypted with you only need to worry about the private key. Do this.

ssh-keygen -f /home/xed/.ssh/xedkey.pem -p

It will prompt you for the old and new passphrase.

It is weird that there is a minimum of 5 characters to this password. It’s weird because 5 is ok, 4 is insufficient, but 0 is ok. Hmm.

Note About Remote Passwords And Account Setup

If you set up an account for someone and give them access to it by placing their public key as described below, you can run into an odd problem where the new account holder can log in but can not change their own password. This is because they never knew it and the passwd program requires it. This is a reminder for the administrator making the account to leave a copy of the password in a file the new user can access. This will allow them to change it. I sometimes make a strong password and then know that it stays that way if the account holder never uses (i.e. complains about) password authentication.

Using Keys To Work With A Cluster

If you ever work with a cluster, it is common that each node on the cluster will have a common shared file server where a user’s home directory is stored. This means that a file changed on node n12 will also be changed on node n17. Often it is useful to set up a key pair with no password so that you can easily control things on any node of the cluster. Since each node is probably on a local network (i.e. not available to the general internet) the security issues are less problematic. Also, if any node is compromised, with a shared file system all nodes are then compromised regardless of what key pairs are lying around.

To set this up is relatively easy since your home and office hosts as used in the previous examples are basically the same machine as far as setup goes. Perhaps the easiest way is to just run:

$ ssh-keygen

And then accept defaults and when it asks for a password, just hit enter (it will ask for confirmation - hit enter again). Then it will create the key pair in ~/.ssh. You can ls -l ~/.ssh to have a look. Now that you have a key pair you can use, you have to tell your account (which is replicated on all the nodes) that you will accept this key pair for log in attempts. A good way to do that would be:

$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

If you had an authorized_keys file you have now just appended the contents of your new public key to it (this example assumes that the new public key was ~/.ssh/id_rsa.pub, but it could be ~/.ssh/id_dsa.pub or some other name you gave it).

Note the permissions section below to sort out any permissions issues.

Once the keys are set up and have the correct permissions, you should be able to log into another machine without a password. Just

$ ssh node22

If node22 has the same home directory source, it should find your authorized_keys file there (even though you set it up "here") and it should log in nicely. If there are problems, try using the -v option. This always is a helpful first step to diagnosing SSH key problems.

Key Permissions

Often there are problems with permissions. Permissions are important because you don’t want all this security only to have some one else log into the system and just read your ssh set up and learn what strategies would best be used to attack your account. This is why permissions can be fussy.

Note that the authorized_keys file should have no permissions for group or other or it will not work. Do this to make sure:

$ chmod 600 ~/.ssh/authorized_keys

You also have to have private keys be private for SSH to take them seriously:

$ chmod 600 ~/.ssh/id_rsa

The public keys can be any way you like; they’re public after all:

$ chmod 644 ~/.ssh/id_rsa.pub

I also think it’s a good idea to do the following:

$ chmod 700 ~/.ssh

This limits everybody but you and the SSH server from seeing how your security is configured.

Changing Passphrase

If your private key is encrypted with a passphrase (a very good idea), here is how you change it:

$ ssh-keygen -p -f ~/.ssh/superkey

Creating Special Key Pairs With Limited Functionality

Here is how to limit what a key can do by origin and by function:

from="ok.xed.ch",command="/bin/tcsh",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAA......keynoisehere.......WHB73nQ== user@example.com

Just manually edit your authorized_keys file and include the keys you want to authorize with any restriction prefixes.

Note that the command= directive won’t block a different command, but rather it will just run the command specified in the authorized_keys file. So if you have this key in host alpha:.ssh/authorized_keys:

command="cal" ssh-rsa AAA........keynoisehere.....WHB73nQ== user@example.com

And do this from beta:

ssh alpha.example.com ls

You’ll get the results of cal. In other words, instead of simply not working, it works differently than you explicitly commanded it to.

See also ForceCommand for how to configure this on the sshd side with no need for keys to be set up.

Checking And Specifying Commands

How do you know what command is really being run remotely? Have SSH monitor it and report it back to you. To do this use a program like this:

#!/bin/sh
# test_ssh_cmd
# Put this in front of the key of interest in .ssh/authorized_keys:
#    command="/fullpath/test_ssh_cmd" ssh-rsa AAAAB3.....x4sBbn62w6sISw== xed@xedshost
# Then check the contents of sshcmdlog for what ssh really saw. That
# is now usable in a command= directive to limit the activity to what
# you really intend.
echo "$SSH_ORIGINAL_COMMAND" >> sshcmdlog
exec $SSH_ORIGINAL_COMMAND

Make sure it’s 755 so that it can be run as a command.

This is useful if you’re trying to set up a key that only allows a specific rsync job. Using test_ssh_cmd, you can run your rsync command and see that it really runs on the sshd server like this:

rsync --server -vnlogDtpr --delete-during --partial .

Then you can clip that out and put it verbatim into a command="" specifier and it should work. To otherwise just guess this can be difficult.

Key Agents

General

Private keys that are used interactively should always be protected with a passphrase. However, it’s reasonable to decrypt the passphrase and leave it available in memory for the duration of a session you might want to use it. This allows you to use the key without entering a passphrase every time. This is accomplished with a key agent. This is a special program that SSH knows how to consult with to see if any of the encrypted keys it needs to use have been decrypted recently and loaded in memory. If so, it just uses it again.

gnome-keyring-daemon

I don’t know the details, but it looks like by default Ubuntu (and probably other distributions' display managers) uses a Gnome key manager that seems pretty easy to use. Basically when you need a private key decrypted, a window pops up and offers you the chance to load it into its manager. Then subsequent usage of this key will use it from the key manager’s copy and not bug you for the decryption passphrase.

ssh-agent

Sometimes you don’t have a GUI and a fancy integrated keyring manager won’t be possible. If you’re trying to get a key agent working with text console logins you have to resort to the standard key agent. The classic key agent is called ssh-agent. It’s kind of an unusual program. It basically needs to be the top level parent process of all the processes that might be running that would require its services. This can be a pain if you’re already running a shell that requires it and the agent wasn’t set up to be a parent process. So you have to plan in advance. This is done by running it and then running your shell as a sub process.

Sub-Process Method

The most basic way to have a process (and its children) know about the key-agent is to make that process a subprocess of the agent. Here are some likely examples of how you’d use it in a simple way:

$ ssh-agent xterm &
$ ssh-agent tcsh
$ ssh-agent screen

Here’s an example I use:

alias sx='( nohup ssh-agent startx & ) ; clear; exit'

This alias is useful if you often use a computer without graphics, but when you do want graphics, this starts it up and makes the entire X Windows session a sub process of the ssh-agent. This means that whatever xterm you might have open, all of it is in touch with the key-agent. (Note that this alias also cleanly shuts down the tty console you were using which I think is a nice security feature so that people can’t Ctrl-Alt-F1 into the shell you started X from when you’ve locked the screen.)

Modify Current Environment Method

Basically the agent works by exporting some important ssh-agent environment variables to the subprocess. You can also just have ssh-agent generate the shell commands to establish the current shell as an environment that knows about the agent. To do this, run (only one of these):

$ eval `ssh-agent -s`  # In bash.

or

> eval `ssh-agent -c`  # In csh.

After this is done the environment variables should be in place for ssh clients to know how to use the agent.

These are untested, but seem reasonable for getting a key agent environment set up automatically from a .bashrc (in Bash):

SSHAGENT=/usr/bin/ssh-agent
SSHAGENTARGS="-s"
if [ -z "$SSH_AUTH_SOCK" -a -x "$SSHAGENT" ]; then
    eval `$SSHAGENT $SSHAGENTARGS`
    trap "kill $SSH_AGENT_PID" 0
fi

Having this in bash allows the new shells to find the agent. To start the agent the first time, use something like this:

$ eval `ssh-agent`

keychain

You can also install the program "keychain" which manages a lot of this mess. Apparently, you run the program:

$ keychain

And it creates files in a directory .keychain:

$ keychain
KeyChain 2.6.8; http://www.gentoo.org/proj/en/keychain/
Copyright 2002-2004 Gentoo Foundation; Distributed under the GPL
 * Initializing /home/xed/.keychain/office.example.com-sh file...
 * Initializing /home/xed/.keychain/office.example.com-csh file...
 * Initializing /home/xed/.keychain/office.example.com-fish file...
 * Starting ssh-agent

Now you just have to source one of the files in ~/.keychain:

$ source ~/.keychain/office.example.com-sh

or

$ source ~/.keychain/office.example.com-csh

Now when you log in from somewhere else, you can use the same key agent by sourcing this keychain script.

To stop a keychain spawned agent (and get rid of the .keychain scripts):

$ keychain -k all

Agent Forwarding

Note
It seems that some people consider SSH Agent Forwarding to be a bad idea. I believe my use case does not fit his proposed alternative, but ProxyCommand is an interesting thing to be aware of.

Having an agent keep your keys in memory is cool, but sometimes you need something fancier. You’re on hostA and you need to log in to hostC from hostB. This could happen, for example, if you’re at home on hostA and hostB is the secure gateway that lets you in through the firewall to hostC. I also use this setup to have a system administration machine (hostB) at work that controls various machines (hostC) from home (hostA).

The first thing to check is that hostB allows agent forwarding.

$ sudo grep AllowAgentForwarding /etc/ssh/sshd_config
AllowAgentForwarding yes

If not, reset this to yes and restart you ssh server. Once this is enabled, hostB pretends to have an agent running even when it doesn’t. When you ask the agent for keys, it basically forwards this request over the already established connection to hostA. Apparently if hostB is messy with untrusted activity, it might be best not to do this since an attacker can hijack all your loaded keys if there is a compromise. Otherwise I think it’s pretty safe.

Also your client on hostA needs to be cool about giving the agent information to your connections that ask for it. You need to set this on hostA:

[hostA]$ grep -B1 ForwardAgent /etc/ssh/ssh_config
Host *
ForwardAgent yes

Note that you can do this per host so that only certain hosts get the forwarded keys stored in the agent. Shown above is how to do it for all hosts. You can also do it on the command line using:

[hostA]$ ssh -o ForwardAgent=yes hostB

To see if things have a shot at working you can use this command to see what keys are loaded on hostA:

ssh-add -l

Use -L to see the full keys. If hostA shows you some keys, then you know your agent is working at least. Next run that same command on hostB. You should see the exact same keys available. If in doubt, you can delete a key from the list back on hostA and check to see if that did the same thing on hostB. Use this to remove a key from the agent’s control:

ssh-add -d hostC_rsa

Once you have keys loaded in an agent, real or forwarded from somewhere else, you should be able to proceed as if it were a simple A to B connection.

Note
If you’re using screen or tmux then you might need to restart the whole session if you’re just turning on agent forwarding. At the very least, do initial testing without this complication. In fact, screen is tricky. Here’s a good discussion of the situation. I was able to just set the environment variables explicitly in the screen session that existed in the normal login. Something like:
export SSH_AUTH_SOCK=/tmp/ssh-ilYGf77777/agent.17777

Host Keys

Host keys are not the same thing as the keys discussed above. What host keys do is ensure that when you contact a machine, it really is the machine you intended to contact. The technology is similar - the server has a private key and you have a public key for that server. When you contact it, you ask it to authenticate itself as the machine you really want and if it really is that machine, it uses its private key to create a message that could only be created by that key. If your version of the public key can decode the message (this all happens automatically - you never read the message per se) then you are well assured that you are talking to the machine you think you are.

Imagine it from another point of view. Let’s say you’d like to steal some log in credentials from ssh users. You figure out how to interject your server on the network in such a way that when users try to log into the official server, they actually get to yours. But just as the server asks the user to authenticate, the user should ask for confirmation that the server is still good.

Now, back on earth, it’s really pretty difficult to pull off this kind of attack even without identification verification of the host key. But if this mechanism were not there, bad people would focus on the resulting weakness, so we need to take care of it. So that’s what host keys are all about.

When you get an SSH session complaining about host keys the odds are overwhelming that it’s because of some kind of misconfiguration or legitimate situation. The two most common situations are when you’ve never logged into that server before (this mechanism is completely ineffective for first time log ins) and when the operating system has been reinstalled. In the latter case, the cryptographic keys are generated anew with the new OS installation. Upgrading your SSH could conceivably cause this too.

In most cases, the situation is that during first time log in, SSH asks you if you are confident that the host is not bogus. Here I show this process.

$ ssh a.xed.ch date
The authenticity of host 'a.xed.ch (137.110.66.66)' can't be established.
RSA key fingerprint is 5e:bd:a3:e0:31:d6:9e:58:a1:69:67:3d:9d:2b:39:df.
Are you sure you want to continue connecting (yes/no)? y
Please type 'yes' or 'no': yes
Warning: Permanently added 'a.xed.ch,137.110.66.66' (RSA) to the list of known hosts.
Password:
Tue Aug  7 17:52:19 PDT 2012
$ cat ~/.ssh/known_hosts
a.xed.ch,10.0.0.111 ssh-rsa AAAAB3NzaC1lp2RNNNNOVjNNNDRNcJNPun43a3AuCMg21JCjWiXZbRSu2zql8bJ
iM7behoN9Tvo7NKe9xbn6DH+yWv6jELJWBJ7Hzwmeiwxmfd34QtRb2ZZwGOKV5xAV7ajqtdk10T4/o24BoG9hdB2nNh
hnuYUp9k7gO8EDacz6sVUvQaXFRmZ3HBkINKANV2xgSsG19ZUv3AjUvwV5xLdSQX+Pu/k3kf0XjLCGxWKl1dQVYga0t
aXHEw8tYiQJiJNcwpDRGqDCm2XpVPZO73ov8RrSmmIT9+UyFJ4CdTo5EPFbT2qWAKmjtb6KekTvOTHd4JmOL8fk62Fy
cAdvXb9Ov4TO3J8JafnpwlJvfrfzpwqCBD==

Notice that you need to type "yes" to have it accept. Then the example shows the host key in the file ~/.ssh/known_hosts, in other words this is a list of hosts you know and trust.

Let’s say I go messing with the known_hosts file and I make a change to the host key for this host. I would see a message like this:

$ ssh a.xed.ch
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
08:74:6b:90:49:39:3b:28:7f:4b:84:41:b2:14:83:38.
Please contact your system administrator.
Add correct host key in /home/xed/.ssh/known_hosts to get rid of this message.
Offending key in /home/xed/.ssh/known_hosts:12
RSA host key for a.xed.ch has changed and you have requested strict checking.
Host key verification failed.

Generally the best way to handle this is to edit the known_hosts file and delete the offending key. This can be tricky with SSH configurations that don’t easily identify the hosts with the key. But the clue is where it says Offending key in.... That line ends in a 12 and that’s the line number of the file where that key is. The way I handle it is to type vim ~/.ssh/known_hosts +12 and then once that loads type dd to delete that line and then ZZ to quit and save. You can use your own editor or an in line sed edit:

`sed -i 12d ~/.ssh/known_hosts`

Because host keys problems are so irksome to me, I wrote a little script that very aggressively cures them.

sssh
#!/bin/bash
# Chris X Edwards - xed.ch - Sat Jun 26 2010
# "Serious SSH". Curing host key annoyances.
# Don't you hate it when you try to use ssh and it balks because the
# host key has changed? Well I often do a lot of things on a lot of
# machines that cause the host key to change (reinstall OSes for
# example). When you are using ssh amongst a large collection of
# machines, this can be quite a pain. Also since ssh saves and checks
# host keys for various ways to specify the same machine, it really
# gets irritating.

# This program is designed so that the user can re-run the failed ssh
# command with `sssh [sshopts] [user@]host [cmd [cmdopts]]` and it
# will then parse out the host, figure out any other likely names
# associated with that host, and then rerun the ssh command (in a way
# that accepts the new host key).

# EXAMPLE:
##  :-> [swan][~]$ ssh ostester
##  Warning: the RSA host key for 'ostester' differs from the key for the IP address '10.0.133.157'
##  Offending key for IP in /home/xed/.ssh/known_hosts:25
##  Matching host key in /home/xed/.ssh/known_hosts:53
##  Are you sure you want to continue connecting (yes/no)? ^C
##  :-< [swan][~]$ sssh ostester
##  KILLING KEY FOR: ostester
##  /home/xed/.ssh/known_hosts updated.
##  Original contents retained as /home/xed/.ssh/known_hosts.old
##  KILLING KEY FOR: 10.0.133.157
##  /home/xed/.ssh/known_hosts updated.
##  Original contents retained as /home/xed/.ssh/known_hosts.old
##  KILLING KEY FOR: ostester.xed.ch
##  /home/xed/.ssh/known_hosts updated.
##  Original contents retained as /home/xed/.ssh/known_hosts.old
##  RUNNING THIS SSH COMMAND:
##  ssh -o 'StrictHostKeyChecking no' ostester
##  Warning: Permanently added 'ostester,10.0.133.157' (RSA) to the list of known hosts.
##  xed@ostester's password:
##  Last login: Sat Jun 26 23:17:10 2010 from netblock-72-25-108-23.dslextreme.com
##  :-> [ostester.xed.ch][~]$

# What this program does:
# Step ONE - parse the ssh command line and extract the host
# Step TWO - find all other ways to address this host
# Step THREE - delete host keys for all hostnames found
# Step FOUR - rerun the command adding the flag to accept new host key

# Returns first word in a list: "returnsthis notthis orthis orthis"
function firstword {
    if [ -z "$1" ]; then return 1; fi
    echo $1
} # End function firstword.

# Step TWO
function find_host_aliases {
    HH=$1
    #ETCHOSTS=`grep " $HH " /etc/hosts`
    ETCHOSTS=`grep "\(^$HH \| $HH \)" /etc/hosts`
    if [ "$ETCHOSTS" ]
    then
        IP=`echo $ETCHOSTS | awk '{print $1}'`
        ALIST=`echo $ETCHOSTS | sed "s/$IP *//"`
        if [ "$IP" == "$HH" ]; then IP="" ; fi
        for A in $ALIST
        do
           if [ "$A" != "$HH" ]
           then ALIASES="$ALIASES $A"
           fi
        done
    else
        IP=`host "$HH" | sed -n 's/.*has address //p'`
        if [ $IP ]; then
            ALIASES=`nslookup "$IP" | sed -n 's/.*name = \(.*\).$/\1/p'`
            if [ "$ALIASES" == "$HH" ]; then ALIASES=""; fi
        fi
    fi
    echo $HH $IP $ALIASES
} # End function find_host_aliases.

# Step ONE
# The whole point here is to reclaim the option string for ssh options
# only and rebuild it to use when resubmitting the ssh command.
while getopts "1246AaCfgKkMNnqsTtVvXxYyb:c:D:e:F:i:L:l:m:O:o:p:R:S:w:" OPT
do
    case ${OPT} in
        1 | 2 | 4 | 6 | A | a | C | f) OL="$OL -$OPT";;
        g | K | k | M | N | n | q | s) OL="$OL -$OPT";;
        T | t | V | v | X | x | Y | y) OL="$OL -$OPT";;
        b | c | D | e | F | i | L | l) OL="$OL -$OPT $OPTARG";;
            m | O | o | p | R | S | w) OL="$OL -$OPT $OPTARG";;
    esac
done

shift $(( $OPTIND -1 ))
CMD=$@
POSSIBLEHOST=`firstword $CMD`
if [[ "$POSSIBLEHOST" =~ "@" ]]
then
    H=${POSSIBLEHOST##*@}
else
    H=${POSSIBLEHOST}
fi
UHOST=${POSSIBLEHOST}
shift
CMD=$@

if [ -z "$H" ]; then echo "ERROR: Can not figure out host."; exit; fi
shift

# Step THREE
for HA in `find_host_aliases $H`
do
    echo KILLING KEY FOR: $HA
    ssh-keygen -R $HA -f $HOME/.ssh/known_hosts
done

# Step FOUR
echo RUNNING THIS SSH COMMAND:
SSHCMD="ssh $OL -o 'StrictHostKeyChecking no' $UHOST $CMD"
echo $SSHCMD
eval $SSHCMD

The Pain Of Host Keys On Read Only File Systems

Now for a very unusual but very irritating problem. Let’s say that for some crazy reason, you want to have a home directory that is not writable. This means that the normal place (~/.ssh/known_hosts) can’t accumulate host keys.

There are some things that can be tried. If you are the admin for a bunch of machines (maybe a compute cluster) you can think about adding all the nodes' host keys to /etc/ssh/ssh_known_hosts which contains a system wide definition of known host keys. This relieves users of having to verify them.

Another possibility that might be even easier is this option:

ssh -o StrictHostKeyChecking=no newhost.xed.ch

That’s a lot to type and I’ve had trouble with it believing my intentions sometimes.

Another trick is to collect the host keys in a non-typical place and put them in the right place some other way (elsewhere maybe). To do that use this option:

$ ssh -o UserKnownHostsFile=/tmp/a_temp_known_hosts_file example.xed.ch

Hashed Host Keys

Sometimes you’re trying to sort things out with the known host keys and you find something like this:

|1|ob6ehezy0ZtO5CZQ3YRHABNH890=|NfJD9i/tBGTGgGDgm2VaCP7ktK0= ssh-rsa AAAAB
333aC1yc2EAAANOVjNNNDRNiZth9r8wWJ4hrfGbWPwfWkGnTPWiXV0JNA6VWwqhT3mdZgfAaA6
3NVJR4qbNsmxgdk3++ADHIb7k9dfogkAClieGcNX6p83sfc174wwho7QkzCyuUYkrBWGGyNW+X
+/D38Ow/kbiqSNTrm0G1WsaP2YpoExV4iEyHzQzCRYBeMmx+PvpLW42c0a51uWb2OewTdHAqsk
KfiR2RlY6UCa4jtMI/Dw97q3s27WVgBkphEIxwnwj/0NVVO11TzPf/SWlAT0gGj/YieBXS//Fg
mPsvnYWU5Ad5eVKtZ2f4cKH13bIThH2u6wAuQBHYG2a9lQfcKCe6Bmp7nMwwwz6Bq7OsD==

This is a hashed known host key. The idea here is that if an attacker compromises an account, the first thing they will attempt is to look at your known hosts file for a record of exactly where else you like to log in. Then they’ll try those credentials that got them into this account on those accounts. This is how SSH viruses propagate. By hashing the host key, you have enough information for verifying a host, but not enough to propagate an attack to other machines SSH is known to have contacted. That said, this is a huge pain to deal with in many cases. The way to turn off (or on) hashed host keys is to set the option:

ssh -o HashKnownHosts=yes example.xed.ch

To fix this permanently, add this option to /etc/ssh/ssh_config. It’s off by default (if not configured) though configured on by default in some distributions.

Also take a look at the -F option of ssh-keygen to deal with unruly hashed keys. If the idea of hashed host keys sounds great to you you can convert a plain host key file with ssh-keygen -H filename.

Fingerprint

Sometimes you need to see an old fashioned MD5 fingerprint of a key. For example, here I’m verifying that my key was installed in Github properly. This is how I got the file to produce the classic fingerprint.

$ ssh-keygen -l -E md5 -f /home/xed/.ssh/github_rsa
2048 MD5:b7:63:d6:03:8c:91:63:04:dd:e3:be:5a:c5:61:1e:1f xed@usb64 (RSA)

VisualHostKeys

Ever seen something like this and wondered what it was?

A bunny?
$ ssh cs.uw.edu
The authenticity of host 'cs.uw.edu (128.208.3.118)' can't be established.
ECDSA key fingerprint is ab:b4:a2:bc:50:e9:44:10:5a:08:f7:73:1e:35:5d:b3.
+---[ECDSA 256]---+
|=oo     o. .o    |
|o+ .   . ..  o   |
|. . o o     E    |
| . . + .         |
|  +   . S        |
| +       .       |
|. .   . .        |
| o  .. o         |
|  +o .o          |
+-----------------+

This is a visual representation of the host key’s fingerprint (also shown in the example). Since humans are pretty terrible at comparing such long strings of numbers (confession - I have only ever looked at the first few and last few characters when checking such things), this rendering is hoped to be better for humans to more easily notice inconsistencies from one occasion to the next. Here’s an early paper describing the concept. Here’s a paper which authoritatively talks about how it’s implemented in OpenSSH.

One thing I found interesting is that the "S" is for "start" and it is always the same (which seems dumb), the middle. The pattern is made of a random walk, a move for each of 64 bit pairs of a 128 bit MD5 hash. Cells visited once get a . while multiple visits get the following: 2="o", 3="+", 4="=", 5="*", 6="B", 7="O", 8="X", 9="@", 10="%", 11="&", 12="#", 13="/", 14="^". "E" is the end of the walk.

If you’re very serious about this, see the file sshkey.c in the source code. Search for the phrase "human brian". A copy of this might be here.

To just see the ascii art for an arbitrary key, use -l -v.

ssh-keygen -v -l -f somekey.pub

If you like the ASCII art stuff, you might also like bubblebabble. Take a look at one of your keys with this.

ssh-keygen -B -f somekey.pub

Port Forwarding

SSH can be used to do port forwarding to get around firewalls and other useful things. For something like a license server you’ll need to know what port it responds on. Using the application that needs a license can often tell you what port it has. Use netstat | grep license.example.com. You should see two ports that don’t change. One of those is the one that you need. Now add something like:

Host machine.example.com
    LocalForward  27000 license_server_hostname:27000
    LocalForward  56775 license_server_hostname:56775

Then set your license to point to localhost:27000.

There is a lot more you can do with forwarding. Here is a good collection of tricks.

Here’s an example I tried where I tried to temporarily move a mail server to be a different machine while backups were being made.

[root@relayhost ~]# ssh -g -L25:realtargethost.example.edu:25 realtargethost.example.edu
root@realtargethost.example.edu's password:
Last login: Sat Jan 23 05:08:32 2016 from relayhost.example.edu
[root@realtargethost ~]#

Now when you do nc relayhost.example.edu 25 you’ll get the server running on realtargethost.example.edu but note that in this case, with port 25, it won’t work since the realtargethost will just send the mail back to relayhost. You can have a chat with the mail server though. Maybe there’s some fancy sendmail magic that can help too.

Here is another example. Say you have some ROS stuff running somewhere and a simulator running somewhere else. To make the simulator think it’s running on the same host as the ROS stuff, you can do something like this from the host where you want to run your simulator.

ssh -g -L4567:roshost:4567 roshost

This says log into roshost and set things up there so that anything that needs port 4567 will tunnel that back to the current (originating) host. The connection sits open like a normal SSH connection but the tunnel will allow the sim to place itself effectively on the remote host.

Reverse Tunneling

Let’s say you have a computer that’s behind some kind of firewall. Perhaps it requires a VPN connection to access and you don’t have an invasive client because you only have access to a system with user SSH accounts (like an ISP), i.e. you can’t change the network. If you can log in to the remote machine some other way initially to set things up (either by being on site or using the VPN from a suitable client machine) you can set up a tunnel which will allow you to later reach back to that hidden machine. Here’s an example.

Imagine that we have a machine called hidingbox that is on a local network. It may be using NAT or it might just have a serious firewall. We want to connect to it from a machine called openbox. First log in to the hidingbox, either by using a VPN or a bastion host. Or visit the hidden machine in person.

:->[hidingbox][~]$ ssh -R 6166:localhost:22 xed@openbox.example.edu
xed@openbox.example.edu's password:
Last login: Tue Feb  7 07:09:02 2017 from 132.32.1.8
:->[openbox][~]$

Note that 6166 is a port I completely made up. Use whatever is convenient.

You might need to run screen or nohup to put this open connection in the background (or leave it just sitting open on the terminal). Or consider the ssh -N option. Essentially, this connection needs to be connected. Also consider settings like this if needed.

ServerAliveInterval 15
ServerAliveCountMax 4

Or even an aggressive dedicated solution to keep the tunnel open. You can also add a command like sleep 50000 to make sure you don’t leave an open usable shell back to your real computer lying around on the remote hidden machine.

Once that is done, you can go to your openbox and log in to the hidingbox like this.

:->[openbox][~]$ ssh -l xed localhost -p 6166
Password:
:->[hidingbox][~]$ ifconfig eth0 | sed -n 2p
        inet 192.168.0.22  netmask 255.255.255.0  broadcast 192.168.0.255

Note that by logging into the tunnel on openbox, I am connected, over the internet, to hidingbox which really doesn’t even have a real IP number.

Proxy Forwarding

Imagine you’re away from home and you want to work on a computer that is on but usually behind a router with a local IP address. Normally while still at home you’d go to (local IP) 192.168.1.1 and configure the router to forward connections to this target machine. Note that the normal way people do this is to leave the router configured to accept outside configuration connections to the router directly. But if you didn’t do this (which is reasonable from a security standpoint, especially because there is a good work around) and you have any machine you can SSH into inside the network (with forwarding to its port 22 already established in the router) you can tunnel in and make the changes using the router’s web interface as if you were inside the network.

Another use is if some service insists that you originate your web session from the last IP address, your home, but you’re on vacation. Maybe it can be used to bypass annoying regional restrictions.

Start by making a tunnel to the IP address that is SSH accessible from the outside. It’s the normal way you’d SSH to it with the normal port you’ve exposed but with a -D option — this setting is dynamic application-level port forwarding.

ssh -p ${NORMALHOMEPORT} -D 9000 ${NORMALHOMEIP}

Now on your "away" machine you can open a browser, go to the Network Settings, and choose Socks Proxy to 127.0.0.1 port 9000. Now requests from this away browser will go through your home machine’s network. This allows you to put in something like 192.168.1.1 in the away browser outside your internal network and see the configuration settings as if you were on the home network. Note you can check this is working by something like an IP location service.

Some possible problems can be that the SSH server needs permissions for some things. Make sure your sshd_config where you log in to has these settings.

AllowTcpForwarding yes
PermitOpen any
TCPKeepAlive yes

You may find that the tunnel connection gets intermittent errors like this.

channel 99: open failed: connect failed: open failed

I looked in sudo tail /var/log/auth.log and found a bunch of connection attempts on IPv6 — so it makes sense to disable that (try setting IPv6 to "ignore" in sudo nmtui). There may be nothing that can really be done about this if you don’t want to humor IPv6. For example, I got calls to an IPv6 number resolving to 1e100.net (n.b. a googol) which is a vehicle for Google’s spycraft.

Fancy Proxy Forwarding

Sometimes you’re in a situation where you do not have access to your target but you have access to another host which has access to it. This happens, for example, on a big internal 172.19.* network which can see all local machines but has no direct contact with the outside world. Most things done on such a machine are done through web proxies, but if you need an SSH connection (or Rsync or scp, etc) here is a potential solution.

ssh -o ProxyCommand="ssh guser@G.example.com nc %h %p 2>/dev/null" tuser@T.example.com [command]

Here G is the gateway host that you have some kind of access too as guser. T is the target host you have access to as tuser. Of course the gateway needs a sensible implementation of netcat, a.k.a. nc. When this command is run it will ask for passwords twice (once for the gateway and again for the target). Keys can be used to smooth all this out.

It seems that the ProxyCommand above with netcat may not work now due to some changes with SSH. After an upgrade this configuration produced a mysterious ssh_exchange_identification: Connection closed by remote host.

Now I’m having better luck with this format.

~/.ssh/config

Host T
    User tuser
    Hostname T.example.gov
    ProxyCommand ssh -q -W %h:%p guser@G.example.edu

Rough Ideas For A General Proxy Server

To set up a general use SSH proxy server think about setting up a special user with an empty password and catching any log in from them with this.

Match User proxyuser
    ForceCommand nc -q0 dest port

The nc command could be replaced with a (Python) script that parsed the SSH_ORIGINAL_COMMAND environment variable looking for all the correct stuff and plugging it into a suitable netcat session. This would get around hard coding the destination host and port in the ForceCommand directive. It could also have a look at SSH_CLIENT or, better, SSH_CONNECTION to check on client socket info (coming from the approved address block?) Here’s how to test what that looks like.

ssh testhost.uni.edu echo '$SSH_CONNECTION'

This kind of wrapper would make an ideal place for logging who used the proxy. Also it can be used for blocking specific things if need be. Obviously configuring the firewall to be well-coordinated with such a scheme would be good too.

SSHFS

Sometimes you need to use a lot of files that are on another machine. Or you need to use a graphical molecule editor (or something like that) on a remote data file using a local program. For normal people the instinct is to use "remote desktop" and that is usually slightly clumsy if you’re lucky. Sending X windows over the wire is quite doable but the performance can be devastatingly bad. (There are better and worse ways to do this and if you really, really feel strongly that you want to do it see my nx notes.)

You could also rsync the files to your local machine and then put them back when you’re done. Although that is extremely reasonable, it can be inconvenient. This is where sshfs comes in very handy. With sshfs, you can mount a directory which you have SSH access to onto your local system. Then working with this apparently local file will actually affect the file remotely with SSH taking care of all the tunnel administration.

Start by getting it. On Debian style systems.

sudo apt-get install sshfs

Here’s how to use it. First, verify that you have access to the remote thing. Here I have a file Control_t001.MOV in my remote directory /data/xed/worms. Note that ws1 is defined in my .ssh/config file as described above. You could also use something like user@remote.example.com:2020 if needed. This shows using ordinary SSH to just see if I have access to this file.

:->[localbox][~]$ ssh ws1 ls /data/xed/worms
xed@199.99.9.110's password:
Control_t001.MOV

Next, create a mount point locally. I’m just doing this temporarily so it reasonably goes in /tmp.

:->[localbox][~]$ mkdir /tmp/myremotedir

Run the sshfs program.

:->[localbox][~]$ sshfs ws1:/data/xed/worms/ /tmp/myremotedir
xed@199.99.9.110's password:

Now I can see this file locally even though it really is still remote. Now programs that require a normal local path can use this file even though it is really elsewhere. Here I’m watching a pretty huge video over the connection and the performance is about as good as such a thing will get. Certainly better than pumping every pixel over the wire.

:->[localbox][~]$ ls /tmp/myremotedir/
Control_t001.MOV
:->[localbox][~]$ mplayer /tmp/myremotedir/Control_t001.MOV
MPlayer2 2.0-728-g2c378c7-4+b1 (C) 2000-2012 MPlayer Team
Playing /tmp/myremotedir/Control_t001.MOV.
Detected file format: QuickTime / MOV (libavformat)
[lavf] stream 0: video (mpeg4), -vid 0
...
Exiting... (Quit)

When you’re done working with those files, you can unmount them but sometimes you can’t do it with the umount command without sudo. But here’s the way to do it. Then the directory returns to being empty.

:->[localbox][~]$ umount /tmp/myremotedir/
umount: /tmp/myremotedir/: Permission denied
:-<[localbox][~]$ fusermount -u /tmp/myremotedir/
:->[localbox][~]$ ls /tmp/myremotedir/
:->[localbox][~]$

One problem that can occur is that you’re trying to install some software to or from the mounted system and when you go to do something like sudo make install you get some kind of annoying "Permission denied" message. This seems wrong because you’re sudo, effectively root. But the thinking here is that the SSH connection you made didn’t agree to you being some other person. Of course if you’re root, you can pretend to be that person so it’s all just security niceties. There is, however, an easy fix. Simply add the -o allow_other fuse option.

sudo sshfs -o allow_other xed@192.168.1.9:/home/xed /home/xed/mini

Note that as sudo you may need to be more explicit about where all the absolute paths are.

Serious Troubleshooting

First, the most critical thing to remember is to add -v.

ssh -v user@host

This will print a bunch of perhaps not too helpful messages. But it may also give hints about what is going wrong.

If you have access to and control of the server you can be more thorough with your debugging efforts.

The first thing to try is looking at logs.

tail -f /var/log/auth.log

Of course the location can be different depending on how your system is setup.

The most interesting trick I know is that if you stop the sshd service, the ssh session you are currently using still continues to function. Careful with this! This means you can do something like this.

service ssh stop

Or if not on a Debian style machine.

/etc/init.d/sshd stop

And your current session will continue to be operational. The reason to stop the ssh server is so you can start it up inside your active troubleshooting ssh session but in debug mode.

/usr/sbin/sshd -d

Then go back to your client and try whatever sad thing you’re trying to get working again. This time, there will be serious and helpful messages on the server host. Note that this only works for one connection attempt. Once you close the connection, the debug mode server quits too. When you are done with the debugging just do the natural start.

service ssh start

Or.

/etc/init.d/sshd start

Actually, a better idea is to start up a different SSH process that you can monitor.

/usr/sbin/sshd -D -d -p 666

Then while watching that, connect to it from the client with ssh -p 666 host.

Things To Check

  • Is your .ssh directory owned by someone else? whoami; id and ls -nl ~/.ssh should tell you that.

  • Is your home directory or your $HOME/.ssh directory writable by the group? This is common because often if your user is xed your group will be xed. The problem is that other people may be members of the xed group and SSH feels, rightly, that’s not secure enough.

  • Is your private key 600? It must not be writable or executable by anyone but the owner. Do a chmod 600 $HOME/.ssh/id_rsa to make sure. Also the public key is, in theory, public, but just to minimize confusion, you can make it 600 too.

  • Maybe use ssh-copy-id to copy your keys. It probably will give you fewer problems because it "…also changes the permissions of the remote users home, ~/.ssh, and ~/.ssh/authorized_keys to remove group writability (which would otherwise prevent you from logging in, if the remote sshd has StrictModes set in its configuration)".

  • Is the home directory containing the keys encrypted? Kids these days.

  • Check the sshd server logs. Often (RH) this can be found at /var/log/secure.

  • Check sshd configuration paramter PubkeyAuthentication yes and AuthorizedKeysFile .ssh/authorized_keys.

  • Is your key named something unusual? If it is ~/.ssh/specialserver_rsa you may need to explicitly try logging in with ssh -i ~/.ssh/specialserver_rsa user@host.

  • Do you need to add new keys to the agent? ssh-add?

  • SELinux. Total downer. :-( Check /etc/selinux/config.

  • Check /etc/hosts.allow and /hosts.deny if you get the cryptic unhelpful message: ssh_exchange_identification: Connection closed by remote host

  • Need to try the password method in case the keys are messed up? Use ssh -o PreferredAuthentications=password remotehost

  • "This account is currently not available"? Check getent passwd <user> and look for the shell.