Using The X Window System Remotely

The "X Window System" is the pedantic name for the graphical interface technology (generally) used by Linux (I will abbreviate this to "XWS") . It consists of two major pieces of software. First is a server which accepts requests to do things to a display that the server has control over. Second, is the low-level C library called Xlib. This is the library that programs use to interact with the X servers. Programs written with Xlib functions are called clients. These clients connect to a running X server and make requests for it to do the desired things.

A classic example of an X Window System client is xclock. This program uses Xlib functions to contact a server and, if successful, request that it draw an analog clock on a display that the server controls. It then periodically makes subsequent requests to the server to redraw the clock so that it shows the correct time.

This is why the applications you, the user, are running are generally called clients as far as the XWS. is concerned even if they provide server functionality with respect to other protocols (where they might rightly be called servers).

Note that the XWS does not specifically mandate or endorse a particular window manager. A window manager is a program that is just another client using Xlib which can contact and interact with a running X server. Its function, as its name suggests, is to manage the properties of other windows (X clients) that may be running on the display. The protocol provides for some special consideration for window managers, but in principle they are just X clients like any other program that uses Xlib. The nice thing about this is that it allows one to choose among many different window managers that are available. KDE and Gnome are the major ones that compete for mass appeal while specialty ones like ratpoison are for people with a clear idea about what they want to do with their displays.

The XWS uses Xlib to talk to an X server. The most common situation of this is when a user spawns a process that uses Xlib to make graphical requests to the X server which is running on the same system. In this simple case, the user, the program doing the user’s business, and the X server which moderates the display are all on the same physical machine.

But the designers of the XWS were under the heavy influence of new and exciting networking technologies. They figured that if you want to make a print server, you wouldn’t tie it to just one machine (the way the overwhelming majority of printers are actually setup). They immediately thought of a network with its arbitrary interface. Their main concern at the time was interoperability of displays and proprietary systems. But they also envisioned the possibility of producing graphical displays on systems that weren’t running the programs that generated what to plot.

More modern X systems seldom use the open network port (TCP 6000). This is for two reasons. First, to just have an X server port open invites all kinds of terrible security mischief where an attacker can basically display arbitrary things on a user’s display (like "Enter your password now:") and then capture a user’s interface events (like the key presses of a password being entered). The second reason that this port is usually no longer open is that modern X systems tend to use direct Unix sockets now to mediate client/server requests and connections. By skipping the overhead of a networking stack designed for arbitrary traffic, the fully local system can be more responsive.

However, this does not mean that X servers are incapable of receiving connections from remote clients. The modern way to use X remotely is to tunnel the traffic to the X server in a secure protocol.

Since an attacker who can eavesdrop on the network traffic of an X session basically can control the clients and server to the same extent that the legitimate user can, remote traffic must be protected. In any case where some arbitrary networking protocol must be wrapped in an additional layer of security, the first thing to think of is SSH (Secure SHell). SSH is normally for creating a secure log in terminal on a remote machine, but it’s modular design allows it to wrap any arbitrary connection with the same high-quality public key security.

Since protecting X traffic is a complex yet well-used feature, SSH has specific provisions for setting up the correct port forwarding needed for secure remote X sessions to work. There are two actually.

The first and classic way to have SSH handle X traffic is to log in with -X like this:

This should set up the necessary things that X clients on the remote host need to forward their traffic over the SSH connection to where there server is (generally where the user is sitting).

The problem with -X is that it uses the X Server "SECURITY" extension which is somewhat fussy in how it does business. It is designed to prevent a remote user from having trouble from malicious users on the same system.

There is another way which is less restrictive and does all the SSH forwarding magic but without worrying too much about the SECURITY extension. This is the -Y option.

Note that using this option implies that you trust pretty much everything the remote host is doing. If it is your box or one used only by your office, it should be fine. This is the recommended option for having the least trouble with connecting and using X clients remotely.

If I log in without -X or -Y and try to run a graphical program, I get Error: Can't open display:. Once I add that, SSH creates a .Xauthority file and also sets the DISPLAY environment variable which helps the X clients know where to connect. As far as the clients are concerned they’re connecting to a local X server but SSH is actually tunneling that traffic to the remote site (remote to the program, local to the human user).

When you think of all the dots on your monitor and all the key press and mouse events that constantly go on during a normal interactive session, you should start to get the feeling that there may be a lot of information involved in accomplishing a remote X session. There is no magic. If a program running on a remote machine changes a dot on the local display, that dot must be part of a message. Worse, that message is encrypted which adds some additional overhead.

Modern computers tend to have more processing power than they need and if you’re running remote X clients, this is almost certain to be true while your computer waits for display information to pass to the server. You can turn on compression of the SSH stream that improves bandwidth at the expense of requiring more CPU time on both computers. This usually fine for the remote user’s (X server’s) host since it is waiting for display information. However, if the client is running on a shared computational node, that could be problematic.

The way to turn on compression is with -C.

In general, it is better to use it when possible.

Let’s say that you’re working on a project with a graphical editor at work and when you go home you want to continue to work. You think it might be worth running the editor on the remote machine and continue as usual. Depending on the latency to your house this could drive you mad. But even if it doesn’t if the work machine has limited bandwidth which is shared with other users, then you could be negatively impacting their less profligate activities.

In a case like this, the classy solution is to use a version control system like CVS (Concurrent Versioning System) or Subversion or Git or Mercurial. This allows you to check out a copy of the project to your home and work on it and merge the changes back into the main repository at work when you’re ready.

Getting used to working like this has numerous advantages:

Maybe you need something more interactive. Maybe you’re graphing data that is being dumped into a set of files on the remote machine. If you can create a Network File System mount you can work on the remote files from far away with the only bandwidth incurred being the necessary remote data. NFS is a natural way to distribute data in work group, but it generally is not very secure and not advised for general purpose long distance remote exporting.

SSHFS may be useful if NFS seems attractive, but you face the following NFS related problems:

As long as you have the ability to log in with SSH (as you would if you were tunneling X) you can use SSHFS. Basically SSHFS is a virtual file system using Linux’s FUSE feature. This allows user space programs to interface with file system features. What this means is that you can mount a remote file system over SSH and see it locally on your machine as if it were a local file. The only network traffic is the file data, the file system management traffic, and the encryption overhead. To use SSHFS do something like the following:

Of course you’ll need to create a mount point directory (here it is remotehost_projects). But once this mount is established you can list that directory and all your remote files will appear. You can work on them with your local graphical software and they’ll be updated (securely) on the remote system. When you unmount the connection, the files will cease to be available on your local system but the remote ones will be as you left them.

Let’s say you really have a problem that really needs a remote machine to produce some graphics and you need those graphics on your local machine. This can be very frustrating if you have a slow connection or a clumsy X client running remotely. NXserver is a proprietary product of Nomachine which basically optimizes the XWS traffic for network portability. It is quite impressive and can really do a lot to help the difficult situations.

Keep in mind, however, it is proprietary and for profit and licensing is not always straight forward in my opinion. The free software world does not sit by long and let a cool thing remain in the hands of the plutocrats who can afford it. Keep an eye on FreeNX which aims to reproduce the impressive results of NXserver as free software.

Oh hey, check it out! Looks like we’ve been saved from NXannoyances by a more recent, less proprietary project.

A new better version of the NX protocol based system.

https://wiki.x2go.org/doku.php

Xpra is "screen for X" so says their website. It looks like it can tunnel X connections over SSH and reestablish sessions later.

I very, very rarely tunnel X over a network. I’m more likely to start an X server on a remote host to do something like watch a movie across the room. But there are times when I’ve found no other choice but to run X clients remotely:

No two remote computing needs are the same. If you’re really interested in how problematic your remote X situation may be, you have to actually check. Since X server events and Xlib calls tend to be user-based real time affairs they can be quite random. In fact, they are literally the way programs which need high entropy (PGP) get it.

To get a good feel for what the network is doing while you’re remotely tunnelling the graphical environment, you can set up a monitor. I used the following program:

Compile this with gcc -o bytecounter bytecounter.c and run the following:

This will watch over network device eth0 and forward copies of all bytes transmitted over that device to the bytecounter program that keeps a running display of what’s going on. This can be very illustrative.

For the first test I was editing text files. The numbers show the amount of network traffic generated. The first row shows the traffic required just to open the editor with no file. The small edit was basically taking a file containing cal 2012 and doing G:r!cal 2013 in vim or gvim and saving. The big edit involved scrolling through an 8MB file containing the works of Plato until I got to 10% of the document and then jumping to the end and then exiting.

As you can see simple things can probably sneak by with gvim but once the interaction gets busy, the overhead is considerable. NXclient is really pretty remarkable in its efficient use of bandwidth. Also note that the SSHFS method had to download enough blocks to display everything I wanted. This made it (using the editor locally) less efficient than just logging in and doing over the network. And on small edits, the file system overhead is also not competitive. For many other applications, however, sshfs is a great option. It also will have the least latency problems for real time operations.

Next I ran the ICM molecular modelling and analysis suite to get results for a busy graphical application. This didn’t have really any file data and was concentrating mostly on the X server traffic generated by the interface. For this test I started by just loading the GUI version of the program. Then I loaded a hepatitis E virus "3hag" protein and displayed it. Next I had ICM construct the 60 facet icosahedral capsid sphere of this virus (60 times more model).

Again the value of SSH compression is evident. And again the bandwidth requirement of the NXserver is quite impressive.

The focus of this exercise will be on getting synthetic steering wheel events into a driving game as if the wheel was manually used. This will be helpful for creating autonomous assists for driving games.

Here is how I have done this (over SSH no less). Start by finding which event number your wheel is using in the /dev/input kernel-generated pseudo tree. Here it is event5

After the evemu-record you can play with the wheel until pressing Ctrl-C. Then fire up the game and get it ready to notice the synthetic input and run the evemu-play. You should see the game behave as if the wheel is being turned like you turned it when recording.

The next step is to truly synthesize novel events programmatically.

This is just not going to be easy. You’ll need to recreate the business end of something like simplescreenrecorder or Open Broadcast System. I believe these use the MIT shared memory extensions to X. See man xshm.

Here’s a Cpp strategy for doing that which integrates with OpenCV.