The Bash Shell

September 2014 a very serious undocumented feature in Bash was published. Here’s how to check for it.

If this returns the word "vulnerable" then it is. If not, then no. Patches cleanly fix this but it’s a big problem for CGI enabled web-servers and DHCP servers.

I often forget useful Bash keyboard tricks because I have other ways of doing things. Here are some worth remembering.

This has a lot of sensible tips about organizing and running more complex Bash programs.

In the /etc directory are some default bash config files that affect bash’s behavior unless countermanded by user set config files.

Other people have also tried to explain this.

One of the extremely important features of the shell which elevates command line work from a nightmare to a joy is command history. If you’ve typed something complicated, you don’t want to type it again. Using the up arrow to revisit previous commands is the normal thing to do. Also typing history <N> will show you what is in the history for the last N lines. Ctrl-R will allow you to start typing something that will be matched with a command that is in history; keep pressing Ctrl-R until you get the matching command you want.

Note that there’s a decent man page for the GNU history C library that provides some interesting insight, man 3 history. And Bash’s built in help history.

Here’s an example - this changes a big long list of x001.gif, x002.gif,etc to this b001.gif, b002.gif, b003.gif etc.

Another example - this converts a series of tif images with the names tb01.tif, tb02.tif,tbwhatever.tif to tb01.gif, tb02.gif, etc. It also changes the images to 15 color grayscale.

I don’t know how long bash has had this, but it wasn’t a useful feature when I learned Bash a long time ago. But these days it is quite handy and elevates Bash to the status of quite a sensible general purpose language. Basically in the distant Unix past one had to use bc and dc and expr to get any kind of arithmetic done, even trivial things like incrementing a variable involved spawning a new shell and casting text strings every time. Now you can pretty much use the (( expression )) syntax and the expression will be evaluated in a pretty sensible way. If you need the expression to stand by itself as a process which produces its own exit code (like the old bc but I’m not sure a new process is actually spawned), then you can do something like this:

If you need to produce the result for further consideration, you need to expand the expression like this:

Note that the performance of these operations can be suboptimal. Often spawning new shells for sub commands beats the arithmetic expansion.

Oh and check out this:

Normally in Bash, it’s good to help the tokenizer by being explicit about where things are, but in this case, spaces just waste time.

Do the action 10 times concurrently:

Bash has perfectly good support for Arrays. Search for /^ *Arrays in the man page for complete information.

Bash arrays start indexing at 0 (like C arrays).

This will print out a random file from the current directory:

Here is an example program that copies a large collection of files in a source directory to many different hosts (say in a cluster for processing) such that each host will receive the same number of files (to the extent possible):

Here’s an example of using associative arrays. The trick is that they need to be declared with the -A option or else they’ll just act kind of funny if you’re unlucky enough to not get an explicit error (which happens).

Note that to access these variables, I believe you need the curly braces style of referring to bash variables. So echo $A[mykey] will not work but echo ${A[mykey]} will. Also note that using keys with periods in them was not working for me. There may be a way to do it though.

This example will download each zip file and save it based on the key ("data_2014q1_xml.zip", etc). This technique can be handy when sorting out a bunch of badly named things as in the example.

Interestingly enough, ${!D[@]} will make an iterable list of the keys even if the keys have spaces. In other words the for won’t iterate on each word of the keys.

Here is a method to massively update lots of files that all contain some bad thing and replace it with some good thing. There might be a smoother way to use the sed command, but redirecting to the same file causes the file to simply disappear. Using a tempfile works just fine.

Here, all files that match the project??.type pattern (project01.type, project1a.type, etc) will be copied into the temporary file temp.temp and then they’ll be pulled back out into their real name line by line by sed which will also make substitutions as requested. In this case, it will change all references of "bad" to "good". Don’t forget to erase the residual temp.temp file when you’re done.

Another note - if the text is long, you need quotes around it so bash doesn’t get wise with it: sed s/"This is very bad."/Nowitsgood/g file

If the text has quotes in it:

Here’s another example-

This only does the substitution after it finds 3 dashs and only until it finds 3 more dashes.

This useful program shows how to use proper mktemp temporary files. It also shows how to have commands run as root using sudo if the user running the script is not root.

Bash functions are quite useful. They are like bash aliases with the added ability to take parameters. Here’s an example of function use that I have in my ~/.bashrc:

Another syntax that can be used is this.

This is what the set command reports and is probably preferable even though it seems kind of "fake C" to me.

The automatic variable ${FUNCNAME} is available in the body of a function. The parameters supplied when the function was called can be accessed with $@ with $1 being the first one, etc.

These both produce the same result.

This will do the expected thing with stdout, but stderr will head off to the trash.

To get all of the garbage a command spits out to go to a file:

Many more fancy things can be done with pipes and redirection. It is possible to use the test command (which is related to the [ … ] syntax since [ is an alias for test) to check for where a file descriptor is from. That’s confusing but this makes it clear:

The exec command replaces the current process with a subshell containing the specified process as in exec takeoverthisjob -a args. An interesting effect and demonstration of the exec command is seen in the following example.

Here is a nice example of how to use named pipes to implement a complete private chat system. Imagine your user name is "primary" and your friend’s is "guest".

This is the named pipe which will contain your messages to the guest. Note that it goes in the guest’s home directory which you’ll need access to. These could go anywhere (somewhere in /tmp might be good).

Here’s the named pipe which you will look at to see the guest’s chat. After you specify where these will go, you need to create these named pipes once with this command:

Next put these macros into both .bashrc files.

Now to chat both parties simply log in (SSH makes the whole thing pretty secure) and each types listen to start listening for the other’s text. And then they type chat to start sending. Now both parties can type and read the other’s typing.

Bash is so foolishly badass that it takes the place of pretty powerful network tools. It provides support for special device files that do things with arbitrary network sockets. The format for these is:

Note that these aren’t really files. They are very strange file-like interfaces. They only really work with redirection!

As an example, don’t have nmap lying around? This will cleverly scan for open ports using nothing but Bash.

Want to just check if an SSH server is up?

Wonder which nodes on your cluster are accepting SSH connections? I found this was more helpful than nmap that whined about insufficient permissions with some simple usages. This one works very quickly but note that the results may not be in order.

Don’t have wget installed? This can work as a complete web client.

In fact Bash makes reverse shells easy without using netcat on the target system. Run something like this on the remote system and connect to it with netcat or whatever you want locally.

To get an idea of what this can do look at the netcat example in my sound notes.

Often you’ll be tempted to use xargs but you will be frightened by very messy syntax. One way to get around such things is to use a bash loop to receive the output. Here’s how it would work:

This will produce a list of check sums for all files in the current directory (and below). This command can be useful to create an inventory of what’s in a directory tree and compare with a directory tree elsewhere to see what has changed.

Here’s example showing how to avoid xargs:

This takes a big list of things and a smaller list of things you wish were not in the big list and it looks through each item in the excludes list and removes it in place from the big list (if you’re using GNU sed, otherwise make your own temp file).

Careful when doing this not to find yourself in a subshell which has different variable scopes. For example, something like this could give unintended results.

I’m sure there are even better ways to handle this.

Actually there are different ways, but this technique of piping to while read could be worse. Here’s an example of it being superior.

Here the directory contains a file that includes a space. The for loop has trouble with this treating the space-separated parts of the filename c d as two items.

If you ever need to log something anywhere or do any kind of error monitoring or debugging, check out man logger. This is very handy and it’s universally available as an old school program yet I had never heard of it.

Here’s a clever way to turn off logging selectively (credit to DaveTaylorOnline.com):

Here’s a nice function that dresses up errors and puts them in stderr if the output is to a terminal.

Trapping a signal is kind of exotic but when it’s a good idea, it’s a good idea. It can be good for things like cleaning up a mess before letting a user cancel an operation with Ctrl-C. For example (more from Dave Taylor):

It can also be used as a timeout enforcer (though there are better ways). Basically spawn another background process that just waits the maximum allowed time and then raises a STOP signal. Then have the main thing that’s being limited trap for that. Here’s how:

Note that this little script will have problems if the main script exits cleanly and its process ID is taken by something else, something important that should not get a SIGALRM. Just keep that in mind.

To turn off a trap for the INT signal (as used in the previous example):

Actually this might be better as it should reset to the way things were instead of clearing the traps.

You can check what’s active with just trap by itself.

The special DEBUG signal is always received when a command is entered. This can be useful for tricks like keeping your own log of commands that you run or perhaps a common history from multiple terminals.

Here’s another example. This little routine calculates (via exit code) whether a number is a power of 2 or not. Apparently sometimes big search engine companies care about such things.