Useful Python Reference

I used to do things like if cool: but that seems to have become uncool in Python3. The correct way apparently is to use is.

Python has an odd but sensible idiom where a program is checked to see if it was run as a real program. The idea is that if it was not, then perhaps nothing should really happen. This is useful for creating modules and other subcomponents of larger projects. This way you can define a library of functions that perform various functions and import them into any code you need them in and they will not run unless explicitly called. However if you run the module as a stand alone program, then you can have a function that tests the functions of interest. This helps in development.

The specific technique is to always put your default code which should be run as a standalone program at the end after a construction like this:

I can never remember the order of the object/function argument.

some_nice_separator_string .join( some_sequence )

some_joined_string .split( divide_at_string )

I am getting better about this and what finally helped it sink in is that both split and join are string functions. Even though join really smells like a function concerned with sequences, it is a string function.

Another handy note is that split() with no arguments will split on whitespace nicely without giving you silly things like multiple consecutive space list entries (e.g. for something like a many spaces b).

The any() function seems totally superfluous to me, but there it is. Feed it an iterable and it will return true if any of the items are true.

It is possible to name slice objects, perhaps to improve clarity.

Or obfuscation!

General String Format:

Example:

Note that to get plus signs always, use + at the beginning. To get leading zeros, just put leading zeros. Note however, that if you want a number like this "+001.300" you need something like %+08.3. The 8 is needed because that is the full length you want reserved. The 3 is how to deal with the fraction and everything else is just decoration.

Of course Python3 had to go and reinvent how this is done. It is more complex in some ways, but simpler in other ways. For example, it is no longer strictly necessary to specify in the template what kinds of types will be showing up; it can just deduce them.

You can scramble the order.

Need full times with leading zeros and milliseconds?

Note that the ints really need to be ints!

So that’s all fine and annoyingly different but not obviously way better. But hang on, it’s all about to sink in why this new system is much better. The important trick is that you can use the format() function but without writing it out explicitly. Basically you can specify a formatted "f-string" where the formatting operations are implied. The braces are not just holding places but rather containing the data itself. Take a look.

This is most useful when used to print variables.

This nice webpage has a very nice catalog of most of the cool things this format system can do helpfully contrasted with the old way of doing things.

Here are some more obscure attributes:

For example:

Here is a comparative example of a simple substitution using core Python functions and regular expressions.

Here’s a string containing the characters "JUNK" followed by 4 unknown characters all of which must be removed.

There are two major 2 ways.

#1: Use find() or index() to figure out where in the string the thing is and then use slices:

#2: Use regular expressions. Just match with "JUNK….".

Despite being a regular expression pro, in Python, I tend to minimize it and, unlike other environments, that’s easy to do (as shown here).

For more information, check the official gory details.

List comprehensions are a nice way to apply some action to a list in such a way that a new list is generated. The syntax is a bit odd at first, but it’s actually pretty reasonable and compact. Note that the functionality is comparable to the map function.

Conditional filtering works too, including a form with else.

Note these are subtly different. The first is a filter. It throws out any that do not match the if condition. This example divides by two but never wants to see a value split in half and rounds up in the case of odd numbers.

There the if x%2 else x/2 acts as if it wraps the myfn() clause. Note that in this construction you must have an else; if you don’t really want a fancy myelsefn(), just use else x.

This similar example is simply filtering out the source list based on how each list item interacts with the mycondition.

My general rule for the whole list comprehension syntax is that I use it only where it reads to humans quite naturally. This occurs surprisingly often but generally precludes overly complex logic.

That said, sometimes you want to be aware of performance benefits, which can be substantial. For example, this program took only 0.100s.

Compare with this equivalent code which took 0.171 seconds.

Note that Y+=[x] takes even longer (0.200s) than the append(x).

The list comprehension can be conditional which basically operates like a filter function.

To be clear, the official Python description of filter says that these are equivalent.

And so are these.

Note that in Python 3+ the filter command produces an iterator instead of a list. This may be helpful for large data sets that need to be processed. Otherwise, you should probably use list comprehensions.

For example, this kind of thing is not cool in Python 3.

But this is and it’s pretty obviously an improvement.

While we’re covering wacky functions that like the lambda construction, here’s a use of the reduce function. I really don’t think this function is very useful, but this was my best attempt to do something more exotic with it than the normal adding a bunch of things.

I think this is as good as it gets judging by this.

If you never understood reduce’s utility you’re in luck! Python 3 removed it. You can still use the reduce function found in functools. But really, probably best to consider it dead to Python.

Generators are very much like list comprehensions except they don’t synthesize the entire list into memory at their location. Instead they produce a generator object which can be iterated, generally with a next() function. Each time it is iterated, the next item in the sequence is generated at that time until the specified objects are exhausted.

The generator syntax is a shorthand for a more verbose style involving the yield keyword. The yield keyword returns an argument just like return (unlike return the argument is mandatory). Then the function’s state is preserved and the next call to it resumes where it left off. This can be reset by a return statement or just a natural end to the function.

The following example illustrates the usage with a function that provides unique incrementing ID numbers.

This produces:

As of Python 2.7 and all of Python 3, there are dict comprehensions which are very, very similar to Python’s list comprehension. The syntax looks like this.

Here’s an example.

I can’t say that this is clearer and better than this equivalent syntax.

Other attributes of set objects:

Python has a hasattr() function.

This guy has some good reasons to think that it’s better to except on AttributeError than do checks this way. However this can mask unknown unknown attribute errors.

He also points out that this works and is a bit more efficient too.

If there is no color attribute for TheObject, the None object is returned.

Python also has setattr() and delattr() if that’s what you need.

Here’s a decorator example that times a function:

This program outputs something like this:

Note that I’m trying for minimal dependency and maximal comprehensibility but there are plenty of official ways to do this like the timeit module.

Sometimes you’re expecting something to happen and you’re not sure how long it will take. You do know that if it goes beyond a certain threshold, you would rather just abort. An example of this is if you are scanning for fast internet mirrors from which to download something. In this case, by definition, there would exist slow mirrors and they may be so slow that they bog down the operation quite a lot. With the following decorator, you can give each mirror a certain amount of time to attempt its operation before cutting your loses and pulling the plug.

Produces:

I think that a situation like this:

…is the same as this:

I could be wrong though. Here’s a working example of a decorator that can be adjusted with an argument.

This program produces the following output.

I was amazed to discover that this is valid code.

That example is some Blender stuff and the @ here is an operator doing some matrix multiplication. This was new in 3.5. Works with numpy too.

Error handling in Python is quite powerful, but it can be a bit complex too. In Python an "exception" is an event that accompanies an error situation and it is valuable to realize that all errors in Python use the exception mechanism. Because of this, it’s pretty useful to know how to deal with them. For example, python doesn’t have a simple "exit" keyword and the ultimate way programs stop is when this effectively happens:

Although understanding exceptions is important, I have a bit of philosophical unease with the idea of planning for things you didn’t plan for. My thinking is that if you expect an exceptional situation, you should take precautions to make it not exceptional. In the Python world this seems to be divided into the "Easier to Ask for Forgiveness than Permission" (EAFP) and "Look Before You Leap" (LBYL) factions.

The classic case is a division by zero error. What is the functional difference between letting a division operator raise a very specific ZeroDivisionError and just checking to see if the denominator is zero before proceeding? I think that sometimes you want such explicit control and sometimes you can tolerate a certain amount of ambiguity. For example, if you’re going to do 100 divisions and the whole set are invalid if any one has a zero denominator, then the code might be easier to write and later understand if you use exception facilities. However, if you know that a certain attribute might be missing from an object, it is very reasonable to do a if hasattr(object,propstr): rather than try: ... except: AttributeError:. In theory these are very similar approaches as hasattr basically calls getattr and catches exceptions. However, if using exceptions directly, looking at the code later will tell you nothing about why you thought to catch an error there, i.e. something about object and propstr. For all your future self knows, you were just covering the "unknown unknowns".

I like to explicitly check for all the things I can think of which will go wrong (LBYL) and then use exceptions to stagger away still breathing if something truly unforeseeable happens. To me it’s like working on a roof while wearing a safety harness - falling off the roof is still to be avoided and jumping off the roof seems to be seriously full of the wrong attitude.

There are other cases, however, where exceptions are strategically preferred. One example is explained on the OS module documentation. It points out that checking to see if a file is readable and then reading it is not as robust as just trying to open it and then catching an exception when it doesn’t work. The thinking is that in the former case, an attacker could devise a way to change the state of the thing being checked between the check and the action.

Code to be monitored for an exception is "tried" with the try keyword. An "exception" is "raised" (not "thrown" as in C++ and Java). An exception is, uh, excepted with except and not "caught" with catch.

The basic syntax looks like:

In the above code the Exception exception catches most sensible things (but not stuff that you need to stop the program). I find I use this for overlooking a few anomalous corrupt input records while processing large quantities of mostly good ones.

Also don’t forget about assert which can raise an AssertionError. This kind of thing can be useful.

That throws an exception and prints that message if the script is run on the wrong kind of platform. Basically, consider assert when you want to raise an exception conditionally.

What exceptions are there to be raised? Here’s an abridged diagram of the exception hierarchy. Note that if you except an EnvironmentError exception then it will catch IOError and OSError since those are subclasses, or types, of that exception.

Here is the full list of built in exceptions and what they mean.

You can find out about exceptions with help:

Or this:

Looks like there has been a change in the syntax used to manage exception information. Formerly it was something like:

And now it is something like:

What this means is that in your except clause you can use the exception instance that is generated with the raise (perhaps as part of a system error). This exception instance contains some handy stuff relating to the error (usually). It seems that the various built in exceptions have a variety of attributes. For example, an IOError will have a filename attribute that you can access. Here is an example of the basic attribute system showing the generic data stored by the exception object and how to find out what exactly the exception object can tell you.

Produces:

This catches any general exception. Presumably to catch KeyboardInterrupt and even higher level exceptions that you’re not expecting, try BaseException.

If none of the built in exception classes seem appropriate or if they lack the necessary attributes, you can create your own exception classes which more specifically do what you need. You can inherit from any exception, but it’s normal to use Exception as an uncluttered base class. Here’s an example of what methods and structure such a user defined exception class entails.

This example program produces simply Idiot programmer alert!.

The common way for paths to be assembled so that they work on that other kind of OS is to use os.path.join(). But there seems to be a newer, better way with another built-in module pathlib.

Path(__file__) gives you the filepath of the file you’re in, so you can do things like this to orient yourself in a tree where things about the workspace or environment might change but you at least know where you should be relative to other things in the hierarchy.

One of the biggest gotchas though is that functions asking for a filepath to do something with it will only sometimes accept a Path object as a viable representation of the path, sometimes you do end up needing to convert it back to a string. But when passing paths around one’s own consistent code, it just makes a lot of little things much cleaner and easier to work with if they’re Paths.

Use the os.access() function to check if the file you are interested in is in the condition you expect.

You can also use os.W_OK and os.X_OK to test for writable and executable.

Also consider os.path for checking on directories.

To read an entire file into a string:

To read each line of a file use something like:

Note that this comes with newlines from the file and from the print. Use sys.stdout.write(l) to avoid this problem.

Other streams besides sys.stdout are sys.stdin and sys.stderr.

Sometimes you need very specific things done with your file writes. If, for example, you need to aggressively prevent buffering, use the third argument of 0 in the open function. But this only works with binary (e.g. wb) modes. When using binary modes, you have to make sure any string types are converted — or encoded — into something at a lower level. Here’s an example that finally did what I wanted it to do.

Note the with/as syntax is described below.

It looks like since version 2.5, the fanciest (and best?) way to open files and read through them is to use the new with and as reserved words.

That also shows how to use the polite os.path.join to make path amalgamations; note the necessary leading slash.

This creates a list of lines in a file.

Need to deal with each line on its own individually? Here’s an example that counts the lines in a file.

The advantage here, apparently, is that the file gets neatly closed even if it’s rudely interrupted and never makes it to your file closing statement. Or something like that.

A better way to think of the with as syntax is that it seems to set up a "context". An object which has an __enter__() and __exit__() method can be used with with as such that the enter method gets called upon entering the block and the exit method gets called, unsurprisingly, upon exit. This is why it’s such a reasonable way to handle file opening because things can be done with the file and the exit function makes sure that, whatever weirdness transpires, the file will get closed.

Another example of when this would be appropriate would be some kind of routine that output some SVG. You may want to adjust the view parameters with a <g transform=...> block. You might start with the opening tag and then do a bunch of stuff and then print a </g> tag from the exit method of an object. This will allow for nested objects and the opening and closing tags will always exist and be correct.

It might be smart to rewrite my HTML tagger with this style.

Another example is one of a transaction "lock". If you need to do something like update a database and it must be locked to prevent access from other actors, you can have the lock be in the __enter__() method and the release be in the __exit__() method. This way, even if bad things happen, the lock will get released properly.

I think the core of the syntax is that this code (with as VAR optional):

Is equivalent to:

The __exit__() function needs to take 4 values, self, type, value, and traceback. I think these are the arguments of a raise statement.

Also newer Pythons (2.7 and 3+) support stuff like this.

Complete mind boggling details can be found here.

Here is a sample where a temporary file is created and then the program turns control over to Vim for the user to compose something and when the user quits, the Python program has all of the input. This is what you would do if you wanted to, for example, recreate the functionality of a mail client like Mutt.

Although normally not something to worry about, sometimes it’s important to remember that Python tends to politely buffer output as a general rule. You can have unbuffered output by invoking Python with the -u run time option. I’ve found this to be important when using tee, named pipes, and other fancy stream situations.

Also note that you can use something like print(x,flush=True). Sometimes I do print('.',end='',flush=True) to print a line of status dots and I don’t want the status to be incorrect for what is actually going on.

Python can deal with compressed files just fine. There are the modules gzip and bz2 which are very similar but not identical. Here’s how to read a gzip compressed file.

Then work with f as a normal file handle. This example is illustrative for writing compressed files:

Here’s an example of how to use bz2. This little program is a (rough) wc program for text files with bzip2 compression.

The fileinput really simplifies getting things from files or standard input.

This program produces this output.

Here’s the recommended way for executing shell commands as of 2013.

Start with getting the lines of output from the simplest kind of command to fill a Python list.

The reason for all that guff is that this check_output command produces a bytes object. Another way to untangle a byte stream object is to decode it.

Here are some more examples.

Note the stdout=subprocess.PIPE value to the Popen constructor. This is required to keep the function from immediately dumping the results on the spot. The function does run immediately when the constructor runs. So if you do a date function, for example, and there’s a lag between the constructor and the n.stdout.read() the time will reflect the initial operation.

Proper Python documentation suggests that it’s good to use the supplied convenience functions when possible. These are call, check_call, and check_output. Here’s how the latter work:

I started having trouble reading lines of standard output from a process with Python 3. Here’s a way that worked.

Here’s an example of an outside command that gets run with some data the Python program knows about being piped to the command and the standard output being captured back into the program.

Note that the second item (pro.communicate()[1]) is the standard error.

To access environment variables from python use this technique:

Also, os.urandom(n) returns n random bytes from /dev/urandom or some other OS specific collection of high quality entropy. This is slower and depends on actual random events having occurred on the system.

Do you have some trivial little thing that needs some very rough unimportant randomness but you don’t want to import random for such a trifle? Here’s a way using Python’s core abilities.

Also a[a<2] gives true values where a < 2.

The traditional idea with arrays is that you reserve the memory you need and that’s that. But sometimes you need to build an array up from smaller parts and it’s more convenient to increase its size than replace parts of it (e.g. you may not know the final size). This happened to me where I needed to read in a sequence of images and store the whole collection as an array (holding each image) of an array (holding each image’s row) of an array (holding each row’s column) of an array (holding each pixel’s RGB). Assume a collection of three 2x2 grayscale images.

Now suppose you have a new image that you want to add.

You might think that having something like [*] would be what you need to add to something like [ [*] [*] [*] [*] ] but in fact, you need something like [ [*] ]. So here’s what works.

There are many fancy ways of sorting things. Often you have a list of lists and you want to sort by some item in the list. Here’s a list of tuples representing (model_number,score) which need to be sorted so that the top 5 scoring models are displayed.

Strangely I haven’t found a cleaner way to do this. Here’s another more complicated example of a two level sort.

Here the result ms1 is sorted by date of ascent (earliest first) and then, if that is the same, by mountain height (highest first). The results look like this:

If you need to "graph" some data, Python can help. The main technique is to use matplotlib. Although a bit overly fancy and likely to spontaneously burst into a GUI, it is powerful and, in some modes, easy:

For more details, check out my complete notes on matplotlib.

Also, check out Pychart.

See pydot which is the Python interface to the mighty Graphviz package.

The original way to parse options draws stylistic inspiration from the C version. Many languages (Bash, Perl) have such a thing and if you’re used to one of them, the Python version won’t be too complicated.

There is a module called optparse which has been deprecated since Python version 2.7. In its place is the newer and pretty awesome argparse module. Official documentation is here. If you’re using an ancient system, check to see if it’s available but these days (e.g. Python 3) it always is.

These are the main steps to using this module.

When defining a parser object, you can use the following optional parameters.

When adding arguments you want the parser to look out for, start with either a name of the positional argument you want or a list of option strings. Then you can add some of these optional parameters to get the exact behavior you want.

Here’s how that can be used. Note that the usage program name shows up as`xyz` and not argtest.py which is how it was really run; this is thanks to the prog= parameter when defining the parser.

Here’s another example showing how to use the parse object as a global variable neatly containing all the user’s preferences. All this stuff is appropriate for a global variable since the sys.argv input itself is global to any executed process.

This technique also allows the user to read in N files, or standard input in the case of none, while keeping the ability to parse complex options.

Here’s another example of how to use it. This example should be pretty much functionally equivalent to the getopt example above.

Argparse is powerful and can do weird things too. Here’s a stranger case where I needed two classes of arguments with unknown quantities. One or more files needs to be supplied for each type of file.

Run with something like this.

This produces something like this for args.V and args.N respectively.

Sometimes your program is not misbehaving but running just fine as far as arguments go but some processing in your code suggests that the user is an idiot who needs to read the instructions. How can you immediately generate the automatically generated usage message?

And here’s another tip when you’re writing instructive descriptions and argparse overly helpfully removes formatting. Here’s how to cure that.

One astonishing thing I discovered is that despite being seemingly competent at parsing any kind of options a sane person could want, there is a glaring deficiency. Let’s say we wanted to emulate the options parsing behavior of the most command line program of command line programs: unix’s ls. Well, this command has options (many) that take an option argument. Not only that, but they take an optional option argument. For example, this is a reasonable command: ls --color *.txt. And so is this ls --color=never *.txt. How can argparse set up this behavior. As far as I can tell, for arbitrary values, the answer is it can not! I was amazed by this. As far as I can tell, you must have the default option (without the modifying option argument) end in an = to get the default value. So if you rewrote ls using argparse you would have to say something like ls --color= *.txt. I do not think there is a way around this.

Here is a demonstration of how to set that up. This imagines setting up options for a program that creates lines of output where you might want an integrated behavior of the unix head command. If you just say --head= you get 10, but if you say --head=100 you get 100. If you leave it out completely you get all the lines. This is just the option setup for such a thing.

Here’s what running looks like.

This inability to match bog standard GNU getopt_long behavior found in commands like ls, grep, tar, etc. left me somewhat disillusioned with Python’s argument processing abilities.

I did discover that you can probably match the ls --color option behavior specifically with something like the following:

However that only worked with a finite set of enumerated possibilities. How one would go about using it with something like a --resize[=VAL] option with an int still a mystery to me.

I often use wget — Apple people like curl. Python unsurprisingly has a perfectly good way to simple web client downloads.

One of the best reasons to use Python for web projects is the cgitb module. This stands for CGI TraceBack and is a diagnostic tool to help you understand what might be going wrong with your Python script run over the web. The nice thing is that this is super easy to use and super useful when activated. Here’s an example showing how to use it (simply import and enable it) and some faulty code which takes advantage of it:

Putting this in a cgi-bin directory and typing its URL in a browser produces this very cool diagnostic (which in this case correctly notices that the function idontexist does not exist):

If you’re looking at the output without HTML rendering, you’ll also notice that this is tacked on to the previous HTML message for maximum intelligent utility:

Note that this works on any Python program run over the web, not just ones that use CGI per se. It is advisable to comment out the enable line when your program is served live to the public to avoid any leaking of sensitive information such as how your code works. But other than that, use this early and often.

Before generating any HTML, every web program will most likely need to send back the HTTP content type. It’s often useful to make a global variable of it.

I personally hate Python code that is filled with HTML. HTML should be in HTML documents and Python should be programming. But sometimes they mix annoyingly. This throws off syntax highlighting and the wholesome goodness of Python’s formatting and style. Here is a technique I use in my Python code to completely obviate the need for any HTML.

This function can be imported into programs requiring the generation of HTML. It allows you to not put HTML in python code. It’s easier to type, easier to think about, and it doesn’t break syntax highlighting. When run as a standalone program, it prints a complete HTML document as a demonstration.

The technique above is useful for generating web-based output. To process web sourced input, the cgi module is helpful. This module is very helpful but it is not magical. I think the most helpful way to illustrate what it does is to not use it and see what that looks like.

Assuming the helpers such as the tag() function as defined as above are in place, the following code is very illustrative:

When run you get a list of environment variables that your CGI program knows about. This sample list may or may not include some of the following you would see: