XKCD dependency

Here are some notes for using make for software development and other more interesting purposes.

What Is Make?

The program make is a utility to manage complex development tasks where recent changes to one part of the system create potentially complex dependency requirements. The normal context of this is in software development where there are a large number of programs or files containing program source code. Many of these files will include other ones. Since compiling the source code into computer executables can be a long tedious process, it is best to only compile exactly what is necessary in light of the changes that were made.

It’s not just classic software development that benefits from make. Many data processing projects also benefit from the organization and structure it facilitates. In this excellent article on the value of make, Mike Bostock points out correctly that: "Makefiles are machine-readable documentation that make your workflow reproducible." That is quite accurate and if you ever have workflow that needs to be reproduced, make might help in more ways that is initially obvious.

Note that make is not magic and it can not know what changes really impact a system. All it can do is know that a certain file is dependent on certain defined other files and if one of those files changes, then it’s best to change the one that depends on it. This is done even if the change was trivial (say changing a comment). All make can do is see that if A depends on B, and B is newer than A, then A needs to be updated to reflect the new B. Because make uses file timestamps to infer what is recent, it is critical that your systems have the correct time set.

Note
 Because make was written by Richard Stallman himself, it has a weak man page. Instead of struggling with the irritating info page, it might just be easier to look at the online version of the official documentation if you need help.

Makefile

The way make wants to be used is by simply typing make and things work great. This is supposed to be a component in a streamlined professional software development cycle. Making the usage complex would defeat the purpose. The way that make becomes almost automagical is by having a very elaborate and, well, sometimes tricky configuration file. You can specify this file with the -f option. However, normally there are some default locations one can use which make will look for automatically. The default make files are GNUmakefile, makefile, and Makefile in that order. Best to leave GNUmakefile for weird cases where the makefile depends on the GNU version. That would be very weird. In practice, Makefile is the file that is used the overwhelming majority of the time.

Rules

Makefiles are composed primarily of "rules" which have the following form:

target : prerequisite1 prerequisite2
    recipe

The target is the file (or action) to bring up to date. The prerequisites are all of the things that are required for target. This means that if a prerequisite is not up to date, it must first be made current before the target can be properly satisfied.

Note
 Perhaps the most irritating thing about make is that it uses the tab character (ctrl-i, in some contexts). The recipe is specified by being preceded by exactly one tab as the first character of the line. If you don’t like this (and I don’t) you can change the tab character to any other using .RECIPEPREFIX := :. The : is recommended since it won’t clash with other rule names (it’s already a separator).

Here’s a very simple example using the : as the recipe initiator character:

.RECIPEPREFIX:=:

all: aprogram

aprogram: aprogram.c
:gcc -o aprogram aprogram.c

Variables

There are two "flavors" of variables. There are recursively evaluated variables and simple ones. Recursive ones look and work like this:

Y = xed$(X)
X = .ch
all:;echo $(Y)

This produces xed.ch. This is because Y is set with a value of a value, yet to be finally determined.

Simple variables look and work like this:

X := .ch
Y := xed$(X)
X := something else

Here Y is set equal to xed.ch since it gets evaluated right away and the fact that it was defined with a variable is forgotten. The variable X can go on to be used elsewhere without confusion. This allows for things like:

FLAGS := $(FLAGS) --additional_flag

With recursive variables, this would create an infinite loop (which make would detect and stop as an error). It’s best to use immediate evaluation assignments (:=) unless you know you’ll need deferred evaluation.

The append operation shown above is for illustration. Make actually has a special operator for appending:

FLAGS += --additional_flag

This actually works for recursive flavored variables by (possibly) delaying evaluation. Just be aware.

Another way to define variables is conditional assignment like this:

DEFAULTS ?= normal

This sets the value of DEFAULTS to normal if DEFAULTS is undefined (a technical condition variables can be in). Hard to say for sure but it looks like this syntax is recursive in behavior.

Another common variable setting technique is pattern substitution. This basically duplicates functionality of the patsubst function. It has the form $(var:a=b) and looks like this:

foo := a.o b.o c.o
bar := $(foo:%.o=%.c)
sets "bar" to "a.c b.c c.c"

Or here’s a way to do rsync excludes:

RSYNCEXCL := .*.swp Makefile $(XEDCONF)
RSYNCOPTS := $(patsubst %,--exclude="%",$(RSYNCEXCL)) -aP

Make variables can be set with command arguments when running make. This happens when an argument to make has an = in it (or := also). These specified variables can be changed as needed with the override directive during assignment. This seems a little backwards to me since I would use the command argument variables to override the normal makefile defaults.

In addition to the override directive, there is the undefine directive that can do what it implies. There’s also a define which is helpful for multiline assignment, but this gets messy.

Note
 It’s important to realize that: "Every environment variable that make sees when it starts up is transformed into a make variable with the same name and value." This can be helpful or problematic. It is sometimes the preferred method for modifying compile behavior for software written by other people. Be aware!

Target specific variables are possible. Also variables specific to targets that match a pattern. It’s like a limited scope. This kind of voodoo can be used to suppress variable inheritance among prerequisites. Try to avoid this kind of thing if you value your sanity.

Special Syntax

|

In a prerequisite clause this separates the normal prerequisites from the "order-only" ones. These don’t worry about updates, just existence.

*

In target and prerequisite clauses this is a wildcard. In recipes the shell does the expansion. Elsewhere (e.g. variable assignment) use the $(wildcard *.txt) function. Note that if you use a wildcard in variable assignment statements it will be a literal * however, if that variable is used in a target or prerequisite, it will be expanded then. Crazy, right?! Also, you need the wildcard function if you’re using it in arguments of another function (say patsubst).

?

Expansion just like *. Also [...] work too.

@

When a recipe starts with this, echoing is suppressed.

+

Marks a recipe line as recursive meaning it will get executed in a shell regardless of other mechanisms to generally suppress this (such as -t).

-

When a recipe starts with this, unsuccessful exit codes are ignored.

Special Variables

$@

The target of the rule.

$?

Represents only the files that have changed.

$^

A list of all the prerequisites of the rule.

$<

The first prerequisite of the rule.

VPATH

List of directories to search for prerequisites and targets. Separate by blanks or colons. Note that the current directory is always searched first. The vpath directive can also impart this functionality and it can be used to set paths for classes of prerequisites (or targets) matching certain patterns.

SHELL

What shell to use to execute recipes. Default is ‘/bin/sh. If you’re messing with this you might also use the `SHELLFLAGS variable. This variable is not picked up from the make process’ execution environment since it’s usually defined for user preferences and not a happy functioning makefile.

CURDIR

Pathname of the current working directory of this execution of make.

MAKE

The file name with which make was invoked. Useful for recursive operations.

MAKELEVEL

This is a number ("0" is the starting level) which indicates which level of recursion make is in. Useful for having a makefile that behaves differently if it is run directly or part of a bigger make process.

.RECIPEPREFIX

Use this to avoid needing tabs to define your recipes.

.VARIABLES

Expands to all the not "undefined" variables. Could be useful for debugging.

Special Target Names

.PHONY

Prerequisites of this target are considered phony targets and will process their recipes in all cases.

.DEFAULT

Any prerequisite file which are not targets will have this executed on its behalf.

.ONESHELL

If this is used as a target anywhere then all lines of each recipes will be evaluated in one shell process. The default is that each line of recipe spawns a new shell. This can be troublesome if you want to preserve state such as the current directory.

.NOTPARALLEL

This pseudo-target will prevent parallel execution that may have been otherwise requested using the -j option.

.DELETE_ON_ERROR

If this target appears then target files that are created but interrupted before that rule can finish are deleted. This prevents up-to-date time stamps on not up-to-date files from occurring. The special target .PRECIOUS is the opposite.

Make Functions

wildcard

Explicit wildcard expansion.

patsubst

Explicit pattern substitution. Here % acts as a wildcard.

subst

Direct substitution of a static thing for another static thing.

shell

Run a Unix command. Let’s say you want the executable to be the same name as your directory so different directories can use the same make file: EXECUTABLE= $(shell basename $${PWD}) Note the double $ which is necessary.

ifeq

Conditional structure. See also ifneq, ifdef, else, and endif.

strip

Strips out white space.

call

Allows you to effectively create your own functions.

findstring

Useful for conditionals that need to detect if a string contains another. Returns found string or nothing as appropriate.

filter

Returns from a whitespaced list of strings only the ones that match a patter. The % wildcard can be used. Note that filter-out is the opposite function.

sort

Sorts a list of words in lexical order. Makes unique too!

words

Like a length function. Counts words.

wordlist

Can be used like a Python slice.

dir

Isolates the directory parts of a list of names. Contrast with notdir which does the opposite.

suffix

The final dot until the end of each word.

basename

Suffix (and dot) removed. Directory remains.

addsuffix

As it sounds. Similar to addprefix.

join

Joins x1 to y1, x2 to y2, etc. If there’s an x3 but no y3, x3 is returned unmodified.

realpath

For each word, returns the filename without . and double // etc.

or

A logical operation. There’s an and too.

foreach

A function mapping function. Similar to a for loop in Bash. Instead of arbitrary stuff the expression is usually some kind of text processing.

origin

Tells where a variable came from - "default", "environment", "command", etc.

flavor

Like origin but for variable flavors - "undefined", "recursive", "simple".

Fancy Make Functions

There is a special function called call which allows one to basically define a custom function. Here’s a small example:

andagain = $(1) $(1)
doubled = $(call andagain, bye)

In this case doubled is set to "bye bye". This can be nested and each one gets its own set of numbered argument variables.

There is an eval. It does what eval statements usually do in languages - cause confusion. Let’s try to avoid that.

Make Options

-C <dir>

Specifies the directory for make to use. Useful for recursive invocations.

-j [integer]

Number of jobs to run concurrently if possible. If value is omitted, attempt to spawn unlimited number as needed.

-n

Synonym for --dry-run which explains what it does.

-t

Touch all files bringing them up to date without actually doing anything to them. Use this to avoid recompilations to systems where trivial changes were made.

--eval=string

Here string is makefile syntax. This allows impromptu command line make with fancy behavior.

-i

Ignore errors. Sounds dangerous.

-r

Synonym for --no-builtin-rules which eliminates implicit (usually traditional C compiling) rules.

-R

Synonym for --no-builtin-variables which is like -r (implies it actually) but stronger.

Multiple Directories

Often if you are interested in using make, your project is complex and if your project is complex, then it probably spans many directories. Here are some techniques which can handle that sort of thing.

There probably is a better way than this but here is a possible way:

SUBS:= ./sub1 ./sub2

all: $(SUBS)
.PHONY: all clean

force:
    true

$(SUBS): force
    echo "Checking subdirectory:" $@
    $(MAKE) -C $@

clean:
    for dir in $(SUBS); do ( cd $$dir; $(MAKE) clean); done

Note that some people feel that Recursive Make Considered Harmful. This sensible analysis says that recursive Makefiles are inherently broken (not make’s fault!). It is better to have one big Makefile that uses include directives so that the dependency graph can be complete and correct.

Here’s a set of files that comprise a large single Makefile spread out over many directories which use include. This, in theory, should never have directed acyclical graph problems. It is moderately convenient but I wish I could figure out how to have the subdirectory variable ($(D)) be correct for all sub files. I tried eval and many other tricks but it seems I need to have actual distinct variable names so that the whole mess can be organized before it begins.

Makefile
#DB := -g
SUBS := vector vector/vectortest
LIBS :=
PROGS :=

%.o : %.cc
    $(CXX) $(DB) -c -Wall $< -o $@

all: allnowdefined

include $(patsubst %, %/Makefile.sub, $(SUBS) )

clean:
    rm $(SUBS:%=%/*.o) $(PROGS) 2>/dev/null || true

allnowdefined: $(PROGS) $(LIBS)

.PHONY: clean all allnowdefined
vector/Makefile.sub
# === Makefile.sub for `vector`.
D2 := vector
LIBS += $(D2)/vector.o $(D2)/line.o

$(D2)/vector.o: $(D2)/vector.cc $(D2)/vector.h
    $(CXX) $(DB) -c -Wall $< -o $@

$(D2)/line.o: $(D2)/line.cc $(D2)/line.h
    $(CXX) $(DB) -c -Wall $< -o $@
vector/vectortest/Makefile.sub
# === Makefile.sub for `vector/vectortest`
D3 := vector/vectortest
D3PROGS := $(D3)/vectortest $(D3)/linetest
PROGS += $(D3PROGS)
VTLIBS := vector/vector.o vector/line.o

$(D3)/vectortest : $(D3)/vectortest.o $(VTLIBS)
    $(CXX) $(DB) -Wall $(VTLIBS) $(D3)/vectortest.o -o $@

$(D3)/linetest : $(D3)/linetest.o $(VTLIBS)
    $(CXX) $(DB) -Wall $(VTLIBS) $(D3)/linetest.o -o $@

clean_vectortest:
    touch $(D3PROGS) $(D3)/__.o && rm $(D3)/*.o $(D3PROGS)

Not Compiling C Programs?

Make is optimized for C program compiling (and things very much like that). This is fine, but make is more versatile than that and often you don’t want to be bothered second guessing what C development features are doing to your set up. Here are some settings to consider when using make for projects that have nothing to do with C.

As shown above, consider the -R option (and it’s little brother -r). This disables implicit rules and variables.

Set .LIBPATTERNS to blank so that no magical library substitutions are used.

Also you might want to add the .SUFFIXES: rule to your makefile to reset any implicit suffixes that might be treated as something meaningful. Any targets are now considered suffixes but with no targets the implicit suffixes are disabled.

Makefile For Simple C Programming

Here is my favorite simple Makefile that will try to compile any C programs you add to the directory (*.c). Note the tabs are tabs.

# Compiles .c and .cxx programs.
# Default compiler variables. Change if needed.
#CC=gcc
#CXX=g++
# Pick your own favorite headers and libraries.
INCDIRS = -I/usr/include/GL
LDLIBS = -lGL -lGLEW -lglut -lGLU
#CFLAGS = $(shell pkg-config --cflags opencv)
#LIBS = $(shell pkg-config --libs opencv)
# Include this for debugging symbols.
FLAGS = -g

ALLCSRCS := $(wildcard *.c)
ALLCPPSRCS := $(wildcard *.cxx)
ALLEXEC := $(ALLCSRCS:.c=) $(ALLCPPSRCS:.cxx=)
all: $(ALLEXEC)
%: %.c
    $(CC) $(FLAGS) $(INCDIRS) -o $@ $< $(LDLIBS)
%: %.cxx
    $(CXX) $(FLAGS) $(INCDIRS) -o $@ $< $(LDLIBS)

clean:
    rm -f *.o $(ALLEXEC)
.PHONY: all clean

This is handy to put in a directory when writing a small program in Vim so you can use Vim’s :make command to compile the program and return you to any error. Note the target make clean can (and presumably should) irrevocably delete files. Make sure you know exactly how that’s going to work and/or back up your data (version control?) before implementing this.

Fancier Tricks

The following Makefile is what I use to organize directories of text documents that need to automatically be converted into HTML (see asciidoc). The problem here is that I don’t know what my targets will be. If I create a file called thelatest.txt then I will want a target for thelatest.html. But I don’t want to have to edit the makefile every time I create a new document. This can be solved with "Substitution References". You can see how this is used in the line that defines ALLDOCSRCS. This also includes a way to exclude certain files that would otherwise match the pattern. Also shown are my notes for thinking through tricky make problems. The way that the PHONY is done is lifted right out of /usr/src/linux/Makefile which is a great source for ideas.

# Makefile for processing help documents.
# Chris X Edwards
# ---------------------------------------------
# WHATISMADE: all HTML files _except_ index.html
# SENSITIVE TO: corresponding text source file (*.txt)
# WHATHAPPENS: converted to HTML with asciidoc.
#
# WHATISMADE: index.txt
# SENSITIVE TO: all HTML files _except_ index.html
# WHATHAPPENS: mkindex script is run
#
# WHATISMADE: index.html
# SENSITIVE TO: index.txt
# WHATHAPPENS: index.txt converted to index.html

XEDCONF := xedhelp_asciidoc.conf
ASCIIDOC := /usr/bin/asciidoc
AFLAGS := --verbose -f $(XEDCONF)
MKIND := ./mkindex

ALLDOCSRCS := $(filter-out index.txt,$(wildcard *.txt))
ALLDOCHTML := $(ALLDOCSRCS:.txt=.html)
ALLSYMLINKS := $(ALLDOCSRCS:.txt=)

PHONY = all
all : index.html symlinks

index.txt: $(MKIND) $(ALLDOCHTML)
        true MAKING INDEX SOURCE FILE
        $(MKIND) > $@

index.html: index.txt $(XEDCONF)
        true MAKING INDEX HTML FILE
        $(ASCIIDOC) $(AFLAGS) --out-file=$@ $< ; chmod a+r $@
        true Do \"make up\" to upload to xed.ch

PHONY += symlinks
symlinks: $(ALLSYMLINKS)

%.html : %.txt
        $(ASCIIDOC) $(AFLAGS) --out-file=$@ $< ; chmod a+r $@

% : %.html
        ln -s $< $@

PHONY += install up
up: install
install:
        rsync --exclude=".*.swp" -aP /home/xed/help fancy.server.com:public_html/

.PHONY: $(PHONY)

Here’s a very simple Makefile I use to process stupid slide show presentations:

all: index.html

clean:
        rm index.html talk.html

talk.html:talk.txt
        /home/xed/src/asciidoc-8.6.3/asciidoc.py -b slidy talk.txt

index.html:talk.html
        ln -sf talk.html index.html

.PHONY: all clean

Troubleshooting

This article talks about a clever rule to allow queries to the values of variables through make targets.

CMake

CMake is not make nor is a alternative to make. It is a stupid infinite regress meta-make. Its "feature" is that it mostly helps poor Windows people persist with a stupid OS. I’m about as sympathetic to that problem as the software world was to cross-platform issues in the late 1990s. The CMake Wikipedia page says "[a CMake pioneer] blended components of [some shit proprietary mess] with his own ideas, striving to mimic the functionality of Unix configure scripts." You know what else has the functionality of Unix configure scripts? That’s right, Unix configuration scripts. Still, we have to deal with it. Without CMake, I suppose many projects would never see the light of a wholesome platform, so we can’t be too unfriendly to it.

The normal thing one does is something like this.

git clone https://github.com/gituser/theproj
cd theproj
mkdir build
cd build
cmake ..
make
make install

Why maintainers who use this system can’t provide an empty build directory is a mystery.

Usually there are dozens to thousands of inscrutable configuration options. How do you know what they are? Certainly not documentation! Try one of these.

cmake -LA

If you add a -H you basically get the slightly abridged contents of the important CMakeCache.txt file. Note that this file contains the following important hint.

# This is the CMakeCache file.
# It was generated by CMake: /usr/bin/cmake
# You can edit this file to change values found and used by cmake.

Note also that it is not the CMakeCache file as stated, but rather the CMakeCache.txt but it does seem like editing it is effective for configuring the build process.

Here is a list of CMake related variables and some clue about what they’re supposed to do.

Here’s one you might want to set to true if you’re having problems or are fond of Gentoo.

CMAKE_VERBOSE_MAKEFILE:BOOL=TRUE

This one is critical if you’re trying to have the latest source compiled version of something not conflict with your distro’s old-fashioned perspective.

CMAKE_INSTALL_PREFIX:PATH=/usr/local

This one looks interesting for the same reason. No idea what it really does but certainly suggestive. Maybe it is only found in OpenCV.

//Enables mangled install paths, that help with side by side installs.
INSTALL_TO_MANGLED_PATHS:BOOL=OFF

Here’s another tip from OpenCV that illustrates how to set these things without using the configuration files.

cmake -D CMAKE_INSTALL_PREFIX=/usr/local -D OPENCV_EXTRA_MODULES_PATH=/tmp/opencv_contrib/modules -D BUILD_EXAMPLES=ON ..