Parallel execution with make

Despite its old age, make is one of the main build automation tools in use. It has an outstanding reputation as a simple yet powerful tool. Software developers and DevOps engineers alike hold it in high regard. Makefiles offer an easy way to orchestrate a build pipeline and the prerequisite steps.

This post will introduce you to the art of writing a makefile. We will discover how to use makes parallelization features. That will help us harness the power of multi processor systems. Before I go ahead, I want to outline a demo scenario.
If you are in a hurry, feel free to jump to the code part and skip the intro.

Scenario

For demo’s sake, we will keep it simple. Imagine a project targeting multiple platforms. Let’s say macOS, linux, and windows for this example. The project uses make to execute build commands for each platform.

# makefile
# sleep represents work
build_job_mac:
	@echo "[MAC] building..."
	@sleep 10
	@echo "[MAC] DONE!"

build_job_linux:
	@echo "[LINUX] building..."
	@sleep 10
	@echo "[LINUX] DONE!"

build_job_win:
	@echo "[WIN] building..."
	@sleep 10
	@echo "[WIN] DONE!"

build: build_job_mac build_job_linux build_job_win

Each build will take ten seconds. That leaves us with 30 seconds to finish the execution of all build targets. That is not necessary since a multi processor system executes the work.

To speed-up the pipeline, we want to execute the build tasks in parallel.

Makefile

Make offers a solid mechanism to control parallel task execution. A look into the documentation helps to identify the needed options.

$ make --help
Usage: make [options] [target] ...
Options:
...
  -j [N], --jobs[=N]          Allow N jobs at once; infinite jobs with no arg.
  -k, --keep-going            Keep going when some targets can't be made.
  -l [N], --load-average[=N], --max-load[=N]
                              Don't start multiple jobs unless load is below N.
...

Parallel execution

The –jobs parameter regulates the number of jobs executed parallel. One job per processor is a good ratio here. To acquire the number of available processors, use one of the following commands:

# linux
grep -c processor /proc/cpuinfo

# macOS
sysctl -n hw.physicalcpu

Now we can extend the initial makefile to execute the pipeline steps in parallel. The following snippet shows the additions for a system running on macOS.

# makefile
...

build_parallel: build_job_mac build_job_linux build_job_win

build:
	@$(MAKE) build_parallel -j$(shell sysctl -n hw.physicalcpu)
	@echo "[ok]"

Limit execution

Heavy system load is a common problem with parallel execution. The more operations a system has to handle, the more the load increases. With increasing load, everything slows down.

Make’s –max-load option is a great way to prevent such a thing from happening. That limits the number of jobs make runs at once, based on system load.

# makefile
...
build:
	@$(MAKE) build_parallel -j$(shell sysctl -n hw.physicalcpu) -l 1.75
	@echo "[ok]"

With this limitation in place, it is time to execute the pipeline. We notice that the execution phase takes only a fraction of the initial time consumed. In the outlined scenario, the savings equal to 2/3 of the initial time. That’s an immense saving in time and thus connected resources as well. What a great success!

Taken together, these results show how powerful make and its parallelization features are.

Here is the complete makefile:

build_job_mac:
	@echo "[MAC] building..."
	@sleep 10
	@echo "[MAC] DONE!"

build_job_linux:
	@echo "[LINUX] building..."
	@sleep 10
	@echo "[LINUX] DONE!"

build_job_win:
	@echo "[WIN] building..."
	@sleep 10
	@echo "[WIN] DONE!"

build_parallel: build_job_mac build_job_linux build_job_win

build:
	@$(MAKE) build_parallel -j$(shell sysctl -n hw.physicalcpu) -l 1.75
	@echo "[ok]"

Congratulations, you did it. You hopefully got a better understanding of make and how to use its parallelization features.

Going one step further

You could improve the above example by:

• using the –keep-going modifier
• add a prerequisites target that executes before the build target
• use order-only targets and normal targets as prerequisites

Let me end this excursion with a quote for your meditation

We stand at the threshold of a many core world. The hardware community is ready to cross this threshold. The parallel software community is not.
— Tim Mattson