Makefile Introduction
Distribute your software with GNU Make tool (Makefile)
GNU Make tool is a task runner that can be used to compile your software to target multiple platforms and architectures. In this article, we are going to learn the basics of Make and Makefile.
I am sure you must have heard about interpreted and compiled languages before. A program written in an interpreted language like JavaScript or Python can be distributed directly without having to compile it first.
An interpreter is a virtual machine that can process raw source code and compile it to the machine language on the fly and then run it on the machine. This way, programs written in interpreted languages don’t need to care about what system and architecture an end-user is using. Everything is taken care of by the interpreter.
This story is a little different for compiled languages. A program written in compiled language has to be compiled first for the architecture of a machine before running it on that machine. Hence when we are distributing software, we generally create compiled binaries for various platforms and a user can pick the correct binary file suitable for him/her.
In this scenario, interpreted languages look like winning the distribution war but when it comes to speed, efficiency, security, and portability, compiled languages are the champions. On the other hand, interpreted languages need an interpreter while compiled programs can directly run by the machine.
A compiled language may take a significant amount of time to compile because it is doing a lot of optimizations under the hood. The final binary distribution file is compact and could be less in size compared to the overall source code, which makes it highly portable.
Both interpreted and compiled languages have their own benefits and drawbacks but when it comes to distributing your software, compiled languages can be tough to work with (well, for some people).
When you want to distribute a piece of software written on interpreted languages, you can just provide a compressed .zip file or a git repository with the original source code with some instructions on how to run it. Then it becomes the responsibility of the end-user to make it work.
When you distribute your software written in a compiled language like C, C++ or Go, you need to first create a binary executable file that contains machine instructions of specific processor architecture and runtime of a specific operating system. The combination of processor type and the operating system can be called a compilation target.
💡 The final binary executable file needs to have the knowledge of what operating system and processor architecture it will run on. The processor architecture decides the machine instructions set to use while the operating system decides the OS APIs to communicate with the machine for I/O operations and other system specific operations during runtime.
There are be multiple combinations of processor types and operating systems that machines around the world are using. That means while distributing software, we need to consider multiple compilation targets and create distributions for each of them (depending on your userbase).
Normally, when you compile a program on a machine using a standard compiler provided by the programming language, you are creating a binary executable for that specific machine architecture and the operating system.
However, some compilers provide the capability to target multiple architectures and operating systems to create distribution files. This process is called cross-compilation and the cross-compilers make it happen.
A compilation target can be selected using some command-line arguments provided by the compiler. That means if you want to change a compilation target, you just need to change the command-line arguments.
Golang Cross-Compilation
We can create a binary executable from a Go program for a specific platform using the combination of GOOS and GOARCH environment variables.
$ GOOS=linux GOARCH=amd64 go build http-server.go
The command above is Go’s way to cross-compiling a Go program for the Linux operating system and AMD64 processor architecture. The GOOS environment variable specifies an operating system/kernel of the platform and the GOARCH environment variable specifies the processor architecture.
💡 Find more about the general purpose and architecture-specific environment variables from this official documentation.
Go out of the box supports cross-compilation for many platforms and they can be listed using the go tool dist list command which lists the os/arch combinations like below.
android/amd64 android/arm64 darwin/386 darwin/amd64
freebsd/amd64 linux/amd64 linux/mips linux/ppc64le
nacl/386 netbsd/386 openbsd/386 openbsd/arm64
plan9/386 solaris/amd64 windows/386 windows/amd64
...more
💡 A more comprehensive list of GOOS and GOARCH list is made available by asukakenji on this Gist but you can rely on go tool dist list. You may also need to provide architecture-specific environment variables in some cases, more from here under section “Architecture-specific environment variables”.
C/C++ Cross-Compilation with Clang
Similarly, the command below can be used to compile a project written in C (or C++) programming language for a specific target.
$ clang --target=x86_64-apple-darwin hello.c
The command above is Clang’s way of cross-compiling a C program for the Darwin operating system(kernal) and x86_64 (AMD64) processor architecture. The --target argument specifies a compilation target based on machine-vendor-os combination, also called a target triplet.
💡 You can see the target triplet of your machine using gcc -dumpmachine command which might result in something like x86_64-apple-darwin. The target triple has the general form of <arch><sub>-<vendor>-<sys>-<abi>. You can learn more about the possible values of the target triplet from this article.
Software Compilation Process
Before we get into Make tool, let’s understand a typical software compilation process. We are going to use C as the programming language to generate some distributions using the GNU GCC compiler.
💡 I have set up a sample C project to demonstrate the use of Make and Makefile. I won’t be able to show each file to you since there are a lot of files, but you can check this GitHub repository for the source code.
In the program above, we have main.c which depends on calculate.c and it is the entry point of the application since it has the main method. It calculates the result by executing the calculate() function and prints to the console.
In the calculate.c, we have the definition of calculate() function and it depends on add.c for the add() function and sub.c for the sub() function.
Let’s try to compile the program manually. You can use any compiler of your choice but I am going to use GNU GCC provided by Xcode (apple machine).
$ gcc -o program.out main.c calculate.c add.c sub.c
The command above creates a binary executable file program.out in the current directory for the current architecture (platform) of the machine by compiling the main.c, calculate.c, add.c, and sub.c files.
We can run this binary executable by using the command below.
$ ./program.out
calculate( 1, 2. '+' ) => 3
The compilation process looks straightforward because we are combining multiple source files to generate one executable file. The way C compiler works under the hood is a little different than that.
source code => preprocessor => compiler => linker => executable
First, our C is code is processed by a preprocessor which creates a stripped-down version of our source code and expands preprocessor directives.
The compiler creates the machine code from the source code and generates object files. These object files may contain external references to other object files or the standard library also called symbol references.
Once the object files are created, the linker links them together to generate an executable file that can be run. The linker will resolve all symbol references and may give an error when a symbol reference can not found.
We can instruct gcc to create object files from individual source code files using -c flag. The object files have the .o extension on Unix systems.
$ gcc -c -o main.o main.c
$ gcc -c -o calculate.o calculate.c
$ gcc -c -o add.o add.c
$ gcc -c -o sub.o sub.c
Once the object files are created, we can instruct gcc to create an executable file by providing these object files. We don’t need to provide a flag to the command, GCC is smart enough to figure out that only the linker needs to be used to combine these object files.
$ gcc -o program.out main.o calculate.o add.o sub.o
If you have seen the source code of these object files from the GitHub repository, we can draw the dependency tree of these files like below.
main.o
├── main.c
└── calculate.o
├── calculate.c
| └── calculate.h
├── add.o
| └── add.c
| └── add.h
└── sub.o
└── sub.c
└── sub.h
GCC under the hood uses the GNU LD linker for the linking. We can also use the command below to generate an executable from multiple object files.
ld -lc -o program.out main.o calculate.o add.o sub.o
The -lc flag provides the standard library (libc.a) during the linking part as we have a prinf external symbol reference from the standard C library.
If we miss an object file during the linking process, ld will throw an error.
$ gcc -o program.out main.o calculate.o add.o
Undefined symbols for architecture x86_64:
"_sub", referenced from:
_calculate in calculate.o
ld: symbol(s) not found for architecture x86_64
In Unix based systems, .out or simply an executable file without an extension is ready for distribution. You might be familiar with .exe executable files in the Windows operating system.
If you are cross-compiling for multiple platforms, you would compile object files for each platform using some command-line arguments suitable for gcc or clang and then link them together using ld.
linux_arm
darwin_amd64.out
windows_i386.exe
windows_x86_64.exe
Make and Makefile
GNU Make is an open-source tool to generate binary executables. In a nutshell, it is a task runner that can run custom tasks but also look for optimizations while running them.
Imagine you need to target 100 platforms for your software distribution. It would be hard to execute each platform-specific command that creates a distribution file through some combination of command-line arguments as we saw earlier. This would be painstakingly hard.
You can write all these 100 commands in a bash script and execute it. That would solve the problem but you wouldn’t able to specifically select a command to run it without running the other 99 commands.
In some cases, you have one program file dependent on another program file. In that case, you would only want to compile programs only if its dependencies change. This way, you don’t have to compile entire software which could take hours to compile.
The Make tool defines a structured way to write tasks and provides a Command Line Interface (CLI) to execute these tasks in a specific order. We can also instruct Make to execute only specific tasks based on our needs and Make is smart enough to figure out which tasks need to be run based on whether or not their dependencies have been changed.
💡 Make is generally installed on most of the systems but you can follow this documentation on how to install it in case you don’t have it already. You can use the make --version command to verify the installation.
A software compilation process is a little complex than just running a command to create distributions. In the previous example, we learned how to create object files and how to link them together to generate an executable.
Let’s imagine if we only changed the implementation of main.c, in that case, we do not need to generate caculate.o, add.o or the sub.o because they do not depend on main.o. This is where Make can be very helpful.
Makefile Introduction
Make tool is a task runner, and like any other task runners, it needs a configuration file to provide the list of tasks to run. This is a text-based file typically with the Makefile or makefile filename. We can run these tasks using the make command provided by the Make tool.
💡 Though any file can be used as a Makefile using -f, --file or --makefile arguments to the make command. However, if this argument is missing, make uses makefile or Makefile file in the current directory.
Makefile is an extension to the Bash script and perhaps may look and feel similar to a Bash script with few differences here and there. The basic structure of listing a task in Makefile is as follows.
# task for target 1
<target_1>: <prerequisites_for_target_1>
<recipe_for_target_1>
# task for target 2
<target_2>: <prerequisites_for_target_2>
<recipe_for_target_2>
...
A Makefile structure is similar to the Bash script, hence we can use # for adding some comments. The main thing to focus on the Makefile is the target, prerequisites and recipe directives.
The target is the filename or the file path (relative to Makefile) that will be eventually generated by the make command. The prerequisites are any files that the target depends on (separated by spaces).
Finally, the receipe is some a Bash program that will eventually generate the taget at the correct location. The taget and prerequisites are separated by a colon :. The recipe script is left-padded by a tab.
Let’s look at the simple task for generating add.o object file.
# generate add.o file
add.o: add.c add.h
gcc -c -o add.o add.c
In the above Make task, our target is add.o file (in the current directory) and it depends on the add.o and add.h files. The recipe is a Bash command that generates the add.o file in the current directory.
To execute this task, we just need to execute the make command where the Makefile is located. Make will execute the first task in the Makefile. If you want to pick a specific task to execute, you can use the make <target> command as stated in the example below.
$ make add.o
As you can see from the Makefile, Make does not compile a C program (or any other program) on its own. It is our job to specify the Bash commands which will generate the target file at the correct location.
So you would ask, why we need prerequisites? In the above tasks, add.o and add.h are the prerequisites which are nothing but the files that add.o target depends on. These dependencies are not necessary for Make to work but they help Make take better decisions for performance optimizations.
For example, with the above Makefile, if you run Make, you will this output.
$ make
gcc -c -o add.o add.c
This will create a add.o in the current directory. Next time, when you run the same command, you will see a surprising output.
$ make
make: `add.o' is up to date.
The next time, when we run make command, Make does not regenerate the add.o file. This is because since we have mentioned the dependencies of add.o in the Makefile, Make looks at the timestamps of these dependencies and checks whether any of these files is newer than the add.o.
If any of the dependencies are newer than the target file itself, that means the source code of these files must have been changed and new build needs to be triggered. If not, then Make skips the task.
To rebuild a target, either we need to change the source of the dependencies or change their timestamps. One way to change the timestamp is to use touch command or use -B or --always-make flag with the make command.
$ touch add.c
$ make
gcc -c -o add.o add.c
Target Dependencies
As we discussed before, we have multiple object files dependent on each other. for example, main.o depends on calculate.o and caculate.o depends on add.o and sub.o. So, like we can mention dependencies of a target in the Makefile, can we list the dependencies of a task?
The answer is Yes and No. There is a hidden easter egg in the prerequisites (dependencies) we mentioned in the Makefile. What if one of the prerequisites does not exist? Let’s change the add.h to add.header in the Makefile.
$ make
make: *** No rule to make target `add.header', needed by `add.o'.
What this error tells us that Make could not find the add.header in the current directory so it went to make one. But since there is no target for add.header file, it could not build it and threw an error.
The takeaway here is that a dependency of a target can also be another target. If that’s the case, Make will try to build the dependency first and once that is done, it will build the main target. This can lead to a long chain where one target depends on another target and so on (we will look into this later).
We can also have some Make tasks that do not have any prerequisites but that’s a rare case. But if we do not list the prerequisites for a target, Make has no way of knowing whether it should be rebuild again. Hence once the target file is generated, it won’t be rebuilt again, ever, unless it is deleted.
Let’s create the final Makefile that will build our complete C program.
# executable
program.out: main.o calculate.o add.o sub.o
gcc -o program.out main.o calculate.o add.o sub.o
# main object file
main.o: main.c
gcc -c -o main.o main.c
# calculate object file
calculate.o: calculate.c calculate.h
gcc -c -o calculate.o calculate.c
# add object file
add.o: add.c add.h
gcc -c -o add.o add.c
# sub object file
sub.o: sub.c sub.h
gcc -c -o sub.o sub.c
The main.o, calculate.o, add.o and sub.o targets in the Makefile do not have any target-level dependencies but program.out target depends on all other targets. Hence, when we run make command, these things will happen.
$ make
gcc -c -o calculate.o calculate.c
gcc -c -o add.o add.c
gcc -c -o sub.o sub.c
gcc -o program.out main.o calculate.o add.o sub.o
As you can see, even tough make command runs the first task in the Makefile, it compiled the other tasks targets first to generate the dependencies that the program.out depends on and then it built the program.out target.
Once these task runs successfully, you will have the following files in your project directory and I admit that it looks like a mess.
make-introduction/c
├── Makefile
├── program.out
├── add.c
├── add.h
├── add.o
├── calculate.c
├── calculate.h
├── calculate.o
├── main.c
├── main.o
├── sub.c
├── sub.h
└── sub.o
Phony Targets
If you want to clean the object files that are no longer needed, you can add a task in your Makefile to just do that.
clean:
rm -f main.o calculate.o add.o sub.o
The clean target is not a real target because it’s not a file that we want to generate but we can run it like a task using make <target> command.
$ make clean
rm -f main.o calculate.o add.o sub.o
But there is a problem here. The clean is not a real target but Make doesn’t know that. Since the recipe of the clean target does not generate clean file, this task will run again whenever invoked using make.
But if somehow we infested our project with a clean file, Make won’t run this task and instead will consider the clean file is up to date.
$ make clean
make: `clean' is up to date.
To avoid this, we need to label the clean target as a special target so that Make will start treating it differently from other real targets. The special target we want to label this task with is a PHONY target.
.PHONY: clean
clean:
rm -f main.o calculate.o add.o sub.o
Once we mark one or more targets as .PHONY, we don’t have to worry about whether or not a clean file made its way into our codebase. Make will run the recipe of a PHONY target unconditionally.
$ make clean
rm -f main.o calculate.o add.o sub.o
If your object files grow, you need to keep changing the recipe of the clean target. The easier thing would be to use * pattern.
.PHONY: clean
clean:
rm -f *.o
💡 You can specify multiple PHONY targets separated by the space on the same line like .PHONY: clean reset export upload. You can read more about other special targets in Makefile from this official documentation.
Optimizing Makefile
While working on a large project, you would like to modularize your source code and separate source from the distribution files. Normally, you would use src directory to host the source code and dist directory for the distribution files. Other configuration files can exist in the parent directory.
make-introduction/c
├── Makefile
├── dist
| ├── add.o
| ├── calculate.o
| ├── main.o
| ├── program.out
| └── sub.o
└── src
├── add.c
├── add.h
├── calculate.c
├── calculate.h
├── main.c
├── sub.c
└── sub.h
As you can see from the above directory structure, we have relocated the project source and distribution files and now only the Makefile exists in the parent directory. That means, we need to change the locations of the targets and prerequisites in the Makefile as well.
# executable
dist/program.out: dist/main.o dist/calculate.o dist/add.o dist/sub.o
gcc -o dist/program.out dist/*.o
# main object file
dist/main.o: src/main.c
gcc -c -o dist/main.o src/main.c
# calculate object file
dist/calculate.o: src/calculate.c src/calculate.h
gcc -c -o dist/calculate.o src/calculate.c
# add object file
dist/add.o: src/add.c src/add.h
gcc -c -o dist/add.o src/add.c
# sub object file
dist/sub.o: src/sub.c src/sub.h
gcc -c -o dist/sub.o src/sub.c
# clean object files
.PHONY: clean
clean:
rm -f dist/*.o
From the above Makefile, we have only modified the correct path of the source and target files. Also, in the first target recipe, we have used the dist/*.o pattern to provide all object files from the dist directory.
If you take a close look at our Makefile and the source code, the prerequisites for program.out target seem little unoptimized. If we are planning to add mult.c, division.c and other program files to the project, we will have a long list of dependencies to manage for the main target.
If you go through the program source code, you will find that calculate.c, add.c and sub.c together forms a library for mathematic operations. Let’s combine calculate.o, add.o and sub.o to create a relocatable object file (static library) that can be linked with main.o later.
...
# lib static libary (relocatable object file)
dist/lib.o: dist/calculate.o dist/add.o dist/sub.o
ld -r -o dist/lib.o dist/calculate.o dist/add.o dist/sub.o
...
From the above Makefile, we have added one more task to generate lib.o static library by combining calculate.o, add.o and sub.o files. We have used ld GNU linker with -r flag to create this relocatable object file.
Since lib.o contains the instructions of calculate.c, add.c and sub.c, we only need main.o and lib.o to generate the final executable. But still, we need to maintain the prerequisites for lib.o target.
One thing we can do is save the prerequisites of lib.o target in a variable and use that variable instead. I personally prefer to list all variables at the top of the file.
# library objects
LIB_OBJECTS = dist/calculate.o dist/add.o dist/sub.o
#--------------------------------------#
...
# lib static libary (relocatable object file)
dist/lib.o: $(LIB_OBJECTS)
ld -r -o dist/lib.o $(LIB_OBJECTS)
...
A variable in a Makefile is defined similarly to a Bash variable. The only difference is that you can not use any quotes because it’s a character in a Makefile. When you want to reference a variable, you need to use $(var) syntax. This isn’t quite similar to Bash but hey, Makefile is not a Bash script.
We can also use the wildcard in the prerequisites of a task.
# calculate object file
dist/calculate.o: src/calculate.*
gcc -c -o dist/calculate.o src/calculate.c
💡 Wildcards do not expand when used in a Makefile variable declaration but we can use wildcard command to expand the wildcard value. Read more about wildcards from this official documentation.
We can also use the $@ automatic variable to get the name of the target inside the recipe so that you don’t have to repeat the target name in the build command itself.
💡 You can read more about variables in Makefile from this official documentation. Makefile supports mutliple automatic variables, read more from this documenation.
After incorporating these optimizations, we have the final Makefile.
# library objects
LIB_OBJECTS = dist/calculate.o dist/add.o dist/sub.o
#--------------------------------------#
# executable
dist/program.out: dist/main.o dist/lib.o
gcc -o $@ dist/main.o dist/lib.o
# main object file
dist/main.o: src/main.*
gcc -c -o $@ src/main.c
# lib static libary (relocatable object file)
dist/lib.o: $(LIB_OBJECTS)
ld -r -o $@ $(LIB_OBJECTS)
# calculate object file
dist/calculate.o: src/calculate.*
gcc -c -o $@ src/calculate.c
# add object file
dist/add.o: src/add.*
gcc -c -o $@ src/add.c
# sub object file
dist/sub.o: src/sub.*
gcc -c -o $@ src/sub.c
# clean object files
.PHONY: clean
clean:
rm -f dist/*.o
Ideal Makefile for Cross-Compilation
I have set up a Go project which accomplishes everything we have done in our C project. You can follow the same GitHub repository for the source code. If you want to learn more about Go, follow my rungo Medium publication.
Though Go goes through the similar complication process as C, we do not generate the object files and link them together. That’s a rare case scenario. Instead, we compile the binary executable file directly from the source.
Unlike GCC or Clang, Go supports cross-compilation for multiple platforms out of the box as discussed in earlier topics. We need to use the appropriate GOOS and GOARCH environment variables with go build command to generate platform-specific executable files.
While setting a Makefile to generate cross-platform executable files, we need a lot of PHONY targets. Normally, you have a PHONY target that compiles executables for all the targets and then you have some PHONY targets that compile executable for individual targets.
I have created a basic Makefile with some PHONY targets and some real targets. As we know, a PHONY target will unconditionally run the recipe, we need a mechanism to prevent that from happening. We need a PHONY target to generate a platform-specific build file but only when the source is newer.
This can be facilitated by providing real targets as the prerequisites for the PHONY target and keeping the recipe empty. When this PHONY target runs, it will try to unconditionally create the real targets but the real target will only run if its prerequisites are newer.
# source files (expand using `wildcard` command)
SOURCE_FILES := $(wildcard src/*.go)
# define `PHONY` targets
.PHONY: all darwin windows
#--------------------------------------#
# build for all platforms
all: darwin windows
#--------------------------------------#
# build for macOS (darwin)
darwin: dist/darwin_amd64.out dist/darwin_386.out
dist/darwin_amd64.out: $(SOURCE_FILES)
GOOS=darwin OSARCH=amd64 go build -o $@ $(SOURCE_FILES)
dist/darwin_386.out: $(SOURCE_FILES)
GOOS=darwin OSARCH=386 go build -o $@ $(SOURCE_FILES)
#--------------------------------------#
# build for windows
windows: dist/windows_x86_64.exe dist/windows_x86.exe
dist/windows_x86_64.exe: $(SOURCE_FILES)
GOOS=windows OSARCH=amd64 go build -o $@ $(SOURCE_FILES)
dist/windows_x86.exe: $(SOURCE_FILES)
GOOS=windows OSARCH=386 go build -o $@ $(SOURCE_FILES)
#--------------------------------------#
# clean `.out` and `.exe` files
.PHONY: clean
clean:
rm -f dist/*.out dist/*.exe
In the above Makefile, we have all, darwin and windows as the PHONY targets. Since all is the first target in the Makefile, when we run make command, this is the target that is going to run by default.
When all target is run, it first runs the darwin and windows targets unconditionally. Since it doesn’t have a recipe, it won’t do anything on its own. These kinds of targets are used to run other PHONY or real targets.
The darwin target is also a PHONY target that means it will also run unconditionally. However, it also doesn’t have a recipe but some real targets as the prerequisites. These real targets will only run if any of the src/*.go files are newer than the target itself.
Let’s run the make command as see what it does.
$ make
GOOS=darwin OSARCH=amd64 go build -o dist/darwin_amd64.out ...
GOOS=darwin OSARCH=386 go build -o dist/darwin_386.out ...
GOOS=windows OSARCH=amd64 go build -o dist/windows_x86_64.exe ...
GOOS=windows OSARCH=386 go build -o dist/windows_x86.exe ...
$ make darwin
make: Nothing to be done for `darwin'.
$ make clean
rm -f dist/*.out dist/*.exe
$ make darwin
GOOS=darwin OSARCH=amd64 go build -o dist/darwin_amd64.out ...
GOOS=darwin OSARCH=386 go build -o dist/darwin_386.out ...
As we have learned, Makefile can be very helpful for creating platform-specific distribution files as well as significantly improving the performance of your overall build process.
Makefile can also be used as a normal task runner. The possibilities are endless. You may have used make && make install command to compile and install a piece of software from the source. The install target in the Makefile is a PHONY target that relocates the generate files at the correct location.
Though Makefile is easy to read and make sense of, some syntax and expression can be hard to work with. To understand more about the syntax and features of Makefile, follow this official documentation.
Distribute your software with GNU Make tool was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.