Proper dependency tracking in GNU make

make is used to build projects, e.g. compile source code into binaries. If the project consists of multiple files, explicit dependencies must be specified to run the command in the correct order.

In addition to that Makefiles can also be used to track implicit dependencies: If one file is modified, only those commands are re-run which are needed. For large projects that can be a big time-saver if incremental changes are done.

But how to do that properly (for a C project)?

The historical way

• Makefile

    main: main.o
    main.o: main.c
  

• main.c

    #include <stdio.h>
    #include "main.h"
  

• main.h

    #include <stdint.h>
  

In the past many projects implemented that themselves. They used the pre-processor cpp to process all #include statements and then used regular expressions to extract the path of all files, which have been read. These dependencies are then converted into a make fragment, which declares that dependency:

main.o: main.h /usr/include/stdio.h /usr/include/stdint.h

The main Makefiles has to include this fragment using something like -include main.d.

This solution has multiple issues.

Vanishing dependencies

Consider, you refactor your code and remove main.h. In that case your automatically generated dependencies show an issue: As main.o depends on main.h, which no longer is there, make will fail as there is no receipt to remake it.

This fix this your dependency generation tool needs to output empty rules for all dependencies:

main.h:
/usr/include/stdio.h:
/usr/include/stdint.h:

There are three cases:

1. if the file still exists and was not updated — it is older than the target — no remake is triggered by this dependency — but others may still trigger one.
2. if the file still exists and was updates — it is newer than the target - a rebuild is triggered for the target.
3. if the file does no longer exist, make invokes the empty receipt to remake it. The will not really create the file, but make will consider it as newer than the target and continue with the previous case 2 above and remake the target.

Without that any developer would have to invoke make clean to remove all targets and dependency files, resulting in a full rebuild:

.PHONY: clean
clean:
    $(RM) main *.o *.d

Maintaining the dependency tool

First of all you must run the pre-processor a 2nd time to generate the input for you dependency extraction tool. For small projects that cost might be negligible, but for larger projects that might add up.

Second you must maintain yet another tool. While the pre-processed output is relatively easy to parse, newer compiler versions may add new features or change the output slightly, which your tool then must handle also.

Third you must make sure to invoke your pre-process run with exactly the same arguments as your real compilation: Any -Ddefine, -Idirectory, -include, -imacros is important as otherwise you might miss or record wrong dependencies.

You must also decide, when to call your tool: Many projects call it before the actual compilation, but that is unneeded: If the target is missing, make must remake it anyway. If the target exists, but you don’t no longer have the dependency information, you must also remake the target as you cannot guarantee, that any (changed) header might not introduce a significant change.

Generating the dependency information afterwards looks okay. But you might get into situations, where you have stale information, for example if you interrupt make between the compilation and dependency-gathering steps.

Best would be to do it at the same time. Luckily that is possible with gcc and other modern compilers like clang.

The gcc way

Luckily modern GCC has built-in support to generate dependency information in make-syntax itself:

• -M enables generating dependency information instead of compiling the file. The output is written to STDOUT unless -o is used to redirect it to a file.
• -MM similar to the above, but system header files are not mentioned.
• -MD and -MMD are variants of -M and -MM respectively, which generate dependency information in addition to the requested action, e.g. -c to compile the unite.
• -MF file writes the information to the given file instead of STDOUT.
• -MP adds additional .PHONY targets for all dependencies to solve the Vanishing dependencies problem from above.
• -MT target allows to overwrite the target name. By default the base-name of the main input file is used, where the suffix is replaced by .o.
• -MQ target is the variant of the above, which also quotes any make meta-characters to make sure, the name is not mangled by make but reaches the shell command as-given.

So let’s rewrite our Makefile and try this:

main: main.o

%.o %.d &: %.c
	$(CC) $(CPPFLAGS) $(CFLAGS) -MMD -MF $*.d -MP -c -o $*.o $<

-include *.d

1. ‘&: tells make`, that the recipt generated both files at the same time. (grouped targets)
2. -MMD tells gcc to both compile and generate dependency information at the same time. System header files are excluded.
3. -MF $*.d tells gcc to write the dependency information into a file with the file name extension .d.
4. -MP tells gcc to generate .PHONY targets for all included file to make the dependency information future-proof in case one of them gets deleted.
5. -c -o $*.o $< to compile the unit.
6. -include *.d includes the dependency information as far as it already exists

First compilation issue

This does not work as expected: make has a built-in mechanism to Remake Makefiles. All files included via include are considered Makefiles and make tries to update them. If there is no file *.d, make applies our rule and will try to compile *.c to *.d :-( (That is why the above rule already uses $*.o instead of $@ as the later would be *.d, which then is passed to both -MF and -o with catastrophic results.)

We can avoid this by explicitly using $(wildcard ) to include only the existing files:

-include $(wildcard *.d)

Second compilation issue

While the solution looks okay, actually it is not: This way dependency information is optional. If you delete all dependency files *.d, modify main.h and re-run make: Nothing will happen. We lost the information, that main.o depends on main.h. Therefore we must change the rule to always require the associated file $*.d to always exist:

%.o: %.c %.d
	$(CC) $(CPPFLAGS) $(CFLAGS) -MMD -MF $*.d -MP -c -o $@ $<
%.d: ;
.NOTINTERMEDIATE: %.d

• the empty rule for %.d is needed for make to handle the case, when the file is missing. For that case we tell make that it should consider that file as remade, so it newer than the target. That will remake the target to actually generate the real dependency information.

• the .NOTINTERMEDIATE is needed as %.d is never mentioned as a real target. make will search its chain of implicit rules main → main.o → main.d and mark it as intermediate. Because of that the file is not remade and/or will be deleted if it is remade. By marking it as non-intermediate we tell make to handle it as a regular file and to keep it afterwards.

  This is only available since GNU make 4.4!

Final version — make 4.4

#!/usr/bin/make -f
# Disable built-in rules and variables
MAKEFLAGS += --no-builtin-rules

main: main.o

CFLAGS := -g
MYCFLAGS := -Wall -Werror
DEPFLAGS = -MMD -MP -MF $*.d -MT $@

COMPILE.c = $(CC) $(DEPFLAGS) $(CPPFLAGS) $(CFLAGS) $(MYCFLAGS) -c

%.o: %.c %.d
	$(COMPILE.c) $(OUTPUT_OPTION) $<
%.d: ;
.NOTINTERMEDIATE: %.d
%: %.o
	$(LINK.o) $^ $(LOADLIBES) $(LDLIBS) -o $@

-include $(wildcard *.d)

.PHONY: clean
clean:
	$(RM) main *.o *.d

Final version — make 4.3

#!/usr/bin/make -f
# Disable built-in rules and variables
MAKEFLAGS += --no-builtin-rules

SRCS := main.c
OBJS := $(SRCS:%.c=%.o)
DEPS := $(SRCS:%.c=%.d)

main: $(OBJS)

CFLAGS := -g
MYCFLAGS := -Wall -Werror
DEPFLAGS = -MMD -MP -MF $*.d -MT $@

COMPILE.c = $(CC) $(DEPFLAGS) $(CPPFLAGS) $(CFLAGS) $(MYCFLAGS) -c

%.o: %.c %.d
	$(COMPILE.c) $(OUTPUT_OPTION) $<
$(DEPS):

-include $(wildcard $(DEPS))

.PHONY: clean
clean:
	$(RM) main $(OBJS) $(DEPS)

The kbuild way

The Linux kernel uses its own build system called kbuild, which is based on a bunch of make receipts. It has some additional requirements:

1. The Linux is heavily configurable. There is a huge .config file, which lists all options. If that file would be used as a pre-dependency, all such files would get rebuilt each time a single option was changed. Therefore kbuild uses some mechanisms to split that big file into smaller chunks, so that each compilation unit can just depend on those options, it really depends on.
2. The above solution does not track the $(…FLAGS) variables or $(CC). Changing them might a complete rebuild to have a consistent kernel again. As such kbuild logs the final command used to compile the target also in the dependency information file. On the next run the commands are compared and the invocation may only be skipped, if they match.

For that kbuild overwrites most of makes dependency mechanism with its own implementation:

1. Most targets have FORCE as their pre-dependency, so that the receipt will always run.
2. The receipt itself will then use some heavy macro magic to read back its dependency information from a file and compare that to the actual run. The command is only executed if any pre-requisite is changed or any relevant configuration option is changed.
3. If a command cannot determine, if it needs to run, it will run by default but will write its output to a temporary file. That file is then compared to the previous version.
   • if the content differs, the temporary file is renamed over the real output file.
   • if the content did not change, the temporary file is deleted. That way the old time stamp is preserved if no change did happen. This is done to prevent needless downstream rebuilds.

Closing word

Much of this was inspired by the article Auto-Dependency Generation from Paul D. Smith. Thank you very much for writing this in the first place. The main difference is, that he uses a variable $(SRCS), which explicitly lists all source C files. That way he can explicitly name the expected *.o and *.d files, which bypasses the problem with intermediate files from my solution above. That version also works for make 4.3 an earlier as .NOTINTERMEDIATE is only available since make 4.4.