depfile as separate build step

[sean-anderson-seco]

I want to use ninja with a tool that generates dependencies as a separate build step. These dependencies are not necessary when building from scratch, but they can be used after an initial (full) build to correctly rebuild targets when source files change. In make, I do something roughly like

# This generates an unpredictable set of objects (one per logical unit, where
# each source file may contain multiple units). It's equivalent to running
# incremental_build once for each source file.
my.lib:
	initial_build -l $@ foo.src baz.src

%.o: | my.lib
	incremental_build -l my.lib $^

%.bin: | my.lib
	link_binary -l my.lib $@

# Note that dependencies cannot be generated until after we link the binaries
my.d: foo.bin qux.bin | my.lib
	generate_dependencies -l my.lib $@

include my.d

Where library.d contains dependencies like

foo.o: foo.src
bar.o: foo.src
baz.o: baz.src
qux.o: qux.src
foo.bin: foo.o bar.o baz.o
qux.bin: qux.o baz.o

On the first build, the incremental_build rule is never executed, but once library is created, incremental_build will be used. This is not an ideal solution, but the tool is rather constraining in this regard.

I would like to replace my makefiles with ninja, as I find it difficult to manage all the different rules. However, I could not figure out the right way to replicate the above setup. I tried something like

rule initial
	command = initial_build -l $out $srcs # Why isn't there an $in_order ?
rule incremental
	command = incremental_build -l my.lib $in # This won't work because it should actually be $in_implicit
rule link
	command = link_binary -l my.lib $out
rule gendeps
	command = generate_dependencies -l my.lib $out

build my.lib: initial
	srcs = foo.src baz.src
# We don't actually know what objects we have until we actually run a build...
build foo.o: incremental || my.dd
	dyndep = my.dd
build bar.o: incremental || my.dd
	dyndep = my.dd
build baz.o: incremental || my.dd
	dyndep = my.dd
build qux.o: incremental || my.dd
	dyndep = my.dd
build foo.bin: link || my.lib
	# Here ninja complains that my.d isn't in the dependencies, but if we add it things are circular
	dyndep = my.dd
build qux.bin: link || my.lib
	dyndep = my.dd
build my.d: gendeps | foo.bin qux.bin || my.lib

So dyndeps won't work here. But we can't use depfile either, since the dependencies aren't generated as part of the build steps. Maybe I can make build.ninja depend on my.d and regenerate the ninja file whenever the dependencies change?

[digit-google]

In a nutshell, what you are trying to do is not directly possible with Ninja.

I.e. depfiles and dyndeps files can only add dependencies between input/output paths (a.k.a. "Nodes" in the source tree) and existing build statements (a.k.a. "Edges"). There is no way to create new edges dynamically, like the Make pattern matching allows. And this is by design, to keep Ninja fast.

[sean-anderson-seco]

OK, so lets try reorganizing this in a different way. If you notice, incremental_build never actually references any of the object files, just the source files. So what if I rewrote the ninja file like

rule initial
	command = initial_build -l $out $srcs && touch $stamps
rule incremental
	command = incremental_build -l my.lib $in && touch $out
rule link
	command = link_binary -l my.lib $out
	depfile = $out.d
rule gendeps
	command = generate_dependencies -l my.lib && postprocess

build my.lib: initial
	srcs = foo.src baz.src qux.src
	stamps = foo.stamp baz.stamp qux.stamp
build foo.stamp: incremental foo.src
build baz.o: incremental baz.src
build qux.o: incremental qux.src
build foo.bin: link || my.lib
build qux.bin: link || my.lib
build foo.bin.d qux.bin.d: gendeps | foo.bin qux.bin || my.lib

And processed the depfile to generate something like

# foo.bin.d
foo.bin: foo.stamp bar.stamp
# qux.bin.d
qux.bin: qux.stamp bar.stamp

I'm away from the project atm, so I can't try this out, but it seems much closer to what ninja can do. Will I run into trouble with generating the depfiles in one step?

Of course, there is the potential downside that (for example) moving a unit from one file to another will break the build (or maybe my.lib would get rebuilt since the arguments changed?). I think this would happen in the original makefile, though.