This is a somewhat crude approximation of dirdeps for gmake. We cannot handle all the bells and whistles that bmake can, but we can still manage the basics to handle orchestrating a large build correctly.
The concept is quite simple. We locate a Makefile.depend* (actually GNUmakefile.depend* for gmake) in each directory to be made.
Each such makefile sets DIRDEPS to a list of paths relative to SRCTOP that need to be build before the current directory. And then includes dirdeps.gmk to do the work.
This happens recursively and in the process we build a tree dependency graph which we hand over to make to build in the optimal order, in parallel with no risk of race conditions - provided DIRDEPS are accurate.
In the bmake version a Makefile.depend file looks like:
# Autogenerated - do NOT edit!
DIRDEPS = \
lib/sjg \
.include <dirdeps.mk>
for gmake it will be a bit more crude. For a start we'll assume that all GNUmakefile.depend* are manually maintained.
There is a vpath PATTERN DIRLIST directive, than sounds like we could have:
vpath %.gmk ${GMAKESYSPATH}
so that:
include dirdeps.gmk
should just work, but it doesn't. Instead we need:
include ${GMKSYSDIR}/dirdeps.gmk
where GMKSYSDIR is either set in environment or by sys.gmk which we can cause to be included before each makefile by setting something like MAKEFILES=${SRCTOP}/gmk/sys.gmk.
Since I tend to use a wrapper around make for conditioning the environment, that's simple.
We could also run gmake with -I ${GMKSYSDIR} which is easy enough with the mk wrapper, but that's a bit ugly for anyone not using it.
Since gmake lacks many of the string manipulation features of bmake (many of which I added over the last 20 years) we will have to rely on more manual coding.
For example if TARGET_SPEC_VARS needs to be more than just the default MACHINE, then things like TARGET_SPEC and DEP_TARGET_SPEC need to be set manually in say local.sys.gmk since we cannot auto generate them from TARGET_SPEC_VARS.
Further, since we cannot expect sys.gmk to be able to decompose a TARGET_SPEC tupple into useful variables.... (gmake seems incapable of dealing with , in variables), we have to ensure that MACHINE at least, is split out and passed by dirdeps.gmk.
Ok, I guess one could use $(shell echo "${var}" | tr ',' ' ') but is it too much to expect something like $(subst \,, ,${var}) to work?
At first glance the lack of a .for loop appeared daunting. But to cut a long story short; using nested macros can get the job done.
Having constructed the fully qualified list of dirdeps for the current directory eg.:
${SRCTOP}/${RELDIR}.${TARGET_SPEC}
We can turn that into a list of qualified depend files to try and include:
${SRCTOP}/${RELDIR}/${DEPENDFILE_PREFIX}.${TARGET_SPEC}
We use the following to do the actual including of the next depend file while setting DEP_RELDIR and DEP_TARGET_SPEC appropriately. We want a qualified depend file if it exists, otherwise an unqualified one should do:
define include-depend
$(eval DEP_TARGET_SPEC=$(subst .,,$(suffix $1)))
$(eval DEP_RELDIR=$(patsubst %/,%,$(subst ${SRCTOP}/,,$(dir $1))))
$(if ${DEBUG_DIRDEPS},$(info Looking for $1 DEP_TARGET_SPEC=${DEP_TARGET_SPEC} DEP_RELDIR=${DEP_RELDIR}),)
$(eval -include $(shell test -s $1 && echo $1 || echo $(basename $1)))
endef
invoked via:
$(eval $(foreach dep,${_more_depends},$(call include-depend,${dep})))
That's actually not too shabby.
Gmake does not appear to provide a means of knowing if a target has been defined, so we need to do that ourselves, otherwise we can get lots of warnings about duplicate targets. The following does the trick:
# we have to do target tracking ourselves to avoid complaints
define if-new-target
$(eval t=_${1}_target)
$(if ${$t},,$1)
$(eval $t=1)
endef
invoked thus:
# add dependencies to the graph
${_this_dirdep}: ${_dirdeps}
# now which ones have not been made targets yet?
_newdeps := $(sort $(foreach dep,${_dirdeps},$(call if-new-target,${dep})))
ifneq "${_newdeps}" ""
${_newdeps}:
$(call build-dirdep)
endif
We want to be able to:
mk -j8 -f dirdeps.gmk some/dir.host other/dir.i386
which should behave exactly as if we had a GNUmakefile.depend file containing:
DIRDEPS= \
some/dir.host \
other/dir.i386 \
include ${GMKSYSDIR}/dirdeps.gmk
Unfortunately gmake's filter does not seem able to match a pattern like %/% so we have to resort to a shell script:
# (shell is needed because filter cannot do %/%)
_dirdeps := $(sort $(shell for f in ${MAKECMDGOALS}; do echo "$$f"; done | grep /))
and we had to use grep since case "$$f" in */*) echo $$f;; esac caused a parsing error - the ) closed the $(shell.
Inefficient, but it's a one-off and it works.
Another cool feature missing in gmake is the ability to control the directory where files will be generated. But we can compensate by having dirdeps.gmk create the necessary objdir and telling gmake to chdir there and with the desired src dir in VPATH.
This of course relies on our previously mentioned sys.gmk to deal with correctly setting _CURDIR, _OBJDIR and RELDIR to the desired values. The lack of elif or elifeq etc make that messier than is desirable but it works - see example below.
The example below (available in http://www.crufty.net/ftp/pub/sjg/gmk-tests.tar.gz ) consists of:
tests/GNUmakefile.inc tests/lib/GNUmakefile.inc tests/lib/a/GNUmakefile tests/lib/a/GNUmakefile.depend tests/lib/a/GNUmakefile.depend.host tests/lib/b/GNUmakefile tests/lib/b/GNUmakefile.depend tests/lib/b/GNUmakefile.depend.host tests/lib/d/GNUmakefile tests/lib/d/GNUmakefile.depend tests/lib/e/GNUmakefile tests/lib/e/GNUmakefile.depend tests/prog/GNUmakefile tests/prog/GNUmakefile.depend tests/tools/GNUmakefile.inc tests/tools/tool/GNUmakefile tests/tools/tool/GNUmakefile.depend.host
of which the following all list tests/tools/tool.host as a dependency:
tests/lib/a/GNUmakefile.depend tests/lib/e/GNUmakefile.depend tests/prog/GNUmakefile.depend
and tests/lib/a depends on tests/lib/b and tests/lib/e, while tests/lib/b depends on tests/lib/d and tests/lib/e.
Since tests/tools/tool/GNUmakefile.depend.host is:
DIRDEPS = \
tests/lib/a \
include ${GMKSYSDIR}/dirdeps.gmk
it needs tests/lib/a built for host. note that tests/lib/a/GNUmakefile.depend.host is just:
DIRDEPS = \
tests/lib/b \
include ${GMKSYSDIR}/dirdeps.gmk
(so tests/lib/b also needs to build for host) while tests/lib/a/GNUmakefile.depend is:
DIRDEPS = \
tests/tools/tool.host \
tests/lib/b \
tests/lib/e
include ${GMKSYSDIR}/dirdeps.gmk
Also note that there is no GNUmakefile.depend.host in tests/lib/b so we will re-use its GNUmakefile.depend.
Finally note that all of the GNUmakefile end up including tests/GNUmakefile.inc which has just:
all:
# sys.gmk already set this
_CURDIR ?= ${CURDIR}
PROG ?= $(notdir ${_CURDIR})
THING ?= ${PROG}
ifneq "${MAKELEVEL}" "0"
all: realbuild
realbuild:
@echo Building ${CURDIR}/${THING}
@echo Finished making ${RELDIR} for ${TARGET_SPEC}
endif
so we don't actually build anything, we just claim to.
As we can see below, everything get's built in the optimal order and nothing is visited more than once:
$ mk-i386 -C tests/prog gmake: Entering directory '/h/sjg/work/gmk/tests/prog' Checking /h/sjg/work/gmk/tests/lib/b for host ... gmake[1]: Entering directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/b' Building /h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/b/libb.a Finished making tests/lib/b for host gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/b' Checking /h/sjg/work/gmk/tests/lib/a for host ... gmake[1]: Entering directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/a' Building /h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/a/liba.a Finished making tests/lib/a for host gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/a' Checking /h/sjg/work/gmk/tests/tools/tool for host ... gmake[1]: Entering directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/tools/tool' Building /h/sjg/work/gmk/obj/freebsd11-amd64/tests/tools/tool/tool Finished making tests/tools/tool for host gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/tools/tool' Checking /h/sjg/work/gmk/tests/lib/d for i386 ... gmake[1]: Entering directory '/h/sjg/work/gmk/obj/i386/tests/lib/d' Building /h/sjg/work/gmk/obj/i386/tests/lib/d/libd.a Finished making tests/lib/d for i386 gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/i386/tests/lib/d' Checking /h/sjg/work/gmk/tests/lib/e for i386 ... gmake[1]: Entering directory '/h/sjg/work/gmk/obj/i386/tests/lib/e' Building /h/sjg/work/gmk/obj/i386/tests/lib/e/libe.a Finished making tests/lib/e for i386 gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/i386/tests/lib/e' Checking /h/sjg/work/gmk/tests/lib/b for i386 ... gmake[1]: Entering directory '/h/sjg/work/gmk/obj/i386/tests/lib/b' Building /h/sjg/work/gmk/obj/i386/tests/lib/b/libb.a Finished making tests/lib/b for i386 gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/i386/tests/lib/b' Checking /h/sjg/work/gmk/tests/lib/a for i386 ... gmake[1]: Entering directory '/h/sjg/work/gmk/obj/i386/tests/lib/a' Building /h/sjg/work/gmk/obj/i386/tests/lib/a/liba.a Finished making tests/lib/a for i386 gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/i386/tests/lib/a' Checking /h/sjg/work/gmk/tests/prog for i386 ... gmake[1]: Entering directory '/h/sjg/work/gmk/obj/i386/tests/prog' Building /h/sjg/work/gmk/obj/i386/tests/prog/prog Finished making tests/prog for i386 gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/i386/tests/prog' gmake: Leaving directory '/h/sjg/work/gmk/tests/prog' $
and it all works just as well in parallel (see below).
Also if we change the origin, the graph changes accordingly - only that which is needed is built:
$ mk-amd64 -j8 -C tests/lib/b gmake: Entering directory '/h/sjg/work/gmk/tests/lib/b' Checking /h/sjg/work/gmk/tests/lib/d for amd64 ... Checking /h/sjg/work/gmk/tests/lib/b for host ... gmake[1]: Entering directory '/h/sjg/work/gmk/obj/amd64/tests/lib/d' gmake[1]: Entering directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/b' Building /h/sjg/work/gmk/obj/amd64/tests/lib/d/libd.a Building /h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/b/libb.a Finished making tests/lib/d for amd64 gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/amd64/tests/lib/d' Finished making tests/lib/b for host gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/b' Checking /h/sjg/work/gmk/tests/lib/a for host ... gmake[1]: Entering directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/a' Building /h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/a/liba.a Finished making tests/lib/a for host gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/a' Checking /h/sjg/work/gmk/tests/tools/tool for host ... gmake[1]: Entering directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/tools/tool' Building /h/sjg/work/gmk/obj/freebsd11-amd64/tests/tools/tool/tool Finished making tests/tools/tool for host gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/tools/tool' Checking /h/sjg/work/gmk/tests/lib/e for amd64 ... gmake[1]: Entering directory '/h/sjg/work/gmk/obj/amd64/tests/lib/e' Building /h/sjg/work/gmk/obj/amd64/tests/lib/e/libe.a Finished making tests/lib/e for amd64 gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/amd64/tests/lib/e' Checking /h/sjg/work/gmk/tests/lib/b for amd64 ... gmake[1]: Entering directory '/h/sjg/work/gmk/obj/amd64/tests/lib/b' Building /h/sjg/work/gmk/obj/amd64/tests/lib/b/libb.a Finished making tests/lib/b for amd64 gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/amd64/tests/lib/b' gmake: Leaving directory '/h/sjg/work/gmk/tests/lib/b' $
Note too that when building for the pseudo machine host we use an objdir named for HOST_TARGET (set by mk) which helps ensure that a tree shared via NFS with lots of different machines will just work.
The mk tool is reading .sandbox-env to condition the environment and identify the top of the tree (sandbox):
export SB_MAKE=gmake export SRCTOP=$SB export GMKSYSDIR=$SB/gmk export MAKEFILES=$GMKSYSDIR/sys.gmk SB_PATH=$PATH
The MAKEFILES=$GMKSYSDIR/sys.gmk is key.
You can run the tests as:
$ mk -j8 -f gmk/dirdeps.gmk tests/prog.i386 tests/prog.amd64
and it will build tests/prog for both i386 and amd64 but all the host bits will still be built only once.
If we add DEBUG_DIRDEPS=1 we can get to see all the plumbing in action:
$ mk DEBUG_DIRDEPS=1 -j8 -f gmk/dirdeps.gmk tests/prog.i386 tests/prog.amd64 INCLUDED_FROM=/h/sjg/work/gmk/gmk/local.sys.gmk dirdeps: /h/sjg/work/gmk/tests/prog.amd64 /h/sjg/work/gmk/tests/prog.i386 _more_depends= /h/sjg/work/gmk/tests/prog/GNUmakefile.depend.amd64 /h/sjg/work/gmk/tests/prog/GNUmakefile.depend.i386 Looking for /h/sjg/work/gmk/tests/prog/GNUmakefile.depend.amd64 DEP_TARGET_SPEC=amd64 DEP_RELDIR=tests/prog -including /h/sjg/work/gmk/tests/prog/GNUmakefile.depend.inc ... dirdeps: DEP_RELDIR=tests/prog DEP_TARGET_SPEC=amd64 DIRDEPS=tests/tools/tool.host tests/lib/a tests/lib/e /h/sjg/work/gmk/tests/prog.amd64: /h/sjg/work/gmk/tests/tools/tool.host /h/sjg/work/gmk/tests/lib/a.amd64 /h/sjg/work/gmk/tests/lib/e.amd64 _more_depends= /h/sjg/work/gmk/tests/tools/tool/GNUmakefile.depend.host /h/sjg/work/gmk/tests/lib/a/GNUmakefile.depend.amd64 /h/sjg/work/gmk/tests/lib/e/GNUmakefile.depend.amd64 Looking for /h/sjg/work/gmk/tests/tools/tool/GNUmakefile.depend.host DEP_TARGET_SPEC=host DEP_RELDIR=tests/tools/tool -including /h/sjg/work/gmk/tests/tools/tool/GNUmakefile.depend.inc ... dirdeps: DEP_RELDIR=tests/tools/tool DEP_TARGET_SPEC=host DIRDEPS=tests/lib/a /h/sjg/work/gmk/tests/tools/tool.host: /h/sjg/work/gmk/tests/lib/a.host ...
and so on.
It is easier to read without that noise though:
$ mk -j8 -f gmk/dirdeps.gmk tests/prog.i386 tests/prog.amd64 Checking /h/sjg/work/gmk/tests/lib/b for host ... Checking /h/sjg/work/gmk/tests/lib/d for amd64 ... Checking /h/sjg/work/gmk/tests/lib/d for i386 ... gmake[1]: Entering directory '/h/sjg/work/gmk/obj/i386/tests/lib/d' gmake[1]: Entering directory '/h/sjg/work/gmk/obj/amd64/tests/lib/d' gmake[1]: Entering directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/b'
Note that since tests/lib/d and tests/lib/b (for host) do not have any dependencies (empty DIRDEPS) they get built first.
Building /h/sjg/work/gmk/obj/i386/tests/lib/d/libd.a Building /h/sjg/work/gmk/obj/amd64/tests/lib/d/libd.a Finished making tests/lib/d for i386 gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/i386/tests/lib/d' Building /h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/b/libb.a Finished making tests/lib/d for amd64 gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/amd64/tests/lib/d' Finished making tests/lib/b for host gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/b' Checking /h/sjg/work/gmk/tests/lib/a for host ... gmake[1]: Entering directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/a' Building /h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/a/liba.a Finished making tests/lib/a for host gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/a' Checking /h/sjg/work/gmk/tests/tools/tool for host ... gmake[1]: Entering directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/tools/tool' Building /h/sjg/work/gmk/obj/freebsd11-amd64/tests/tools/tool/tool Finished making tests/tools/tool for host gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/tools/tool' Checking /h/sjg/work/gmk/tests/lib/e for amd64 ... Checking /h/sjg/work/gmk/tests/lib/e for i386 ... gmake[1]: Entering directory '/h/sjg/work/gmk/obj/amd64/tests/lib/e' gmake[1]: Entering directory '/h/sjg/work/gmk/obj/i386/tests/lib/e' Building /h/sjg/work/gmk/obj/amd64/tests/lib/e/libe.a Building /h/sjg/work/gmk/obj/i386/tests/lib/e/libe.a Finished making tests/lib/e for amd64 gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/amd64/tests/lib/e' Finished making tests/lib/e for i386 gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/i386/tests/lib/e' Checking /h/sjg/work/gmk/tests/lib/b for amd64 ... Checking /h/sjg/work/gmk/tests/lib/b for i386 ... gmake[1]: Entering directory '/h/sjg/work/gmk/obj/amd64/tests/lib/b' gmake[1]: Entering directory '/h/sjg/work/gmk/obj/i386/tests/lib/b' Building /h/sjg/work/gmk/obj/amd64/tests/lib/b/libb.a Building /h/sjg/work/gmk/obj/i386/tests/lib/b/libb.a Finished making tests/lib/b for amd64 gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/amd64/tests/lib/b' Finished making tests/lib/b for i386 gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/i386/tests/lib/b' Checking /h/sjg/work/gmk/tests/lib/a for amd64 ... Checking /h/sjg/work/gmk/tests/lib/a for i386 ... gmake[1]: Entering directory '/h/sjg/work/gmk/obj/amd64/tests/lib/a' gmake[1]: Entering directory '/h/sjg/work/gmk/obj/i386/tests/lib/a' Building /h/sjg/work/gmk/obj/amd64/tests/lib/a/liba.a Building /h/sjg/work/gmk/obj/i386/tests/lib/a/liba.a Finished making tests/lib/a for amd64 gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/amd64/tests/lib/a' Finished making tests/lib/a for i386 gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/i386/tests/lib/a' Checking /h/sjg/work/gmk/tests/prog for i386 ... Checking /h/sjg/work/gmk/tests/prog for amd64 ... gmake[1]: Entering directory '/h/sjg/work/gmk/obj/i386/tests/prog' gmake[1]: Entering directory '/h/sjg/work/gmk/obj/amd64/tests/prog' Building /h/sjg/work/gmk/obj/i386/tests/prog/prog Building /h/sjg/work/gmk/obj/amd64/tests/prog/prog Finished making tests/prog for i386 gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/i386/tests/prog' Finished making tests/prog for amd64 gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/amd64/tests/prog' gmake: Nothing to be done for 'tests/prog.amd64'. $
This makefile handles the logic normally found in a top-level makefile to decide what to build.
For any given target dirdeps-targets.gmk looks for a directory of that name under ${SRCTOP}/targets/ and anything else in DIRDEPS_TARGETS_DIRS, that contains a GNUmakefile.depend* file.
Building on our example above:
$ cat targets/tests/GNUmakefile.depend
DIRDEPS= tests/prog
include ${GMKSYSDIR}/dirdeps.gmk
is all we need to:
$ mk -f gmk/dirdeps-targets.gmk tests Checking /homes/sjg/work/gmk/tests/lib/b for host ... gmake[1]: Entering directory '/homes/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/b' Building /homes/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/b/libb.a Finished making tests/lib/b for host .. .. gmake[1]: Entering directory '/homes/sjg/work/gmk/obj/amd64/tests/prog' Building /homes/sjg/work/gmk/obj/amd64/tests/prog/prog Finished making tests/prog for amd64 gmake[1]: Leaving directory '/homes/sjg/work/gmk/obj/amd64/tests/prog' $
This is really just a proof of concept.
I haven't used gmake seriously for over 20 years, so it is quite possible others could improve on my implementation.
The real dirdeps (with bmake) works incredibly well. This implementation provides only the basic functionality but is far more capable that I expected would be possible.
A key missing feature is the ability to filter DIRDEPS in weird and wonderful ways, and this implementation cannot automatically adapt to different requirements by simply tweaking TARGET_SPEC_VARS.
Filtering I address by including local.dirdeps-filter.gmk at the appropriate point. The implementation is left as an exercise for the suitably motivated reader ;-)
If interested; you can download this version from https://www.crufty.net/ftp/pub/sjg/gmk.tar.gz
Consider dirdeps.gmk and sys.gmk as the key bits, the rest are really just examples so I could exercise dirdeps.gmk
Improvements are always welcome.
| Author: | sjg@crufty.net |
|---|---|
| Revision: | $Id: gmake-dirdeps.txt,v d860ac70f438 2022-03-13 23:13:11Z sjg $ |