This doc is all about building using dirdeps.mk, which consumes lists of DIRDEPS from Makefile.depend files scattered throughout the build tree.
It is worth noting that while we generally talk of dirdeps in the context of a meta mode build, the two concepts are actually orthogonal. In the FreeBSD build two separate options are used; DIRDEPS_BUILD and META_MODE.
Using bmake(1) in meta mode, allows for more accurate update builds, and provides a convenient means of automatically updating DIRDEPS, but dirdeps.mk itself does not care about any of that. You can use manually (or semi-manually) maintained dirdeps just as well. For this doc we will assume that Makefile.depend files are manually updated.
First off, just what are DIRDEPS? Very simply it is a list of directories (relative to SRCTOP), which must be built prior to the current directory.
For example here is bin/cat/Makefile.depend:
# Autogenerated - do NOT edit!
DIRDEPS = \
gnu/lib/libgcc \
include \
include/xlocale \
lib/${CSU_DIR} \
lib/libc \
.include <dirdeps.mk>
.if ${DEP_RELDIR} == ${_DEP_RELDIR}
# local dependencies - needed for -jN in clean tree
.endif
The variable DIRDEPS lists all the directories (such as lib/libc) which must be built before we can build this directory (bin/cat).
Before including each Makefile.depend file, dirdeps.mk sets DEP_RELDIR to the path relative to SRCTOP where it was found.
It sets _DEP_TARGET_SPEC to the suffix of the target of interest, as well as DEP_MACHINE and any other DEP_* variables according to TARGET_SPEC_VARS, in case the Makefile.depend file uses them.
It also resets DIRDEPS just in case a manually edited Makefile.depend file fails to do so.
In earlier versions dirdeps.mk relied on the Makefile.depend file to set DEP_RELDIR based on .PARSEDIR, but doing it in dirdeps.mk defends against bugs in manually edited Makefile.depend files.
dirdeps.mk will use DEP_RELDIR and DIRDEPS to build a dependency graph of the tree.
As in the previous example, a Makefile.depend file will exist in each directory we need to build, and lists the DIRDEPS for that particular directory. This allows us to ensure the tree is always built in the correct order.
We use the term Makefile.depend in the generic sense - but there can be a number of files of this format depending on how complex the build is, and in fact Makefile.depend is just the default value of .MAKE.DEPENDFILE_PREFIX. See sys.dependfile.mk
The last part of the example above, is a feature of auto-generated Makefile.depend* where we capture local dependencies - of generated srcs such as those generated by yacc and lex. Such dependencies are often forgotten by developers and are a common cause of inablility to build in parallel.
A downside is that those local dependencies can vary per architecture, either causing churn or a need to use MACHINE specific depend files like Makefile.depend.amd64 etc.
This makefile is read via sys.mk, and looks in the current directory (.CURDIR) to find the first entry in .MAKE.DEPENDFILE_PREFERENCE that exists. The default list is:
# All depend file names should start with this
.MAKE.DEPENDFILE_PREFIX ?= Makefile.depend
# The order of preference: we will use the first one of these we find
# It usually makes sense to order from most specific to least.
.MAKE.DEPENDFILE_PREFERENCE ?= \
${.CURDIR}/${.MAKE.DEPENDFILE_PREFIX}.${MACHINE} \
${.CURDIR}/${.MAKE.DEPENDFILE_PREFIX}
# Normally the 1st entry is our default choice
# Another useful default is ${.MAKE.DEPENDFILE_PREFIX}
.MAKE.DEPENDFILE_DEFAULT ?= ${.MAKE.DEPENDFILE_PREFERENCE:[1]}
That is if a Makefile.depend qualified with ${MACHINE} exists that will be used in preference to the more generic name. A more complex preference list might be:
.MAKE.DEPENDFILE_PREFERENCE ?= \
${.CURDIR}/${.MAKE.DEPENDFILE_PREFIX}.auto.${MACHINE} \
${.CURDIR}/${.MAKE.DEPENDFILE_PREFIX}.manual.${MACHINE} \
${.CURDIR}/${.MAKE.DEPENDFILE_PREFIX}.auto \
${.CURDIR}/${.MAKE.DEPENDFILE_PREFIX}.manual \
${.CURDIR}/${.MAKE.DEPENDFILE_PREFIX}
which could be useful if SCM triggers are to allow Makefile.depend.auto* to be committed with less restriction than manually edited ones.
In the FreeBSD build, a plain Makefile.depend suffices for the vast majority of the tree - with lib/${CSU_DIR} hiding the most common machine specific dependency.
For a small number of directories though, either DIRDEPS or the local dependencies are quite machine specific, and for these a Makefile.depend.${MACHINE} is more appropriate.
Sometimes we want to build some of the tree for both the target machine and the build host itself. We use the pseudo machine host to represent the build host, and invariably use a qualified file Makefile.depend.host because the DIRDEPS needed is usually a small subset of that needed when building for the target.
Note: when building for host I use an objdir named for HOST_TARGET a variable derrived from the OS name and architecture such as netbsd5-i386, freebsd10-amd64 or linux3-x86_64. This makes it safe to simultaneously build a tree shared via NFS on different hosts. It is also intended to not be confused with any objdir derived from TARGET_SPEC_VARS.
In some cases we want to build part of the tree for the build host, but the code perhaps only works 32bit. In this case we use the pseudo machine host32 and HOST_TARGET32 (eg. freebsd10-i386 if HOST_TARGET is freebsd10-amd64).
Due to this corner case, we use:
.if ${MACHINE:Nhost*} == ""
in Makefiles that want to test if we are building for either host or host32.
This is a rather complicated makefile, though what it does is conceptually quite simple. As our example above shows, each Makefile.depend will read dirdeps.mk.
Most of what dirdeps.mk does is only interesting to bmake level 0 (ie. the first instance of bmake). In fact the dirdeps build works best if level 0 is reserved for orchestration only.
In simple terms; each time dirdeps.mk is read, we have DIRDEPS set by Makefile.depend and we have DEP_RELDIR, _DEP_TARGET_SPEC and _DEP_MACHINE passed by the previous instance of dirdeps.mk (or we deduce them from current environment). All DEP_* variables are set according to _DEP_TARGET_SPEC. Some of the discussion below assumes that is only DEP_MACHINE.
The first step is to split DIRDEPS into qualified and unqualified entries. An unqualified dirdep is one which should be built for any value of ${MACHINE}. A qualified dirdep is one which specifies the machine it should be built for. This is common for host tools or pseudo targets which trigger the building of things for multiple target machines.
As noted above we use the pseudo machine host when building for the build host itself. Another pseudo machine we often use is common; which is handy for machine independent code generation and similar targets, which need only be processed once regardless of the number of architectures they need to be built for.
All the DIRDEPS are prefixed with ${SRCTOP}/ and the unqualified entries have a .${TARGET_SPEC} qualifier added for each tuple that we are building this directory for.
Thus after the first step we have created a list of absolute paths with some .${TARGET_SPEC} value appended.
We then create a dependency for ${SRCTOP}/${DEP_RELDIR}.${DEP_TARGET_SPEC} (which represents the directory and machine we are currently considering DIRDEPS for) on that list, and associate all of them with the _DIRDEP_USE target which will arrange to build them.
We then look at each directory we need to build, and look for a Makefile.depend file there to continue the process. We skip any directory that we have already examined for a given TARGET_SPEC. We always set _DEP_MACHINE according to what we want, so that regardless of the entry found via .MAKE.DEPENDFILE_PREFERENCE the next dirdeps.mk will know the correct value for DEP_MACHINE. Since DEP_RELDIR is reset each time we read a Makefile.depend file which then reads dirdeps.mk the process continues and thus builds a complete graph of dependencies with respect to the directory we started in.
For those that need concrete examples, we can look at the dirdeps.cache for bin/cat (on NetBSD):
# Autogenerated - do NOT edit!
BUILD_DIRDEPS=no
.include <dirdeps.mk>
# bin/cat.i386
dirdeps: \
${SRCTOP}/bin/cat.i386 \
${SRCTOP}/gnu/lib/crtstuff4.i386 \
${SRCTOP}/gnu/lib/libgcc4/libgcc.i386 \
${SRCTOP}/gnu/lib/libgcc4/libgcc_eh.i386 \
${SRCTOP}/include.i386 \
${SRCTOP}/lib/csu/i386_elf.i386 \
${SRCTOP}/lib/libc.i386 \
${SRCTOP}/lib/libpthread.i386 \
${SRCTOP}/sys/arch/i386/include.i386 \
${SRCTOP}/sys/arch/x86/include.i386 \
${SRCTOP}/sys/sys.i386
${SRCTOP}/bin/cat.i386: _DIRDEP_USE
${SRCTOP}/gnu/lib/crtstuff4.i386: _DIRDEP_USE
${SRCTOP}/gnu/lib/libgcc4/libgcc.i386: _DIRDEP_USE
${SRCTOP}/gnu/lib/libgcc4/libgcc_eh.i386: _DIRDEP_USE
${SRCTOP}/include.i386: _DIRDEP_USE
${SRCTOP}/lib/csu/i386_elf.i386: _DIRDEP_USE
${SRCTOP}/lib/libc.i386: _DIRDEP_USE
${SRCTOP}/lib/libpthread.i386: _DIRDEP_USE
${SRCTOP}/sys/arch/i386/include.i386: _DIRDEP_USE
${SRCTOP}/sys/arch/x86/include.i386: _DIRDEP_USE
${SRCTOP}/sys/sys.i386: _DIRDEP_USE
${SRCTOP}/bin/cat.i386: \
${SRCTOP}/gnu/lib/crtstuff4.i386 \
${SRCTOP}/gnu/lib/libgcc4/libgcc.i386 \
${SRCTOP}/gnu/lib/libgcc4/libgcc_eh.i386 \
${SRCTOP}/include.i386 \
${SRCTOP}/lib/csu/i386_elf.i386 \
${SRCTOP}/lib/libc.i386 \
${SRCTOP}/lib/libpthread.i386 \
${SRCTOP}/sys/arch/i386/include.i386 \
${SRCTOP}/sys/arch/x86/include.i386 \
${SRCTOP}/sys/sys.i386
skip a bit ....
# lib/libc.i386
dirdeps: \
${SRCTOP}/common/include/prop.i386 \
${SRCTOP}/include.i386 \
${SRCTOP}/include/rpc.i386 \
${SRCTOP}/lib/libpthread.i386 \
${SRCTOP}/sys/arch/i386/include.i386 \
${SRCTOP}/sys/arch/x86/include.i386 \
${SRCTOP}/sys/sys.i386
${SRCTOP}/include/rpc.i386: _DIRDEP_USE
${SRCTOP}/lib/libc.i386: \
${SRCTOP}/common/include/prop.i386 \
${SRCTOP}/include.i386 \
${SRCTOP}/include/rpc.i386 \
${SRCTOP}/lib/libpthread.i386 \
${SRCTOP}/sys/arch/i386/include.i386 \
${SRCTOP}/sys/arch/x86/include.i386 \
${SRCTOP}/sys/sys.i386
.info ${.newline}${TRACER}Makefiles read: total=41 depend=14 dirdeps=14
we can see that ${SRCTOP}/include.${MACHINE} would need to be built before ${SRCTOP}/lib/libc.${MACHINE} which is built before ${SRCTOP}/bin/cat.${MACHINE}.
The time it takes for the level 0 bmake to compute the tree graph is directly proportional to the number of Makefile.depend files to be read. It can be as little as a second, or as much as several minutes (dirdeps.mk outputs stats so we can track this):
Makefiles read: total=38 depend=11 dirdeps=16 seconds=0 Makefiles read: total=11609 depend=11573 dirdeps=13560 seconds=200
If we enable using DIRDEPS_CACHE, then the computed graph is saved and can be re-used see example above.
Naturally there is a .meta files so if any of the Makefile.depend files are updated we have to re-compute the graph, but since saving the cache adds only a few seconds overhead and can save you several minutes when it is up to date, and as a bonus it can be invaluable for diagnosing why things you didn't expect to be built, were.
It is generally considered evil to set anything but DIRDEPS via a Makefile.depend file. For any rule however there are exceptions.
For such cases, special handling is needed when DIRDEPS_CACHE is being generated, since this happens in a sub-make, and thus any .export will not be seen by the rest of the build.
Fortunately dirdeps.mk handles this for us, just add the variables to DEP_EXPORT_VARS:
WITH_SOME_KNOB=1 # we need to enable SOME_KNOB
DEP_EXPORT_VARS+= WITH_SOME_KNOB
# incase we have old version
.export ${DEP_EXPORT_VARS}
This target macro is associated with each expanded dirdep - absolute path plus .${TARGET_SPEC} extension.
The target's extension describes the .${TARGET_SPEC} it should be built for. We can thus set MACHINE and TARGET_SPEC to the extension of the target before running bmake in the appropriate directory.
Also since the target will never exist (because of the .${TARGET_SPEC} extension) it is always out of date and will always be visited. Which is exactly what we want.
As noted above, we normally only care about setting ${MACHINE} for each sub-make. The default TARGET_SPEC_VARS is just MACHINE, but other variables can be added such that TARGET_SPEC is set to a tuple for each sub-make, though we expect that MACHINE is always the first element of that tuple.
In this case sys.mk is expected to decompose ${TARGET_SPEC} and set the component variables appropriately. The coordination is not hard since dirdeps.mk is using the TARGET_SPEC_VARS list set by sys.mk.
While this makes dirdeps.mk more complex, it adds great flexibility. For example building for multiple target operating systems at the same time (and multiple machines for each TARGET_OS).
If MK_DIRDEPS_CACHE is "yes", and NO_DIRDEPS is not defined then dirdeps.mk uses a sub-make to process all the Makefile.depend* files and capture the result in DIRDEPS_CACHE which defaults to ${OBJTOP}/dirdeps-cache.${.TARGETS} (suitably transformed).
We rely on meta mode for this to work well since the .meta file allows us to know if the cache is out-of-date, and thus regenerate it.
For a large target that might take several minutes to compute the tree dependencies, an up-to-date cache allows the build to get going in only a few seconds.
Sometimes, a tree is so big that computing the dependency graph for some targets can take an unacceptible amount of time.
One option to mitigate this is use of a STATIC_DIRDEPS_CACHE. For example:
targets/pseudo/production/Makefile.dirdeps.cache
is essentially a copy of:
$OBJTOP/dirdeps.cache.production
When dirdeps-targets.mk finds the initial DIRDEP targets/pseudo/production for the target production it checks if a Makefile.dirdeps.cache file exists in that directory and sets STATIC_DIRDEPS_CACHE to point to it.
If MK_STATIC_DIRDEPS_CACHE is yes, then STATIC_DIRDEPS_CACHE will be used instead of DIRDEPS_CACHE which can save considerable build time.
Managing a STATIC_DIRDEPS_CACHE however can be a pain so dirdeps-cache-update.mk takes care of it to the extent possible.
Orchestrating a complex build - eg for multiple MACHINE and TARGET_OS tuples at the same time is made far simpler with dirdeps.mk.
The dirdeps build works best when level 0 (the 1st instance of bmake) is reserved for orchestration.
This implies that sys.mk and friends should avoid setting anything which is specific to MACHINE or any of the other TARGET_SPEC_VARS.
An exception would be things specific to the build host which will not change throughout the build.
We should be especially careful that anything we .export from level 0 is safe for the entire build.
Anything which is specific to any of TARGET_SPEC_VARS should only be exported when level > 0, since by definition it can only affect a single sub-make at that point.
Each time dirdeps.mk is read, it sets DEP_RELDIR and DEP_* for TARGET_SPEC_VARS (just MACHINE by default), in all but the 1st case, these are passed in via _DEP_TARGET_SPEC set by the previous dirdeps.mk.
It then reads local.dirdeps.mk which provides an opportunity to tweak DIRDEPS for the currently considered directory. This can make it feasible to share one set of Makefile.depend files for different architectures.
The main goal of local.dirdeps.mk is to avoid any temptation to touch dirdeps.mk.
Like dirdeps.mk, local.dirdeps.mk can use a test for the target _DIRDEP_USE to know wether we are being read for the first time or not.
We can set generic things like DIRDEPS_FILTER or we can set things like DIRDEPS_FILTER.host which will only be applicable when DEP_MACHINE is "host" (ie. we are building something for the build host itself).
For example:
.if !target(_DIRDEP_USE)
# first time here, do one off initialization
# never build these for "host"
SKIP_DIR.host = \
Nlib/csu* \
Nlib/libc
# CSU_DIR.i386 etc are set in *sys.mk if necessary
# and CSU_DIR = ${CSU_DIR.${MACHINE_ARCH}}
# but here we need it to be selected by DEP_MACHINE_ARCH
# these allow us to use ${CSU_DIR} in DIRDEPS
DEP_MACHINE_ARCH = ${MACHINE_ARCH.${DEP_MACHINE}}
CSU_DIR.${DEP_MACHINE_ARCH} ?= csu/${DEP_MACHINE_ARCH}
CSU_DIR = ${CSU_DIR.${DEP_MACHINE_ARCH}}
As above, variables used by dirdeps.mk are qualified with DEP_* which are reset for the current DEP_TARGET_SPEC we are processing.
If we were doing something really complicated like building for multiple TARGET_OS at the same time, we might want to have local.dirdeps.mk include something like local.dirdeps.${DEP_TARGET_OS}.mk to reset any complex filters needed:
.endif # !target(_DIRDEP_USE)
# this section re-processed for each DEP_TARGET_SPEC
# pick up per TARGET_OS filters etc
.-include <local.dirdeps.${DEP_TARGET_OS}.mk>
Each time we read dirdeps.mk we reset DEP_DIRDEPS_FILTER and DEP_SKIP_DIR to include any variables qualified with DEP_*. For example:
DEP_SKIP_DIR = ${SKIP_DIR} \
${SKIP_DIR.${DEP_TARGET_SPEC}:U} \
${TARGET_SPEC_VARS:@v@${SKIP_DIR.${DEP_$v}:U}@} \
${SKIP_DIRDEPS.${DEP_TARGET_SPEC}:U} \
${TARGET_SPEC_VARS:@v@${SKIP_DIRDEPS.${DEP_$v}:U}@}
In a generic code base like FreeBSD, there can be many optional features which can be enabled via knobs. This poses a problem for dirdeps.mk
The solution is dirdeps-options.mk which allows for a Makefile.depend.options to add optional DIRDEPS and to supress the capture of those in Makefile.depend.
This makefile is to be included by a top-level makefile, to handle the task of finding an initial DIRDEP corresponding to the target requested.
It looks in the directories listed in DIRDEPS_TARGETS_DIRS (default: targets targets/pseudo) So for the target production it might find:
targets/pseudo/production
and the Makefile.depend file there is used to start the dirdeps computation.
As noted above a Makefile.dirdeps.cache might also exist and be used to skip the overhead of computing dirdeps.
Managing a static dirdeps cache (Makefile.dirdeps.cache) can be painful.
It can of course be done manually, but that tends to be unreliable. This makefile provides for two means of automatically updating such a cache. The behavior is controlled by a set of options:
MK_STATIC_DIRDEPS_CACHE_UPDATE
If yes we want to update the STATIC_DIRDEPS_CACHE.
MK_STATIC_DIRDEPS_CACHE
If this is also yes then we are driving the build from the STATIC_DIRDEPS_CACHE and we want to generate an update in parallel with the build.
This is done leveraging the cache update magic in dirdeps.mk, we simply need to launch it appropriately.
If MK_STATIC_DIRDEPS_CACHE is no but MK_DIRDEPS_CACHE is yes, then a dynamic cache will be generated at the start of the build and we simply need to update STATIC_DIRDEPS_CACHE from it when the build gets to the final DIRDEP which of course is where the STATIC_DIRDEPS_CACHE is found.
To do all that dirdeps-cache-update.mk needs to be included from local.dirdeps.mk and from a Makefile or Makefile.inc when the final target directory is visited. Eg. local.dirdeps.mk:
.if !target(_DIRDEP_USE)
# we are the 1st makefile
.if defined(STATIC_DIRDEPS_CACHE) && exists(${STATIC_DIRDEPS_CACHE})
.include <dirdeps-cache-update.mk>
.endif
and targets/Makefile.inc:
.if ${.MAKE.LEVEL} > 0 && defined(STATIC_DIRDEPS_CACHE)
.include <dirdeps-cache-update.mk>
.endif
Using dirdeps.mk makes it simple to have a very complex build just work. There are no tree walks involved, the needed leaf directories are visited directly in the correct order. The build can be initiated from anywhere and gain the full benefit.
| Author: | sjg@crufty.net |
|---|---|
| Revision: | $Id: dirdeps.txt,v c592e473486a 2022-04-23 22:15:49Z sjg $ |