This doc describes building NetBSD using bmake in meta mode leveraging the work done building FreeBSD in meta mode
Junos (which is based on FreeBSD) has been built this way for over 10 years, and it works very well.
Note: while I normally talk of meta mode and using dirdeps.mk together, either can be used without the other.
In FreeBSD two separate controls are provided DIRDEPS_BUILD and META_MODE. Enabling DIRDEPS_BUILD implies META_MODE, but many developers use the latter even with the traditional build targets.
I first did this exercise with NetBSD in about 2015, and again in 2023, which was as much as anything a test of new a bootstrapping method. This was prompted by a failure to build NetBSD/current on an older NetBSD host.
There isn't a lot wrong with the NetBSD build as it is. At least for doing release builds and such. It builds in parallel ok, and can be cross-built on a number of different hosts. Thus I've not previously been seriously tempted to interfere with it.
However the prospect of doing a dev project on NetBSD at work was enough to prompt me to see what it would take to get the DIRDEPS_BUILD (which is hard to live without once you've tried it ;-) working on NetBSD.
The short answer is very little. Frankly despite planning to leverage work already done for FreeBSD, I was very pleasantly surprised to find how little effort it took to get bin/cat (and everything it needs) building. Most of that time was spend finding the various places that needed to be bootstrapped to link cat. See bootstrapping below to see how this step is now much simpler.
After building for i386 I then build for something else to see what the impact on the dependencies are. For FreeBSD I found with just parameterizing CSU_DIR a single Makefile.depend worked for 99% of the tree, and most places where Makefile.depend.${MACHINE} was needed it was due to generated files being named differently.
For NetBSD some of the dependencies varied quite a bit between i386 and evbarm, but nothing that cannot be handled by a little filtering - which the meta mode infrastructure provides for.
Fair question.... It is a better way of building a large tree of software.
Most of this is covered in building FreeBSD in meta mode But I'll briefly rehash the story here.
My goal is to be able to start a build from anywhere in a freshly checked out tree (eg. bin/cat) and have it just work. That means building everything else in the tree needed, in the right order, in parallel. To be able to do that for multiple target machines at the same time and even multiple target operating systems at the same time. All in a single pass of the tree.
All the above is handled by using dirdeps.mk (hence the DIRDEPS_BUILD option in FreeBSD) which is used by the initial make instance to compute a graph of tree dependencies from the current origin, and then go build them all in parallel.
The tree wide graph ensures that all leaf directories are visited in the correct order - no tree walks. In fact dirdeps.mk suppresses the behavior of bsd.subdir.mk.
Using dirdeps.mk we can greatly simplify the top-level build logic, which apart from share/mk/ is where we typically find a concentration of complexity:
NetBSD (2015):
$ wc -l Makefile build.sh
530 Makefile
2309 build.sh
2839 total
NetBSD (2023):
$ wc -l Makefile build.sh
540 Makefile
2589 build.sh
3129 total
That's a rather modest increase in almost 10 years!
FreeBSD (2015):
$ wc -l Makefile Makefile.inc1
525 Makefile
2196 Makefile.inc1
2721 total
FreeBSD (2023):
$ wc -l Makefile Makefile.inc1
808 Makefile
3633 Makefile.inc1
4441 total
Much more substantial.
The pre-meta mode Junos build had over 5000 lines of very dense makefile at the top-level. The original top-level makefiles for the meta mode build that replaced all that was less than 300 lines!
NetBSD/FreeBSD meta mode (2015):
$ wc -l targets/Ma* share/mk/dirdeps.mk
172 targets/Makefile
52 targets/Makefile.inc
46 targets/Makefile.xtras
644 share/mk/dirdeps.mk
914 total
FreeBSD meta mode (2023):
$ wc -l targets/Makefile* share/mk/dirdeps*.mk
116 targets/Makefile
54 targets/Makefile.inc
77 targets/Makefile.xtras
111 share/mk/dirdeps-options.mk
173 share/mk/dirdeps-targets.mk
959 share/mk/dirdeps.mk
1490 total
I included dirdeps.mk in the line count because that's where most of the complexity is, but of course we leverage it for all builds not just top-level ones.
dirdeps-targets.mk handles the task of finding a suitable Makefile.depend under targets/ and dirdeps-options.mk handles the dependency complication introduced by dozens of optional features.
In fact targets/Makefile isn't necessary at all. The inclusion of dirdeps-targets.mk can be done from local.sys.mk which is what I did in the more recent NetBSD exercise.
With the DIRDEPS_BUILD adding a new top-level target is as simple as creating a directory somewhere under targets/ and putting a Makefile.depend file there that lists the directories needed, the Makefile is often trivial - leveraging common packaging logic. These makefiles though are no more complex than any leaf makefile.
When make is run in meta mode, it creates a .meta file for each target being built. The meta file captures information that allows make to do a much better job.
Firstly we capture the expanded command line. This allows make to compare the command it might need to run with what was done last time, if anything changed the target is out-of-date.
Now sometimes you want to suppress that behavior so there are knobs to do so for a whole target or just one line.
This feature alone makes a huge difference to build reliability. It is no longer necessary to make targets depend on every makefile that might influence them. This allows us to avoid building things when nothing relevant has changed, while ensuring we never fail to rebuild when they have.
The captured command is also invaluable for debugging.
The next thing captured is any output from the command. This is mainly for human debugging purposes.
This extremely useful when you have 1000's of users who fail to log their builds and still want to you help them understand why their build failed.
The filemon module provides /dev/filemon which make will use if available.
When make runs a child, it opens /dev/filemon and gives it a temp file and the pid of the child. All successful syscalls (interesting to make) made by the child and its progeny will be recorded in the temp file. When the child exits, make appends the data to the meta file.
This data has two main uses. Firstly make itself uses it to better check if a target needs update. Secondly we can post-process the data for all sorts of purposes including extracting directory dependencies - which allows us to automate the collection of the data needed by dirdeps.mk.
For example bin/cat/Makefile.depend:
# Autogenerated - do NOT edit!
DIRDEPS = \
include \
lib/libpthread \
sys/arch/amd64/include \
sys/arch/x86/include \
sys/sys \
.include <dirdeps.mk>
.if ${.MAKE.LEVEL} > 0
# local dependencies - needed for -jN in clean tree
.endif
As we read the files read or executed, any we see in an object directory which is not ${.OBJDIR} indicates a directory that needs to be built before ${.CURDIR}. Being able to map object dirs to src dirs is important.
The local dependencies section is where we record any dependencies involving locally generated files - also gleaned from the syscall trace. This allows us to reliably build a clean tree in parallel while skipping any explict depend step.
The syscall trace is also invaluable for debugging weird build issues. For example spotting that your supposed captive build is reading headers from /usr/include or linking libs from /usr/local/lib. Or running a perl or python binary other than the one intended. Being able to look under the covers after a build has failed is immensely useful.
When make is checking if a target needs update and the normal makefile rules indicate it is up-to-date, we delve into the meta file. Here is cat.o.meta:
# Meta data file /h/NetBSD/5.X/obj/i386/bin/cat/cat.o.meta CMD @echo '# ' "compile " cat/cat.o CMD /var/obj/NetBSD/5.X/tools/NetBSD-5.2_STABLE-i386/bin/i386--netbsdelf-gcc -O2 -Wall -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wno-sign-compare -Wno-traditional -Wreturn-type -Wswitch -Wshadow -Wcast-qual -Wwrite-strings -Wextra -Wno-unused-parameter -nostdinc -I/h/NetBSD/5.X/obj/stage/i386/usr/include -nostdinc -isystem /h/NetBSD/5.X/obj/stage/i386/usr/include -c /h/NetBSD/5.X/src/bin/cat/cat.c CWD /h/NetBSD/5.X/obj/i386/bin/cat TARGET cat.o -- command output -- # compile cat/cat.o -- filemon acquired metadata -- # filemon version 4 # Target pid 2050 V 4 E 8142 /bin/sh R 8142 /etc/ld.so.conf R 8142 /lib/libedit.so.2 R 8142 /lib/libtermcap.so.0 R 8142 /lib/libc.so.12 X 8142 0 # Bye bye E 5426 /var/obj/NetBSD/5.X/tools/NetBSD-5.2_STABLE-i386/bin/i386--netbsdelf-gcc R 5426 /etc/ld.so.conf R 5426 /usr/lib/libc.so.12 R 5426 /var/tmp//ccKCUD9I.s W 5426 /var/tmp//ccKCUD9I.s E 15146 /h/obj/NetBSD/5.X/tools/NetBSD-5.2_STABLE-i386/bin/../libexec/gcc/i386--netbsdelf/4.1.3/cc1 R 15146 /etc/ld.so.conf R 15146 /usr/lib/libc.so.12 R 15146 /h/NetBSD/5.X/src/bin/cat/cat.c R 15146 /var/tmp//ccKCUD9I.s W 15146 /var/tmp//ccKCUD9I.s R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/cdefs.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/cdefs.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/cdefs_elf.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/param.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/null.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/inttypes.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/stdint.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/int_types.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/int_mwgwtypes.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/int_limits.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/int_const.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/wchar_limits.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/int_fmtio.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/types.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/types.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/ansi.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/ansi.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/endian.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/endian.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/types.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/endian_machdep.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/bswap.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/byte_swap.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/bswap.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/bswap.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/fd_set.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/pthread_types.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/syslimits.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/signal.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/sigtypes.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/satypes.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/siginfo.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/signal.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/trap.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/x86/trap.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/ucontext.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/mcontext.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/param.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/limits.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/stat.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/time.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/select.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/time.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/time.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/ctype.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/err.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/errno.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/errno.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/fcntl.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/locale.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/stdio.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/stdlib.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/string.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/strings.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/string.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/unistd.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/unistd.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h X 15146 0 E 18126 /h/obj/NetBSD/5.X/tools/NetBSD-5.2_STABLE-i386/bin/../lib/gcc/i386--netbsdelf/4.1.3/../../../../i386--netbsdelf/bin/as R 18126 /etc/ld.so.conf R 18126 /usr/lib/libc.so.12 R 18126 cat.o W 18126 cat.o R 18126 /var/tmp//ccKCUD9I.s X 18126 0 D 5426 /var/tmp//ccKCUD9I.s X 5426 0 # Bye bye
I bet you didn't know it takes all those headers to compile cat.o. That's 79 headers (54 unique), not what you would expect from just looking at cat.c.
Anyway, as noted above the first thing we do is check if the command line changed, any change to CFLAGS would do it. If it did - we are done - out-of-date.
Otherwise we look at the syscall trace. This is formatted so that it is trivial to parse by even a shell script.
In fact I have a collection of shell functions for extracting useful data from meta files - like extracting the command so it can be re-run easily, spliting the command into individual words one per line so it is much easier to compare with another using diff(1). Extracting the list of files read - all resolved to absoulte paths, and so on.
In the above example it is simple, we just need to look at all the E (exec) and R (read) entries; if any of the files read/executed are newer the target is out-of-date.
Sometimes the syscall trace is more complex, so make needs to follow the F (fork) and C (chdir) syscalls by pid so it knows what the cwd of each is so if it sees a file opened by a relative path it can find it.
If a file previously written is missing (and the syscall trace does not show it being D deleted), the target is out-of-date.
Of course make will ignore some paths (eg. anything in /tmp) and you can provide it a list of path prefixes to ignore.
The ability for make to look under the covers like this allows it to do a much better job of ensuring things are re-built when they need to be.
You can use the debug flag -dM to have make tell you what it is about a meta file that made it decide the target is out-of-date.
The filemon module is currently available in NetBSD and FreeBSD and a version for Linux is available from https://github.com/trixirt/filemon-linux.git
In Jan 2020 NetBSD introduced filemon_ktrace as an inteface to fktrace(2) for use by make eliminating the need for filemon(4). The old api is retained as filemon_dev for use on FreeBSD and others.
filemon_ktrace adds a little overhead in the form of an extra file descriptor per job.
With luck the filemon_ktrace model can be leveraged by other syscall trace mechanisms on other systems - especially for Linux where there is currently no good alternative.
Initially I only had to touch one makefile outside of share/mk and that was include/Makefile because I wanted an obj dir:
Index: include/Makefile
===================================================================
RCS file: /cvsroot/src/include/Makefile,v
retrieving revision 1.140
diff -u -p -r1.140 Makefile
--- include/Makefile 11 Dec 2013 01:24:08 -0000 1.140
+++ include/Makefile 13 May 2015 18:53:29 -0000
@@ -3,7 +3,9 @@
# Doing a make includes builds /usr/include
+.if ${MK_DIRDEPS_BUILD:Uno} == "no"
NOOBJ= # defined
+.endif
# Missing: mp.h
by contrast I had to quite heavily tweak the FreeBSD equivalent. This was a pleasant surprise.
Though since include/Makefile isn't making some of the symlinks in ${INCSDIR} that are obviously needed we had to compensate in local.final.mk (see below).
In fact for the 2023 revisit apart from the above change to include/Makefile the only existing files touched were:
share/mk/bsd.files.mk share/mk/bsd.init.mk share/mk/bsd.sys.mk share/mk/sys.mk external/gpl3/gcc/lib/libgcc/Makefile.inc
in each case only one or two lines such as:
.-include <local.dirdeps-build.mk>
local.dirdeps-build.mk is normally included by dirdeps.mk at level 1+ (building), but while bootstrapping there may not be any Makefile.depend files to include dirdeps.mk.
I should mention that I consider building toolchains out of scope for this exercise. Building toolchains is a corner case, and should happen rarely enough to not be worth optimizing - or even integrating.
BTW for 5.X I hacked build.sh to not build the old make, since my /usr/bin/make is always the latest from current.
But in general I just use the toolchain as built by the normal build.
Our goal is to build the tree in a single pass. This means we cannot have any circular dependencies.
In NetBSD current of 2015 I had to break cycles involving libpthread which is achieved by adding:
.if ${MK_DIRDEPS_BUILD:Uno} == "yes"
# avoid a circular dependency with libpthread
CPPFLAGS+= -I${SRCTOP}/lib/libpthread
.endif
to each of:
external/gpl3/gcc/lib/libgcc/Makefile.inc lib/csu/Makefile lib/libc/Makefile
Without the direct -I each of these needs libpthread built first so that pthread_types.h is staged, but libpthread cannot be built without all the above.
In 2023 I did not find that necessary. Though I did have to fix external/gpl3/gcc/lib/libgcc/Makefile.inc to avoid multiple variants of libgcc from trying to stage the same header:
Index: external/gpl3/gcc/lib/libgcc/Makefile.inc
===================================================================
RCS file: /cvsroot/src/external/gpl3/gcc/lib/libgcc/Makefile.inc,v
retrieving revision 1.52
diff -u -p -r1.52 Makefile.inc
--- external/gpl3/gcc/lib/libgcc/Makefile.inc 22 Jul 2022 21:59:11 -0000 1.52
+++ external/gpl3/gcc/lib/libgcc/Makefile.inc 11 May 2023 06:12:46 -0000
@@ -255,3 +255,7 @@ CPPFLAGS+= -I${BINBACKENDOBJ}
#.if !empty(LIBGCC_MACHINE_ARCH:Mearm*)
COPTS.unwind-dw2.c+= -Wno-discarded-qualifiers
#.endif
+
+.if ${MK_STAGING} == "yes" && ${LIB:Mgcc*} != "" && ${LIB} != "gcc"
+INCS=
+.endif
Back in 2015 it took only a couple of days effort to have most of userland building (400+ apps and all the libs they need) as well as building kernels. Of course I was able to re-use targets/* from FreeBSD.
For the 2023 exercise it took less than a day to get bin/cat building, and I didn't do more, as I'd achieved my purpose.
| Author: | sjg@crufty.net |
|---|---|
| Revision: | $Id: netbsd-meta-mode.txt,v 8c676d3327df 2023-05-11 23:00:35Z sjg $ |