more reliable update builds

Provided filemon is being used, the .meta files created during a build, allow bmake much greater visibility into what happens during the creation of a target. This allows it to be more thorough when deciding if a target is out-of-date.

captured command line

Even without filemon, the .meta file captures the expanded command line used last time, and by default any change to that command line causes the target to be out-of-date.

The comparison of command lines can be controlled at multiple levels; per build, makefile, target or even line within a target script. The last is perhaps the most useful.

Any command that that references the special variable .OODATE cannot be usefully compared and so isn't. This can be leveraged by using .OODATE in such a way that it does not affect the command: ${.OODATE:M.no.cmp.} will expand to nothing (change the match string if .no.cmp. might actually appear ;-)

captured output

The .meta files capture the output from the commands that generate a target.

This allows a much cleaner - and easier to read build log, and even for people who refuse to log the output of builds, the vital clues are captured when something goes wrong.

tree dependencies

When filemon is used, the .meta files capture lots of useful data, which can be used in many ways, but one of the most obvious is learning the dependencies of one directory on others.

simplify the build

This is one of the key benefits. This is mostly a benefit of using dirdeps.mk, though many individual makefiles can be simplified by use of meta mode.

The use of meta mode also allows easy automation of the data needed by dirdeps.mk, so we usually just refer to this as the meta mode build. The dirdeps.mk or meta mode build is very easy to understand.

Look at the top-level makefiles of any BSD system today, and try to understand exactly what they do. It can be a challenge indeed. Ignoring comments and blank lines, FreeBSD 10's top-level Makefile and Makefile.inc1 have more than 1800 lines.

The pre-meta-mode Junos build had over 5000 lines in its top-level makefiles.

By contrast, Makefile, pkgs/Makefile and pkgs/Makefile.inc which drive the Junos meta mode build were until recently less than 130 lines.

Of course I am not counting more than 10k autogenerated Makefile.depend* files which make it possible, but;

a/ they are autogenerated and each by itself is easy to understand

b/ they are not only used for top-level builds

In fact in this model, the top-level build is little different from any other.

Not to mention that top-level builds can often be dispensed with.

We use the example below of make -j8 -C bin/cat in a clean tree.

tree dependencies

Everyone is familiar with doing make depend to capture the dependencies that a directory has on the files that it uses, to help ensure that re-running make will do the right thing.

What we need is the same level of information for the tree as a whole. As detailed in Building BSD with meta mode I've implemented that a couple of ways in the Junos build, and meta mode is the current method used.

meta files

In meta mode, for most targets bmake creates a .meta file to capture information like the expanded command used, any command output (useful for debugging), and a record of all the successful system calls that are interesting to make:

# Meta data file /var/obj/projects-bmake/amd64/bin/cat/cat.o.meta
CMD cc -O2 -pipe  -nostdinc -isystem /var/obj/projects-bmake/stage/amd64/usr/include -isystem /var/obj/projects-bmake/stage/amd64/usr/include/clang/3.2 -std=gnu99 -Qunused-arguments -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -Wno-empty-body -Wno-string-plus-int -c /b/sjg/work/FreeBSD/projects-bmake/src/bin/cat/cat.c
CMD ctfconvert -L VERSION cat.o
CWD /var/obj/projects-bmake/amd64/bin/cat
TARGET cat.o
-- command output --
-- filemon acquired metadata --
# filemon version 4
# Target pid 42504
# Start 1363631731.803698
V 4
F 42504 42529
E 42529 /bin/sh
R 42529 /var/run/ld-elf.so.hints
R 42529 /lib/libedit.so.7
R 42529 /lib/libncurses.so.8
R 42529 /lib/libc.so.7
S 42529 .
S 42529 /var/obj/projects-bmake/amd64/bin/cat
S 42529 /usr/bin/cc
F 42529 42530
E 42530 /usr/bin/cc
S 42530 /usr/bin/cc
S 42530 /usr/bin/cc
F 42530 42533
E 42533 /usr/bin/cc
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/clang/3.2
R 42533 cat.o-24d10fa0
W 42533 cat.o-24d10fa0
S 42533 /b/sjg/work/FreeBSD/projects-bmake/src/bin/cat
R 42533 /b/sjg/work/FreeBSD/projects-bmake/src/bin/cat/cat.c
S 42533 /b/sjg/work/FreeBSD/projects-bmake/src/bin/cat/cat.c
R 42533 /b/sjg/work/FreeBSD/projects-bmake/src/bin/cat/cat.c
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/cdefs.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/param.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_null.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/types.h
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/endian.h
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/x86
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/x86/endian.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_types.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/_types.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/x86/_types.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_pthreadtypes.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_stdint.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/select.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_sigset.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_timeval.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/timespec.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_timespec.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/syslimits.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/signal.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/_limits.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/x86/_limits.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/signal.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/trap.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/x86/trap.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/param.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/_align.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/x86/_align.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/limits.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/stat.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/time.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/time.h
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/xlocale
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/xlocale/_time.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/socket.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_iovec.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_sockaddr_storage.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/un.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/errno.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/ctype.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/_ctype.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/runetype.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/xlocale/_ctype.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/err.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/fcntl.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/locale.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/xlocale/_locale.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/stddef.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/stdio.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/stdlib.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/string.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/strings.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/xlocale/_string.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/unistd.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/unistd.h
M 42533 '/var/obj/projects-bmake/amd64/bin/cat/cat.o-24d10fa0' 'cat.o'
X 42533 0
X 42530 0
S 42529 /usr/bin/ctfconvert
F 42529 42541
E 42541 /usr/bin/ctfconvert
R 42541 /var/run/ld-elf.so.hints
R 42541 /lib/libctf.so.2
R 42541 /usr/lib/libdwarf.so.3
R 42541 /usr/lib/libelf.so.1
R 42541 /lib/libz.so.6
R 42541 /lib/libthr.so.3
R 42541 /lib/libc.so.7
R 42541 cat.o
R 42541 cat.o
R 42541 cat.o.ctf
W 42541 cat.o.ctf
M 42541 'cat.o.ctf' 'cat.o'
X 42541 0
X 42529 0
# Stop 1363631732.322849
# Bye bye

Note: in the NetBSD version of filemon we don't bother recording the stat calls (S).

From this we can derive not only the equivalent data of make depend, but we can derive tree based dependencies.

By that I mean that any file read from an object directory which is not .OBJDIR represents a directory which needs to be built before .CURDIR, it's that simple.

Of course having some consistency in the relationship between object dirs and src dirs helps. Even when that isn't possible though we can work around it.

In the example above, a number of headers are read from /var/obj/projects-bmake/stage/amd64/usr/include/. This is where we stage headers as we build the tree.

/var/obj/projects-bmake/stage/amd64/usr/include/xlocale/_string.h
/var/obj/projects-bmake/stage/amd64/usr/include/xlocale/_string.h.dirdep

include/xlocale.amd64

meta2deps

Originally a shell script meta2deps.sh was used to extract the useful information from .meta files. It works fine. I should say worked fine - it hasn't had any attention recently.

If Python is available though we will use meta2deps.py which produces the same result, but much more efficiently (5-10 times faster) and can gather additional data without overhead.

Makefile.depend

After processing the .meta files for bin/cat we end up with the following in ${.MAKE.DEPENDFILE}:

# Autogenerated - do NOT edit!

DEP_RELDIR := ${_PARSEDIR:S,${SRCTOP}/,,}

DIRDEPS = \
        gnu/lib/libgcc \
        include \
        include/xlocale \
        lib/${CSU_DIR} \
        lib/libc \
        lib/libcompiler_rt \

.include <dirdeps.mk>

.if ${DEP_RELDIR} == ${_DEP_RELDIR}
# local dependencies - needed for -jN in clean tree
.endif

which shows a number of interesting things.

.dirdep:
        @echo ${RELDIR}.${MACHINE} > $@

.if defined(.PARSEDIR)
# we are bmake ...
.endif

DIRDEPS = \
        include \
        include/xlocale \
        usr.sbin/config.host \

buildworld -DWITH_META_MODE

Currently we can do buildworld -DWITH_META_MODE to create meta files in a build that otherwise behaves like normal buildworld. These meta files are very useful for comparing against those in the meta mode build - especially when after a sync, things seem to be broken.

We can probably leverage these meta files for bootstrapping too though that is basically done already.

bootstrapping new content

Once the bulk of the tree can build in meta mode, dealing with new dependencies is not that big a deal.

Eg. after last sync, usr.bin/kdump needs libcapsicum and libnv:

mk -C lib/libcapsicum
mk -C lib/libnv
mk -C usr.bin/kdump

is all that is needed - of course there will now be Makefile.depend files in the new libs which need to be added to SVN.

targets/pseudo/

The targets/Makefile which we use as a top-level makefile looks for subdirs under targets/ and targets/pseudo/ that match the target to be built.

Subdirs of targets/pseudo/ are not expected to build anything themselves, just be place holders for Makefile.depend files (which are invariably manually maintained). Thus their makefiles do nothing - except to say not to update Makefile.depend.

From our bootstrapping exercise above we got:

targets/pseudo/bin/Makefile.depend
targets/pseudo/cddl/Makefile.depend
targets/pseudo/clang/Makefile.depend
targets/pseudo/games/Makefile.depend
targets/pseudo/gnu/Makefile.depend
targets/pseudo/include/Makefile.depend
targets/pseudo/kerberos5/Makefile.depend
targets/pseudo/lib/Makefile.depend
targets/pseudo/libexec/Makefile.depend
targets/pseudo/misc/Makefile.depend
targets/pseudo/sbin/Makefile.depend
targets/pseudo/secure/Makefile.depend
targets/pseudo/share/Makefile.depend
targets/pseudo/usr.bin/Makefile.depend
targets/pseudo/usr.sbin/Makefile.depend

which we simply list in targets/pseudo/userland/Makefile.depend:

# This file is not autogenerated - take care!

DEP_RELDIR := ${_PARSEDIR:S,${SRCTOP}/,,}

DIRDEPS = \
        targets/pseudo/bin \
        targets/pseudo/cddl \
        targets/pseudo/games \
        targets/pseudo/gnu \
        targets/pseudo/include \
        targets/pseudo/kerberos5 \
        targets/pseudo/lib \
        targets/pseudo/libexec \
        targets/pseudo/sbin \
        targets/pseudo/secure \
        targets/pseudo/share \
        targets/pseudo/usr.bin \
        targets/pseudo/usr.sbin \

.include <dirdeps.mk>

which is in turn referenced by targets/pseudo/the-lot/Makefile.depend:

# This file is not autogenerated - take care!

DEP_RELDIR := ${_PARSEDIR:S,${SRCTOP}/,,}

DIRDEPS = \
        targets/pseudo/kernel \
        targets/pseudo/toolchain \
        targets/pseudo/toolchain.host \
        targets/pseudo/userland \


.include <dirdeps.mk>

The compilers are built via toolchain for both the host and target machines.

For the embedded world we don't normally build compilers at all.

machine dependent dirdeps

The DIRDEPS.amd64 lines from our bootstrap output, lead us to run a separate extraction process for each machine type:

getMdirdeps() {
    for mf in "$@"
    do
        case "$mf" in
        *targets/*|*~) continue;;
        esac
        d=${mf%/*}
        m=${mf##*.}
        case "$m" in
        ""|inc|orig|rej|bak|old) continue;;
        esac
        MACHINE=$m ${MAKE:-make} -C $d -f bsd.arch.inc.mk gen-dirdeps 2>&1
    done
}

which we can then add to targets/pseudo/sbin/Makefile.depend etc:

DIRDEPS.amd64= sbin/bsdlabel sbin/fdisk sbin/nvmecontrol
DIRDEPS.arm= sbin/bsdlabel sbin/fdisk
DIRDEPS.i386= sbin/bsdlabel sbin/fdisk sbin/nvmecontrol sbin/sconfig
DIRDEPS.ia64= sbin/mca
DIRDEPS.mips= sbin/bsdlabel sbin/fdisk
DIRDEPS.pc98= sbin/bsdlabel sbin/fdisk_pc98 sbin/sconfig
DIRDEPS.sparc64= sbin/bsdlabel sbin/sunlabel

DIRDEPS+= ${DIRDEPS.${MACHINE}:U}

.include <dirdeps.mk>

Note that any directory listed in DIRDEPS.amd64 should be removed from the generic DIRDEPS - bsd.subdir.mk would have entered them and caused them to list themselves.

Also note, that the DIRDEPS.amd64 lines output by bsd.arch.inc.mk are not always leaf dirs. For instance in targets/pseudo/usr.sbin/Makefile.depend we originally see:

DIRDEPS = \
        usr.sbin/acpi/acpiconf \
        usr.sbin/acpi/acpidb \
        usr.sbin/acpi/acpidump \
        usr.sbin/acpi/iasl \

but we see:

DIRDEPS.amd64=  usr.sbin/acpi

which means we need to move usr.sbin/acpi/* from DIRDEPS to DIRDEPS.amd64 and any other that references usr.sbin/acpi.

package manifests

Of course all this is moot once you have a set of manifests or mtree specs used to build the install packages/media. These simply list the applications and shared libs needed, and from those we learn the dependencies we need.

In the Junos build we produce packages that contain isofs images. These are built in targets/* and unlike the targets/pseudo/*, auto updated dependencies based on what went into the package.

targets/pseudo/kernel

The exception to the rule is targets/pseudo/kernel which actually builds a kernel in the FreeBSD way (GENERIC by default).

In the Junos build we need to build multiple kernels for multiple machines and want to automatically capture dependencies for them all. So there is a src directory that corresponds to each kernel.

In keeping with the traditional FreeBSD kernel build though, this makefile just does:

# $FreeBSD: projects/bmake/targets/pseudo/kernel/Makefile 248288 2013-03-14 22:04:25Z sjg $

# Build the kernel ${KERNCONF}
KERNCONF?= ${KERNEL:UGENERIC}

TARGET?= ${MACHINE}
# keep this compatible with peoples expectations...
KERN_OBJDIR= ${OBJTOP}/sys/compile/${KERNCONF}
KERN_CONFDIR= ${SRCTOP}/sys/${TARGET}/conf

CONFIG= ${STAGE_HOST_OBJTOP}/usr/sbin/config

${KERNCONF}.config: .MAKE .META
        mkdir -p ${KERN_OBJDIR:H}
        (cd ${KERN_CONFDIR} && \
        ${CONFIG} ${CONFIGARGS} -d ${KERN_OBJDIR} ${KERNCONF})
        (cd ${KERN_OBJDIR} && ${.MAKE} depend)
        @touch $@

# we need to pass curdirOk=yes to meta mode, since we want .meta files
# in ${KERN_OBJDIR}
${KERNCONF}.build: .MAKE ${KERNCONF}.config
        (cd ${KERN_OBJDIR} && META_MODE="${.MAKE.MODE} curdirOk=yes" ${.MAKE})

.if ${.MAKE.LEVEL} > 0
all: ${KERNCONF}.build
.endif

UPDATE_DEPENDFILE= no

.include <bsd.prog.mk>

circular dependencies

Our goal is to be able to build the tree in a single pass. That is; visiting dirs in the correct order - once, and have the default (all) target do all that is required, such as building objects (libs, progs) installing (staging) headers, libs even progs if -DWITH_STAGING_PROG, and finally updating any dependencies.

This requires that there be no circular dependencies within the tree. Currently in head, lib/libc depends on headers from both lib/libutil and lib/msun, and since libc.so needs to be scanned when linking shared libs, we have a potential circular dependency.

In the case of lib/libutil this is trivial to avoid, since it has only two headers and they are both public, so adding:

CFLAGS+= -I${.CURDIR:H}/libutil

to lib/libc/Makefile is fine.

Things are not quite as neat for lib/msun which not only has headers in a machine specific subdir, but also has math.h in a directory which contains both public and private headers. The following in lib/libc/Makefile avoids the dependency:

MSUN_ARCH_SUBDIR != ${MAKE} -B -C ${.CURDIR:H}/msun -V ARCH_SUBDIR
# unfortunately msun/src contains both private and public headers
CFLAGS+= -I${.CURDIR:H}/msun/${MSUN_ARCH_SUBDIR} -I${.CURDIR:H}/msun/src

but is not as clean as the previous case.

Of course lib/msun/Makefile does something similar to grab headers from libc:

# Location of fpmath.h and _fpmath.h
LIBCDIR=        ${.CURDIR}/../libc
.if exists(${LIBCDIR}/${MACHINE_ARCH})
LIBC_ARCH=${MACHINE_ARCH}
.else
LIBC_ARCH=${MACHINE_CPUARCH}
.endif
CFLAGS+=        -I${.CURDIR}/src -I${LIBCDIR}/include \
        -I${LIBCDIR}/${LIBC_ARCH}

Keeping public headers separate from private headers helps ensure that the above sort of dance is harmless.

Another example is lib/libproc and lib/librtld_db each of which includes a header staged by the other - resulting in a circular dependency. The fix is simple:

Index: lib/libproc/Makefile
===================================================================
--- lib/libproc/Makefile        (revision 242545)
+++ lib/libproc/Makefile        (working copy)
@@ -14,6 +14,8 @@
 INCS=  libproc.h

 CFLAGS+=       -I${.CURDIR}
+# avoid cyclic dependency
+CFLAGS+=       -I${.CURDIR:H}/librtld_db

 .if ${MK_LIBCPLUSPLUS} != "no"
 LDADD+=                -lcxxrt

MACHINE specific depend files

Having previously built the tree for i386, and now building for amd64 we can detect directories where a single Makefile.depend is probably not ideal:

Index: usr.bin/truss/Makefile.depend
===================================================================
--- usr.bin/truss/Makefile.depend       (revision 242503)
+++ usr.bin/truss/Makefile.depend       (working copy)
@@ -8,7 +8,6 @@
        gnu/lib/libgcc \
        include \
        include/arpa \
-       include/rpc \
        include/xlocale \
        lib/${CSU_DIR} \
        lib/libc \
@@ -18,10 +17,12 @@

 .if ${DEP_RELDIR} == ${_DEP_RELDIR}
 # local dependencies - needed for -jN in clean tree
-i386-fbsd.o: syscalls.h
-i386-fbsd.po: syscalls.h
-i386-linux.o: linux_syscalls.h
-i386-linux.po: linux_syscalls.h
+amd64-fbsd.o: syscalls.h
+amd64-fbsd.po: syscalls.h
+amd64-fbsd32.o: freebsd32_syscalls.h
+amd64-fbsd32.po: freebsd32_syscalls.h
+amd64-linux32.o: linux32_syscalls.h
+amd64-linux32.po: linux32_syscalls.h
 ioctl.o: ioctl.c
 ioctl.po: ioctl.c
 .endif

There are very few of these cases (so far), where machine specific local dependencies need to be captured. The simplest solution is for these to create Makefile.depend.${MACHINE}.

The vast majority of the tree seems to be fine with a simple Makefile.depend.

We configure .MAKE.DEPENDFILE_PREFERENCE such that once a Makefile.depend.${MACHINE} exists, it will be used.

We set .MAKE.DEPENDFILE_DEFAULT to the plain Makefile.depend.

Once a directory contains a machine specific depend file, we will automatically follow suit when building for other machines.

Staging conflicts

The latest version of meta.stage.mk throws an error when it detects that a different directory has already staged the file that it wants to.

Most of the cases found so far are simple - and easily fixed. For example lib/ncurses/ncursesw/Makefile includes lib/ncurses/ncurses/Makefile and thus attempts to install the same headers in the same location. This is easily fixed:

Index: lib/ncurses/ncurses/Makefile
===================================================================
--- lib/ncurses/ncurses/Makefile        (revision 242503)
+++ lib/ncurses/ncurses/Makefile        (working copy)
@@ -304,6 +304,7 @@
 SYMLINKS+=     libncurses${LIB_SUFFIX}_p.a
 ${LIBDIR}/libtinfo${LIB_SUFFIX}_p.a
 .endif

+.if ${.CURDIR:T} == "ncurses"
 DOCSDIR=       ${SHAREDIR}/doc/ncurses
 DOCS=          ncurses-intro.html hackguide.html

@@ -311,6 +312,7 @@
 .PATH: ${NCURSES_DIR}/doc/html
 FILESGROUPS=   DOCS
 .endif
+.endif

 # Generated source
 .ORDER: names.c codes.c

beforeinstall

When building WITH_STAGING_PROG we want to stage pretty much everything. There are a number of makefiles however that rely on the target beforeinstall to prepare files that we would want to stage.

Rather than re-write all these makefiles, we leverage beforeinstall and set DESTDIR=${STAGE_OBJTOP} which is very similar to what we did for src/include/.

targets/pseudo/stage

Many of those beforeinstall targets rely on mtree having been run in DESTDIR. The makefile in targets/pseudo/stage ensures that this is done, and is inserted as a dependency on every directory except itself.

bootstrap-tools

Anyone remember porting gcc in the early days? It was a 3 stage dance.

1. build stage1-gcc with cc
2. build stage2-gcc with stage1-gcc
3. build gcc with stage2-gcc

now we could do even that without breaking our single pass through the tree goal - by having separate makefiles for the three stages.

But since the whole toolchain topic needs is own overhaul, for now I'm happy to largely punt it, and even leverage the bootstrap targets from src/Makefile.inc1.

Changing behavior based on clean/update build

A very common cause is a makefile using .if exists(something) and changing its behavior as a result. Thus the result from a clean tree build can differ from an update build. This should be fixed.

Use of SRCTOP, OBJTOP, OBJROOT etc, can greatly help canonicalize references to locations within the tree which can eliminate many such issues.

Conflicting makefiles

Whenever two (or more) makefiles try to do the same thing, eg include/arpa/Makefile and lib/libtelnet/Makefile both installing usr/include/arpa/telnet.h. In this case meta.stage.mk will throw an error so the issue is quickly resolved.

Other cases can be more subtle. In general it is dangerous for any makefile to stomp on anything outside of its .OBJDIR.

Foreign builds

If part of the build is done with say gmake, our ability to reliably capture what it does is limited to clean tree builds. There is explicit mechanism in meta.autodep.mk to cope with this.

Current state of FreeBSD

Update: as of June 2017, thanks to lots of work from Bryan Drewery, building WITH_META_MODE may soon be enabled by default.

For WITH_DIRDEPS_BUILD instead of simply using ${MACHINE} to distinguish targets we use TARGET_SPEC_VARS concatenated with , eg. ${MACHINE},${MACHINE_ARCH}

Environment

I should note that I generally run make via a wrapper script (mk) which first reads a file $SB/.sandbox-env (where SB is typically the parent dir to src/) to condition the environment. I have dozens of active trees for Junos, NetBSD and FreeBSD and multiple branches and this allows me use the same user interface for all.

For the DIRDEPS_BUILD the key environment items are:

export SRCTOP=$SB/src
export OBJROOT=/var/obj/projects-bmake/
export OBJTOP="$OBJROOT\${MACHINE}"
export MAKESYSPATH=$SRCTOP/share/mk
export MAKEOBJDIR='${.CURDIR:S,${SRCTOP},${OBJTOP},}'
export HOST_TARGET=freebsd10-amd64
export HOST_OBJTOP="$OBJROOT$HOST_TARGET"

Not all are strictly necessary as *sys.mk can set them, but putting such logic in the makefiles constrains choices.

Note that MAKEOBJDIR value is single quoted, this defers expansion until bmake sees it.

HOST_TARGET is set by the wrapper script, based on uname output of the host machine. For example freebsd10-amd64 or freebsd7-i386. When building for the pseudo machine host we use ${HOST_OBJTOP} rather than ${OBJTOP}. This allows us to avoid trouble when the same tree is mounted via NFS and built from incompatible machines.

I have local.sys.mk set a number of options:

WITH_INSTALL_AS_USER= yes
WITH_AUTO_OBJ= yes
WITH_META_MODE= yes
WITH_STAGING= yes
WITH_STAGING_PROG= yes

WITH_AUTO_OBJ is important, since it causes the equivalent of make obj to be done automatically and early, so that as the makefile is read .OBJDIR and .CURDIR have their correct values which matters in the gcc build where we see:

.PATH: ../cc_tools

which is supposed to add ${.OBJDIR}/../cc_tools to .PATH, but if ${.OBJDIR} does not exist at the time that line is read, the right thing will not happen.