Building FreeBSD with bmake

This doc describes the process of converting the FreeBSD build to use bmake as a prerequisite for building with meta mode.

The initial commit of bmake went into FreeBSD head in October 2012. FreeBSD 10 is the first release to use bmake by default.

To avoid? confusion I will use fmake when referring to the older FreeBSD make and bmake regardless of which is installed as make.

Why bmake?

Bmake is make from NetBSD with some extra portability glue. It has been used on everything from UTS mainframes to Minix. There has been a lot of active development of make in NetBSD over the last 15 years, I've added a lot of functionality to it to better support the Junos build. While bmake and fmake have a common ancestor, they have diverged in subtle (and not so subtle) ways.

My ultimate goal is building with meta mode, and since one of the key makefiles requires every single fancy feature of bmake; the first step is to be able to build FreeBSD using bmake as a replacement for /usr/bin/make. This is actually quite simple.

Conflicting modifiers

Several years ago FreeBSD make adopted :U to upper case a value, and :L to lowercase it, from OpenBSD. At about the same time I added a raft of modifiers from ODE to NetBSD's make. These included:

     This is the loop expansion mechanism from the OSF Development Envi-
     ronment (ODE) make.  Unlike .for loops expansion occurs at the time
     of reference.  Assign temp to each word in the variable and evaluate
     string.  The ODE convention is that temp should start and end with a
     period.  For example.
           ${LINKS:@.LINK.@${LN} ${TARGET} ${.LINK.}@}

     However a single character variable is often more readable:

     If the variable is undefined newval is the value.  If the variable
     is defined, the existing value is returned.  This is another ODE
     make feature.  It is handy for setting per-target CFLAGS for
     If a value is only required if the variable is undefined, use:

     If the variable is defined newval is the value.

:L   The name of the variable is the value.

:P   The path of the node which has the same name as the variable is the
     value.  If no such node exists or its path is null, then the name of
     the variable is used.  In order for this modifier to work, the name
     (node) must at least have appeared on the rhs of a dependency.

At a later point NetBSD added:

:tl  Converts variable to lower-case letters.

:tu  Converts variable to upper-case letters.

While the FreeBSD ports collection used :U and :L quite a bit, the base system as of 2010 did not use them at all.

Thus building FreeBSD itself using bmake is mostly a matter of adding .NOPATH statements in a couple of places; to curb its enthusiasm for finding things via .PATH.

The diff to allow bmake -jN buildworld to work was less than 300 lines (not counting comments).

Other changes

While converting the base system to be able to use bmake, was simple, converting FreeBSD to use bmake by default was more involved.

The ports collection provided several challenges detailed in a later section.

Solutions to some issues required changes to bmake. In most cases these were made to NetBSD's make, in some cases (especially where the change was deemed short term) local changes were made to bmake in FreeBSD.

The good-will and support of developers in both projects was key to success.


While the original goal was to achieve cutover within a release cycle, this proved impractical, so a BMAKE option was added to select which make should be installed as /usr/bin/make. The knob defaults to "yes".

The top-level makefiles, have to pay attention to WITH[OUT]_BMAKE and ensure that the correct make is used.

This is a short term thing, since to keep WITHOUT_BMAKE working, requires anyone making changes to makefiles (at least src/share/mk/* and src/Makefile*) to test both ways.

job tokens

When you do make -j8, the goal is that at most 8 jobs will run in parallel. To achieve this most modern make's use a shared token pool, each sub-make waits for a token before running a job.

In fmake the token pool is managed as a named FIFO. If fmake does not find the appropriate variable in its environment, it creates and initializes the FIFO and exports its name.

In bmake the token pool is a pipe, with the open descriptors passed to sub-makes only for targets that are tagged with .MAKE. Sub-makes know that they are such due to the -J in,out arg they get. If the descriptors named are invalid, bmake behaves as for -B (compat mode).

Both designs have their benefits. The difference though can lead to problems when mixing the two.

If the first make instance is fmake, it creates its FIFO and passes -jN to sub-makes. If these are bmake the lack of -J causes them to create their own token pool - oversubscription.

Similarly if the first make is bmake and sub-makes are fmake problems can arise since bmake does not export the name of FIFO, so each sub-make will create its own. This case is less likely to happen accidentally, since -J will cause an error for fmake and thus needs to be explicitly filtered. Which implies the mixing is deliberate. An example of this is in src/Makefile when building WITHOUT_BMAKE.

.MAKE source

As noted above, bmake only passes the descriptors for its token pool if the target is tagged .MAKE:

.MAKE     Execute the commands associated with this target even if the -n
          or -t options were specified.  Normally used to mark recursive

The fmake man page says the same, but .MAKE appears not to work and thus isn't used all the places it should be.

This causes problems for bmake. While it is not a huge problem to fix the base source, this does not help all the makefiles others have written.

The solution was another local change, a knob .MAKE.ALWAYS_PASS_JOB_QUEUE to cause bmake to pass the descriptors of its token pool to all jobs.


Logic was recently added to NetBSD make to pass the token pool descriptors to sub-makes even without .MAKE.


Normally on failure during a jobs-mode build, bmake puts an error token into the job token pool. This causes the build to be abandoned immediately, which is normally exactly what is desired.

For make universe however it is desired that each branch of the build that can continue, do so. This is facilitated by the following in src/Makefile:

.if make(universe)
# we do not want a failure of one branch abort all.


Fmake runs target scripts with set -e in effect. This means that the script fails if any statement in a multi-statement command line fails. This means that the command:

cd /no/such/dir; rm -rf *

will not do any harm.

By contrast, bmake runs scripts such that the command line rather than individual statements is the unit of failure. This means that the above command should be written as:

cd /no/such/dir && rm -rf *

while fixing all the makefiles in base is feasible, fixing ports is another matter, and then there is the question of makefiles written by FreeBSD users.

The practical solution is to provide a .SHELL description that replicates the fmake behavior:

# By default bmake does *not* use set -e
# when running target scripts, this is a problem for many makefiles here.
# So define a shell that will do what FreeBSD expects.
.SHELL: name=sh \
        quiet="set -" echo="set -v" filter="set -" \
        hasErrCtl=yes check="set -e" ignore="set +e" \
        echoFlag=v errFlag=e \


Making FreeBSD use bmake requires dealing with ports. The conflicting modifiers mentioned above are used quite a lot within ports. There are a few other issues too, detailed below.

Quoted strings in .for loops

The following:

.for i in ${OPTIONS}

is expected to iterate 3 times. Quoted strings isn't something NetBSD's make (hence bmake) supported. This has been added. Using bmake the above could be coded as:

.for opt description default in ${OPTIONS}

allowing a much easier to follow makefile.

Bootstrapping bmake for old versions of FreeBSD

A bigger issue for ports though is that it isn't branched and must remain buildable on the oldest supported release. Thus it is important to not use enhancements such as multiple iterators in .for loops until the oldest supported release uses bmake.

The reason being that in order to build ports on an old release, which neither has bmake nor a make which has the compatibility modifiers (:tu etc), we need ports to be able to build and install bmake for itself automatically.

This can be done with the following in Mk/

.if !defined(.PARSEDIR) && ${.MAKEFLAGS:M-V} == ""
.MAIN: all
# We are not bmake
# bmake-sh will invoke bmake (installing it if needed).
all ${.TARGETS}: use-bmake
.if !target(use-bmake)
        +sh ${PORTSDIR}/Mk/bmake-sh ${.MAKEFLAGS} ${.TARGETS}
.else                           # Not bmake

The script Mk/bmake-sh will simply invoke bmake if it exists, and otherwise will build/install it - using a copy of Mk/*mk converted back to the old make.

I won't bore you with the details, since this approach was not adopted.

Hack to allow old modifiers

Turns out that the way portmgr build ports cannot accommodate the approach outlined above. So a knob was added to allow to:

# tell bmake we use the old :L :U modifiers
.MAKE.FreeBSD_UL= yes

support for this will removed once 8.3 is EOL and portmgr convert ports to use :tl and :tu.

With this as part of a simple patch to, portmgr were able to get ports building with bmake, and thus close this issue.

Portmgr were very helpful during this process.

Default -V behavior

Ports uses make -V FOO extensively. The old FreeBSD make outputs a fully expanded value for FOO so:

FOO_VAR!= ${MAKE} -C foo -V VAR

is commonly used.

However, with the above, bmake will show its literal value, which might be ${DESTDIR}foo - not what is desired.

To ensure full expansion bmake -V '${FOO}' is used. In a makefile that typically needs to be coded as:

FOO_VAR!= ${MAKE} -C foo -V '$${VAR}'

But those doubled $$ get complex fast when commands get put in other variables. Thus NetBSD pkgsrc uses:

FOO_VAR!= ${MAKE} -C foo show-var VARIABLE=VAR

to avoid complexities when such things need to be used indirectly. Using this model, will work equally well regardless of make version, but only handles a single variable.

Changing -V behavior

As useful as the ability to see the literal value of a variable is for debugging (I use it a lot), it isn't a useful behavior from the build's perspective.

It would be very nice to not have to frob all the uses of make -V in the FreeBSD ports collection and elsewhere.

My preferred solution was to change the default behavior and add a debug flag to allow seeing the literal value:

$ bmake -V FOO_DIR
$ bmake -dV -V FOO_DIR

A compromise was adding a knob that could be set via FreeBSD's (or anywhere else) that controls the default behavior. This allows both teams to see the behavior they expect:


.for loop iterators

With fmake used a rather complex dance involving nested .for loops with heavily escaped iterator variables to handle building the MLINKS list for multiple languages.

In bmake loop iterators are substituted as ${:Uvalue} which avoids lots of issues but won't work with the above. Fortunately a one line nested inline-loop does the job:

.if defined(.PARSEDIR)
# inline loops are simpler


There are two loops there, one with iterator m for ${MANLANG} resulting in values like man man/fr etc., and another p for ${MLINKS}, thus for:

MANLANG= "" fr
MLINKS= foo.1 goo.2

we get:


exactly the same as the 20 or so original lines produce.


As noted above the initial import of bmake to FreeBSD happened in early October 2012.

Progress on ports stalled for some time waiting for an exp-run, but portmgr signed off on bmake early May 2013.

By the time FreeBSD 10 was branched, bmake was installed as /usr/bin/make by default.

Support for fmake is being deprecated shortly. For those that have a need, there is a port.

I note however that a lot of changes have gone into head since stable/10 that would make building with fmake impossible.
Revision:$Id: freebsd-bmake.txt,v 0cc0e47d3e1f 2015-05-31 17:02:44Z sjg $