The Mozilla Build System has, for most of its history, not changed much. But, for a couple years now, we’ve been, slowly and incrementally, modifying it in quite extensive ways. This post summarizes the progress so far, and my personal view on where we’re headed.

Recursive make

The Mozilla Build System has, all along, been implemented as a set of recursively traversed Makefiles. The way it has been working for a very long time looks like the following:

• For each tier (group of source directories) defined at the top-level:
  • For each subdirectory in current tier:
    • Build the export target recursively for each subdirectory defined in Makefile.
    • Build the libs target recursively for each subdirectory defined in Makefile.

The typical limitation due to the above is that some compiled tests from a given directory would require a library that’s not linked until after the given directory is recursed, so another target was later added on top of that (tools).

There was not much room for parallelism, except in individual directories, where multiple sources could be built in parallel, but never would sources from multiple directories be built at the same time. So, for a bunch of directories where it was possible, special rules were added to allow that to happen, which led to interesting recursions:

• For each of export, libs, and tools:
  • Build the target in the subdirectories that can be built in parallel.
  • Build the target in the current directory.
  • Build the target in the remaining subdirectories.

This ensured some extra fun with dependencies between (sub)directories.

Apart from the way things were recursed, all sorts of custom build rules had piled up, some of which relied on things in other directories having happened beforehand, and the build system implementation itself relied on some quite awful things (remember allmakefiles.sh?)

Gradual Overhaul

Around two years ago, we started a gradual overhaul of the build system.

One of the goals was to move away from Makefiles. For various reasons, we decided to go with our own kind-of-declarative (but really, sandboxed python) format (moz.build) instead of using e.g. gyp. The more progress we make on the build system, and the more I think this was the right choice.

Anyways, while we’ve come a long way and converted a lot of Makefiles to moz.build, we’re not quite there yet:

One interesting thing to note in the above graph is that we’ve also been reducing the overall number of moz.build files we use, by consolidating some declarations. For example, some moz.build files now declare source or test files from their subdirectories directly, instead of having one file per directory declare sources and test files local to their own directory.

Pseudo derecursifying recursive Make

Neologism aside, one of the ideas to help with the process of converting the build system to something that can be parallelized more massively was to reduce the depth of recursion we do with Make. So that instead of a sequence like this:

• Entering directory A
  • Entering directory A/B
  • Leaving directory A/B
  • Entering directory A/C
    • Entering directory A/C/D
    • Leaving directory A/C/D
    • Entering directory A/C/E
    • Leaving directory A/C/E
    • Entering directory A/C/F
    • Leaving directory A/C/F
  • Leaving directory A/C
  • Entering directory A/G
  • Leaving directory A/G
• Leaving directory A
• Entering directory H
• Leaving directory H

2014-12-03 02:01:43+0900

p.m.o

You can leave a response, or trackback from your own site.