- \n
- For each subdirectory in current tier<\/em>:\n
- \n
- Build the
export<\/code> target recursively for each subdirectory defined in Makefile.<\/li>\n- Build the
libs<\/code> target recursively for each subdirectory defined in Makefile.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\nThe typical limitation due to the above is that some compiled tests from a given directory would require a library that's not linked until after the given directory is recursed, so another target was later added on top of that (
tools<\/code>)<\/a>.<\/p>\nThere was not much room for parallelism, except in individual directories, where multiple sources could be built in parallel, but never would sources from multiple directories be built at the same time. So, for a bunch of directories where it was possible, special rules were added to allow that to happen<\/a>, which led to interesting recursions:<\/p>\n
- \n
- For each of
export<\/code>,libs<\/code>, andtools<\/code>:\n- \n
- Build the target in the subdirectories that can be built in parallel.<\/li>\n
- Build the target in the current directory.<\/li>\n
- Build the target in the remaining subdirectories.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
This ensured some extra fun with dependencies between (sub)directories.<\/p>\n
Apart from the way things were recursed, all sorts of custom build rules had piled up, some of which relied on things in other directories having happened beforehand, and the build system implementation itself relied on some quite awful things (remember
allmakefiles.sh<\/code>?)<\/p>\nGradual Overhaul<\/h2>\n
Around two years ago, we started a gradual overhaul of the build system.<\/p>\n
One of the goals was to move away from Makefiles<\/a>. For various reasons, we decided to go with our own kind-of-declarative (but really, sandboxed python) format (
moz.build<\/code>) instead of using e.g. gyp<\/a>. The more progress we make on the build system, and the more I think this was the right choice.<\/p>\nAnyways, while we've come a long way and converted a lot of Makefiles to
moz.build<\/code>, we're not quite there yet:
\n![](\"\/blog\/wp-content\/uploads\/2014\/11\/makefiles.png\")
<\/p>\nOne interesting thing to note in the above graph is that we've also been reducing the overall number of
moz.build<\/code> files we use, by consolidating some declarations. For example, somemoz.build<\/code> files now declare source or test files from their subdirectories directly, instead of having one file per directory declare sources and test files local to their own directory.<\/p>\nPseudo derecursifying recursive Make<\/h2>\n
Neologism aside, one of the ideas to help with the process of converting the build system to something that can be parallelized more massively was to reduce the depth of recursion<\/a> we do with Make. So that instead of a sequence like this:<\/p>\n
- \n
- Entering directory A\n
- \n
- Entering directory A\/B<\/li>\n
- Leaving directory A\/B<\/li>\n
- Entering directory A\/C\n
- \n
- Entering directory A\/C\/D<\/li>\n
- Leaving directory A\/C\/D<\/li>\n
- Entering directory A\/C\/E<\/li>\n
- Leaving directory A\/C\/E<\/li>\n
- Entering directory A\/C\/F<\/li>\n
- Leaving directory A\/C\/F<\/li>\n<\/ul>\n<\/li>\n
- Leaving directory A\/C<\/li>\n
- Entering directory A\/G<\/li>\n
- Leaving directory A\/G<\/li>\n<\/ul>\n<\/li>\n
- Leaving directory A<\/li>\n
- Entering directory H<\/li>\n
- Leaving directory H<\/li>\n<\/ul>\n
We would have a sequence like this:<\/p>\n
- \n
- Entering directory A<\/li>\n
- Leaving directory A<\/li>\n
- Entering directory A\/B<\/li>\n
- Leaving directory A\/B<\/li>\n
- Entering directory A\/C<\/li>\n
- Leaving directory A\/C<\/li>\n
- Entering directory A\/C\/D<\/li>\n
- Leaving directory A\/C\/D<\/li>\n
- Entering directory A\/C\/E<\/li>\n
- Leaving directory A\/C\/E<\/li>\n
- Entering directory A\/C\/F<\/li>\n
- Leaving directory A\/C\/F<\/li>\n
- Entering directory A\/G<\/li>\n
- Leaving directory A\/G<\/li>\n
- Entering directory H<\/li>\n
- Leaving directory H<\/li>\n<\/ul>\n
For each directory there would be a directory-specific target per top-level target, such as
A\/B\/export<\/code>,A\/B\/libs<\/code>, etc. Essentially those targets are defined as:<\/p>\n\n
%\/$(target):\r\n $(MAKE) -C $* $(target)\r\n<\/pre>\n<\/blockquote>\n
And each top-level target is expressed as a set of dependencies, such as, in the case above:<\/p>\n
\n
A\/B\/libs: A\/libs\r\nA\/C\/libs: A\/B\/libs\r\nA\/C\/D\/libs: A\/C\/libs\r\nA\/C\/E\/libs: A\/C\/D\/libs\r\nA\/C\/F\/libs: A\/C\/E\/libs\r\nA\/G\/libs: A\/C\/F\/libs\r\nH\/libs: A\/G\/libs\r\nlibs: H\/libs\r\n<\/pre>\n<\/blockquote>\n
That dependency list, instead of being declared manually, is generated from the \"traditional\" recursion declaration we had in
moz.build<\/code>, through the various*_DIRS<\/code> variables. I'll skip the gory details about how this replicated the weird subdirectory orders I mentioned above when you had bothPARALLEL_DIRS<\/code> andDIRS<\/code>. They are irrelevant today anyways.<\/p>\nYou'll also note that this also removed the notion of tier<\/em>, and the entirety of the tree is dealt with for each target, instead of iterating all targets for each group of directories.<\/p>\n
[But note that, confusingly, but for practical reasons related to the amount of code changes required, and to how things were incrementally set in place, those targets are now called tiers]<\/p>\n
From there, parallelizing parts of the build only involves reorganizing those Make dependencies such that Make can go in several subdirectories at once, the ultimate goal being:<\/p>\n
\n
libs: A\/libs A\/B\/libs A\/C\/libs A\/C\/D\/libs A\/C\/E\/libs A\/C\/F\/libs A\/G\/libs A\/H\/libs\r\n<\/pre>\n<\/blockquote>\n
But then, things in various directories may need other things built in different directories first, such that additional dependencies can be needed, for example:<\/p>\n
\n
A\/B\/libs A\/C\/libs: A\/libs\r\nA\/H\/libs: A\/G\/libs\r\n<\/pre>\n<\/blockquote>\n
The trick is that those dependencies can in many cases be deduced from the contents of
moz.build<\/code>. That's where we are today, for example, for everything related to compilation: We added acompile<\/code> target that deals with everything C\/C++ compilation, and we have dependencies between directories building objects and directories building libraries and executables that depend on those objects. You can have a taste yourself:<\/p>\n\n
\r\n$ mach clobber\r\n$ mach configure\r\n$ mach build export\r\n$ mach build -C \/ toolkit\/library\/target\r\n<\/pre>\n<\/blockquote>\n
[Note, the
compile<\/code> target is actually composed of two sub-targets,target<\/code> andhost<\/code>.]<\/p>\nThe third command runs the
export<\/code> target on the whole tree, because that's a prerequisite that is not expressed as a make dependency yet (it would be too broad of a dependency anyways).<\/p>\nThe last command runs the
toolkit\/library\/target<\/code> target at the top level (as opposed tomach build toolkit\/library\/target<\/code>, which runs thetarget<\/code> target in thetoolkit\/library<\/code> directory). That will build libxul, and everything needed to link it, but nothing else unrelated.<\/p>\nAll the other top-level targets have also received this treatment to some extent:<\/p>\n
- \n
- The
export<\/code> target was made entirely parallel, although it keeps some cross-directory dependencies derived from the historical traversal data.<\/li>\n- The
libs<\/code> target is still entirely sequential, because of all the horrible rules in it that may or may not depend on the historical traversal order.<\/li>\n- A parallelized
misc<\/code> target was recently created<\/a> to receive all the things that are currently done as part of thelibs<\/code> target that are actually safe to be parallelized (another reason for creating this new target is thatlibs<\/code> is now a misnomer, since everything related to libraries is now part ofcompile<\/code>). Please feel free to participate in the effort to movelibs<\/code> rules there.<\/li>\n- The
tools<\/code> target is still sequential, but see below.<\/li>\n<\/ul>\nAlthough, some interdependencies that can't be derived yet are currently hardcoded in
config\/recurse.mk<\/code><\/a>.<\/p>\nAnd since the mere fact of entering a directory, figuring out if there's anything to do at all, and leaving a directory takes some time on builds when only a couple things changed, we also skip some directories entirely by making them not have a
directory\/target<\/code> target at all:<\/p>\n- \n
export<\/code> andlibs<\/code> only traverse directories where there is aMakefile.in<\/code> file, or where themoz.build<\/code> file sets variables that do require something to be done in those targets.<\/li>\ncompile<\/code> only traverses directories where there is something to compile or link.<\/li>\nmisc<\/code> only traverses directories with an explicitHAS_MISC_RULE = True<\/code> when they have aMakefile.in<\/code>, or with amoz.build<\/code> setting variables affecting themisc<\/code> target.<\/li>\ntools<\/code> only traverses directories that contain a Makefile.in containingtools::<\/code> (there's actually a regexp, but in essence, that's it).<\/li>\n<\/ul>\nThose skipping rules allow to only traverse, taking numbers from a local Linux build, as of a couple weeks ago:<\/p>\n
- \n
- 186 directories during
libs<\/code>,<\/li>\n- 472 directories during
compile<\/code>,<\/li>\n- 161 directories during
misc<\/code>,<\/li>\n- 382 directories during
libs<\/code>,<\/li>\n- 2 directories during
tools<\/code>.<\/li>\n<\/ul>\ninstead of 850 for each.<\/p>\n
In the near future<\/b>, we may want to change
export<\/code> andlibs<\/code> to opt-ins, likemisc<\/code>, instead of opt-outs.<\/p>\nAlternative build backends<\/h2>\n
With more and more declarations in
moz.build<\/code>, files, we've been able to build up some alternative build backends for Eclipse and Microsoft Visual Studio. While they still are considered experimental, they seem to work well enough for people to take advantage of them in some useful ways. They however still rely on the \"traditional\" Make backend to build various things (to the best of my knowledge).<\/p>\nUltimately, we would like to support entirely different build systems such as ninja or tup, but at the moment, many things still heavily rely on the \"traditional\" Make backend. We're getting close to having everything related to compilation available from the
moz.build<\/code> declarations (ignoring third-party code, but see further below), but there is still a long way to go for other things.<\/p>\nIn the near future<\/b>, we may want to implement hybrid build backends where compilation would be driven by ninja or tup, and the rest of the build would be handled by Make. However, my feeling is that the Make backend is fast enough for compilation and ninja doesn't bring enough other than performance that it's not worth investing in a ninja backend. Tup is different because it does solve some<\/a> of the problems<\/a> with incremental builds<\/a>.<\/p>\n
While on the topic of being close to having everything related to compilation available from
moz.build<\/code> declarations, closing in on this topic would allow, more than such hybrid build systems, to better integrate tools such as code static analyzers.<\/p>\nUnified C\/C++ sources<\/h2>\n
Compiling C code, and even more compiling C++ code, involves reading the same headers an important number of times. In C++, that usually also means instantiating the same templates and compiling the same inline methods numerous times.<\/p>\n
So we've worked around this by creating \"unified sources\" that just
#include<\/code> the actual source files, grouping them 16 by 16 (except when specified otherwise).<\/p>\nThis reduced build times drastically, and, interestingly, reduced the size of DWARF debugging symbols<\/a> as well. This does have a couple downsides, though. It allows
#include<\/code> impurity in the code in such a way that e.g. changes in the file groups can lead to subtle build failures. It also makes incremental builds significantly slower in parts of the tree where compiling one file is already somehow slow, so doing 16 at a time can be a drag for people working on that code. This is why we're considering decreasing the number for the javascript engine<\/a> in the near future<\/b>. We should probably investigate where else building one unified source is slow enough to be a concern. I don't think we can rely on people actively complaining about it, they've been too used to slow build times to care to file bugs about it.<\/p>\nRelatedly, our story with
#include<\/code> is suboptimal to say the least, and several have been untangled, but there's still a long tail<\/a>. It's a hard problem to solve, even with tools like IWYU<\/a>.<\/p>\nFake libraries<\/h2>\n
For a very long time, the build system was building intermediate static libraries, and then was linking them together to form shared libraries. This is how the main components were built in the old days before libxul (remember libgklayout?), and was how libxul built when it was created. For various reasons, ranging from disk space waste to linker inefficiencies, this was replaced by building fake libraries<\/a>, that only reference the objects that would normally be contained in the static library that used to be built. This later was changed to use a more complex system allowing more flexibility<\/a>.<\/p>\n
Fast-forward to today, and with all the knowledge from
moz.build<\/code>, one of the usecases of that more complex system was made moot<\/a>, and in the future<\/b>, those fake libraries could be generated as build backend files instead of being created during the build.<\/p>\nSpecialized incremental builds<\/h2>\n
When iterating C\/C++ code patches, one usually needs to (re)compile often. With the build system having the overhead it has, and rebuilding with no change taking many seconds (it has been around a minute for a long time on my machine and is now around half of that, although, sadly, I had got it down to 20 seconds but that regressed recently, damn
libs<\/code> rules), we also added a special rule that only handles header changes and rebuilding objects, libraries and executables.<\/p>\nThat special rule is invoked as:<\/p>\n
\n
$ mach build binaries<\/pre>\n<\/blockquote>\n
And takes about 3.5s on my machine when there are no changes. It used to be faster thanks to clever tricks, but that was regressed on purpose. That's a trade-off, but linking libxul, which most code changes require, takes much longer than that anyways. If deemed necessary, the same clever tricks could be restored.<\/p>\n
While we work to improve the overall build experience, in the near future<\/b> we should have one or more special rules for non-compilation use-cases. For Firefox frontend developers, the following command may do part of the job:<\/p>\n
\n
$ mach build -C \/ chrome<\/pre>\n<\/blockquote>\n
but we should have some better and more complete commands for them and e.g. Firefox Android developers.<\/p>\n
We also currently have a build option allowing to entirely skip everything that is compilation related (
--disable-compile-environment<\/code>), but it is currently broken<\/a> and is only really useful in few use cases. In the near future<\/b>, we need build modes that allow to use e.g. nightly builds as if they were the result of compiling C++ source. This would allow some classes of developers to skip compilations altogether, which are an unnecessary overhead for them at the moment, since they need to compile at least once (and with all the auto-clobbers we have, it's much more than that).<\/p>\nLocalization<\/h2>\n
Related to the above, the experience of building locale packs and repacks for Firefox is dreadful. Even worse than that, it also relies on a big pile of awful Make rules. It probably is, along with the code related to the creation of Firefox tarballs and installers, the most horrifying part of the build system. Its entanglement with release automation also makes improving the situation unnecessarily difficult.<\/p>\n
While there are some sorts of tests running on every push, there are many occasions where those tests fail to catch regressions that lead to broken localized builds for nightlies, or worse on beta or release (which, you'll have to admit, is a sadly late a moment to find such regressions).<\/p>\n
Something really needs to be done about localization, and hopefully the discussions we'll have this week in Portland will lead to improvements in the short to medium term<\/b>.<\/p>\n
Install manifests<\/h2>\n
The build system copies many files during the build. From the source directory to the \"object\" directory. Sometimes in
$(DIST)\/somedir<\/code>, sometimes elsewhere. Sometimes in both or more. On non-Windows systems, copies are replaced by symbolic links. Sometimes not. There are also files that are preprocessed during the build.<\/p>\nAll those used to be handled by Make rules invoking
$(NSINSTALL)<\/code> on every build. Even when the files hadn't changed. Most of these were replaced by some Makefile<\/a> magic<\/a>, but many are now covered with so-called \"install manifests\"<\/a>.<\/p>\nOthers, defined in
jar.mn<\/code> files, used to be added to jar files during the build. While those jars are not created anymore because of omni.ja, the corresponding content is still copied\/symlinked and defined injar.mn<\/code>.<\/p>\nIn the near future<\/b>, all those should be switched to install manifests somehow, and that is greatly tied to solving part of the localization problem: currently, localization relies on Make overrides that
moz.build<\/code> can't know about, preventing install manifests being created and used for the corresponding content.<\/p>\nFaster configure<\/h2>\n
One of the very first things the build system does when a build starts from scratch is to run
configure<\/code>. That's a part of the build system that is based on the antiquatedautoconf<\/code> 2.13, with 15+ years of accumulated linear m4 and shell gunk. That's what detects what kind of compiler you use, how broken it is, how broken its headers are, what options you requested, what application you want to build, etc.<\/p>\nTopping that, it also invokes
configure<\/code> from third-party software that happen to live in the tree, like ICU or jemalloc 3. Those are also based onautoconf<\/code>, but in more recent versions than 2.13. They are also third-party, so we're essentially only importing them, as opposed to actively making them bigger for those that are ours.<\/p>\nWhile it doesn't necessarily look that bad when running on e.g. Linux, the time it takes to run all this pile of shell scripts is painfully horrible on Windows (like, taking more than 5 minutes on automation). While there's still a lot to do, various improvements were recently made:<\/p>\n
- \n
- Some classes of changes (such as modifying
configure.in<\/code>) make the build system re-runconfigure<\/code>. It used to trigger everyconfigure<\/code> to run again, but now only re-runs a relevant subset.<\/li>\n- They used to all run sequentially, but apart from the top-level one, which still needs to run before all the others, they now all run in parallel. This cut
configure<\/code> times almost in half on Windows clobber builds on automation.<\/li>\n<\/ul>\nIn the future<\/b>, we want to get rid of
autoconf<\/code> 2.13 and use smart lazy python code to only run the tests that are relevant to the configure options. How this would all exactly work has, as of writing, not been determined. It's been on my list of things to investigate for a while, but hasn't reached the top. In the near future, though, I would like to move all ourautoconf<\/code> code related to the build toolchain (compiler and linker) to some form of python code. <\/p>\nZaphod beeblebuild<\/h2>\n
There are essentially two main projects in the mozilla-central repository: Firefox\/Gecko and the Javascript engine. They use the same build system in many ways. But for a very long time, they actually relied on different copies of the same build system files, like
config\/rules.mk<\/code> orbuild\/autoconf\/*.m4<\/code>. And we had acheck-sync-dirs<\/code> script verifying that both projects were indeed using the same file contents. Countless times, we've had landings forgetting to synchronize the files and leading to acheck-sync-dirs<\/code> error during the build. I plead guilty to have landed such things multiple times, and so did many other people.<\/p>\nThose days are now long gone, but we currently rely on dirty tricks that still keep the Firefox\/Gecko and Javascript engine build systems half separate<\/a>. So we kind of replaced a conjoined-twins system with a biheaded system. In the future<\/b>, and this is tied to the section above, both build systems would be completely merged<\/a>.<\/p>\n
Build system interface<\/h2>\n
Another goal of the build system changes was to make the build and test experience better. Especially, running tests was not exactly the most pleasant experience.<\/p>\n
- They used to all run sequentially, but apart from the top-level one, which still needs to run before all the others, they now all run in parallel. This cut
- 472 directories during
- The
- The
- Entering directory A\n
- Build the
- Build the