Archive for May, 2014

A year and a half ago (already), I landed replace-malloc, a feature that allows more memory allocator flexibility in Firefox. It enabled building tools such as the dark matter detector (aka DMD) more easily. It also allows to replace our default allocator, (moz)jemalloc.

Until now, you had to explicitly enable the feature with --enable-replace-malloc. As of writing, it is enabled by default on all builds except Windows builds with jemalloc disabled (but that's due to change too). Note Windows debug builds, as well as local Windows builds have jemalloc disabled by default.

It currently is only on mozilla-inbound, but should propagate quickly to other branches. It won't, however, ride the trains: it will stay disabled on aurora, beta, release, esr.

Relatedly, two years ago (already), I landed jemalloc 3.0.0, and updated it until 3.2.0 six months later. It was and still is disabled by default. Sadly, it hasn't seen an update since then. The recent increase in activity around improving the memory footprint of our own fork of jemalloc (dating back to before version 1.0) made me want to update it at last.

This is now done, and the tree contains a (slightly patched) copy of jemalloc 3.6.0. And combined with replace-malloc, it is now possible to test it on nightly builds (well, starting from the one after the next mozilla-inbound merge to mozilla-central) with the following:

• On GNU/Linux:
  

  $ LD_PRELOAD=/path/to/libreplace_jemalloc.so firefox

• On OSX:
  

  $ DYLD_INSERT_LIBRARIES=/path/to/libreplace_jemalloc.dylib firefox

• On Windows:
  

  $ MOZ_REPLACE_MALLOC_LIB=drive:\path\to\replace_jemalloc.dll firefox

• On Android:
  

  $ am start -a android.activity.MAIN -n org.mozilla.fennec/.App --es env0 MOZ_REPLACE_MALLOC_LIB=/path/to/libreplace_jemalloc.so

No, you don't need to rebuild Firefox to test it with jemalloc 3.6.0. The relevant library is now shipped in the nightly builds. Except on Android, as I haven't figured where to put it, but you can take the .so file from a local build and use it with a nightly build.

I would appreciate if several people could start using jemalloc 3.x this way. There is still work to do to make it the default. In fact, the list of dependencies of the tracking bug still has the same bugs I filed a long time ago. Hopefully, the ease of use will make someone want to scratch those itches. Please ping me if you want to take one of those bugs.

I just saw two related gists about how some people are building Firefox.

Both are doing the same mistake, which is not really surprising, since one is based on the other. As I'm afraid people might pick that up, I'm posting this:

Don't ever use in-tree mozconfigs

If your mozconfig contains something like

. $topsrcdir/something

Then remove it. Now.

Those mozconfigs are for use in automated builds. They make many assumptions on the build environment being the one from the build slaves. Local developers shouldn't need anything but minimalistic, self contained mozconfigs. If there are things that can be changed in the build system to accommodate developers, file bugs (I could certainly see the .noindex thing automatically added to MOZ_OBJDIR by default on mac)

Corollary: if you can't build Firefox without a mozconfig (for a reason other than your build environment missing build requirements), file a bug.

If you're following this blog, you may be aware of the recent work on shared compilation cache. This has been deployed with great results on Mozilla's try server for all platforms (except a few build types, like ASAN or valgrind), and is being tested for Linux/Android builds on b2g-inbound (more on that in subsequent posts).

A side effect of the work to make it run on all platforms is that it now works to build Firefox on Windows, although it requires a specific setup. And since recently, it's also possible to use it with local storage instead of S3. This means we now have a (basic) ccache for Windows that works to build Firefox.

If you wish to try it, here is what you need to do:

• Clone the repository from github:
  

  $ git clone https://github.com/glandium/sccache

• Add the following to your mozconfig:
  

  ac_add_options "--with-compiler-wrapper=python2.7 path/to/sccache/sccache.py"
export _DEPEND_CFLAGS='-deps$(MDDEPDIR)/$(@F).pp'
mk_add_options "export CC_WRAPPER="
mk_add_options "export CXX_WRAPPER="
mk_add_options "export COMPILE_PDB_FLAG="
mk_add_options "export HOST_PDB_FLAG="
mk_add_options "export MOZ_DEBUG_FLAGS=-Z7"


  Update: Currently, path/to/sccache/sccache.py needs to be a windows-like path (as opposed to msys/cygwin path) with forward slashes.

• Then set the SCCACHE_DIR environment variable to some local directory.
• And build happily.

A few things to note:

• As of writing, sccache doesn't support cleaning up the storage directory, so it will grow indefinitely (until you clean it up yourself).
• Because the MSVC preprocessor is not exactly fast, and because sccache doesn't have a direct mode like ccache, it doesn't make as much difference as ccache does.
• It also works on non-windows, but doesn't require all the mozconfig changes, except for the --with-compiler-wrapper line.

Play with it and feel free to fork it on github, and improve it. Pull requests encouraged.