A new Jemalloc has landed

... disabled by default.

Firefox 3, released close to 4 years ago, came with its own memory allocator: Jemalloc. Jason Evans (original author, and to this day, still upstream maintainer) and our own Stuart Parmenter worked hard to have it work in Firefox for Windows and Linux. Sadly, a lot of this work stayed Firefox-only. Time passed, we added support for OSX, fixed various issues and added various features. Mostly, this all stayed Firefox-only.

By then, the original Jemalloc was a 0.x version. And while we have been busy growing our own fork, Jason has been busy growing Jemalloc. We've sometimes retrofitted new things from the new Jemalloc into ours, but all in all, both grew in very separate ways, and it was hard to benefit from each other's work.

During the past weeks, I've been working on getting the upstream Jemalloc development branch in shape so that we can use it in Firefox. It involved porting that code-base to Windows, fixing it for OSX, and fix some other issues found on the way. All this work was incorporated upstream and is part of the latest Jemalloc 3.0.0 release.

What landed today on mozilla-central is a pristine copy of Jemalloc 3.0.0 and the necessary bits to build and link it in Firefox. It is however disabled by default until we fix all remaining issues. See the dependencies of bug 762449 for what is left to be done.

If you want to give a hand to make Jemalloc 3 the default, you first need to enable it at build time, which is achieved by adding the following to your .mozconfig:

export MOZ_JEMALLOC=1

Once Jemalloc 3 is the default, it will be straightforward to update our copy to use the latest from upstream.

2012-06-07 17:37:44+0900

p.m.o

Responses are currently closed, but you can trackback from your own site.

12 Responses to “A new Jemalloc has landed”

  1. Benoit Girard Says:
    2012-06-07 19:22:20+0900

    Excellent!

  2. scottjad Says:
    2012-06-07 21:29:27+0900

    gcc -o arena.o -c -I../../dist/system_wrappers -include /var/tmp/mozilla-central/config/gcc_hidden.h -DMOZ_GLUE_IN_PROGRAM -D_GNU_SOURCE -I/var/tmp/m
    ozilla-central/memory/jemalloc/src/include -Isrc/include -I/var/tmp/mozilla-central/memory/jemalloc -I. -I../../dist/include -I../../dist/include/ns
    prpub -I/usr/include/nspr -I/usr/include/nss -fPIC -pedantic -Wall -Wpointer-arith -Wdeclaration-after-statement -Werror=return-type -Wtype-li
    mits -Wempty-body -Wno-unused -Wno-overlength-strings -Wcast-align -Wno-long-long -march=native -fno-strict-aliasing -ffunction-sections -fdata-secti
    ons -pthread -pipe -DNDEBUG -DTRIMMED -Os -freorder-blocks -fomit-frame-pointer -std=gnu99 -include ../../mozilla-config.h -DMOZILLA_CLIENT -MD –
    MF .deps/arena.pp /var/tmp/mozilla-central/memory/jemalloc/src/src/arena.c
    In file included from src/include/jemalloc/internal/jemalloc_internal.h:470:0,
    from /var/tmp/mozilla-central/memory/jemalloc/src/src/chunk_dss.c:2:
    /var/tmp/mozilla-central/memory/jemalloc/src/include/jemalloc/internal/ctl.h:46:28: error: ‘NBINS’ undeclared here (not in a function)
    …

  3. glandium Says:
    2012-06-07 21:40:10+0900

    scottjad: Did configure run?

  4. scottjad Says:
    2012-06-07 21:54:07+0900

    Yes:
    …
    configuring in memory/jemalloc/src
    …
    ===============================================================================
    jemalloc version : 3.0.0-0-gfc9b1dbf69f59d7ecfc4ac68da9847e017e1d046
    library revision : 1
    …

  5. Manoj Mehta Says:
    2012-06-07 21:56:17+0900

    Awesome!

    What does this landing bring for regular users (not developers)? Better memory management? Less fragmentation? Something else?

    Thanks.

  6. scottjad Says:
    2012-06-07 21:58:37+0900

    Ah I see what is wrong:
    ./config.status: line 1123: /var/tmp/mozilla-central/memory/jemalloc/src/include/jemalloc/internal/size_classes.sh: Permission denied

    chmod +x /var/tmp/mozilla-central/memory/jemalloc/src/include/jemalloc/internal/size_classes.sh
    fixed the issue.
    I’m a git user BTW.

  7. glandium Says:
    2012-06-07 22:17:40+0900

    scottjad: interesting… in mercurial, the executable bit is recorded, i’d expect git to pick it… except if the git/mercurial bridge is broken.

    Manoj Mehta: since it’s disabled, nothing.

  8. pd Says:
    2012-06-08 18:08:15+0900

    @glandium: what will this landing – when eventually enabled – bring for end users?

  9. glandium Says:
    2012-06-08 22:36:05+0900

    pd: hopefully, a faster firefox, and less memory usage. We’re not there yet after tab close, but peak memory usage is already slightly lower than with current jemalloc. The main point is that we’re going to be able to pick upstream improvements, and we’ll be able to share our improvements with upstream. A win-win situation.

  10. Jason Evans Says:
    2012-06-11 03:52:18+0900

    @glandium did a lot of great work to make this possible — thanks! To my mind, one of the biggest benefits is that all jemalloc users will be using a more broadly tested version of the code. There were distinct bugs in the FreeBSD, Mozilla, and stand-alone versions (some known, some previously unknown); those bugs have been fixed thanks to the three unique testing environments.

  11. TK Says:
    2012-06-12 16:59:04+0900

    Are these potential memory allocator security issues fixed in all Firefox versions (seems so, blog tells jemalloc was fixed in 2006-2008)?

    http://seclists.org/oss-sec/2012/q2/461
    http://kqueue.org/blog/2012/03/05/memory-allocator-security-revisited/

  12. antistress Says:
    2012-09-12 01:35:10+0900

    @glandium : Would it be possible to see your full mozconfig file for building iceweasel on Debian ? Merci d’avance !