Archive for the 'p.m.o' Category

Starting with Firefox 68, the Linux builds shipped by Mozilla should be reproducible (it is not currently automatically validated that it definitely is, but 68.0 is). These builds are optimized with Profile Guided Optimization, and the profile data was not kept and published until recently, which is why they weren't reproducible until now.

The following instructions require running Docker on a Linux host (this may or may not work on a non-Linux host, I don't know what e.g. Docker for Mac does, and if the docker support in the mach command works with it). I'll try to make them generic enough that they may apply to any subsequent release of Firefox.

• Clone either the mozilla-unified or mozilla-release repository. You can use Mercurial or Git (with git-cinnabar), it doesn't matter.
• Checkout the FIREFOX_68_0_RELEASE tag and find out what its Mercurial changeset id is (it is 353628fec415324ca6aa333ab6c47d447ecc128e).
• Open the Taskcluster index tool in a browser tab.
• In the input field type or copy/paste gecko.v2.mozilla-release.shippable.revision.353628fec415324ca6aa333ab6c47d447ecc128e.firefox.linux64-opt and press the Enter key. (replace 353628fec415324ca6aa333ab6c47d447ecc128e with the right revision if you're trying for another release)
• This will fill the "Indexed Task" pane, where you will find a TaskId. Follow the link there, it will bring you to the corresponding Task Run Logs
• Switch to the Task Details
• Scroll down to the "Dependencies" list, and check the task name that begins with "build-docker-image". For the Firefox 68 build task, it is build-docker-image-debian7-amd64-build.
• Take that name, remove the "build-docker-image-" prefix, and run the following command, from inside the repository, to download the corresponding docker image:

  $ ./mach taskcluster-load-image debian7-amd64-build

  Obviously, replace debian7-amd64-build with whatever you found in the task dependencies. The image can also be built from the source tree, but this is out of scope for this post.

• The command output will give you a docker run -ti ... command to try. Run it. It will open a shell in the docker image.
• From the docker shell, run the following commands:

  $ echo no-api-key > /builds/mozilla-desktop-geoloc-api.key
  $ echo no-api-key > /builds/sb-gapi.data
  $ echo no-api-key > /builds/gls-gapi.data
  

  Or replace no-api-key with the actual keys if you have them.

• Back to the Task Details, check the env part of the "Payload". You'll need to export all these variables with the corresponding values. e.g.

  $ export EXTRA_MOZHARNESS_CONFIG='{"update_channel": "release", "mozconfig_variant": "release"}'
  $ export GECKO_BASE_REPOSITORY='https://hg.mozilla.org/mozilla-unified'
  $ export GECKO_HEAD_REPOSITORY='https://hg.mozilla.org/releases/mozilla-release'
  ...

• Set the missing TASKCLUSTER_ROOT_URL environment variable:

  $ export TASKCLUSTER_ROOT_URL='https://taskcluster.net'

• Change the value of MOZHARNESS_ACTIONS to:

  $ export MOZHARNESS_ACTIONS='build'

  The original value contains get-secrets, which will try to download from http://taskcluster/, which will fail with a DNS error, and check-test, which runs make check, which is not necessary to get a working Firefox.

• Take command part of the "Payload", and run that in the docker shell:

  $ /builds/worker/bin/run-task --gecko-checkout /builds/worker/workspace/build/src -- /builds/worker/workspace/build/src/taskcluster/scripts/builder/build-linux.sh

• Once the build is finished, in another terminal, check what the container id of your running docker container is, and extract the build artifact from there:

  $ docker ps
  CONTAINER ID        IMAGE                                                                                  COMMAND             CREATED             STATUS              PORTS               NAMES
  d234383ba9c7        debian7-amd64-build:be96d1b734e1a152a861ce786861fca6e70bcb996bf67347f5af4f146db157ec   "bash"              2 hours ago         Up 2 hours                              nifty_hermann
  $ docker cp d234383ba9c7:/builds/worker/artifacts/target.tar.bz2 .

  (replace d234383ba9c7 with your container id)

• Now you can exit the docker shell. That will remove the container.

After all the above, you can finally compare your target.tar.bz2 to the Linux64 Firefox 68 release. You will find a few inevitable differences:

• The .chk files will be different, because they are self-signatures for FIPS mode that are generated with one-time throw-away keys.
• The Firefox 68 release contains .sig files that your build won't contain. They are signature files, which aren't reproducible outside Mozilla automation for obvious reasons.
• Consequently, the precomplete file contains instructions for the .sig files in the Firefox 68 release that won't be in your build.
• The omni.ja files are different. If you extract them (they are uncompressed zip files with a few tweaks to the format), you'll see the only difference is in modules/AppConstants.jsm, for the three API keys you created a file for earlier.

Everything else is identical bit for bit.

All the above is a rather long list of manual steps. Ideally, most of it would be automated. We're not there yet. We only recently got to the point where the profile data is available to make it possible at all. In other words, this is a starting point. It's valuable to know it does work but requires manual steps and what those are.

It is also worth noting that while the above downloads and uses pre-built compilers and other tools, it is also possible to rebuild those, although they likely won't be bit-for-bit identical. But differences in those shouldn't incur differences in Firefox. Replacing the pre-built ones with ones you'd build yourself unfortunately currently requires some more manual work.

As for Windows and Mac builds, long story short, they are not reproducible as of writing. Mac builds are not optimized with PGO, but Windows builds are, and their profile data won't be available until Firefox 69. Both platforms require SDKs that Mozilla can't redistribute per their license (but are otherwise available for download from Microsoft or Apple, respectively), which makes the setup more complex. And in all likeliness, for both platforms, the toolchains are not deterministic yet (that's at least true for Mac). Also, binary signatures would need to be tripped off the executables and libraries before any comparison.

How is that for clickbait?

With the now released git-cinnabar 0.5.2, the cinnabarclone feature is enabled by default, which means it doesn't need to be enabled manually anymore.

Cinnabarclone is to git-cinnabar what clonebundles is to Mercurial (to some extent). Clonebundles allow Mercurial to download a pre-generated bundle of a repository, which reduces work on the server side. Similarly, Cinnabarclone allows git-cinnabar to download a pre-generated bundle of the git form of a Mercurial repository.

Thanks to Connor Sheehan, who deployed the necessary extension and configuration on the server side, cinnabarclone is now enabled for mozilla-central and mozilla-unified, making git-cinnabar clone faster than ever for these repositories. In fact, under some conditions (mostly depending on network bandwidth), cloning with git-cinnabar is now faster than cloning with Mercurial:

$ time git clone hg::https://hg.mozilla.org/mozilla-unified mozilla-unified_git
Cloning into 'mozilla-unified_git'...
Fetching cinnabar metadata from https://index.taskcluster.net/v1/task/github.glandium.git-cinnabar.bundle.mozilla-unified/artifacts/public/bundle.git
Receiving objects: 100% (12153616/12153616), 2.67 GiB | 41.41 MiB/s, done.
Resolving deltas: 100% (8393939/8393939), done.
Reading 172 changesets
Reading and importing 170 manifests
Reading and importing 758 revisions of 570 files
Importing 172 changesets
It is recommended that you set "remote.origin.prune" or "fetch.prune" to "true".
git config remote.origin.prune true
or
git config fetch.prune true

Run the following command to update tags:
git fetch --tags hg::tags: tag "*"
Checking out files: 100% (279688/279688), done.

real    4m57.837s
user    9m57.373s
sys     0m41.106s

$ time hg clone https://hg.mozilla.org/mozilla-unified
destination directory: mozilla-unified
applying clone bundle from https://hg.cdn.mozilla.net/mozilla-unified/5ebb4441aa24eb6cbe8dad58d232004a3ea11b28.zstd-max.hg
adding changesets
adding manifests
adding file changes
added 537259 changesets with 3275908 changes to 523698 files (+13 heads)
finished applying clone bundle
searching for changes
adding changesets
adding manifests
adding file changes
added 172 changesets with 758 changes to 570 files (-1 heads)
new changesets 8b3c35badb46:468e240bf668
537259 local changesets published
updating to branch default
(warning: large working directory being used without fsmonitor enabled; enable fsmonitor to improve performance; see "hg help -e fsmonitor")
279688 files updated, 0 files merged, 0 files removed, 0 files unresolved

real    21m9.662s
user    21m30.851s
sys     1m31.153s

To be fair, the Mozilla Mercurial repos also have a faster "streaming" clonebundle that they only prioritize automatically if the client is on AWS currently, because they are much larger, and could take longer to download. But you can opt-in with the --stream command line argument:

$ time hg clone --stream https://hg.mozilla.org/mozilla-unified mozilla-unified_hg
destination directory: mozilla-unified_hg
applying clone bundle from https://hg.cdn.mozilla.net/mozilla-unified/5ebb4441aa24eb6cbe8dad58d232004a3ea11b28.packed1.hg
525514 files to transfer, 2.95 GB of data
transferred 2.95 GB in 51.5 seconds (58.7 MB/sec)
finished applying clone bundle
searching for changes
adding changesets
adding manifests
adding file changes
added 172 changesets with 758 changes to 570 files (-1 heads)
new changesets 8b3c35badb46:468e240bf668
updating to branch default
(warning: large working directory being used without fsmonitor enabled; enable fsmonitor to improve performance; see "hg help -e fsmonitor")
279688 files updated, 0 files merged, 0 files removed, 0 files unresolved

real    1m49.388s
user    2m52.943s
sys     0m43.779s

If you're using Mercurial and can download 3GB in less than 20 minutes (in other words, if you can download faster than 2.5MB/s), you're probably better off with the streaming clone.

Bonus fact: the Git clone is smaller than the Mercurial clone

The Mercurial streaming clone bundle contains data in a form close to what Mercurial puts on disk in the .hg directory, meaning the size of .hg is close to that of the clone bundle. The Cinnabarclone bundle contains a git pack, meaning the size of .git is close to that of the bundle, plus some more for the pack index file that unbundling creates.

The amazing fact is that, to my own surprise, the git pack, containing the repository contents along with all git-cinnabar needs to recreate Mercurial changesets, manifests and files from the contents, takes less space than the Mercurial streaming clone bundle.

And that translates in local repository size:

$ du -h -s --apparent-size mozilla-unified_hg/.hg
3.3G    mozilla-unified_hg/.hg
$ du -h -s --apparent-size mozilla-unified_git/.git
3.1G    mozilla-unified_git/.git

And because Mercurial creates so many files (essentially, two per file that ever was in the repository), there is a larger difference in block size used on disk:

$ du -h -s mozilla-unified_hg/.hg
4.7G    mozilla-unified_hg/.hg
$ du -h -s mozilla-unified_git/.git
3.1G    mozilla-unified_git/.git

It's even more mind blowing when you consider that Mercurial happily creates delta chains of several thousand revisions, when the git pack's longest delta chain is 250 (set arbitrarily at pack creation, by which I mean I didn't pick a larger value because it didn't make a significant difference). For the casual readers, Git and Mercurial try to store object revisions as a diff/delta from a previous object revision because that takes less space. You get a delta chain when that previous object revision itself is stored as a diff/delta from another object revision itself stored as a diff/delta ... etc.

My guess is that the difference is mainly caused by the use of line-based deltas in Mercurial, but some Mercurial developer should probably take a deeper look. The fact that Mercurial cannot delta across file renames is another candidate.

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

These release notes are also available on the git-cinnabar wiki.

What's new since 0.5.1?

• Updated git to 2.22.0 for the helper.
• cinnabarclone support is now enabled by default. See details in README.md and mercurial/cinnabarclone.py.
• cinnabarclone now supports grafted repositories.
• git cinnabar fsck now does incremental checks against last known good state.
• Avoid git cinnabar sometimes thinking the helper is not up-to-date when it is.
• Removing bookmarks on a Mercurial server is now working properly.

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

These release notes are also available on the git-cinnabar wiki.

What's new since 0.5.0?

• Updated git to 2.21.0 for the helper.
• Experimental native mercurial support (used when mercurial libraries are not available) now has feature parity.
• Try to read the git system config from the same place as git does. This fixes native HTTPS support with Git on Windows.
• Avoid pushing more commits than necessary in some corner cases (see e.g. https://bugzilla.mozilla.org/show_bug.cgi?id=1529360).
• Added an --abbrev argument for git cinnabar {git2hg,hg2git} to display shortened sha1s.
• Can now pass multiple revisions to git cinnabar fetch.
• Don't require the requests python module for git cinnabar download.
• Fixed git cinnabar fsck file checks to actually report errors.
• Properly return an error code from git cinnabar rollback.
• Track last fsck'ed metadata and allow git cinnabar rollback --fsck to go back to last known good metadata directly.
• git cinnabar reclone can now be rolled back.
• Added support for git bundles as cinnabarclone source.
• Added alternate styles of remote refs.
• More resilient to interruptions when HTTP Range requests are supported.
• Fixed off-by-one when storing mercurial heads.
• Better handling of mercurial branchmap tips.
• Better support for end of parts in bundle v2.
• Improvements handling urls to local mercurial repositories.
• Fixed compatibility with (very) old mercurial servers when using mercurial 5.0 libraries.
• Converted Continuous Integration scripts to Python 3.

As of last nightly (20181115100051), Firefox now supports Wayland on Linux, thanks to the work from Martin Stransky and Jan Horak, mostly.

Before that, it was possible to build your own Firefox with Wayland support (and Fedora does it), but now the downloads from mozilla.org come with Wayland support out of the box for the first time.

However, being experimental and all, the Wayland support is not enabled by default, meaning by default, you'll still be using XWayland. To enable wayland support, first set the GDK_BACKEND environment variable to wayland.

To verify whether Wayland support is enabled, go to about:support, and check "WebGL 1 Driver WSI Info" and/or "WebGL 2 Driver WSI Info". If they say something about GLX, Wayland support is not enabled. If they say something about EGL, it is. I filed a bug to make it more obvious what is being used.

It's probably still a long way before Firefox enables Wayland support on Wayland by default, but we reached a major milestone here. Please test and report any bug you encounter.

Update: I should mention that should you build your own Firefox, as long as your Gtk+ headers come with Wayland support, you'll end up with the same Wayland support as the one shipped by Mozilla.

You might have read that Mozilla recently switched Windows builds to clang-cl. More recently, those Windows builds have seen both PGO (Profile-Guided Optimization) and LTO (Link-Time Optimization) enabled.

As of next nightly (as of writing, obviously), all tier-1 platforms are now built with clang with LTO enabled. Yes, this means Linux, Mac and Android arm, aarch64 and x86. Linux builds also have PGO enabled.

Mac and Android builds were already using clang, so the only difference is LTO being enabled, which brought some performance improvements.

The most impressive difference, though, was on Linux, where we're getting more than 5% performance improvements on most Talos tests (up to 18% (!) on some tests) compared to GCC 6.4 with PGO. I must say I wasn't expecting switching from GCC to clang would make such a difference. And that is with clang 6. A quick test with upcoming clang 7 suggests we'd additionally get between 2 and 5% performance improvement from an upgrade, but our static analysis plugin doesn't like it.

This doesn't mean GCC is being unsupported. As a matter of fact, we still have automated jobs using GCC for some static analysis, and we also have jobs ensuring everything still builds with a baseline of GCC 6.x.

You might wonder if we tried LTO with GCC, or tried upgrading to GCC 8.x. As a matter of fact, I did. Enabling LTO turned up linker errors, and upgrading to GCC 7.x turned up breaking binary compatibility with older systems, and if I remember correctly had some problems with our test suite. GCC 8.1 was barely out when I was looking into this, and we all know to stay away from any new major GCC version until one or two minor updates. Considering the expected future advantages from using clang (cross-language inlining with Rust, consistency between platforms), it seemed a better deal to switch to clang than to try to address those issues.

Update: As there's been some interest on reddit and HN, and I failed to mention it originally, it's worth noting that comparing GCC+PGO vs. clang+LTO or GCC+PGO vs. clang+PGO was a win for clang overall in both cases, although GCC was winning on a few benchmarks. If I remember correctly, clang without PGO/LTO was also winning against GCC without PGO.

Anyways, what led me on this quest was a casual conversation at our last All Hands, where we were discussing possibly turning on LTO on Mac, and how that should roughly just be about turning a switch.

Famous last words.

At least, that's a somehow reasonable assumption. But when you have a codebase the size of Firefox, you're up for "interesting" discoveries.

This involved compiler bugs, linker bugs (with a special mention for a bug in ld64 that Apple has apparently fixed in Xcode 9 but hasn't released the source of), build system problems, elfhack issues, crash report problems, clang plugin problems (would you have guessed that __attribute__((annotate("foo"))) can affect the generated machine code?), sccache issues, inline assembly bugs (getting inputs, outputs and clobbers correctly is hard), binutils bugs, and more.

I won't bother you with all the details, but here we are, 3 months later with it all, finally, mostly done. Counting only the bugs assigned to me, there are 77 bugs on bugzilla (so, leaving out anything in other bug trackers, like LLVM's). Some of them relied on work from other people (most notably, Nathan Froyd's work to switch to clang and then non-NDK clang on Android). This spread over about 150 commits on mozilla-central, 20 of which were backouts. Not everything went according to plan, obviously, although some of those backouts were on purpose as a taskcluster trick.

Hopefully, this sticks, and Firefox 64 will ship built with clang with LTO on all tier-1 platforms as well as PGO on some. Downstreams are encouraged to do the same if they can. The build system will soon choose clang by default on all builds, but won't enable PGO/LTO.

As a bonus, as of a few days ago, Linux builds are also finally using Position Independent Executables, which improves Address Space Layout Randomization for the few things that are in the executables instead of some library (most notably, mozglue and the allocator). This was actually necessary for LTO, because clang doesn't build position independent code in executables that are not PIE (but GCC does), and that causes other problems.

Work is not entirely over, though, as more inline assembly bugs might be remaining only not causing visible problems by sheer luck, so I'm now working on a systematic analysis of inline assembly blocks with our clang plugin.

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

These release notes are also available on the git-cinnabar wiki.

What's new since 0.4.0?

• git-cinnabar-helper is now mandatory. You can either download one with git cinnabar download on supported platforms or build one with make.
• Performance and memory consumption improvements.
• Metadata changes require to run git cinnabar upgrade.
• Mercurial tags are consolidated in a separate (fake) repository. See the README file.
• Updated git to 2.18.0 for the helper.
• Improved memory consumption and performance.
• Improved experimental support for pushing merges.
• Support for clonebundles for faster clones when the server provides them.
• Removed support for the .git/hgrc file for mercurial specific configuration.
• Support any version of Git (was previously limited to 1.8.5 minimum)
• Git packs created by git-cinnabar are now smaller.
• Fixed incompatibilities with Mercurial 3.4 and >= 4.4.
• Fixed tag cache, which could lead to missing tags.
• The prebuilt helper for Linux now works across more distributions (as long as libcurl.so.4 is present, it should work)
• Properly support the pack.packsizelimit setting.
• Experimental support for initial clone from a git repository containing git-cinnabar metadata.
• Now can successfully clone the pypy and GNU octave mercurial repositories.
• More user-friendly errors.

Development process changes

It took about 6 months between version 0.3 and 0.4. It took more than 18 months to reach version 0.5 after that. That's a long time to wait for a new version, considering all the improvements that have happened under the hood.

From now on, the release branch will point to the last tagged release, which is roughly the same as before, but won't be the default branch when cloning anymore.

The default branch when cloning will now be master, which will receive changes that are acceptable for dot releases (0.5.x). These include:

• Changes in behavior that are backwards compatible (e.g. adding new options which default to the current behavior).
• Changes that improve error handling.
• Changes to existing experimental features, and additions of new experimental features (that require knobs to be enabled).
• Changes to Continuous Integration/Tests.
• Git version upgrades for the helper.

The next branch will receive changes for the next "major" release, which as of writing is planned to be 0.6.0. These include:

• Changes in behavior.
• Changes in metadata.
• Stabilizing experimental features.
• Remove backwards compability with older metadata (< 0.5.0).

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

These release notes are also available on the git-cinnabar wiki.

What's new since 0.5.0 beta 3?

• Fixed incompatibility with Mercurial 3.4.
• Performance and memory consumption improvements.
• Work around networking issues while downloading clone bundles from Mozilla CDN with range requests to continue past failure.
• Miscellaneous metadata format changes.
• The prebuilt helper for Linux now works across more distributions (as long as libcurl.so.4 is present, it should work)
• Updated git to 2.18.0 for the helper.
• Properly support the pack.packsizelimit setting.
• Experimental support for initial clone from a git repository containing git-cinnabar metadata.
• Changed the default make rule to only build the helper.
• Now can successfully clone the pypy and GNU octave mercurial repositories.
• More user-friendly errors.

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

These release notes are also available on the git-cinnabar wiki.

What's new since 0.5.0 beta 2?

• Fixed incompatibilities with Mercurial >= 4.4.
• Miscellaneous metadata format changes.
• Move more operations to the helper, hopefully making things faster.
• Updated git to 2.17.0 for the helper.
• Properly handle clones with bundles when the repository doesn't contain anything newer than the bundle.
• Fixed tag cache, which could lead to missing tags.

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

These release notes are also available on the git-cinnabar wiki.

What's new since 0.5.0 beta 1?

• Enabled support for clonebundles for faster clones when the server provides them.
• Git packs created by git-cinnabar are now smaller.
• Added a new git cinnabar upgrade command to handle metadata upgrade separately from fsck.
• Metadata upgrade is now significantly faster.
• git cinnabar fsck also faster.
• Both now also use significantly less memory.
• Updated git to 2.13.1 for git-cinnabar-helper.