Archive for the 'p.m.o' Category

On this very day five years ago, I committed the initial code of what later became git-cinnabar. It is kind of an artificial anniversary, because I didn't actually publish anything until 3 weeks later, and I also had some prototypes months earlier.

The earlier prototypes of what I'll call "pre-git-cinnabar" could handle doing git clone hg::https://hg.mozilla.org/mozilla-central (that is, creating a git clone of a Mercurial repository), but they couldn't git pull later. That pre-git-cinnabar initial commit, however, was the first version that did.

The state of the art back then was similar git helpers, the most popular choice being Felipec's git-remote-hg, or the opposite tool: hg-git, a mercurial plugin that allows to push to a git repository.

They both had the same caveats: they were slow to handle a repository the size of mozilla-central back then, and both required a local mercurial repository (hidden in the .git directory in the case of Felipec's git-remote-hg).

This is what motivated me to work on pre-git-cinnabar, which was also named git-remote-hg back then because of how git requires a git-remote-hg executable to handle hg::-prefixed urls.

Fast forward five years, mozilla-central has grown tremendously, and another mozilla-unified repository was created that aggregates the various mozilla branches (esr*, release, beta, central, integration/*).

git-cinnabar went through multiple versions, multiple changes to the metadata it keeps, and while I actually haven't cumulatively worked all that much on it considering the number of years, a lot of progress has been made.

But let's go back to the 19th of November 2014. Thankfully, Mercurial allows to strip everything past a certain date, artificially allowing to restore the state of the repository at that date. Unfortunately, pre-git-cinnabar only supports the old Mercurial bundle format, which both the mozilla-central and mozilla-unified repositories now don't allow. So pre-git-cinnabar can't clone them out of the box anymore. It's still possible to allow it in mirror repositories, but because they now use generaldelta, that incurs a server-side conversion that is painfully slow (the hg.mozilla.org server rejects clients that don't support the new format for this reason).

So for testing purposes, I setup a nginx reverse-proxy and cache, such that the conversion only happens once, and performed clones multiple times, removing any bundling and conversion cost out of the equation. And tested the current version of Felipec's git-remote-hg, the current version of hg-git, pre-git-cinnabar, and last git-cinnabar release (0.5.2 as of writing), on some AWS instances, with Xeon Platinum 8124M 3Ghz CPUs. That's a different CPU from what I had back in 2014, yielding some different results from what I wrote in that first announcement.

I've thus cloned both mozilla-central (denoted m-c) and mozilla-unified (denoted m-u), with simulated old states of the repositories. Mozilla-unified didn't exist before 2016, but it's still interesting to simulate its content as if it had existed because it allows to learn how the tools perform with the additional branches it contains, with the implication they have on how the data is stored in the repository.

Note: I didn't test older versions of git-remote-hg or hg-git to see how they performed at the time, and how things evolved for them.

There are multiple things of note in the results above:

• I wrote back then that cloning took 12 hours with git-remote-hg and 30 minutes with pre-git-cinnabar on the machine I used. And while cloning with pre-git-cinnabar on more modern hardware was much faster (16 minutes), cloning with git-remote-hg wasn't. The pre-git-cinnabar improvement could, though, be attributed in part to improvements in git-fast-import itself (I haven't tested older versions). But it's also not impossible that git-remote-hg regressed. Only further testing would tell.
• mozilla-unified is bigger than mozilla-central, because it is a superset, and that reflects on the clone times, but hg-git and pre-git-cinnabar are both much slower to clone mozilla-unified than you'd expect from the difference in size, especially hg-git. git-cinnabar made a lot of progress in that regard.
• I hadn't tested hg-git back then, but while it's not as slow as git-remote-hg, it's still horribly slow for a repository this size.

Let's now look at the .git sizes:

Those are the sizes for the .git directory fresh after cloning. In all cases, git gc --aggressive would make the clone smaller, at the cost of CPU time (although not significantly smaller in the git-cinnabar case). And after you spent 12 hours cloning, are you really going to spend another large number of hours on a git gc to save disk space?

It is worth noting that in the case of hg-git, this doesn't include the size of the mercurial repository required to maintain the git repository, while it is included for git-remote-hg, where it is hidden in .git, as mentioned earlier. That puts them about on par w.r.t size.

It's interesting how close hg-git and git-remote-hg are in disk usage, when the former uses dulwich, a pure Python implementation of Git, and the latter uses git-fast-import. pre-git-cinnabar used git-fast-import too, but optimized the data it sent to git-fast-import to allow for a more compact .git. Recent git-cinnabar made it even better, although it doesn't use git-fast-import directly, instead using a helper derived
from git-fast-import.

But that was 2014. Let's look at how things evolved over time, by taking "snapshots" of the repositories at one year interval, starting in November 2007.

Of note:

• pre-git-cinnabar somehow invalidated the nginx cache for years >= 2016 for mozilla-unified, which didn't allow to get reliable measures.
• Things went well out of hand with git-remote-hg and hg-git, so much so that I wasn't able to get results for git-remote-hg clones for 2019 in time for this post. They're now getting clone times that count in days rather than hours.
• Things are getting much worse with mozilla-unified, relatively to mozilla-central, for hg-git than they do for git-remote-hg or git-cinnabar, while they were really bad with pre-git-cinnabar.
• pre-git-cinnabar clone times for mozilla-central are indistinguishable from git-cinnabar's at this scale (but see further below).
• the progression is not linear, but the progression in repository size wasn't linear either. In order to get a slightly better picture, it is better to look at the clone times vs. the size of the repositories. One measure of that size is number of objects (changeset, manifests and file revisions they contain).

The progression here looks more linear, but still not quite linear. The difference between the mozilla-central and mozilla-unified clone times is the most damning, especially for hg-git and pre-git-cinnabar. At this scale things don't look so bad for git-cinnabar, but looking closer, they aren't actually much better:

mozilla-central clone times have slightly improved since pre-git-cinnabar days, at least more than the comparison with hg-git and git-remote-hg suggested. mozilla-unified clone times, however, have dramatically improved (notwithstanding the fact that it's currently not possible to clone with pre-git-cinnabar at all directly from hg.mozilla.org).

But clone times are starting to get a little out of hand, especially for mozilla-unified, which is why I've recently added support for "clone bundles". But I also have work in progress that I expect will make non-bundle clones faster too, and hopefully more linear.

As for .git sizes:

• hg-git and git-remote-hg are still hand in hand.
• Here the progression is mostly linear, with almost no difference between mozilla-central and mozilla-unified, as one could expect.
• I think the larger increase in size between what would be 2017 and 2018 is largely due to the testing/web-platform/meta/MANIFEST.json file.
• People who try to clone the Mozilla repositories with hg-git or git-remote-hg at this point better have a lot of time and a lot of free disk space.

While git-cinnabar is demonstrably significantly faster than both git-remote-hg and hg-git by a large margin for the Mozilla repositories (more than an order of magnitude), looking at the data more closely revealed something interesting that can be pictured in the following graph, plotting how much slower than git-cinnabar the other tools are.

The ratio is not constant, and has surprisingly been decreasing steadily since 2016, correlating with the observation that clone times are getting slower more quickly than the repositories are growing. But they are doing more so with git-cinnabar than with the other tools. Clone times with git-cinnabar have multiplied by more than 5 in five years, for a repository that only has 2.5 times more objects. At this pace, in five more years, clones will take well above 10 hours, and that's not counting for the accelerated slowdown. Hopefully, the current work in progress will help.

It's also interesting to see how the ratios changed after 2011 between mozilla-central and mozilla-unified. 2011 is when Firefox 4 was released and the release process switched to multiple repositories, which mozilla-unified, well, unified in a single repository. So mozilla-unified and mozilla-central were largely identical when stripped of commits after 2011 and started diverging afterwards.

To conclude this rather long post, pre-git-cinnabar improved the state of the art to clone large Mercurial repositories, and git-cinnabar went further in the past five years. But without more work, things will get out of hand. And that only accounts for clone times. I haven't spent much time working on other aspects, like negotiation times (the time it takes to agree with a Mercurial server what the git clone has in common with it), or bundling times (the time it takes to generate a bundle to send a Mercurial server). Both are the relevant parts of push times.

Starting with Firefox 68, the Linux builds shipped by Mozilla should be reproducible (it is not currently automatically validated that it definitely is, but 68.0 is). These builds are optimized with Profile Guided Optimization, and the profile data was not kept and published until recently, which is why they weren't reproducible until now.

The following instructions require running Docker on a Linux host (this may or may not work on a non-Linux host, I don't know what e.g. Docker for Mac does, and if the docker support in the mach command works with it). I'll try to make them generic enough that they may apply to any subsequent release of Firefox.

• Clone either the mozilla-unified or mozilla-release repository. You can use Mercurial or Git (with git-cinnabar), it doesn't matter.
• Checkout the FIREFOX_68_0_RELEASE tag and find out what its Mercurial changeset id is (it is 353628fec415324ca6aa333ab6c47d447ecc128e).
• Open the Taskcluster index tool in a browser tab.
• In the input field type or copy/paste gecko.v2.mozilla-release.shippable.revision.353628fec415324ca6aa333ab6c47d447ecc128e.firefox.linux64-opt and press the Enter key. (replace 353628fec415324ca6aa333ab6c47d447ecc128e with the right revision if you're trying for another release)
• This will fill the "Indexed Task" pane, where you will find a TaskId. Follow the link there, it will bring you to the corresponding Task Run Logs
• Switch to the Task Details
• Scroll down to the "Dependencies" list, and check the task name that begins with "build-docker-image". For the Firefox 68 build task, it is build-docker-image-debian7-amd64-build.
• Take that name, remove the "build-docker-image-" prefix, and run the following command, from inside the repository, to download the corresponding docker image:

  $ ./mach taskcluster-load-image debian7-amd64-build

  Obviously, replace debian7-amd64-build with whatever you found in the task dependencies. The image can also be built from the source tree, but this is out of scope for this post.

• The command output will give you a docker run -ti ... command to try. Run it. It will open a shell in the docker image.
• From the docker shell, run the following commands:

  $ echo no-api-key > /builds/mozilla-desktop-geoloc-api.key
  $ echo no-api-key > /builds/sb-gapi.data
  $ echo no-api-key > /builds/gls-gapi.data
  

  Or replace no-api-key with the actual keys if you have them.

• Back to the Task Details, check the env part of the "Payload". You'll need to export all these variables with the corresponding values. e.g.

  $ export EXTRA_MOZHARNESS_CONFIG='{"update_channel": "release", "mozconfig_variant": "release"}'
  $ export GECKO_BASE_REPOSITORY='https://hg.mozilla.org/mozilla-unified'
  $ export GECKO_HEAD_REPOSITORY='https://hg.mozilla.org/releases/mozilla-release'
  ...

• Set the missing TASKCLUSTER_ROOT_URL environment variable:

  $ export TASKCLUSTER_ROOT_URL='https://taskcluster.net'

• Change the value of MOZHARNESS_ACTIONS to:

  $ export MOZHARNESS_ACTIONS='build'

  The original value contains get-secrets, which will try to download from http://taskcluster/, which will fail with a DNS error, and check-test, which runs make check, which is not necessary to get a working Firefox.

• Take command part of the "Payload", and run that in the docker shell:

  $ /builds/worker/bin/run-task --gecko-checkout /builds/worker/workspace/build/src -- /builds/worker/workspace/build/src/taskcluster/scripts/builder/build-linux.sh

• Once the build is finished, in another terminal, check what the container id of your running docker container is, and extract the build artifact from there:

  $ docker ps
  CONTAINER ID        IMAGE                                                                                  COMMAND             CREATED             STATUS              PORTS               NAMES
  d234383ba9c7        debian7-amd64-build:be96d1b734e1a152a861ce786861fca6e70bcb996bf67347f5af4f146db157ec   "bash"              2 hours ago         Up 2 hours                              nifty_hermann
  $ docker cp d234383ba9c7:/builds/worker/artifacts/target.tar.bz2 .

  (replace d234383ba9c7 with your container id)

• Now you can exit the docker shell. That will remove the container.

After all the above, you can finally compare your target.tar.bz2 to the Linux64 Firefox 68 release. You will find a few inevitable differences:

• The .chk files will be different, because they are self-signatures for FIPS mode that are generated with one-time throw-away keys.
• The Firefox 68 release contains .sig files that your build won't contain. They are signature files, which aren't reproducible outside Mozilla automation for obvious reasons.
• Consequently, the precomplete file contains instructions for the .sig files in the Firefox 68 release that won't be in your build.
• The omni.ja files are different. If you extract them (they are uncompressed zip files with a few tweaks to the format), you'll see the only difference is in modules/AppConstants.jsm, for the three API keys you created a file for earlier.

Everything else is identical bit for bit.

All the above is a rather long list of manual steps. Ideally, most of it would be automated. We're not there yet. We only recently got to the point where the profile data is available to make it possible at all. In other words, this is a starting point. It's valuable to know it does work but requires manual steps and what those are.

It is also worth noting that while the above downloads and uses pre-built compilers and other tools, it is also possible to rebuild those, although they likely won't be bit-for-bit identical. But differences in those shouldn't incur differences in Firefox. Replacing the pre-built ones with ones you'd build yourself unfortunately currently requires some more manual work.

As for Windows and Mac builds, long story short, they are not reproducible as of writing. Mac builds are not optimized with PGO, but Windows builds are, and their profile data won't be available until Firefox 69. Both platforms require SDKs that Mozilla can't redistribute per their license (but are otherwise available for download from Microsoft or Apple, respectively), which makes the setup more complex. And in all likeliness, for both platforms, the toolchains are not deterministic yet (that's at least true for Mac). Also, binary signatures would need to be tripped off the executables and libraries before any comparison.

How is that for clickbait?

With the now released git-cinnabar 0.5.2, the cinnabarclone feature is enabled by default, which means it doesn't need to be enabled manually anymore.

Cinnabarclone is to git-cinnabar what clonebundles is to Mercurial (to some extent). Clonebundles allow Mercurial to download a pre-generated bundle of a repository, which reduces work on the server side. Similarly, Cinnabarclone allows git-cinnabar to download a pre-generated bundle of the git form of a Mercurial repository.

Thanks to Connor Sheehan, who deployed the necessary extension and configuration on the server side, cinnabarclone is now enabled for mozilla-central and mozilla-unified, making git-cinnabar clone faster than ever for these repositories. In fact, under some conditions (mostly depending on network bandwidth), cloning with git-cinnabar is now faster than cloning with Mercurial:

$ time git clone hg::https://hg.mozilla.org/mozilla-unified mozilla-unified_git
Cloning into 'mozilla-unified_git'...
Fetching cinnabar metadata from https://index.taskcluster.net/v1/task/github.glandium.git-cinnabar.bundle.mozilla-unified/artifacts/public/bundle.git
Receiving objects: 100% (12153616/12153616), 2.67 GiB | 41.41 MiB/s, done.
Resolving deltas: 100% (8393939/8393939), done.
Reading 172 changesets
Reading and importing 170 manifests
Reading and importing 758 revisions of 570 files
Importing 172 changesets
It is recommended that you set "remote.origin.prune" or "fetch.prune" to "true".
git config remote.origin.prune true
or
git config fetch.prune true

Run the following command to update tags:
git fetch --tags hg::tags: tag "*"
Checking out files: 100% (279688/279688), done.

real    4m57.837s
user    9m57.373s
sys     0m41.106s

$ time hg clone https://hg.mozilla.org/mozilla-unified
destination directory: mozilla-unified
applying clone bundle from https://hg.cdn.mozilla.net/mozilla-unified/5ebb4441aa24eb6cbe8dad58d232004a3ea11b28.zstd-max.hg
adding changesets
adding manifests
adding file changes
added 537259 changesets with 3275908 changes to 523698 files (+13 heads)
finished applying clone bundle
searching for changes
adding changesets
adding manifests
adding file changes
added 172 changesets with 758 changes to 570 files (-1 heads)
new changesets 8b3c35badb46:468e240bf668
537259 local changesets published
updating to branch default
(warning: large working directory being used without fsmonitor enabled; enable fsmonitor to improve performance; see "hg help -e fsmonitor")
279688 files updated, 0 files merged, 0 files removed, 0 files unresolved

real    21m9.662s
user    21m30.851s
sys     1m31.153s

To be fair, the Mozilla Mercurial repos also have a faster "streaming" clonebundle that they only prioritize automatically if the client is on AWS currently, because they are much larger, and could take longer to download. But you can opt-in with the --stream command line argument:

$ time hg clone --stream https://hg.mozilla.org/mozilla-unified mozilla-unified_hg
destination directory: mozilla-unified_hg
applying clone bundle from https://hg.cdn.mozilla.net/mozilla-unified/5ebb4441aa24eb6cbe8dad58d232004a3ea11b28.packed1.hg
525514 files to transfer, 2.95 GB of data
transferred 2.95 GB in 51.5 seconds (58.7 MB/sec)
finished applying clone bundle
searching for changes
adding changesets
adding manifests
adding file changes
added 172 changesets with 758 changes to 570 files (-1 heads)
new changesets 8b3c35badb46:468e240bf668
updating to branch default
(warning: large working directory being used without fsmonitor enabled; enable fsmonitor to improve performance; see "hg help -e fsmonitor")
279688 files updated, 0 files merged, 0 files removed, 0 files unresolved

real    1m49.388s
user    2m52.943s
sys     0m43.779s

If you're using Mercurial and can download 3GB in less than 20 minutes (in other words, if you can download faster than 2.5MB/s), you're probably better off with the streaming clone.

Bonus fact: the Git clone is smaller than the Mercurial clone

The Mercurial streaming clone bundle contains data in a form close to what Mercurial puts on disk in the .hg directory, meaning the size of .hg is close to that of the clone bundle. The Cinnabarclone bundle contains a git pack, meaning the size of .git is close to that of the bundle, plus some more for the pack index file that unbundling creates.

The amazing fact is that, to my own surprise, the git pack, containing the repository contents along with all git-cinnabar needs to recreate Mercurial changesets, manifests and files from the contents, takes less space than the Mercurial streaming clone bundle.

And that translates in local repository size:

$ du -h -s --apparent-size mozilla-unified_hg/.hg
3.3G    mozilla-unified_hg/.hg
$ du -h -s --apparent-size mozilla-unified_git/.git
3.1G    mozilla-unified_git/.git

And because Mercurial creates so many files (essentially, two per file that ever was in the repository), there is a larger difference in block size used on disk:

$ du -h -s mozilla-unified_hg/.hg
4.7G    mozilla-unified_hg/.hg
$ du -h -s mozilla-unified_git/.git
3.1G    mozilla-unified_git/.git

It's even more mind blowing when you consider that Mercurial happily creates delta chains of several thousand revisions, when the git pack's longest delta chain is 250 (set arbitrarily at pack creation, by which I mean I didn't pick a larger value because it didn't make a significant difference). For the casual readers, Git and Mercurial try to store object revisions as a diff/delta from a previous object revision because that takes less space. You get a delta chain when that previous object revision itself is stored as a diff/delta from another object revision itself stored as a diff/delta ... etc.

My guess is that the difference is mainly caused by the use of line-based deltas in Mercurial, but some Mercurial developer should probably take a deeper look. The fact that Mercurial cannot delta across file renames is another candidate.

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

These release notes are also available on the git-cinnabar wiki.

What's new since 0.5.1?

• Updated git to 2.22.0 for the helper.
• cinnabarclone support is now enabled by default. See details in README.md and mercurial/cinnabarclone.py.
• cinnabarclone now supports grafted repositories.
• git cinnabar fsck now does incremental checks against last known good state.
• Avoid git cinnabar sometimes thinking the helper is not up-to-date when it is.
• Removing bookmarks on a Mercurial server is now working properly.

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

These release notes are also available on the git-cinnabar wiki.

What's new since 0.5.0?

• Updated git to 2.21.0 for the helper.
• Experimental native mercurial support (used when mercurial libraries are not available) now has feature parity.
• Try to read the git system config from the same place as git does. This fixes native HTTPS support with Git on Windows.
• Avoid pushing more commits than necessary in some corner cases (see e.g. https://bugzilla.mozilla.org/show_bug.cgi?id=1529360).
• Added an --abbrev argument for git cinnabar {git2hg,hg2git} to display shortened sha1s.
• Can now pass multiple revisions to git cinnabar fetch.
• Don't require the requests python module for git cinnabar download.
• Fixed git cinnabar fsck file checks to actually report errors.
• Properly return an error code from git cinnabar rollback.
• Track last fsck'ed metadata and allow git cinnabar rollback --fsck to go back to last known good metadata directly.
• git cinnabar reclone can now be rolled back.
• Added support for git bundles as cinnabarclone source.
• Added alternate styles of remote refs.
• More resilient to interruptions when HTTP Range requests are supported.
• Fixed off-by-one when storing mercurial heads.
• Better handling of mercurial branchmap tips.
• Better support for end of parts in bundle v2.
• Improvements handling urls to local mercurial repositories.
• Fixed compatibility with (very) old mercurial servers when using mercurial 5.0 libraries.
• Converted Continuous Integration scripts to Python 3.

As of last nightly (20181115100051), Firefox now supports Wayland on Linux, thanks to the work from Martin Stransky and Jan Horak, mostly.

Before that, it was possible to build your own Firefox with Wayland support (and Fedora does it), but now the downloads from mozilla.org come with Wayland support out of the box for the first time.

However, being experimental and all, the Wayland support is not enabled by default, meaning by default, you'll still be using XWayland. To enable wayland support, first set the GDK_BACKEND environment variable to wayland.

To verify whether Wayland support is enabled, go to about:support, and check "WebGL 1 Driver WSI Info" and/or "WebGL 2 Driver WSI Info". If they say something about GLX, Wayland support is not enabled. If they say something about EGL, it is. I filed a bug to make it more obvious what is being used.

It's probably still a long way before Firefox enables Wayland support on Wayland by default, but we reached a major milestone here. Please test and report any bug you encounter.

Update: I should mention that should you build your own Firefox, as long as your Gtk+ headers come with Wayland support, you'll end up with the same Wayland support as the one shipped by Mozilla.

You might have read that Mozilla recently switched Windows builds to clang-cl. More recently, those Windows builds have seen both PGO (Profile-Guided Optimization) and LTO (Link-Time Optimization) enabled.

As of next nightly (as of writing, obviously), all tier-1 platforms are now built with clang with LTO enabled. Yes, this means Linux, Mac and Android arm, aarch64 and x86. Linux builds also have PGO enabled.

Mac and Android builds were already using clang, so the only difference is LTO being enabled, which brought some performance improvements.

The most impressive difference, though, was on Linux, where we're getting more than 5% performance improvements on most Talos tests (up to 18% (!) on some tests) compared to GCC 6.4 with PGO. I must say I wasn't expecting switching from GCC to clang would make such a difference. And that is with clang 6. A quick test with upcoming clang 7 suggests we'd additionally get between 2 and 5% performance improvement from an upgrade, but our static analysis plugin doesn't like it.

This doesn't mean GCC is being unsupported. As a matter of fact, we still have automated jobs using GCC for some static analysis, and we also have jobs ensuring everything still builds with a baseline of GCC 6.x.

You might wonder if we tried LTO with GCC, or tried upgrading to GCC 8.x. As a matter of fact, I did. Enabling LTO turned up linker errors, and upgrading to GCC 7.x turned up breaking binary compatibility with older systems, and if I remember correctly had some problems with our test suite. GCC 8.1 was barely out when I was looking into this, and we all know to stay away from any new major GCC version until one or two minor updates. Considering the expected future advantages from using clang (cross-language inlining with Rust, consistency between platforms), it seemed a better deal to switch to clang than to try to address those issues.

Update: As there's been some interest on reddit and HN, and I failed to mention it originally, it's worth noting that comparing GCC+PGO vs. clang+LTO or GCC+PGO vs. clang+PGO was a win for clang overall in both cases, although GCC was winning on a few benchmarks. If I remember correctly, clang without PGO/LTO was also winning against GCC without PGO.

Anyways, what led me on this quest was a casual conversation at our last All Hands, where we were discussing possibly turning on LTO on Mac, and how that should roughly just be about turning a switch.

Famous last words.

At least, that's a somehow reasonable assumption. But when you have a codebase the size of Firefox, you're up for "interesting" discoveries.

This involved compiler bugs, linker bugs (with a special mention for a bug in ld64 that Apple has apparently fixed in Xcode 9 but hasn't released the source of), build system problems, elfhack issues, crash report problems, clang plugin problems (would you have guessed that __attribute__((annotate("foo"))) can affect the generated machine code?), sccache issues, inline assembly bugs (getting inputs, outputs and clobbers correctly is hard), binutils bugs, and more.

I won't bother you with all the details, but here we are, 3 months later with it all, finally, mostly done. Counting only the bugs assigned to me, there are 77 bugs on bugzilla (so, leaving out anything in other bug trackers, like LLVM's). Some of them relied on work from other people (most notably, Nathan Froyd's work to switch to clang and then non-NDK clang on Android). This spread over about 150 commits on mozilla-central, 20 of which were backouts. Not everything went according to plan, obviously, although some of those backouts were on purpose as a taskcluster trick.

Hopefully, this sticks, and Firefox 64 will ship built with clang with LTO on all tier-1 platforms as well as PGO on some. Downstreams are encouraged to do the same if they can. The build system will soon choose clang by default on all builds, but won't enable PGO/LTO.

As a bonus, as of a few days ago, Linux builds are also finally using Position Independent Executables, which improves Address Space Layout Randomization for the few things that are in the executables instead of some library (most notably, mozglue and the allocator). This was actually necessary for LTO, because clang doesn't build position independent code in executables that are not PIE (but GCC does), and that causes other problems.

Work is not entirely over, though, as more inline assembly bugs might be remaining only not causing visible problems by sheer luck, so I'm now working on a systematic analysis of inline assembly blocks with our clang plugin.

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

These release notes are also available on the git-cinnabar wiki.

What's new since 0.4.0?

• git-cinnabar-helper is now mandatory. You can either download one with git cinnabar download on supported platforms or build one with make.
• Performance and memory consumption improvements.
• Metadata changes require to run git cinnabar upgrade.
• Mercurial tags are consolidated in a separate (fake) repository. See the README file.
• Updated git to 2.18.0 for the helper.
• Improved memory consumption and performance.
• Improved experimental support for pushing merges.
• Support for clonebundles for faster clones when the server provides them.
• Removed support for the .git/hgrc file for mercurial specific configuration.
• Support any version of Git (was previously limited to 1.8.5 minimum)
• Git packs created by git-cinnabar are now smaller.
• Fixed incompatibilities with Mercurial 3.4 and >= 4.4.
• Fixed tag cache, which could lead to missing tags.
• The prebuilt helper for Linux now works across more distributions (as long as libcurl.so.4 is present, it should work)
• Properly support the pack.packsizelimit setting.
• Experimental support for initial clone from a git repository containing git-cinnabar metadata.
• Now can successfully clone the pypy and GNU octave mercurial repositories.
• More user-friendly errors.

Development process changes

It took about 6 months between version 0.3 and 0.4. It took more than 18 months to reach version 0.5 after that. That's a long time to wait for a new version, considering all the improvements that have happened under the hood.

From now on, the release branch will point to the last tagged release, which is roughly the same as before, but won't be the default branch when cloning anymore.

The default branch when cloning will now be master, which will receive changes that are acceptable for dot releases (0.5.x). These include:

• Changes in behavior that are backwards compatible (e.g. adding new options which default to the current behavior).
• Changes that improve error handling.
• Changes to existing experimental features, and additions of new experimental features (that require knobs to be enabled).
• Changes to Continuous Integration/Tests.
• Git version upgrades for the helper.

The next branch will receive changes for the next "major" release, which as of writing is planned to be 0.6.0. These include:

• Changes in behavior.
• Changes in metadata.
• Stabilizing experimental features.
• Remove backwards compability with older metadata (< 0.5.0).

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

These release notes are also available on the git-cinnabar wiki.

What's new since 0.5.0 beta 3?

• Fixed incompatibility with Mercurial 3.4.
• Performance and memory consumption improvements.
• Work around networking issues while downloading clone bundles from Mozilla CDN with range requests to continue past failure.
• Miscellaneous metadata format changes.
• The prebuilt helper for Linux now works across more distributions (as long as libcurl.so.4 is present, it should work)
• Updated git to 2.18.0 for the helper.
• Properly support the pack.packsizelimit setting.
• Experimental support for initial clone from a git repository containing git-cinnabar metadata.
• Changed the default make rule to only build the helper.
• Now can successfully clone the pypy and GNU octave mercurial repositories.
• More user-friendly errors.

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

These release notes are also available on the git-cinnabar wiki.

What's new since 0.5.0 beta 2?

• Fixed incompatibilities with Mercurial >= 4.4.
• Miscellaneous metadata format changes.
• Move more operations to the helper, hopefully making things faster.
• Updated git to 2.17.0 for the helper.
• Properly handle clones with bundles when the repository doesn't contain anything newer than the bundle.
• Fixed tag cache, which could lead to missing tags.