Archive for April, 2015

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

What's new since 0.2.0?

Not much, but this felt important enough to warrant a release, even though the issue has been there since before 0.1.0:

Mercurial can be slower when cloning or pulling a list of "heads" that contain non-topological heads. On repositories like the mercurial repository, it's not so much of a big deal, taking 7s instead of 4s. But on big repositories like mozilla-central, it's taking 23 minutes instead of 2 minutes and 20s (on my machine). And that's with 100% CPU use on the server side.

The problem is that mozilla-central recently merged some old closed heads, such that it now has branch heads that aren't topological heads. Git-cinnabar, until this release, would request those branch heads, leading the server to use the slow path mentioned above. This release works around the issue.

It also fixes an issue pushing to a remote empty mercurial repository.

Confusingly, GCC has a new, weird, version scheme. The first release of GCC 5 will be 5.1. It is due soon (next week). For that reason, and because it's better to find compiler bugs before it's released, I started looking into building Firefox with it.

The first round of builds I did was with 5-20150405. That got me to find a small bunch of issues:

• The new libstdc++ ABI breaks our compatibility with older libstdc++, which we've been going into great length to keep. Fortunately, it's still possible to build with some level of compatibility by compiling with -D_GLIBCXX_USE_CXX11_ABI=0.
• Some build failure in pkix gtests because of unused gtest-generated variables in an anonymous namespace.
• New false positive hits on -Warray-bounds, which, with -Werror, led the build to fail.
• New libstdc++ compatibility issue that prevented from running elfhack during the build (we don't set LD_LIBRARY_PATH to use the libstdc++ that comes with GCC). Incidentally, I found that the check to ensure we don't introduce dependencies on unwanted symbols from libstdc++ has been broken (by myself) for a few months. Fortunately, nothing slipped in between.
• An Internal Compiler Error (ICE) on --enable-debug builds, which turned out to be already fixed.

So that got me to do a second round with the first 5.1 RC, which had the fix for that ICE.

With all the above fixed, I could finally get builds out of try, and tests running, which revealed two more issues:

• Another (quickly fixed) Internal Compiler Error on 32-bits PGO builds (but only for a nightly setup, with --enable-profiling, not for a release setup, which doesn't have it).
• JS engine assertions during some JIT tests on 64-bits builds (with or without PGO), which Dan Gohman kindly tracked down and reduced to a small test case allowing to file a GCC bug and bisect to pinpoint at the GCC upstream commit that broke it (yay git bisect run on a 36-CPU EC2 instance).

Preliminary results are promising, with benchmarks improving up to 16%, but the comparison wasn't entirely fair, because they compared GCC 4.8 builds with frame pointers and JS engine diagnostics to GCC 5.1 builds without.

I'll also give a spin to LTO, possibly finding more GCC bugs in the process.

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

What's new since 0.1.1?

• git cinnabar git2hg and git cinnabar hg2git commands that allow to translate (possibly abbreviated) git sha1s to mercurial sha1s and vice-versa.
• A "native" helper that makes some operations faster. It is not required for git-cinnabar to work, but it can improve performance significantly. Check the Setup instructions in the README file.
• Do not store mercurial metadata when pushing to non-publishing repositories. For Mozilla developers, this means not storing that metadata when pushing to try, which is a good thing when you know each of them makes pulling slower. This behavior can be changed if necessary. Future releases will allow to remove metadata that was created by previous releases but that wouldn't be created with 0.2.0.
• Made the discovery phase of pushes require less round trips (the phase that finds what is common between the local and remote repositories), hopefully making pushing faster.
• Improved logging, which now doesn't require fiddling with the code to get extra logging.
• Made fsck validate more things, and act on more errors.
• Fixed a few edge cases.
• Better handle files with weird names, and that git quotes in its output.
• Extensively tested on the following repositories: mozilla-central, mozilla-beta, mercurial, hg-git, cpython.

What to expect next?

• Allow to push merge commits.
• Improve memory footprint for pushes (currently, it's fairly catastrophic on big repositories ; don't try to push multiple hundreds of commits of a Mozilla-sized repository if you don't have multiple gigabytes of memory available).
• As mentioned above, allow to remove some metadata.
• And more...

If you want to follow the improvements more closely, I encourage you to switch to the `next` branch. I won't push anything there that hasn't been extensively tested on the above mentioned repositories.

And as always, please report any issue you run into.