June 10th, 2015

PSA: mach now stores its log automatically

It’s been a frustration for a long time: oftentimes, you need to dig the log of the build or test run you just did, but didn’t redirect it. When you have enough backlog in your terminal, that can work itself out, but usually, what happens is that you rerun the command, redirecting it to a log file, hoping what you’re looking for will happen again.

For people working on the build system, it’s even worse because it involves someone else: someone comes and say “my build fails with $message”, and usually, thanks to parallel builds, the actual error message is buried deep in their terminal history… when they have a terminal history (Windows console, I’m looking at you).

Anyways, as of bug 985857, as long as you don’t already redirect mach‘s output to a file, it will now save it for you automatically.

At the moment, the feature is basic, and will store the last log of the last command (mostly). So if you run mach build followed by mach xpcshell-test, only the log for the latter will be available, and the log of the former will be lost.

The log is stored in the same format as mach uses when you give it the -l argument, which is an aggregate of many json pieces, and not very user friendly. Which is why mach now also has a new command to read those logs:

mach show-log

By default, it will display the last log mach stored, but you can also give it the path to any log you got out of mach with the -l argument.

So you can do either:

mach <command>
mach show-log

or

mach -l log-file.json <command>
mach show-log log-file.json

Note that show-log will spawn less automatically, so that you have paging and search abilities.

2015-06-10 06:01:13+0200

p.m.o | 2 Comments »

May 25th, 2015

Dogfooding Firefox GTK+3

Thanks to Lee Salzman, the state of GTK+3 support in Firefox got better. Unit tests went from looking like this:

To looking like this:

elm-after

There’s obviously some work left to make those look even better, but we’ve come a long way.

Ludovic Hirlimann recently asked if there were builds to dogfood and that prompted me to attempt making the builds from the elm branch auto-update. Which, after several attempts, I managed to get working with gross (but small) hacks of the build system.

So here we are, if you want to dogfood GTK+3 Firefox, here is what you can do::

  • In a normal Linux nightly, go to about:config and create the following string preferences (right-click, New, String):
    • app.update.url.override” with the value “https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/elm-linux/latest/update.xml” for 32-bits builds, or “https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/elm-linux64/latest/update.xml” for 64-bits builds,
    • app.update.certs.3.issuerName” with the value “CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US“,
    • app.update.certs.3.commonName” with the value “ftp.mozilla.org“.
  • Open the burger menu, click the “?” icon, then choose “About Nightly”. This should check for an update, find one, and download it. This will upgrade to a GTK+3 build.

Alternatively, you can just download and install the elm builds directly (32-bits, 64-bits).

If for some reason, you want to go back to a normal GTK+2 nightly, go to about:config, find the “app.update.url.override” preference and set it to an empty value. Triggering the update from “About Nightly” won’t, however, work until the next nightly is available, so give it a day.

As mentioned in my previous post about GTK+3, if you’re interested in making those builds work better, you are welcome to help:

2015-05-25 03:37:37+0200

p.m.o | 3 Comments »

May 4th, 2015

Gnome shell Hello world

Gnome Shell, besides providing the main user interface for GNOME 3, is a Javascript shell with bindings to many native interfaces that allow e.g. Window manipulation, graphics rendering and animations, compositing, etc. It also allows developers to write extensions changing Gnome Shell’s behavior.

Less known is that it is possible to replace the entire Javascript code base that Gnome Shell uses. It can be useful to hack on Gnome Shell itself (no need to fiddle with system files, or, since 3.12, no need to rebuild libgnome-shell.so), but it can also be used to implement a completely new User Interface in Javascript.

I’m starting to experiment with the latter, because I want to try building a window manager that fits my needs, while keeping away the boring details of EWMH, xinerama, and other X11 things. And because it’s fun.

But baby steps, first: let’s bootstrap a Hello world with Gnome shell.

  • Create a directory that will hold your code.
  • In that directory, create a ui subdirectory.
  • In that ui directory, create a environment.js file, with the following contents:
    const Shell = imports.gi.Shell;
    
    function init() {
      window.global = Shell.Global.get();
    }
    
  • In the same directory, create a main.js file, with the following contents:
    const St = imports.gi.St;
    
    function start() {
      let text = new St.Label({ text: "Hello, world!" });
      global.stage.add_actor(text);
      global.stage.show();
    }
    
  • Run Gnome Shell with your code:
    $ GNOME_SHELL_JS=/path/to/parent/of/ui gnome-shell
    

    You may want to run this in a separate X server (I use Xephyr)

I tested this with Gnome Shell 3.14. Trying various older versions, I got different results for reasons I don’t know. 3.4 doesn’t display anything unless, paradoxically, global.stage.show() is removed, and 3.8 doesn’t display anything no matter what.

I guess the next step is to go through some Clutter tutorials and transpose them to Javascript.

Update: On the other hand, a lot of the window managing is still done by mutter under the hood, which doesn’t leave a lot of space for something really different.

2015-05-04 04:10:17+0200

miscellaneous, p.d.o | 1 Comment »

May 1st, 2015

Using a git clone of gecko-dev to push to mercurial

The next branch of git-cinnabar now has minimal support for grafting, enough to allow to graft a clone of gecko-dev to mozilla-central and other Mozilla branches. This will be available in version 0.3 (which I don’t expect to release before June), but if you are interested, you can already try it.

There are two ways you can work with gecko-dev and git-cinnabar.

Switching to git-cinnabar

This is the recommended setup.

There are several reasons one would want to start from gecko-dev instead of a fresh clone of mozilla-central. One is to get the full history from before Mozilla switched to Mercurial, which gecko-dev contains. Another is if you already have a gecko-dev clone with local branches, where rebasing against a fresh clone would not be very convenient (but not impossible).

The idea here is to use gecko-dev as a start point, and from there on, use cinnabar to pull and push from/to Mercurial. The main caveat is that new commits pulled from Mercurial after this will not have the same SHA-1s as those on gecko-dev. Only the commits until the switch will. This also means different people grafting gecko-dev at different moments will have different SHA-1s for new commits. Eventually, I hope we can switch gecko-dev itself to use git-cinnabar instead of hg-git, which would solve this issue.

Assuming you already have a gecko-dev clone, and git-cinnabar next installed:

  • Change the remote URL:
    $ git remote set-url origin hg::https://hg.mozilla.org/mozilla-central

    (replace origin with the remote name for gecko-dev if it’s not origin)

  • Add other mercurial repositories you want to track as well:
    $ git remote add inbound hg::https://hg.mozilla.org/integration/mozilla-inbound
    $ git remote add aurora hg::https://hg.mozilla.org/releases/mozilla-aurora
    (...)
  • Pull with grafting enabled:
    $ git -c cinnabar.graft=true -c cinnabar.graft-refs=refs/remotes/origin/* remote update

    (replace origin with the remote name for gecko-dev if it’s not origin)

  • Finish the setup by setting push urls for all those remotes:
    $ git remote set-url --push origin hg::ssh://hg.mozilla.org/mozilla-central
    $ git remote set-url --push inbound hg::ssh://hg.mozilla.org/integration/mozilla-inbound
    $ git remote set-url --push aurora hg::ssh://hg.mozilla.org/releases/mozilla-aurora
    (...)

Now, you’re mostly done setting things up. Check out my git workflow for Gecko development for how to work from there.

Following gecko-dev

This setup allows to keep pulling from gecko-dev and thus keep the same SHA-1s. It relies on everything pulled from Mercurial existing in gecko-dev, which makes it cumbersome, which is why I don’t recommend using it. For instance, you may end up pulling from Mercurial before the server-side mirroring made things available in gecko-dev, and that will fail. Some commands such as git pull --rebase will require you to ensure gecko-dev is up-to-date first (that makes winning push races essentially impossible). And more importantly, in some cases, what you push to Mercurial won’t have the same commit SHA-1 in gecko-dev, so you’ll have to manually deal with that (fortunately, it most cases, this shouldn’t happen).

Assuming you already have a gecko-dev clone, and git-cinnabar next installed:

  • Add a remote for all mercurial repositories you want to track:
    $ git remote add central hg::https://hg.mozilla.org/mozilla-central
    $ git remote add inbound hg::https://hg.mozilla.org/integration/mozilla-inbound
    $ git remote add aurora hg::https://hg.mozilla.org/releases/mozilla-aurora
    (...)
  • For each of those remotes, set a push url:
    $ git remote set-url --push origin hg::ssh://hg.mozilla.org/mozilla-central
    $ git remote set-url --push inbound hg::ssh://hg.mozilla.org/integration/mozilla-inbound
    $ git remote set-url --push aurora hg::ssh://hg.mozilla.org/releases/mozilla-aurora
    (...)
  • For each of those remotes, set a refspec that limits to pulling the tip of the default branch:
    $ git config remote.central.fetch +refs/heads/branches/default/tip:refs/remotes/central/default
    $ git config remote.inbound.fetch +refs/heads/branches/default/tip:refs/remotes/inbound/default
    $ git config remote.aurora.fetch +refs/heads/branches/default/tip:refs/remotes/aurora/default
    (...)

    Other branches can be added, but that must be done with care, because not all branches are exposed on gecko-dev.

  • Set git-cinnabar grafting mode permanently:
    $ git config cinnabar.graft only
  • Make git-cinnabar never store metadata when pushing:
    $ git config cinnabar.data never
  • Make git-cinnabar only graft with gecko-dev commits:
    $ git config cinnabar.graft-refs refs/remotes/origin/*

    (replace origin with the remote name for gecko-dev if it’s not origin)

  • Then pull from all remotes:
    $ git remote update

    Retry as long as you see errors about “Not allowing non-graft import”. This will keep happening until all Mercurial changesets you’re trying to pull make it to gecko-dev.

With this setup, you can now happily push to Mercurial and see your commits appear in gecko-dev after a while. As long as you don’t copy or move files, their SHA-1 should be the same as what you pushed.

2015-05-01 07:32:22+0200

p.m.o | 2 Comments »

Announcing git-cinnabar 0.2.2

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

What’s new since 0.2.1?

  • Don’t require core.ignorecase to be set to false on the repository when using a case-insensitive file system. If you did set core.ignorecase to false because git-cinnabar told you to, you can now set it back to true.
  • Raise an exception when git update-ref or git fast-import return an error. Silently ignoring those errors could lead to bad repositories after an upgrade from pre-0.1.0 versions on OS X, where the default maximum number of open files is low (256), and where git update-ref uses a lot of lock files for large transactions.
  • Updated git to 2.4.0, when building with the native helper.
  • When doing git cinnabar reclone, skip remotes with remote.$remote.skipDefaultUpdate set to true.

2015-05-01 06:55:20+0200

cinnabar, p.m.o | No Comments »

April 15th, 2015

Announcing git-cinnabar 0.2.1

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

What’s new since 0.2.0?

Not much, but this felt important enough to warrant a release, even though the issue has been there since before 0.1.0:

Mercurial can be slower when cloning or pulling a list of “heads” that contain non-topological heads. On repositories like the mercurial repository, it’s not so much of a big deal, taking 7s instead of 4s. But on big repositories like mozilla-central, it’s taking 23 minutes instead of 2 minutes and 20s (on my machine). And that’s with 100% CPU use on the server side.

The problem is that mozilla-central recently merged some old closed heads, such that it now has branch heads that aren’t topological heads. Git-cinnabar, until this release, would request those branch heads, leading the server to use the slow path mentioned above. This release works around the issue.

It also fixes an issue pushing to a remote empty mercurial repository.

2015-04-15 04:20:10+0200

cinnabar, p.m.o | No Comments »

Getting ready for GCC 5.1

Confusingly, GCC has a new, weird, version scheme. The first release of GCC 5 will be 5.1. It is due soon (next week). For that reason, and because it’s better to find compiler bugs before it’s released, I started looking into building Firefox with it.

The first round of builds I did was with 5-20150405. That got me to find a small bunch of issues:

So that got me to do a second round with the first 5.1 RC, which had the fix for that ICE.

With all the above fixed, I could finally get builds out of try, and tests running, which revealed two more issues:

  • Another (quickly fixed) Internal Compiler Error on 32-bits PGO builds (but only for a nightly setup, with --enable-profiling, not for a release setup, which doesn’t have it).
  • JS engine assertions during some JIT tests on 64-bits builds (with or without PGO), which Dan Gohman kindly tracked down and reduced to a small test case allowing to file a GCC bug and bisect to pinpoint at the GCC upstream commit that broke it (yay git bisect run on a 36-CPU EC2 instance).

Preliminary results are promising, with benchmarks improving up to 16%, but the comparison wasn’t entirely fair, because they compared GCC 4.8 builds with frame pointers and JS engine diagnostics to GCC 5.1 builds without.

I’ll also give a spin to LTO, possibly finding more GCC bugs in the process.

2015-04-15 03:59:43+0200

p.m.o | 3 Comments »

April 7th, 2015

Announcing git-cinnabar 0.2.0

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

What’s new since 0.1.1?

  • git cinnabar git2hg and git cinnabar hg2git commands that allow to translate (possibly abbreviated) git sha1s to mercurial sha1s and vice-versa.
  • A “native” helper that makes some operations faster. It is not required for git-cinnabar to work, but it can improve performance significantly. Check the Setup instructions in the README file.
  • Do not store mercurial metadata when pushing to non-publishing repositories. For Mozilla developers, this means not storing that metadata when pushing to try, which is a good thing when you know each of them makes pulling slower. This behavior can be changed if necessary. Future releases will allow to remove metadata that was created by previous releases but that wouldn’t be created with 0.2.0.
  • Made the discovery phase of pushes require less round trips (the phase that finds what is common between the local and remote repositories), hopefully making pushing faster.
  • Improved logging, which now doesn’t require fiddling with the code to get extra logging.
  • Made fsck validate more things, and act on more errors.
  • Fixed a few edge cases.
  • Better handle files with weird names, and that git quotes in its output.
  • Extensively tested on the following repositories: mozilla-central, mozilla-beta, mercurial, hg-git, cpython.

What to expect next?

  • Allow to push merge commits.
  • Improve memory footprint for pushes (currently, it’s fairly catastrophic on big repositories ; don’t try to push multiple hundreds of commits of a Mozilla-sized repository if you don’t have multiple gigabytes of memory available).
  • As mentioned above, allow to remove some metadata.
  • And more…

If you want to follow the improvements more closely, I encourage you to switch to the `next` branch. I won’t push anything there that hasn’t been extensively tested on the above mentioned repositories.

And as always, please report any issue you run into.

2015-04-07 04:18:14+0200

cinnabar, p.m.o | No Comments »

March 13th, 2015

Announcing git-cinnabar 0.1.1

0.1.1 is a one-bugfix release fixing an issue where git-cinnabar could be confused if commands were started from a subdirectory of the repository. It might be the cause for corruptions leading to issues such as the impossibility to push.

If you do encounter failures please report them. Please also paste the output of git cinnabar fsck.

2015-03-13 02:17:25+0200

cinnabar, p.m.o | No Comments »

February 11th, 2015

Announcing git-cinnabar 0.1.0

As you may or may not know, I have been working on this project for quite some time, but actually really started the most critical parts a couple months ago. After having looked for (and chosen) a new name for what was a prototype project, it’s now time for a very first release.

So what is this all about?

Cinnabar is the common natural form in which mercury can be found on Earth. It contains mercury sulfide and its powder is used to make the vermillion pigment.

What does that have to do with git?

Hint: mercury.

Git-cinnabar is a git remote helper (you can think of that as a plugin) to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Numerous such tools already exist. Where git-cinnabar stands out is that it doesn’t use a local mercurial clone under the hood (unlike all the existing other such tools), and is close to an order of magnitude faster to clone a repository like mozilla-central than the git-remote-hg that used to be shipped as a contrib to git.

I won’t claim it is exempt of problems and limitations, which is why it’s not a 1.0. I’m however confident enough with its state to make the first “official” release.

Get it on github.

If you’ve been using the prototype, you can actually continue to use that clone and update it. Github conveniently keeps things working after a rename. You can update the remote url if you feel like it, though.

If you are a Gecko developer, you can take a look at a possible workflow.

2015-02-11 08:51:46+0200

cinnabar, p.m.o | 3 Comments »