February 8th, 2016

SSH through jump hosts, revisited

Close to 7 years ago, I wrote about SSH through jump hosts. Twice. While the method used back then still works, Openssh has grown an new option in version 5.3 that allows it to be simplified a bit, by not using nc.

So here is an updated rule, version 2016:

Host *+*
ProxyCommand ssh -W $(echo %h | sed 's/^.*+//;s/^\([^:]*$\)/\1:22/') $(echo %h | sed 's/+[^+]*$//;s/\([^+%%]*\)%%\([^+]*\)$/\2 -l \1/;s/:\([^:+]*\)$/ -p \1/')

The syntax you can use to connect through jump hosts hasn’t changed compared to previous blog posts:

  • With one jump host:
    $ ssh login1%host1:port1+host2:port2 -l login2
  • With two jump hosts:
    $ ssh login1%host1:port1+login2%host2:port2+host3:port3 -l login3
  • With three jump hosts:
    $ ssh login1%host1:port1+login2%host2:port2+login3%host3:port3+host4:port4 -l login4
  • etc.

Logins and ports can be omitted.

Update: Add missing port to -W flag when one is not given.

2016-02-08 00:26:53+0900

p.d.o, p.m.o | 5 Comments »

February 5th, 2016

Going beyond NS_ProcessNextEvent

If you’ve been debugging Gecko, you’ve probably hit the frustration of having the code you’re inspecting being called asynchronously, and your stack trace rooting through NS_ProcessNextEvent, which means you don’t know at first glance how your code ended up being called in the first place.

Events running from the Gecko event loop are all nsRunnable instances. So at some level close to NS_ProcessNextEvent, in your backtrace, you will see Class::Run. If you’re lucky, you can find where the nsRunnable was created. But that requires the stars to be perfectly aligned. In many cases, they’re not.

There comes your savior: rr. If you don’t know it, check it out. The downside is that you must first rr record a Firefox session doing what you’re debugging. Then, rr replay will give you a debugger with the capabilities of a time machine.

Note, I’m kind of jinxed, I don’t do much C++ debugging these days, so every time I use rr replay, I end up hitting a new error. Tip #1: try again with rr’s current master. Tip #2: roc is very helpful. But my takeaway is that it’s well worth the trouble. It is a game changer for debugging.

Anyways, once you’re in rr replay and have hit your crasher or whatever execution path you’re interested in, and you want to go beyond that NS_ProcessNextEvent, here is what you can do:

(rr) break nsEventQueue.cpp:60
(rr) reverse-continue

(Adjust the line number to match wherever the *aResult = mHead->mEvents[mOffsetHead++]; line is in your tree).

(rr) disable
(rr) watch -l mHead->mEvents[mOffsetHead]
(rr) reverse-continue
(rr) disable

And there you are, you just found where the exact event that triggered the executed code you were looking at was put on the event queue. (assuming there isn’t a nested event loop processed during the first reverse-continue)

Rinse and repeat.

2016-02-05 07:19:41+0900

p.m.o | 1 Comment »

January 30th, 2016

Enabling TLS on this blog

Long overdue, I finally enabled TLS on this blog. It went almost like a breeze.

I used simp_le to get the certificate from Let’s Encrypt, along Mozilla’s Web Server Configuration generator. SSL Labs now reports a rating of A+.

I just had a few issues:

  • I had some hard-coded http:// links in my wordpress theme, that needed changes,
  • Since my wordpress instance is reverse-proxied and the real server not behind HTTPS, I had to adjust the wordpress configuration so that it doesn’t do an infinite redirect loop,
  • Nginx’s config for multiple virtualhosts needs SSL configuration to be repeated. Fortunately, one can use include statements,
  • Contrary to the suggested configuration, setting ssl_session_tickets off; makes browsers unhappy (at least, it made my Firefox unhappy, with a SSL_ERROR_RX_UNEXPECTED_NEW_SESSION_TICKET error message).

I’m glad that there are tools helping to get a proper configuration of SSL. It is sad, though, that the defaults are not better and that we still need to tweak at all. Setting where the certificate and the private key files are should, in 2016, be the only thing to do to have a secure web server.

2016-01-30 07:22:42+0900

p.d.o, p.m.o, website | 2 Comments »

January 16th, 2016

Announcing git-cinnabar 0.3.1

This is a brown paper bag release. It turns out I managed to break the upgrade path only 10 commits before the release.

What’s new since 0.3.0?

  • git cinnabar fsck doesn’t fail to upgrade metadata.
  • The remote.$remote.cinnabar-draft config works again.
  • Don’t fail to clone an empty repository.
  • Allow to specify mercurial configuration items in a .git/hgrc file.

2016-01-16 12:26:45+0900

cinnabar, p.m.o | No Comments »

January 15th, 2016

Announcing git-cinnabar 0.3.0

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

These release notes are also available on the git-cinnabar wiki.

Development had been stalled for a few months, with many improvements in the next branch without any new release. I used some time during the new year break and after in order to straighten things up in order to create a new release, delaying many of the originally planned changes to a future 0.4.0 release.

What’s new since 0.2.2?

  • Speed and memory usage were improved when doing git push.
  • Now works on Windows, at least to some extent. See details.
  • Support for pre-0.1.0 git-cinnabar repositories was removed. You must first use a git-cinnabar version between 0.1.0 and 0.2.2 to upgrade its metadata.
  • It is now possible to attach/graft git-cinnabar metadata to existing commits matching mercurial changesets. This allows to migrate from some other hg-to-git tool to git-cinnabar while preserving the existing git commits. See an example of how this works with the git clone of the Gecko mercurial repository
  • Avoid mercurial printing its progress bar, messing up with git-cinnabar’s output.
  • It is now possible to fetch from an incremental mercurial bundle (without a root changeset).
  • It is now possible to push to a new mercurial repository without -f.
  • By default, reject pushing a new root to a mercurial repository.
  • Make the connection to a mercurial repository through ssh respect the GIT_SSH and GIT_SSH_COMMAND environment variables.
  • git cinnabar now has a proper argument parser for all its subcommands.
  • A new git cinnabar python command allows to run python scripts or open a python shell with the right sys.path to import the cinnabar module.
  • All git-cinnabar metadata is now kept under a single ref (although for convenience, other refs are created, but they can be derived if necessary).
  • Consequently, a new git cinnabar rollback command allows to roll back to previous metadata states.
  • git-cinnabar metadata now tracks the manifests DAG.
  • A new git cinnabar bundle command allows to create mercurial bundles, mostly for debugging purposes, without requiring to hit a mercurial server.
  • Updated git to 2.7.0 for the native helper.

Development process changes

Up to before this release closing in, the master branch was dedicated to releases, and development was happening on the next branch, until a new release happens.

From now on, the release branch will take dot-release fixes and new releases, while the master branch will receive all changes that are validated through testing (currently semi-automatically tested with out-of-tree tests based on four real-life mercurial repositories, with some automated CI based on in-tree tests used in the future).

The next branch will receive changes to be tested in CI when things will be hooked up, and may have rewritten history as a consequence of wanting passing tests on every commit on master.

2016-01-15 09:56:40+0900

cinnabar, p.m.o | 2 Comments »

July 24th, 2015

Firefox nightlies for Linux are now using Gtk+3

As of last nightly (2015-07-23), Firefox for Linux is using Gtk+3 instead of Gtk+2.

Thanks to the recent efforts of Andrew Comminos, all remaining test failures are gone and mozilla-central now defaults to Gtk+3 builds. Some jobs on treeherder are still not converted, but this will come soon (bug 1186748).

If you’ve been using elm builds for dogfooding, you should be automatically switched to standard nightlies today or tomorrow. The elm branch will be recycled to do Gtk+2 builds so that they keep working. Those builds won’t be auto-updating, so don’t use them.

2015-07-24 02:25:37+0900

p.m.o | 2 Comments »

June 10th, 2015

PSA: mach now stores its log automatically

It’s been a frustration for a long time: oftentimes, you need to dig the log of the build or test run you just did, but didn’t redirect it. When you have enough backlog in your terminal, that can work itself out, but usually, what happens is that you rerun the command, redirecting it to a log file, hoping what you’re looking for will happen again.

For people working on the build system, it’s even worse because it involves someone else: someone comes and say “my build fails with $message”, and usually, thanks to parallel builds, the actual error message is buried deep in their terminal history… when they have a terminal history (Windows console, I’m looking at you).

Anyways, as of bug 985857, as long as you don’t already redirect mach‘s output to a file, it will now save it for you automatically.

At the moment, the feature is basic, and will store the last log of the last command (mostly). So if you run mach build followed by mach xpcshell-test, only the log for the latter will be available, and the log of the former will be lost.

The log is stored in the same format as mach uses when you give it the -l argument, which is an aggregate of many json pieces, and not very user friendly. Which is why mach now also has a new command to read those logs:

mach show-log

By default, it will display the last log mach stored, but you can also give it the path to any log you got out of mach with the -l argument.

So you can do either:

mach <command>
mach show-log


mach -l log-file.json <command>
mach show-log log-file.json

Note that show-log will spawn less automatically, so that you have paging and search abilities.

2015-06-10 06:01:13+0900

p.m.o | 2 Comments »

May 25th, 2015

Dogfooding Firefox GTK+3

Thanks to Lee Salzman, the state of GTK+3 support in Firefox got better. Unit tests went from looking like this:

To looking like this:


There’s obviously some work left to make those look even better, but we’ve come a long way.

Ludovic Hirlimann recently asked if there were builds to dogfood and that prompted me to attempt making the builds from the elm branch auto-update. Which, after several attempts, I managed to get working with gross (but small) hacks of the build system.

So here we are, if you want to dogfood GTK+3 Firefox, here is what you can do::

  • In a normal Linux nightly, go to about:config and create the following string preferences (right-click, New, String):
    • app.update.url.override” with the value “https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/elm-linux/latest/update.xml” for 32-bits builds, or “https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/elm-linux64/latest/update.xml” for 64-bits builds,
    • app.update.certs.3.issuerName” with the value “CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US“,
    • app.update.certs.3.commonName” with the value “ftp.mozilla.org“.
  • Open the burger menu, click the “?” icon, then choose “About Nightly”. This should check for an update, find one, and download it. This will upgrade to a GTK+3 build.

Alternatively, you can just download and install the elm builds directly (32-bits, 64-bits).

If for some reason, you want to go back to a normal GTK+2 nightly, go to about:config, find the “app.update.url.override” preference and set it to an empty value. Triggering the update from “About Nightly” won’t, however, work until the next nightly is available, so give it a day.

As mentioned in my previous post about GTK+3, if you’re interested in making those builds work better, you are welcome to help:

2015-05-25 03:37:37+0900

p.m.o | 3 Comments »

May 4th, 2015

Gnome shell Hello world

Gnome Shell, besides providing the main user interface for GNOME 3, is a Javascript shell with bindings to many native interfaces that allow e.g. Window manipulation, graphics rendering and animations, compositing, etc. It also allows developers to write extensions changing Gnome Shell’s behavior.

Less known is that it is possible to replace the entire Javascript code base that Gnome Shell uses. It can be useful to hack on Gnome Shell itself (no need to fiddle with system files, or, since 3.12, no need to rebuild libgnome-shell.so), but it can also be used to implement a completely new User Interface in Javascript.

I’m starting to experiment with the latter, because I want to try building a window manager that fits my needs, while keeping away the boring details of EWMH, xinerama, and other X11 things. And because it’s fun.

But baby steps, first: let’s bootstrap a Hello world with Gnome shell.

  • Create a directory that will hold your code.
  • In that directory, create a ui subdirectory.
  • In that ui directory, create a environment.js file, with the following contents:
    const Shell = imports.gi.Shell;
    function init() {
      window.global = Shell.Global.get();
  • In the same directory, create a main.js file, with the following contents:
    const St = imports.gi.St;
    function start() {
      let text = new St.Label({ text: "Hello, world!" });
  • Run Gnome Shell with your code:
    $ GNOME_SHELL_JS=/path/to/parent/of/ui gnome-shell
    You may want to run this in a separate X server (I use Xephyr)

I tested this with Gnome Shell 3.14. Trying various older versions, I got different results for reasons I don’t know. 3.4 doesn’t display anything unless, paradoxically, global.stage.show() is removed, and 3.8 doesn’t display anything no matter what.

I guess the next step is to go through some Clutter tutorials and transpose them to Javascript.

Update: On the other hand, a lot of the window managing is still done by mutter under the hood, which doesn’t leave a lot of space for something really different.

2015-05-04 04:10:17+0900

miscellaneous, p.d.o | 1 Comment »

May 1st, 2015

Using a git clone of gecko-dev to push to mercurial

[Update: An up-to-date version of this document is available on the git-cinnabar wiki]

The next branch of git-cinnabar now has minimal support for grafting, enough to allow to graft a clone of gecko-dev to mozilla-central and other Mozilla branches. This will be available in version 0.3 (which I don’t expect to release before June), but if you are interested, you can already try it.

There are two ways you can work with gecko-dev and git-cinnabar.

Switching to git-cinnabar

This is the recommended setup.

There are several reasons one would want to start from gecko-dev instead of a fresh clone of mozilla-central. One is to get the full history from before Mozilla switched to Mercurial, which gecko-dev contains. Another is if you already have a gecko-dev clone with local branches, where rebasing against a fresh clone would not be very convenient (but not impossible).

The idea here is to use gecko-dev as a start point, and from there on, use cinnabar to pull and push from/to Mercurial. The main caveat is that new commits pulled from Mercurial after this will not have the same SHA-1s as those on gecko-dev. Only the commits until the switch will. This also means different people grafting gecko-dev at different moments will have different SHA-1s for new commits. Eventually, I hope we can switch gecko-dev itself to use git-cinnabar instead of hg-git, which would solve this issue.

Assuming you already have a gecko-dev clone, and git-cinnabar next installed:

  • Change the remote URL:
    $ git remote set-url origin hg::https://hg.mozilla.org/mozilla-central
    (replace origin with the remote name for gecko-dev if it’s not origin)
  • Add other mercurial repositories you want to track as well:
    $ git remote add inbound hg::https://hg.mozilla.org/integration/mozilla-inbound
    $ git remote add aurora hg::https://hg.mozilla.org/releases/mozilla-aurora
  • Pull with grafting enabled:
    $ git -c cinnabar.graft=true -c cinnabar.graft-refs=refs/remotes/origin/* remote update
    (replace origin with the remote name for gecko-dev if it’s not origin)
  • Finish the setup by setting push urls for all those remotes:
    $ git remote set-url --push origin hg::ssh://hg.mozilla.org/mozilla-central
    $ git remote set-url --push inbound hg::ssh://hg.mozilla.org/integration/mozilla-inbound
    $ git remote set-url --push aurora hg::ssh://hg.mozilla.org/releases/mozilla-aurora

Now, you’re mostly done setting things up. Check out my git workflow for Gecko development for how to work from there.

Following gecko-dev

This setup allows to keep pulling from gecko-dev and thus keep the same SHA-1s. It relies on everything pulled from Mercurial existing in gecko-dev, which makes it cumbersome, which is why I don’t recommend using it. For instance, you may end up pulling from Mercurial before the server-side mirroring made things available in gecko-dev, and that will fail. Some commands such as git pull --rebase will require you to ensure gecko-dev is up-to-date first (that makes winning push races essentially impossible). And more importantly, in some cases, what you push to Mercurial won’t have the same commit SHA-1 in gecko-dev, so you’ll have to manually deal with that (fortunately, it most cases, this shouldn’t happen).

Assuming you already have a gecko-dev clone, and git-cinnabar next installed:

  • Add a remote for all mercurial repositories you want to track:
    $ git remote add central hg::https://hg.mozilla.org/mozilla-central
    $ git remote add inbound hg::https://hg.mozilla.org/integration/mozilla-inbound
    $ git remote add aurora hg::https://hg.mozilla.org/releases/mozilla-aurora
  • For each of those remotes, set a push url:
    $ git remote set-url --push origin hg::ssh://hg.mozilla.org/mozilla-central
    $ git remote set-url --push inbound hg::ssh://hg.mozilla.org/integration/mozilla-inbound
    $ git remote set-url --push aurora hg::ssh://hg.mozilla.org/releases/mozilla-aurora
  • For each of those remotes, set a refspec that limits to pulling the tip of the default branch:
    $ git config remote.central.fetch +refs/heads/branches/default/tip:refs/remotes/central/default
    $ git config remote.inbound.fetch +refs/heads/branches/default/tip:refs/remotes/inbound/default
    $ git config remote.aurora.fetch +refs/heads/branches/default/tip:refs/remotes/aurora/default
    Other branches can be added, but that must be done with care, because not all branches are exposed on gecko-dev.
  • Set git-cinnabar grafting mode permanently:
    $ git config cinnabar.graft only
  • Make git-cinnabar never store metadata when pushing:
    $ git config cinnabar.data never
  • Make git-cinnabar only graft with gecko-dev commits:
    $ git config cinnabar.graft-refs refs/remotes/origin/*
    (replace origin with the remote name for gecko-dev if it’s not origin)
  • Then pull from all remotes:
    $ git remote update
    Retry as long as you see errors about “Not allowing non-graft import”. This will keep happening until all Mercurial changesets you’re trying to pull make it to gecko-dev.

With this setup, you can now happily push to Mercurial and see your commits appear in gecko-dev after a while. As long as you don’t copy or move files, their SHA-1 should be the same as what you pushed.

2015-05-01 07:32:22+0900

p.m.o | 2 Comments »