Archive for the 'p.d.o' Category

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

These release notes are also available on the git-cinnabar wiki.

Development had been stalled for a few months, with many improvements in the next branch without any new release. I used some time during the new year break and after in order to straighten things up in order to create a new release, delaying many of the originally planned changes to a future 0.4.0 release.

What's new since 0.2.2?

• Speed and memory usage were improved when doing git push.
• Now works on Windows, at least to some extent. See details.
• Support for pre-0.1.0 git-cinnabar repositories was removed. You must first
  use a git-cinnabar version between 0.1.0 and 0.2.2 to upgrade its metadata.
• It is now possible to attach/graft git-cinnabar metadata to existing commits
  matching mercurial changesets. This allows to migrate from some other
  hg-to-git tool to git-cinnabar while preserving the existing git commits.
  See an example of how this works with the git clone of the Gecko mercurial
  repository
• Avoid mercurial printing its progress bar, messing up with git-cinnabar's
  output.
• It is now possible to fetch from an incremental mercurial bundle (without
  a root changeset).
• It is now possible to push to a new mercurial repository without -f.
• By default, reject pushing a new root to a mercurial repository.
• Make the connection to a mercurial repository through ssh respect the
  GIT_SSH and GIT_SSH_COMMAND environment variables.
• git cinnabar now has a proper argument parser for all its subcommands.
• A new git cinnabar python command allows to run python scripts or open a python shell with the right sys.path to import the cinnabar module.
• All git-cinnabar metadata is now kept under a single ref (although for
  convenience, other refs are created, but they can be derived if necessary).
• Consequently, a new git cinnabar rollback command allows to roll back to
  previous metadata states.
• git-cinnabar metadata now tracks the manifests DAG.
• A new git cinnabar bundle command allows to create mercurial bundles,
  mostly for debugging purposes, without requiring to hit a mercurial server.
• Updated git to 2.7.0 for the native helper.

Development process changes

Up to before this release closing in, the master branch was dedicated to
releases, and development was happening on the next branch, until a new
release happens.

From now on, the release branch will take dot-release fixes and new
releases, while the master branch will receive all changes that are
validated through testing (currently semi-automatically tested with
out-of-tree tests based on four real-life mercurial repositories, with
some automated CI based on in-tree tests used in the future).

The next branch will receive changes to be tested in CI when things
will be hooked up, and may have rewritten history as a consequence of
wanting passing tests on every commit on master.

Gnome Shell, besides providing the main user interface for GNOME 3, is a Javascript shell with bindings to many native interfaces that allow e.g. Window manipulation, graphics rendering and animations, compositing, etc. It also allows developers to write extensions changing Gnome Shell's behavior.

Less known is that it is possible to replace the entire Javascript code base that Gnome Shell uses. It can be useful to hack on Gnome Shell itself (no need to fiddle with system files, or, since 3.12, no need to rebuild libgnome-shell.so), but it can also be used to implement a completely new User Interface in Javascript.

I'm starting to experiment with the latter, because I want to try building a window manager that fits my needs, while keeping away the boring details of EWMH, xinerama, and other X11 things. And because it's fun.

But baby steps, first: let's bootstrap a Hello world with Gnome shell.

• Create a directory that will hold your code.
• In that directory, create a ui subdirectory.
• In that ui directory, create a environment.js file, with the following contents:
  

  const Shell = imports.gi.Shell;
  
  function init() {
    window.global = Shell.Global.get();
  }
  

• In the same directory, create a main.js file, with the following contents:
  

  const St = imports.gi.St;
  
  function start() {
    let text = new St.Label({ text: "Hello, world!" });
    global.stage.add_actor(text);
    global.stage.show();
  }
  

• Run Gnome Shell with your code:
  

  $ GNOME_SHELL_JS=/path/to/parent/of/ui gnome-shell
  

  You may want to run this in a separate X server (I use Xephyr)

I tested this with Gnome Shell 3.14. Trying various older versions, I got different results for reasons I don't know. 3.4 doesn't display anything unless, paradoxically, global.stage.show() is removed, and 3.8 doesn't display anything no matter what.

I guess the next step is to go through some Clutter tutorials and transpose them to Javascript.

Update: On the other hand, a lot of the window managing is still done by mutter under the hood, which doesn't leave a lot of space for something really different.

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

What's new since 0.2.1?

• Don't require core.ignorecase to be set to false on the repository when using a case-insensitive file system. If you did set core.ignorecase to false because git-cinnabar told you to, you can now set it back to true.
• Raise an exception when git update-ref or git fast-import return an error. Silently ignoring those errors could lead to bad repositories after an upgrade from pre-0.1.0 versions on OS X, where the default maximum number of open files is low (256), and where git update-ref uses a lot of lock files for large transactions.
• Updated git to 2.4.0, when building with the native helper.
• When doing git cinnabar reclone, skip remotes with remote.$remote.skipDefaultUpdate set to true.

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

What's new since 0.2.0?

Not much, but this felt important enough to warrant a release, even though the issue has been there since before 0.1.0:

Mercurial can be slower when cloning or pulling a list of "heads" that contain non-topological heads. On repositories like the mercurial repository, it's not so much of a big deal, taking 7s instead of 4s. But on big repositories like mozilla-central, it's taking 23 minutes instead of 2 minutes and 20s (on my machine). And that's with 100% CPU use on the server side.

The problem is that mozilla-central recently merged some old closed heads, such that it now has branch heads that aren't topological heads. Git-cinnabar, until this release, would request those branch heads, leading the server to use the slow path mentioned above. This release works around the issue.

It also fixes an issue pushing to a remote empty mercurial repository.

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

What's new since 0.1.1?

• git cinnabar git2hg and git cinnabar hg2git commands that allow to translate (possibly abbreviated) git sha1s to mercurial sha1s and vice-versa.
• A "native" helper that makes some operations faster. It is not required for git-cinnabar to work, but it can improve performance significantly. Check the Setup instructions in the README file.
• Do not store mercurial metadata when pushing to non-publishing repositories. For Mozilla developers, this means not storing that metadata when pushing to try, which is a good thing when you know each of them makes pulling slower. This behavior can be changed if necessary. Future releases will allow to remove metadata that was created by previous releases but that wouldn't be created with 0.2.0.
• Made the discovery phase of pushes require less round trips (the phase that finds what is common between the local and remote repositories), hopefully making pushing faster.
• Improved logging, which now doesn't require fiddling with the code to get extra logging.
• Made fsck validate more things, and act on more errors.
• Fixed a few edge cases.
• Better handle files with weird names, and that git quotes in its output.
• Extensively tested on the following repositories: mozilla-central, mozilla-beta, mercurial, hg-git, cpython.

What to expect next?

• Allow to push merge commits.
• Improve memory footprint for pushes (currently, it's fairly catastrophic on big repositories ; don't try to push multiple hundreds of commits of a Mozilla-sized repository if you don't have multiple gigabytes of memory available).
• As mentioned above, allow to remove some metadata.
• And more...

If you want to follow the improvements more closely, I encourage you to switch to the `next` branch. I won't push anything there that hasn't been extensively tested on the above mentioned repositories.

And as always, please report any issue you run into.

0.1.1 is a one-bugfix release fixing an issue where git-cinnabar could be confused if commands were started from a subdirectory of the repository. It might be the cause for corruptions leading to issues such as the impossibility to push.

If you do encounter failures please report them. Please also paste the output of git cinnabar fsck.

As you may or may not know, I have been working on this project for quite some time, but actually really started the most critical parts a couple months ago. After having looked for (and chosen) a new name for what was a prototype project, it's now time for a very first release.

So what is this all about?

Cinnabar is the common natural form in which mercury can be found on Earth. It contains mercury sulfide and its powder is used to make the vermillion pigment.

What does that have to do with git?

Hint: mercury.

Git-cinnabar is a git remote helper (you can think of that as a plugin) to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Numerous such tools already exist. Where git-cinnabar stands out is that it doesn't use a local mercurial clone under the hood (unlike all the existing other such tools), and is close to an order of magnitude faster to clone a repository like mozilla-central than the git-remote-hg that used to be shipped as a contrib to git.

I won't claim it is exempt of problems and limitations, which is why it's not a 1.0. I'm however confident enough with its state to make the first "official" release.

Get it on github.

If you've been using the prototype, you can actually continue to use that clone and update it. Github conveniently keeps things working after a rename. You can update the remote url if you feel like it, though.

If you are a Gecko developer, you can take a look at a possible workflow.

If you've been following this blog, you know I've been working on a (fast) git remote helper to access mercurial without a local mercurial clone, with the main goal to make it work for Gecko developers.

The way git remote helpers work forces how their executable is named: For a foo:: remote prefix, the executable must be named git-remote-foo. So for hg::, it's git-remote-hg.

As you may know, there already exists a project with that name. And when I picked the name for this new helper, I didn't really care to find a separate name, especially considering its prototype nature.

Now that I'm satisfied enough with it that I'm close to release it with a version number (which will be 0.1.0), I'm thinking that the confusion with the other project with that name is not really helpful, and an unfortunate implementation detail.

So I'm looking for a new project name... and have no good idea.

Dear lazy web, do you have good ideas?

One of the main differences in how mercurial and git track files is that mercurial does rename and copy tracking and git doesn't. So in the case of mercurial, users are expected to explicitly rename or copy the files through the mercurial command line so that mercurial knows what happened. Git simply doesn't care, and will try to detect after the fact when you ask it to.

The consequence is that my git-remote-hg, being currently a limited prototype, doesn't make the effort to inform mercurial of renames or copies.

This week, Ehsan, as a user of that tool, pushed some file moves, and subsequently opened an issue, because some people didn't like it.

It was a conscious choice on my part to make git-remote-hg public without rename/copies detection, because file renames/copies are not happening often, and can just as much not be registered by mercurial users.

In fact, they haven't all been registered for as long as Mozilla has been using mercurial (see below, I didn't actually know I was so spot on when I wrote this sentence), and people haven't been pointed at for using broken tools (and I'll skip the actual language that was used when talking about Ehsan's push).

And since I'd rather not make unsubstantiated claims, I dug in all of mozilla-central and related repositories (inbound, b2g-inbound, fx-team, aurora, beta, release, esr*) and here is what I found, only accounting files that have been copied or renamed without being further modified (so, using git diff-tree -r -C100%, and eliminating empty files), and correlating with the mercurial rename/copy metadata:

• There have been 45069 file renames or copies in 1546 changesets.
• Mercurial doesn't know 5482 (12.1%) of them, from 419 (27.1%) changesets.
• 72 of those changesets were backouts.
• 19 of those backouts were of changesets that didn't have rename/copy information, so 53 of those backouts didn't actually undo what mercurial knew of those backed out changesets.
• Those 419 changesets were from 144 distinct authors (assuming I didn't miss some duplicates from people who changed email).
• Fun fact, the person with colorful language, and that doesn't like git-remote-hg, is part of them. I am too, and that was with mercurial.
• The most recent occurrence of renames/copies unknown to mercurial is already not Ehsan's anymore.
• The oldest occurrence is in the 19th (!) mercurial changeset.

And that's not counting all the copies and renames with additional modifications.

Fun fact, this is what I found in the Mercurial mercurial repository:

• There have been 255 file renames or copies in 41 changesets.
• Mercurial doesn't know about 38 (14.9%) of them, from 4 (9.7%) changesets.
• One of those changesets was from Matt Mackall himself (creator and lead developer of mercurial).

There are 1061 files in mercurial, versus 115845 in mozilla-central, so there is less occasion for renames/copies there, still, even they forget to use "hg move" and break their history as a result.

I think this shows how requiring explicit user input simply doesn't pan out.

Meanwhile, I have prototype copy/rename detection for git-remote-hg working, but I need to tweak it a little bit more before publishing.

This push to try was not created by mercurial.

I just landed initial support for pushing to mercurial from git. Considering the scary fact that it's possible to screw up a repository with bundles with missing content (and, guess what, I figured out the hard way), I have restricted it to local mercurial repositories until I am more confident.

As such, I would need volunteers to use and test it on local mercurial repositories. On top of being limited to local mercurial repositories, it doesn't support pushing merges that would have been created by git, nor does it support pushing a root commit (one with no parent).

Here's how you can use it:

$ git clone https://github.com/glandium/git-remote-hg
$ export PATH=$PATH:$(pwd)/git-remote-hg
$ git clone hg::/path/to/mercurial-repository
$ # work work, commit, commit
$ git push

[ Note: you can still pull from remote mercurial repositories ]

This will push to your local repository, where it would be useful if you could check the push didn't fuck things up.

$ cd /path/to/mercurial-repository
$ hg verify

That's the long, thorough version. You may just want to simply do this:

$ cd /path/to/mercurial-repository
$ hg log --stat

Hopefully, you won't see messages like:

abort: data/build/mozconfig.common.override.i@56d6fdb13666: no match found!

Update: You can also add the following to /path/to/mercurial-repository/.hg/hgrc, which should prevent corruptions from entering the mercurial repository at all:

[server]
validate = True

Update 2: The above setting is now unnecessary, git-remote-hg will set it itself for its push session.

Then you can push with mercurial.

$ hg push

Please note that this is integrated in git in such a way that it's possible to pass refspecs to git push and do other fancy stuff. Be aware that there are still rough edges on that part, but that your commits will be pushed, even if the resulting state under refs/remotes/ is not very consistent.

I'm planning a replay of several repositories to fully validate pushes don't send broken bundles, but it's going to take some time before I can set things up. I figured I'd rather crowdsource until then.