clonebundles<\/a> is to Mercurial (to some extent). Clonebundles allow Mercurial to download a pre-generated bundle of a repository, which reduces work on the server side. Similarly, Cinnabarclone allows git-cinnabar to download a pre-generated bundle of the git form of a Mercurial repository.<\/p>\nThanks to Connor Sheehan, who deployed the necessary extension and configuration on the server side, cinnabarclone is now enabled for mozilla-central and mozilla-unified, making git-cinnabar clone faster than ever for these repositories. In fact, under some conditions (mostly depending on network bandwidth), cloning with git-cinnabar is now faster than cloning with Mercurial:<\/p>\n
$ time git clone hg::https:\/\/hg.mozilla.org\/mozilla-unified mozilla-unified_git\nCloning into 'mozilla-unified_git'...\nFetching cinnabar metadata from https:\/\/index.taskcluster.net\/v1\/task\/github.glandium.git-cinnabar.bundle.mozilla-unified\/artifacts\/public\/bundle.git\nReceiving objects: 100% (12153616\/12153616), 2.67 GiB | 41.41 MiB\/s, done.\nResolving deltas: 100% (8393939\/8393939), done.\nReading 172 changesets\nReading and importing 170 manifests\nReading and importing 758 revisions of 570 files\nImporting 172 changesets\nIt is recommended that you set "remote.origin.prune" or "fetch.prune" to "true".\ngit config remote.origin.prune true\nor\ngit config fetch.prune true\n\nRun the following command to update tags:\ngit fetch --tags hg::tags: tag "*"\nChecking out files: 100% (279688\/279688), done.\n\nreal 4m57.837s\nuser 9m57.373s\nsys 0m41.106s\n\n$ time hg clone https:\/\/hg.mozilla.org\/mozilla-unified\ndestination directory: mozilla-unified\napplying clone bundle from https:\/\/hg.cdn.mozilla.net\/mozilla-unified\/5ebb4441aa24eb6cbe8dad58d232004a3ea11b28.zstd-max.hg\nadding changesets\nadding manifests\nadding file changes\nadded 537259 changesets with 3275908 changes to 523698 files (+13 heads)\nfinished applying clone bundle\nsearching for changes\nadding changesets\nadding manifests\nadding file changes\nadded 172 changesets with 758 changes to 570 files (-1 heads)\nnew changesets 8b3c35badb46:468e240bf668\n537259 local changesets published\nupdating to branch default\n(warning: large working directory being used without fsmonitor enabled; enable fsmonitor to improve performance; see "hg help -e fsmonitor")\n279688 files updated, 0 files merged, 0 files removed, 0 files unresolved\n\nreal 21m9.662s\nuser 21m30.851s\nsys 1m31.153s<\/code><\/pre>\nTo be fair, the Mozilla Mercurial repos also have a faster "streaming" clonebundle that they only prioritize automatically if the client is on AWS currently, because they are much larger, and could take longer to download. But you can opt-in with the --stream<\/code> command line argument:<\/p>\n$ time hg clone --stream https:\/\/hg.mozilla.org\/mozilla-unified mozilla-unified_hg\ndestination directory: mozilla-unified_hg\napplying clone bundle from https:\/\/hg.cdn.mozilla.net\/mozilla-unified\/5ebb4441aa24eb6cbe8dad58d232004a3ea11b28.packed1.hg\n525514 files to transfer, 2.95 GB of data\ntransferred 2.95 GB in 51.5 seconds (58.7 MB\/sec)\nfinished applying clone bundle\nsearching for changes\nadding changesets\nadding manifests\nadding file changes\nadded 172 changesets with 758 changes to 570 files (-1 heads)\nnew changesets 8b3c35badb46:468e240bf668\nupdating to branch default\n(warning: large working directory being used without fsmonitor enabled; enable fsmonitor to improve performance; see "hg help -e fsmonitor")\n279688 files updated, 0 files merged, 0 files removed, 0 files unresolved\n\nreal 1m49.388s\nuser 2m52.943s\nsys 0m43.779s<\/code><\/pre>\nIf you're using Mercurial and can download 3GB in less than 20 minutes (in other words, if you can download faster than 2.5MB\/s), you're probably better off with the streaming clone.<\/p>\n
Bonus fact: the Git clone is smaller than the Mercurial clone<\/h2>\n
The Mercurial streaming clone bundle contains data in a form close to what Mercurial puts on disk in the .hg<\/code> directory, meaning the size of .hg<\/code> is close to that of the clone bundle. The Cinnabarclone bundle contains a git pack, meaning the size of .git<\/code> is close to that of the bundle, plus some more for the pack index file that unbundling creates.<\/p>\nThe amazing fact is that, to my own surprise, the git pack, containing the repository contents along with all git-cinnabar needs to recreate Mercurial changesets, manifests and files from the contents, takes less space than the Mercurial streaming clone bundle.<\/p>\n
And that translates in local repository size:<\/p>\n
$ du -h -s --apparent-size mozilla-unified_hg\/.hg\n3.3G mozilla-unified_hg\/.hg\n$ du -h -s --apparent-size mozilla-unified_git\/.git\n3.1G mozilla-unified_git\/.git<\/code><\/pre>\nAnd because Mercurial creates so many files (essentially, two per file that ever was in the repository), there is a larger difference in block size used on disk:<\/p>\n
$ du -h -s mozilla-unified_hg\/.hg\n4.7G mozilla-unified_hg\/.hg\n$ du -h -s mozilla-unified_git\/.git\n3.1G mozilla-unified_git\/.git<\/code><\/pre>\nIt's even more mind blowing when you consider that Mercurial happily creates delta chains of several thousand revisions, when the git pack's longest delta chain is 250 (set arbitrarily at pack creation, by which I mean I didn't pick a larger value because it didn't make a significant difference). For the casual readers, Git and Mercurial try to store object revisions as a diff\/delta from a previous object revision because that takes less space. You get a delta chain when that previous object revision itself is stored as a diff\/delta from another object revision itself stored as a diff\/delta ... etc.<\/p>\n
My guess is that the difference is mainly caused by the use of line-based deltas in Mercurial, but some Mercurial developer should probably take a deeper look. The fact that Mercurial cannot delta across file renames is another candidate.<\/p>\n","protected":false},"excerpt":{"rendered":"
How is that for clickbait? With the now released git-cinnabar 0.5.2, the cinnabarclone feature is enabled by default, which means it doesn’t need to be enabled manually anymore. Cinnabarclone is to git-cinnabar what clonebundles is to Mercurial (to some extent). Clonebundles allow Mercurial to download a pre-generated bundle of a repository, which reduces work on […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[25],"tags":[23],"class_list":["post-3913","post","type-post","status-publish","format-standard","hentry","category-planet-mozilla","tag-en"],"_links":{"self":[{"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/3913","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3913"}],"version-history":[{"count":10,"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/3913\/revisions"}],"predecessor-version":[{"id":4323,"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/3913\/revisions\/4323"}],"wp:attachment":[{"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3913"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3913"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3913"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}