{"id":182,"date":"2008-02-24T22:42:11","date_gmt":"2008-02-24T20:42:11","guid":{"rendered":"http:\/\/glandium.org\/blog\/?p=182"},"modified":"2010-01-27T08:52:29","modified_gmt":"2010-01-27T07:52:29","slug":"testing-the-crazy-idea","status":"publish","type":"post","link":"https:\/\/glandium.org\/blog\/?p=182","title":{"rendered":"Testing the crazy idea"},"content":{"rendered":"

So, MadCoder<\/a> had doubts about the crazy idea<\/a>, so I took a little time to do a test with a package I maintain, namely, xulrunner.<\/p>\n

First, take all the deb files in the history of the package (at least, everything that is available on snapshot.debian.net<\/a> today).<\/p>\n

wget -O - -q http:\/\/snapshot.debian.net\/archive\/pool\/x\/xulrunner\/binary-amd64\/Packages.gz | gzip -cd > list
\nawk '\/^Filename:\/{print $2}' list | xargs -I{} wget http:\/\/snapshot.debian.net\/archive\/{}<\/code><\/p><\/blockquote>\n

Next, commit all these in a different repo per package name :<\/p>\n

perl -e 'use Dpkg::Version qw(vercmp); sub v { my $f = $_[0]; $f =~ s\/.*_(.*)_.*\/$1\/; $f } print sort { vercmp(v($a), v($b)); } map { s\/^Filename: .*\\\/\/\/; $_ } grep { \/^Filename:\/ } <>;' list | while read f; do
\n    pkg=${f%_*_*}
\n    [ ! -d $pkg ] && mkdir $pkg && ( cd $pkg ; git init )
\n    cd $pkg
\n    ar -x ..\/$f
\n    mkdir data control
\n    tar -C data -zxf data.tar.gz
\n    tar -C control -zxf control.tar.gz
\n    git add data control
\n    git commit -q -m $f
\n    rm -rf data control data.tar.gz control.tar.gz debian-binary
\n    cd ..
\ndone<\/code><\/p><\/blockquote>\n

Finally, evaluate sizes for each package, respectively, of all .deb files, their content imported in git (only the .git directory, including the index ; some space could be gained removing it), and the \"optimized\" git repository (after git gc<\/code>, without modifying delta depth or window size, which may even improve the result)<\/p>\n

awk '\/^Package:\/{print $2}' list | sort -u | while read p; do
\n    du -c --si ${p}_*.deb | tail -1
\n    du -s --si $p
\n    cd $p; git gc ; cd ..
\n    echo
\ndone<\/code><\/p>\n

17M\ttotal
\n16M\tlibmozjs0d
\n4.7M\tlibmozjs0d<\/code><\/p>\n

34M\ttotal
\n31M\tlibmozjs0d-dbg
\n14M\tlibmozjs0d-dbg<\/code><\/p>\n

4.7M\ttotal
\n7.1M\tlibnspr4-0d
\n1.3M\tlibnspr4-0d<\/code><\/p>\n

9.2M\ttotal
\n12M\tlibnspr4-0d-dbg
\n2.8M\tlibnspr4-0d-dbg<\/code><\/p>\n

25M\ttotal
\n28M\tlibnss3-0d
\n4.6M\tlibnss3-0d<\/code><\/p>\n

81M\ttotal
\n61M\tlibnss3-0d-dbg
\n21M\tlibnss3-0d-dbg<\/code><\/p>\n

15M\ttotal
\n16M\tlibnss3-tools
\n3.1M\tlibnss3-tools<\/code><\/p>\n

288M\ttotal
\n265M\tlibxul0d
\n146M\tlibxul0d<\/code><\/p>\n

2.0G\ttotal
\n1.8G\tlibxul0d-dbg
\n1.1G libxul0d-dbg<\/code><\/p>\n

5.1M\ttotal
\n6.8M\tpython-xpcom
\n1.1M\tpython-xpcom<\/code><\/p>\n

2.5M\ttotal
\n5.6M\tspidermonkey-bin
\n861k\tspidermonkey-bin<\/code><\/p>\n

13M\ttotal
\n14M\txulrunner
\n2.0M\txulrunner<\/code><\/p>\n

3.3M\ttotal
\n5.9M\txulrunner-gnome-support
\n979k\txulrunner-gnome-support
\n<\/code><\/p><\/blockquote>\n

So these packages, stored in git, take between 15 and roughly 50 percent of the .deb size, which may be a nice improvement. Il would be interesting to know how these numbers evolve with time. Some files, such as the changelog.Debian.gz files, would also benefit from being stored in plain text instead of gzipped form.<\/p>\n

Note git gc<\/code> took a while and a lot of memory for libxul0d-dbg. Also note these don't include delta files that would be necessary to recreate the original .deb file, but this shouldn't make a huge difference.<\/p>\n","protected":false},"excerpt":{"rendered":"

So, MadCoder had doubts about the crazy idea, so I took a little time to do a test with a package I maintain, namely, xulrunner. First, take all the deb files in the history of the package (at least, everything that is available on snapshot.debian.net today). wget -O – -q http:\/\/snapshot.debian.net\/archive\/pool\/x\/xulrunner\/binary-amd64\/Packages.gz | gzip -cd > […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1,5],"tags":[23],"class_list":["post-182","post","type-post","status-publish","format-standard","hentry","category-misc","category-pdo","tag-en"],"_links":{"self":[{"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/182","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=182"}],"version-history":[{"count":1,"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/182\/revisions"}],"predecessor-version":[{"id":692,"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/182\/revisions\/692"}],"wp:attachment":[{"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=182"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=182"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=182"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}