I uploaded xulrunner 1.9~b5-1 yesterday, and today was webkit’s turn, with an upload of svn revision 31841, which happens to be the last revision on trunk as of writing.

Thanks to Mark Rowe, the most problematic crashers I experienced with webkit while preparing this upload have been fixed in revisions 31821 and 31787. This had the side effect to delay the upload enough that we now have the benefit of improved SVG animations support, and CSS gradients.

For those who like numbers, you can take a look at the Sunspider results for xulrunner 1.9~b5 (running mybrowser ; very similar results to what I got with 1.9~b4), and Sunspider results for webkit r31842 (when you compare to previous results for webkit r27674, there are significant improvements).

For another kind of numbers:

~/git/xulrunner$ git diff upstream/1.9... | filterdiff -x b/debian/* -x a/configure | diffstat | tail -1
36 files changed, 507 insertions(+), 291 deletions(-)

Compared to:
~/git/xulrunner$ git diff upstream/1.9...1.9+b4-1 | filterdiff -x b/debian/"*" -x a/configure | diffstat | tail -1
53 files changed, 932 insertions(+), 466 deletions(-)

~/git/webkit$ git diff upstream... | filterdiff -x b/debian/* | diffstat | tail -1
1 file changed, 4 deletions(-)

Compared to:
~/git/webkit$ git diff upstream...0+svn27674-4 | filterdiff -x b/debian/* | diffstat | tail -1
17 files changed, 61 insertions(+), 18 deletions(-)

Somehow, I prefer to work on webkit…

I’m preparing a new upload for WebKit, which will be targetted at unstable. It is much easier to deal with than Gecko, fortunately, so it won’t take several months to get something in shape. The main “difficulties” here is that I’m dropping the Qt WebKit package, since this will be provided along Qt, and the upstream build system for the Gtk port switched from qmake to autotools, which is not a really bad thing ; so, nothing impossible.

Note that switching to autotools also means using libtool, which means no way to use -Wl,–as-needed anymore :-/. Yes, libtool, by trying to be smart, puts it almost at the last position in the arguments list, making it useless.

It finally happened, sorry for the delay. 1.9b4 is currently being uploaded to experimental. Yes, I know, this is not 1.9b5, which was released a few days ago. This is because I wanted to do some more work on 1.9b5 and didn’t want to delay the upload any longer.

Now, for the uninteresting statistics, following are the diffstats, excluding directory debian/ and configure:

for version 1.8.1.13-1: 113 files changed, 1393 insertions(+), 824 deletions(-)
for version 1.9~b4-1: 53 files changed, 932 insertions(+), 466 deletions(-)
for work in progress version 1.9~b5-1: 39 files changed, 848 insertions(+), 423 deletions(-)

(the latter will obviously evolve)

It smells like an april fool, it looks like an april fool, it tastes like an april fool, but it’s not an april fool.

Would there have been a better day than today to announce such a great news ?

I appear to have underestimated the remaining work needed to get xulrunner in a pleasant enough shape for an upload. Which means the package won’t be ready this week-end. It’s always when you come closer to the goal that it gets farther…

And I still haven’t decided once and for all if I would still version the libxul library. The problem is the following: there are two different ways to link or load libxul: dependent glue or standalone glue. The first one dynamically links embedding applications to both libxpcom and libxul, while the second links to a static library (well, dynamic, in Debian, because there is no reason why we should need to binNMU all reverse dependencies whenever we fix something in the glue), which dlload()s libxul. From Mozilla POV, embedding applications are supposed to use the standalone glue.

Considering we will more than probably have both schemes in use within reverse dependencies, I’m not sure I still want to bother diverging from upstream by keeping SO versioning on libxpcom and libxul… That unfortunately means that we will go back to the previously sucky situation where reverse dependencies have to put a dependency on the Gecko runtime themselves. A debhelper might help, though.

I will keep SO versioning on libmozjs, though, because it has some reverse dependencies, and a changing ABI.

The good news, anyways, is that I was able to build and run Iceweasel on top of the xulrunner pre-package.

I also ran sunspider on both this Iceweasel 3.0b4 and Iceweasel 2.0 ; the difference is really impressive: 3.0b4 is almost 4 times as fast !

For reference, the Webkit currently in unstable (which is quite old, actually), gives these results. The one in experimental unfortunately crashes. By the way, I’m planning to package a new Webkit snapshot soon after I’m done with xulrunner and Iceweasel, we’ll see then how it performs.

While speaking of tests, both Iceweasel 3.0b4 and Webkit from experimental pass the Acid 2 test (contrary to Iceweasel 2.0), and have both quite good results on the Acid 3 test: 61 for Webkit from experimental, when it doesn’t crash (but current trunk has been reported to score 91 !), and 67 for Iceweasel 3.0b4 (compared to 52 for Iceweasel 2.0).

Update: Interestingly, built with upstream optimization flags (-Os-freorder-blocks -fno-reorder-functions) instead of -O2, it is slighly slower, though it might be better on some older hardware, or other architectures (I’m testing on x86-64).

Carlo Wood wrote a nice piece of documentation about how to recover files from ext3, based on his experience after deleting 3GB.

This actually happened to me 5 years ago, now, and is how ext3rminator came to exist. I’ve been meaning to update it for a while (and journal parsing has been on the top of the TODO list for a while), but package maintenance is unfortunately time consuming. And I have a big tendancy to do something else, too (like coding on git).

It’s been a while I’ve not talked about my packages. I’ve been preparing a xulrunner trunk upload for a while now, and I must say that while I’m not totally happy with evolutions in Mozilla codebase, I do appreciate that a significant amount of the patches we were applying have been applied upstream. And even more will be applied when Firefox 3.0beta5 is released. I even got patches that were *not* in Debian to be applied. Thanks have to go to Reed Loden for all this.

Another nice thing on Mozilla trunk is that they finally stopped patching sqlite, which means we can now have Mozilla linked against the system one. It is even more important now that libnss uses sqlite, too.

And now that we have libcairo 1.5 in experimental, and recent enough nss and nspr in unstable (uploaded a few days ago, creating some mess with iceweasel, iceape and xulrunner), we have all the necessary bits to have (most of) the bundled libraries replaced by system ones.

The sad thing is they did implement APNG (animated png) in their bundled libpng, so, to have APNG support, we need to either patch libpng or use the bundled one. I’ll see if I can just disable APNG for the moment.

I hope to be able to upload the package to experimental on saturday or sunday, based on the firefox 3.0pre4 codebase. Iceweasel should follow a few days later.

So, MadCoder had doubts about the crazy idea, so I took a little time to do a test with a package I maintain, namely, xulrunner.

First, take all the deb files in the history of the package (at least, everything that is available on snapshot.debian.net today).

wget -O - -q http://snapshot.debian.net/archive/pool/x/xulrunner/binary-amd64/Packages.gz | gzip -cd > list
awk '/^Filename:/{print $2}' list | xargs -I{} wget http://snapshot.debian.net/archive/{}

Next, commit all these in a different repo per package name :

perl -e 'use Dpkg::Version qw(vercmp); sub v { my $f = $_[0]; $f =~ s/.*_(.*)_.*/$1/; $f } print sort { vercmp(v($a), v($b)); } map { s/^Filename: .*\///; $_ } grep { /^Filename:/ } <>;’ list | while read f; do
    pkg=${f%_*_*}
    [ ! -d $pkg ] && mkdir $pkg && ( cd $pkg ; git init )
    cd $pkg
    ar -x ../$f
    mkdir data control
    tar -C data -zxf data.tar.gz
    tar -C control -zxf control.tar.gz
    git add data control
    git commit -q -m $f
    rm -rf data control data.tar.gz control.tar.gz debian-binary
    cd ..
done

Finally, evaluate sizes for each package, respectively, of all .deb files, their content imported in git (only the .git directory, including the index ; some space could be gained removing it), and the “optimized” git repository (after git gc, without modifying delta depth or window size, which may even improve the result)

awk '/^Package:/{print $2}' list | sort -u | while read p; do
    du -c --si ${p}_*.deb | tail -1
    du -s --si $p
    cd $p; git gc ; cd ..
    echo
done

17M total
16M libmozjs0d
4.7M libmozjs0d

34M total
31M libmozjs0d-dbg
14M libmozjs0d-dbg

4.7M total
7.1M libnspr4-0d
1.3M libnspr4-0d

9.2M total
12M libnspr4-0d-dbg
2.8M libnspr4-0d-dbg

25M total
28M libnss3-0d
4.6M libnss3-0d

81M total
61M libnss3-0d-dbg
21M libnss3-0d-dbg

15M total
16M libnss3-tools
3.1M libnss3-tools

288M total
265M libxul0d
146M libxul0d

2.0G total
1.8G libxul0d-dbg
1.1G libxul0d-dbg

5.1M total
6.8M python-xpcom
1.1M python-xpcom

2.5M total
5.6M spidermonkey-bin
861k spidermonkey-bin

13M total
14M xulrunner
2.0M xulrunner

3.3M total
5.9M xulrunner-gnome-support
979k xulrunner-gnome-support


So these packages, stored in git, take between 15 and roughly 50 percent of the .deb size, which may be a nice improvement. Il would be interesting to know how these numbers evolve with time. Some files, such as the changelog.Debian.gz files, would also benefit from being stored in plain text instead of gzipped form.

Note git gc took a while and a lot of memory for libxul0d-dbg. Also note these don’t include delta files that would be necessary to recreate the original .deb file, but this shouldn’t make a huge difference.

I often have a bunch of somewhat crazy ideas, and I don’t have any time available to test or implement them, which is sad. So just in case these crazy ideas would scratch someone’s itch, I’m going to throw them in the wild.

I’ve been using git for a few months, now, and used it not only for source code management, but for efficient storage, too. *VERY* efficient. I’ll have to write about that some day.

Anyways, while installing pristine-tar, today, I just thought it would be neat to have an equivalent pristine-deb, to store deb files efficiently. I’m pretty sure someone else thought about this possibility, but it’s still better that such ideas come to the ears (eyes, actually) of someone that could implement them.

Such a pristine-deb tool could be used to… store packages from snapshot.debian.net. That would reduce the amount of space required for the archive dramatically, IMHO. I’m pretty sure old packages are not requested that much, so they could be generated on-the-fly from a CGI script placed as a GET action, so that urls wouldn’t change.

The same could probably be applied to archive.debian.org. It could even save enough space that archive.debian.org could host snapshot.debian.net. But that depends on the average package content and its average evolution, which I have absolutely no idea about.

Update: It would also be interesting to have the .diff.gz files in there, too ; it would obviously allow to have an easy view of the contents, such as copyright files, changelogs, and other bits of information available on packages.debian.org.

Update 2: Actually, pristine-deb would as easy as storing 2 pristine-tars (one for control.tar.gz and one for data.tar.gz), and a debian-binary file. The .deb can be aggregated with

ar -rc file.deb debian-binary control.tar.gz data.tar.gz