Author Archive

I appear to have underestimated the remaining work needed to get xulrunner in a pleasant enough shape for an upload. Which means the package won't be ready this week-end. It's always when you come closer to the goal that it gets farther...

And I still haven't decided once and for all if I would still version the libxul library. The problem is the following: there are two different ways to link or load libxul: dependent glue or standalone glue. The first one dynamically links embedding applications to both libxpcom and libxul, while the second links to a static library (well, dynamic, in Debian, because there is no reason why we should need to binNMU all reverse dependencies whenever we fix something in the glue), which dlload()s libxul. From Mozilla POV, embedding applications are supposed to use the standalone glue.

Considering we will more than probably have both schemes in use within reverse dependencies, I'm not sure I still want to bother diverging from upstream by keeping SO versioning on libxpcom and libxul... That unfortunately means that we will go back to the previously sucky situation where reverse dependencies have to put a dependency on the Gecko runtime themselves. A debhelper might help, though.

I will keep SO versioning on libmozjs, though, because it has some reverse dependencies, and a changing ABI.

The good news, anyways, is that I was able to build and run Iceweasel on top of the xulrunner pre-package.

I also ran sunspider on both this Iceweasel 3.0b4 and Iceweasel 2.0 ; the difference is really impressive: 3.0b4 is almost 4 times as fast !

For reference, the Webkit currently in unstable (which is quite old, actually), gives these results. The one in experimental unfortunately crashes. By the way, I'm planning to package a new Webkit snapshot soon after I'm done with xulrunner and Iceweasel, we'll see then how it performs.

While speaking of tests, both Iceweasel 3.0b4 and Webkit from experimental pass the Acid 2 test (contrary to Iceweasel 2.0), and have both quite good results on the Acid 3 test: 61 for Webkit from experimental, when it doesn't crash (but current trunk has been reported to score 91 !), and 67 for Iceweasel 3.0b4 (compared to 52 for Iceweasel 2.0).

Update: Interestingly, built with upstream optimization flags (-Os-freorder-blocks -fno-reorder-functions) instead of -O2, it is slighly slower, though it might be better on some older hardware, or other architectures (I'm testing on x86-64).

Carlo Wood wrote a nice piece of documentation about how to recover files from ext3, based on his experience after deleting 3GB.

This actually happened to me 5 years ago, now, and is how ext3rminator came to exist. I've been meaning to update it for a while (and journal parsing has been on the top of the TODO list for a while), but package maintenance is unfortunately time consuming. And I have a big tendancy to do something else, too (like coding on git).

It's been a while I've not talked about my packages. I've been preparing a xulrunner trunk upload for a while now, and I must say that while I'm not totally happy with evolutions in Mozilla codebase, I do appreciate that a significant amount of the patches we were applying have been applied upstream. And even more will be applied when Firefox 3.0beta5 is released. I even got patches that were *not* in Debian to be applied. Thanks have to go to Reed Loden for all this.

Another nice thing on Mozilla trunk is that they finally stopped patching sqlite, which means we can now have Mozilla linked against the system one. It is even more important now that libnss uses sqlite, too.

And now that we have libcairo 1.5 in experimental, and recent enough nss and nspr in unstable (uploaded a few days ago, creating some mess with iceweasel, iceape and xulrunner), we have all the necessary bits to have (most of) the bundled libraries replaced by system ones.

The sad thing is they did implement APNG (animated png) in their bundled libpng, so, to have APNG support, we need to either patch libpng or use the bundled one. I'll see if I can just disable APNG for the moment.

I hope to be able to upload the package to experimental on saturday or sunday, based on the firefox 3.0pre4 codebase. Iceweasel should follow a few days later.

So, MadCoder had doubts about the crazy idea, so I took a little time to do a test with a package I maintain, namely, xulrunner.

First, take all the deb files in the history of the package (at least, everything that is available on snapshot.debian.net today).

wget -O - -q http://snapshot.debian.net/archive/pool/x/xulrunner/binary-amd64/Packages.gz | gzip -cd > list
awk '/^Filename:/{print $2}' list | xargs -I{} wget http://snapshot.debian.net/archive/{}

Next, commit all these in a different repo per package name :

perl -e 'use Dpkg::Version qw(vercmp); sub v { my $f = $_[0]; $f =~ s/.*_(.*)_.*/$1/; $f } print sort { vercmp(v($a), v($b)); } map { s/^Filename: .*\///; $_ } grep { /^Filename:/ } <>;' list | while read f; do
    pkg=${f%_*_*}
    [ ! -d $pkg ] && mkdir $pkg && ( cd $pkg ; git init )
    cd $pkg
    ar -x ../$f
    mkdir data control
    tar -C data -zxf data.tar.gz
    tar -C control -zxf control.tar.gz
    git add data control
    git commit -q -m $f
    rm -rf data control data.tar.gz control.tar.gz debian-binary
    cd ..
done

Finally, evaluate sizes for each package, respectively, of all .deb files, their content imported in git (only the .git directory, including the index ; some space could be gained removing it), and the "optimized" git repository (after git gc, without modifying delta depth or window size, which may even improve the result)

awk '/^Package:/{print $2}' list | sort -u | while read p; do
    du -c --si ${p}_*.deb | tail -1
    du -s --si $p
    cd $p; git gc ; cd ..
    echo
done

17M total
16M libmozjs0d
4.7M libmozjs0d

34M total
31M libmozjs0d-dbg
14M libmozjs0d-dbg

4.7M total
7.1M libnspr4-0d
1.3M libnspr4-0d

9.2M total
12M libnspr4-0d-dbg
2.8M libnspr4-0d-dbg

25M total
28M libnss3-0d
4.6M libnss3-0d

81M total
61M libnss3-0d-dbg
21M libnss3-0d-dbg

15M total
16M libnss3-tools
3.1M libnss3-tools

288M total
265M libxul0d
146M libxul0d

2.0G total
1.8G libxul0d-dbg
1.1G libxul0d-dbg

5.1M total
6.8M python-xpcom
1.1M python-xpcom

2.5M total
5.6M spidermonkey-bin
861k spidermonkey-bin

13M total
14M xulrunner
2.0M xulrunner

3.3M total
5.9M xulrunner-gnome-support
979k xulrunner-gnome-support


So these packages, stored in git, take between 15 and roughly 50 percent of the .deb size, which may be a nice improvement. Il would be interesting to know how these numbers evolve with time. Some files, such as the changelog.Debian.gz files, would also benefit from being stored in plain text instead of gzipped form.

Note git gc took a while and a lot of memory for libxul0d-dbg. Also note these don't include delta files that would be necessary to recreate the original .deb file, but this shouldn't make a huge difference.

I often have a bunch of somewhat crazy ideas, and I don't have any time available to test or implement them, which is sad. So just in case these crazy ideas would scratch someone's itch, I'm going to throw them in the wild.

I've been using git for a few months, now, and used it not only for source code management, but for efficient storage, too. *VERY* efficient. I'll have to write about that some day.

Anyways, while installing pristine-tar, today, I just thought it would be neat to have an equivalent pristine-deb, to store deb files efficiently. I'm pretty sure someone else thought about this possibility, but it's still better that such ideas come to the ears (eyes, actually) of someone that could implement them.

Such a pristine-deb tool could be used to... store packages from snapshot.debian.net. That would reduce the amount of space required for the archive dramatically, IMHO. I'm pretty sure old packages are not requested that much, so they could be generated on-the-fly from a CGI script placed as a GET action, so that urls wouldn't change.

The same could probably be applied to archive.debian.org. It could even save enough space that archive.debian.org could host snapshot.debian.net. But that depends on the average package content and its average evolution, which I have absolutely no idea about.

Update: It would also be interesting to have the .diff.gz files in there, too ; it would obviously allow to have an easy view of the contents, such as copyright files, changelogs, and other bits of information available on packages.debian.org.

Update 2: Actually, pristine-deb would as easy as storing 2 pristine-tars (one for control.tar.gz and one for data.tar.gz), and a debian-binary file. The .deb can be aggregated with

ar -rc file.deb debian-binary control.tar.gz data.tar.gz

Someone showed me some beautiful code, that happens to be procmail's. I'm really impressed :

goto jiasc;
do
{ *++to= *++from;
jiasc:;
}
while(--count);


I suggest anyone to take a look at procmail's source code, it's really a thrill.

Well, it's been a few days, already, but I'm back from vacation, and the website too. It happened that for some reason the ADSL connection decided to die a few days only after my departure, leaving the site unavailable for 3 weeks.

There are cases where you don't want to bring a laptop with you, but still want to have your data or, even better, your system and configuration. It is actually not so difficult to do if you are using LVM. With my setup, it was quite straightforward. So, if you happen to have a similar setup (2 partitions : one for /boot, and the other one for the LVM physical volume), here are the steps to do the same (assuming the usb disk is /dev/sda, the laptop disk /dev/hda, the LVM volume group vg, and you're using grub as a bootloader).

• Create equivalent partitions on the usb disk. In my case, the usb disk happens to be much bigger, so the LVM physical volume is going to be more than half free).
• Create an LVM physical volume on the second partition:
  

  /sbin/pvcreate /dev/sda2

• Extend the existing volume group with the new physical volume:
  

  /sbin/vgextend vg /dev/sda2

• Move all data from the laptop disk to the usb disk:
  

  /sbin/pvmove -v /dev/hda2

• Now you have plenty of time during the pvmove, create a new filesystem for /boot in the first partition:
  

  /sbin/mkfs.ext3 /dev/sda1

• Copy the current /boot files to the new filesystem:
  

  mount /dev/sda1 /mnt
cp -a /boot/. /mnt
umount /mnt

• As you will want to mount the new /boot at boot time, edit /etc/fstab to replace the device for the /boot entry by the new device. Since you probably don't know in advance what the device will be (you may use the disk on any computer, possibly with a SATA disk that will be on sda), it is preferred to use the UUID of the filesystem to mount it. Get it with:
  

  dumpe2fs -h /dev/sda1 | grep UUID

  Then replace the device name (in my case /dev/hda1) with UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

At this point (well, you obviously still need to wait for the pvmove to end), you can reboot right now. Your boot loader setup will use your old /boot partition and the kernel will use the root filesystem on the usb disk.
Be aware of a few things, though.

• Your initrd image (which you must have, if you have your root partition under LVM) must include everything that is necessary to load usb devices before mounting the root partition

(mkinitramfs in Debian does)

• Current mkinitramfs in Debian has a quite annoying bug, that makes impossible to have an LVM root partition on an usb disk: most of the time, the usb disk is detected after initialization of LVM. If that happens to you, you're not screwed: you can add break=mount to the kernel command line, and while you have a shell prompt, wait for the usb disk to come up and just exit the shell, it will continue the boot scripts and initialize LVM properly. On a recent enough initramfs, you can also specify rootdelay=10, replacing 10 with the number of seconds you want the boot to be delayed to wait for usb devices.

You still need to do two things to finish your setup, now:

• Remove the laptop disk physical volume from the volume group:
  

  vgreduce vg /dev/hda2

• Setup grub on the usb disk, so that you can boot off the disk directly if the BIOS supports it:
  

  /usr/sbin/grub
grub> root (hd1,0)
 Filesystem type is ext2fs, partition type 0x83
grub> setup --prefix=/grub (hd1)
 Checking if "/grub/stage1" exists... yes
 Checking if "/grub/stage2" exists... yes
 Checking if "/grub/e2fs_stage1_5" exists... yes
 Running "embed /grub/e2fs_stage1_5 (hd1)"... 15 sectors are embedded.
succeeded
 Running "install /grub/stage1 (hd1) (hd1)1+15 p (hd1,0)/grub/stage2 /grub/menu.lst"... succeeded
Done.
grub> quit


  You probably won't have to change the menu.lst file, because it must already be using hd0, which will be what grub sees for the usb disk when booting off it. Also note your usb disk might not be hd1 under grub, depending on the number of disks on your system.

Now you can have any computer run your system, booting off the usb disk. It had the nice side effect that my laptop is much faster now, as the internal disk is an old 4200 rpm disk, vs. 7200 rpm for the usb disk...
You may also want to try to remove the Xorg configuration file (/etc/X11/xorg.conf) so that the video driver and resolution are not hardcoded ; recent enough Xorg do runtime detection, and that worked pretty well on the two laptops I booted with the usb disk.