Archive for the 'p.d.o' Category

Firefox in Debian?

Got your attention? Don’t hold your breath, we’re not there yet, but we’re a step closer: it’s now possible to build Firefox from the Iceweasel package, since version 17.0.1-2 in experimental as of writing, 18.0~b6-1 from the iceweasel-beta repository, or 19.0~a2+20121228042015-1 from the iceweasel-aurora repository.

Before letting you know how you can get yourself a packaged Firefox based on the Iceweasel source, I’ll remind you that redistribution of Firefox packages requires a trademark license from Mozilla, so please keep the packages you build for yourself for now.

That being said, now it’s clear that such Firefox packages are not official, you can still test them for yourself. First download the Iceweasel source version of your liking, and extract it, then rename all source files from iceweasel_* to firefox_* (rename s/iceweasel/firefox/ iceweasel_* should do it). Edit debian/changelog so that the first line reads:

firefox (x.y.z-r) distribution; urgency=low

instead of:

iceweasel (x.y.z-r) distribution; urgency=low

and run the following command:

$ debian/rules debian/control

Now you’re all set. You can build the package the usual way.

Note there are a few differences between the xulrunner packages you get from building Iceweasel vs. from building Firefox that need to be addressed, and a few other details to sort out.

2012-12-29 11:00:21+0100

firefox | 1 Comment »

Debian EFI mode boot on a Macbook Pro, without rEFIt

Diego’s post got me to switch from grub-pc to grub-efi to boot Debian on my Macbook Pro. But I wanted to go further: getting rid of rEFIt.

rEFIt is a pretty useful piece of software, but it’s essentially dead. There is the rEFInd fork, which keeps it up-to-date, but it doesn’t really help with FileVault. Moreover, the boot sequence for a Linux distro with rEFIt/rEFInd looks like: Apple EFI firmware → rEFIt/rEFInd → GRUB → Linux kernel. Each intermediate step adding its own timeout, so rEFIt/rEFInd can be seen as not-so-useful intermediate step.

Thankfully, Matthew Garrett did all the research to allow to directly boot GRUB from the Apple EFI firmware. Unfortunately, his blog post didn’t have much actual detail on how to do it.

So here it is, for a Debian system:

  • Install a few packages you’ll need in this process:
    # apt-get install hfsprogs icnsutils
  • Create a small HFS+ partition. I have a 9MB one, but it’s only filled by about 500K, so even smaller should work too. If, like me, you were previously using grub-pc, you probably have a GRUB partition, you can repurpose it. In gdisk, it looks like this:
    Number  Start (sector)    End (sector)  Size       Code  Name
       5       235284480       235302943   9.0 MiB     AF00  Apple HFS/HFS+
    
    Partition GUID code: 48465300-0000-11AA-AA11-00306543ECAC (Apple HFS/HFS+)
    Partition unique GUID: AD1F5465-B777-4178-AC4D-1DE8B2EB1B4B
    First sector: 235284480 (at 112.2 GiB)
    Last sector: 235302943 (at 112.2 GiB)
    Partition size: 18464 sectors (9.0 MiB)
    Attribute flags: 0000000000000000
    Partition name: 'Apple HFS/HFS+'
    
  • Create a HFS+ filesystem on that partition:

    # mkfs.hfsplus /dev/sda5 -v Debian

    (replace /dev/sda5 with whatever your partition is)

  • Add a fstab entry for that filesystem:
    # echo $(blkid -o export -s UUID /dev/sda5) /boot/efi auto defaults 0 0 >> /etc/fstab
  • Mount the filesystem:
    # mkdir /boot/efi
    # mount /boot/efi
    
  • Edit /usr/sbin/grub-install, look for « xfat », and remove the block of code that looks like:
    if test "x$efi_fs" = xfat; then :; else
        echo "${efidir} doesn't look like an EFI partition." 1>&2
        efidir=
    fi
    
  • Run grub-install. At this point, there should be a /boot/efi/EFI/debian/grubx64.efi file (if using grub-efi-amd64).
  • Create a /boot/efi/System/Library/CoreServices directory:
    # mkdir -p /boot/efi/System/Library/CoreServices
  • Create a hard link:
    # ln /boot/efi/EFI/debian/grubx64.efi /boot/efi/System/Library/CoreServices/boot.efi
  • Create a dummy mach_kernel file:
    # echo "This file is required for booting" > /boot/efi/mach_kernel
  • Grab the mactel-boot source code, unpack and build it:
    # wget http://www.codon.org.uk/~mjg59/mactel-boot/mactel-boot-0.9.tar.bz2
    # tar -jxf mactel-boot-0.9.tar.bz2
    # cd mactel-boot-0.9
    # make PRODUCTVERSION=Debian
    
  • Copy the SystemVersion.plist file:
    # cp SystemVersion.plist /boot/efi/System/Library/CoreServices/
  • Bless the boot file:
    # ./hfs-bless /boot/efi/System/Library/CoreServices/boot.efi
  • (optional) Add an icon:
    # rsvg-convert -w 128 -h 128 -o /tmp/debian.png /usr/share/reportbug/debian-swirl.svg
    # png2icns /boot/efi/.VolumeIcon.icns /tmp/debian.png
    # rm /tmp/debian.png
    

Now, the Apple Boot Manager, shown when holding down the option key when booting the Macbook Pro, looks like this:

And the Startup disk preferences dialog under OSX, like this:

2012-11-18 11:18:14+0100

debian, p.m.o | 35 Comments »

Building a Linux kernel module without the exact kernel headers

Imagine you have a Linux kernel image for an Android phone, but you don’t have the corresponding source, nor do you have the corresponding kernel headers. Imagine that kernel has module support (fortunately), and that you’d like to build a module for it to load. There are several good reasons why you can’t just build a new kernel from source and be done with it (e.g. the resulting kernel lacks support for important hardware, like the LCD or touchscreen). With the ever-changing Linux kernel ABI, and the lack of source and headers, you’d think you’re pretty much in a dead-end.

As a matter of fact, if you build a kernel module against different kernel headers, the module will fail to load with errors depending on how different they are. It can complain about bad signatures, bad version or other different things.

But more on that later.

Configuring a kernel

The first thing is to find a kernel source for something close enough to the kernel image you have. That’s probably the trickiest part with getting a proper configuration. Start from the version number you can read from /proc/version. If, like me, you’re targeting an Android device, try Android kernels from Code Aurora, Linaro, Cyanogen or Android, whichever is closest to what is in your phone. In my case, it was msm-3.0 kernel. Note you don’t necessarily need the exact same version. A minor version difference is still likely to work. I’ve been using a 3.0.21 source, which the kernel image was 3.0.8. Don’t however try e.g. using a 3.1 kernel source when the kernel you have is 3.0.x.

If the kernel image you have is kind enough to provide a /proc/config.gz file, you can start from there, otherwise, you can try starting from the default configuration, but you need to be extra careful, then (although I won’t detail using the default configuration because I was fortunate enough that I didn’t have to, there will be some details further below as to why a proper configuration is important).

Assuming arm-eabi-gcc is in your PATH, and that you have a shell opened in the kernel source directory, you need to start by configuring the kernel and install headers and scripts:

$ mkdir build
$ gunzip -c config.gz > build/.config # Or whatever you need to prepare a .config
$ make silentoldconfig prepare headers_install scripts ARCH=arm CROSS_COMPILE=arm-eabi- O=build KERNELRELEASE=`adb shell uname -r`

The silentoldconfig target is likely to ask you some questions about whether you want to enable some things. You may want to opt for the default, but that may also not work properly.

You may use something different for KERNELRELEASE, but it needs to match the exact kernel version you’ll be loading the module from.

A simple module

To create a dummy module, you need to create two files: a source file, and a Makefile.

Place the following content in a hello.c file, in some dedicated directory:

#include <linux/module.h>       /* Needed by all modules */
#include <linux/kernel.h>       /* Needed for KERN_INFO */
#include <linux/init.h>         /* Needed for the macros */
static int __init hello_start(void)
{
  printk(KERN_INFO "Hello world\n");
  return 0;
}
static void __exit hello_end(void)
{
  printk(KERN_INFO "Goodbye world\n");
}
module_init(hello_start);
module_exit(hello_end);

Place the following content in a Makefile under the same directory:

obj-m = hello.o

Building such a module is pretty straightforward, but at this point, it won’t work yet. Let me enter some details first.

The building of a module

When you normally build the above module, the kernel build system creates a hello.mod.c file, which content can create several kind of problems:

MODULE_INFO(vermagic, VERMAGIC_STRING);

VERMAGIC_STRING is derived from the UTS_RELEASE macro defined in include/generated/utsrelease.h, generated by the kernel build system. By default, its value is derived from the actual kernel version, and git repository status. This is what setting KERNELRELEASE when configuring the kernel above modified. If VERMAGIC_STRING doesn’t match the kernel version, loading the module will lead to the following kind of message in dmesg:

hello: version magic '3.0.21-perf-ge728813-00399-gd5fa0c9' should be '3.0.8-perf'

Then, there’s the module definition.

struct module __this_module
__attribute__((section(".gnu.linkonce.this_module"))) = {
 .name = KBUILD_MODNAME,
 .init = init_module,
#ifdef CONFIG_MODULE_UNLOAD
 .exit = cleanup_module,
#endif
 .arch = MODULE_ARCH_INIT,
};

In itself, this looks benign, but the struct module, defined in include/linux/module.h comes with an unpleasant surprise:

struct module
{
        (...)
#ifdef CONFIG_UNUSED_SYMBOLS
        (...)
#endif
        (...)
        /* Startup function. */
        int (*init)(void);
        (...)
#ifdef CONFIG_GENERIC_BUG
        (...)
#endif
#ifdef CONFIG_KALLSYMS
        (...)
#endif
        (...)
(... plenty more ifdefs ...)
#ifdef CONFIG_MODULE_UNLOAD
        (...)
        /* Destruction function. */
        void (*exit)(void);
        (...)
#endif
        (...)
}

This means for the init pointer to be at the right place, CONFIG_UNUSED_SYMBOLS needs to be defined according to what the kernel image uses. And for the exit pointer, it’s CONFIG_GENERIC_BUG, CONFIG_KALLSYMS, CONFIG_SMP, CONFIG_TRACEPOINTS, CONFIG_JUMP_LABEL, CONFIG_TRACING, CONFIG_EVENT_TRACING, CONFIG_FTRACE_MCOUNT_RECORD and CONFIG_MODULE_UNLOAD.

Start to understand why you’re supposed to use the exact kernel headers matching your kernel?

Then, the symbol version definitions:

static const struct modversion_info ____versions[]
__used
__attribute__((section("__versions"))) = {
	{ 0xsomehex, "module_layout" },
	{ 0xsomehex, "__aeabi_unwind_cpp_pr0" },
	{ 0xsomehex, "printk" },
};

These come from the Module.symvers file you get with your kernel headers. Each entry represents a symbol the module requires, and what signature it is expected to have. The first symbol, module_layout, varies depending on what struct module looks like, i.e. depending on which of the config options mentioned above are enabled. The second, __aeabi_unwind_cpp_pr0, is an ARM ABI specific function, and the last, is for our printk function calls.

The signature for each function symbol may vary depending on the kernel code for that function, and the compiler used to compile the kernel. This means that if you have a kernel you built from source, modules built for that kernel, and rebuild the kernel after modifying e.g. the printk function, even in a compatible way, the modules you built initially won’t load with the new kernel.

So, if we were to build a kernel from the hopefully close enough source code, with the hopefully close enough configuration, chances are we wouldn’t get the same signatures as the binary kernel we have, and it would complain as follows, when loading our module:

hello: disagrees about version of symbol symbol_name

Which means we need a proper Module.symvers corresponding to the binary kernel, which, at the moment, we don’t have.

Inspecting the kernel

Conveniently, since the kernel has to do these verifications when loading modules, it actually contains a list of the symbols it exports, and the corresponding signatures. When the kernel loads a module, it goes through all the symbols the module requires, in order to find them in its own symbol table (or other modules’ symbol table when the module uses symbols from other modules), and check the corresponding signature.

The kernel uses the following function to search in its symbol table (in kernel/module.c):

bool each_symbol_section(bool (*fn)(const struct symsearch *arr,
                                    struct module *owner,
                                    void *data),
                         void *data)
{
        struct module *mod;
        static const struct symsearch arr[] = {
                { __start___ksymtab, __stop___ksymtab, __start___kcrctab,
                  NOT_GPL_ONLY, false },
                { __start___ksymtab_gpl, __stop___ksymtab_gpl,
                  __start___kcrctab_gpl,
                  GPL_ONLY, false },
                { __start___ksymtab_gpl_future, __stop___ksymtab_gpl_future,
                  __start___kcrctab_gpl_future,
                  WILL_BE_GPL_ONLY, false },
#ifdef CONFIG_UNUSED_SYMBOLS
                { __start___ksymtab_unused, __stop___ksymtab_unused,
                  __start___kcrctab_unused,
                  NOT_GPL_ONLY, true },
                { __start___ksymtab_unused_gpl, __stop___ksymtab_unused_gpl,
                  __start___kcrctab_unused_gpl,
                  GPL_ONLY, true },
#endif
        };

        if (each_symbol_in_section(arr, ARRAY_SIZE(arr), NULL, fn, data))
                return true;

        (...)

The struct used in this function is defined in include/linux/module.h as follows:

struct symsearch {
        const struct kernel_symbol *start, *stop;
        const unsigned long *crcs;
        enum {
                NOT_GPL_ONLY,
                GPL_ONLY,
                WILL_BE_GPL_ONLY,
        } licence;
        bool unused;
};

Note: this kernel code hasn’t changed significantly in the past four years.

What we have above is three (or five, when CONFIG_UNUSED_SYMBOLS is defined) entries, each of which contains the start of a symbol table, the end of that symbol table, the start of the corresponding signature table, and two flags.

The data is static and constant, which means it will appear as is in the kernel binary. By scanning the kernel for three consecutive sequences of three pointers within the kernel address space followed by two integers with the values from the definitions in each_symbol_section, we can deduce the location of the symbol and signature tables, and regenerate a Module.symvers from the kernel binary.

Unfortunately, most kernels these days are compressed (zImage), so a simple search is not possible. A compressed kernel is actually a small bootstrap binary followed by a compressed stream. It is possible to scan the kernel zImage to look for the compressed stream, and decompress it from there.

I wrote a script to do decompression and extraction of the symbols info automatically. It should work on any recent kernel, provided it is not relocatable and you know the base address where it is loaded. It takes options for the number of bits and endianness of the architecture, but defaults to values suitable for ARM. The base address, however, always needs to be provided. It can be found, on ARM kernels, in dmesg:

$ adb shell dmesg | grep "\.init"
<5>[01-01 00:00:00.000] [0: swapper]      .init : 0xc0008000 - 0xc0037000   ( 188 kB)

The base address in the example above is 0xc0008000.

If like me you’re interested in loading the module on an Android device, then what you have as a binary kernel is probably a complete boot image. A boot image contains other things besides the kernel, so you can’t use it directly with the script. Except if the kernel in that boot image is compressed, in which case the part of the script that looks for the compressed image will find it anyways.

If the kernel is not compressed, you can use the unbootimg program as outlined in this old post of mine to get the kernel image out of your boot image. Once you have the kernel image, the script can be invoked as follows:

$ python extract-symvers.py -B 0xc0008000 kernel-filename > Module.symvers

Symbols and signature info could also be extracted from binary modules, but I was not interested in that information so the script doesn’t handle that.

Building our module

Now that we have a proper Module.symvers for the kernel we want to load our module in, we can finally build the module:

(again, assuming arm-eabi-gcc is in your PATH, and that you have a shell opened in the kernel source directory)

$ cp /path/to/Module.symvers build/
$ make M=/path/to/module/source ARCH=arm CROSS_COMPILE=arm-eabi- O=build modules

And that’s it. You can now copy the resulting hello.ko onto the device and load it.

and enjoy

$ adb shell
# insmod hello.ko
# dmesg | grep insmod
<6>[mm-dd hh:mm:ss.xxx] [id: insmod]Hello world
# lsmod
hello 586 0 - Live 0xbf008000 (P)
# rmmod hello
# dmesg | grep rmmod
<6>[mm-dd hh:mm:ss.xxx] [id: rmmod]Goodbye world

2012-08-06 15:11:41+0100

p.d.o, p.m.o | 12 Comments »

What is a Web App?

Is it this, this or that?

2012-07-20 11:04:59+0100

p.d.o, p.m.o | 6 Comments »

Comment spam

Three weeks ago, I slightly modified the comment system on this blog for an experiment. This blog is a standard wordpress installation. Comments are normally directed to the wp-comments-post.php script by the HTML form. What I did is:

  • Create a comments-post.php wrapper script that just includes wp-comments-post.php (this allows things to still work properly after wordpress upgrades),
  • Make the HTML form direct to a comments-post.php script,
  • Add a usedForm=1 parameter to the HTML form action, such that comments-post.php is supposed to always be called with it,
  • Add a simple javascript that adds a hasJS=1 parameter to the HTML form action when the page is loaded, and a Submit=1 parameter when the form is submitted.

During the past three weeks, on this blog, there were 7170 comments, 8 of which were actual comments. 7162 were spam (~99.9%).

  • 3165 spams (~44.1%) were sent to the original WordPress comment handler (wp-comments-post.php) from 1589 unique IP addresses.
  • 0 spam were sent to the new comment handler without a query string (comments-post.php), but 1 was sent with an empty query string (comments-post.php?).
  • 18 spams were sent to the new comment handler with a lowercased query string (comments-post.php?usedform=1) from 6 unique IP addresses.
  • 3971 spams (~55.4%) were sent to the new comment handler with the form query string (comments-post.php?usedForm=1) from 1153 unique IP addresses.
  • 7 spams (~0.1%) were sent to the new comment handler with the full query string, including what is added through javascript (comments-post.php?usedForm=1&hasJS=1&Submit=1) from 5 unique IP addresses.

This means a large portion of spammers didn’t care about actually checking the comment forms and used the standard wordpress url, and another large portion don’t run javascript on their bots, although a very few do.

2012-07-15 11:35:54+0100

p.d.o, p.m.o, website | 1 Comment »

Attempting to close a LinkedIn account

Following the trend, I attempted to close my LinkedIn account. Closing a LinkedIn account involves confirming and confirming and confirming again. Once it’s all done, you’d expect to, well, be done with it.

I’m outraged at the result:

  • My public profile is still there. I can’t be sure but I guess people with a connection to me can still see the full profile.
  • I’m still receiving LinkedIn connection emails (You, know, those “Learn about xxxxxxx, your new connection…” emails ; I must have had pending outgoing invitations).
  • I can still reset my password.
  • I can still login.

The only upside is that after I login, I can only see a page saying “Your LinkedIn account has been temporarily restricted”. “Contact our customer service team to get this resolved as soon as possible.”

Update: After contacting their customer service, the account was closed and the public profile is now unavailable.

2012-06-09 14:52:53+0100

p.d.o, p.m.o | 8 Comments »

Iceweasel ESR in squeeze-backports

In case this went unnoticed, Iceweasel ESR has been available in squeeze-backports for a few weeks, now. I highly recommend anyone using Iceweasel on the Debian stable release to upgrade to that version, at the very least. Even newer versions are available through the pkg-mozilla archive.

2012-06-02 11:33:08+0100

firefox | 8 Comments »

Announcing vmfs-tools version 0.2.5

The last release of vmfs-tools (0.2.1) was almost 2 years ago. It was about time to bring some of the changes that have been available in the git repository in an official tarball. So here it is.

It brings some limited VMFS 5 support and experimental extent removal, as well as some deep changes to the debugvmfs tool and various fixes.

Next release (0.2.6) will have a fixed fsck, which, while it still won’t fix file system inconsistencies, should at least report actual inconsistencies (which is far from being true currently). I won’t give any estimation as to when this will happen, though.

2012-03-25 11:02:03+0100

vmfs-tools | 12 Comments »

libgcc.a symbol visibility considered harmful

I recently got to rebuild an Android NDK with a fresh toolchain again, and hit an interesting problem. I actually had hit it before, but only this time I fully analyzed what’s going on. [As a side note, if you build such a NDK, don’t use mpfr 3.1.0, as there is a bug in the libtool it ships]

Linking an application or a library pulls many things, that aren’t part of the code being built. One of these many things is the libgcc static library. Part of libgcc consists in an implementation of the platform ABI. On Android systems, this means the ARM EABI. GCC, when compiling some instructions, will generate ABI calls. For example, integer divisions may call __aeabi_idiv.

Consider the following minimized real world scenario:

$ echo "int foo(int a) { return 42 % a; }" > foo.c
$ arm-linux-androideabi-gcc -o libfoo.so -shared foo.c -mandroid

GCC will emit a call to __aeabi_idivmod for the % operation. With GCC 4.6.3, this function is in _divsi3.o under libgcc.a. That function itself calls __aeabi_idiv0, which lives in _dvmd_lnx.o under libgcc.a.

When statically linking, ld will thus include foo.o, _divsi3.o and _dvmd_lnx.o, meaning it will include all functions from these object files. That is, foo, __divsi3, __aeabi_idiv, __aeabi_idivmod, __aeabi_idiv0 and __aeabi_ldiv0. And more than being included, these functions are exported, because symbol visibility in libgcc.a is default. So while we expect exporting foo from our library, we’re actually exporting much more, including functions that just happened to be near the ones that our code (indirectly) uses.

Now, let’s say we want to build another library, using that foo function from libfoo:

$ cat > bar.c <<EOF
extern int foo(int a);
long long bar(long long a) { return foo(a) % a; }
EOF
$ arm-linux-androideabi-gcc -o libbar.so -shared bar.c -mandroid

(The code above has absolutely no meaning, it just triggers the same function calls as what I was getting in the actual real world case)

When statically linking the above code, GCC will generate a call to __aeabi_ldivmod, which calls __aeabi_ldiv0, and many other things, directly or indirectly. When linking as above, nothing particularly nasty is going to happen. However, linking as above is actually wrong: the resulting library has an undefined reference to the foo symbol, and doesn’t depend on libfoo. At runtime, if libfoo wasn’t already loaded somehow, loading libbar would fail.

The proper way to link is the following:

$ arm-linux-androideabi-gcc -o libbar.so -shared bar.c -mandroid -L. -lfoo

A feature of ELF static linking is that when it resolves undefined symbols, the linker will choose to use the first occurrence of a symbol it finds in the various objects and libraries given on its command line. So with the command line above, for each __aeabi_* symbol, it will first look in libfoo if there isn’t one. And while __aeabi_ldivmod is not in libfoo, __aeabi_ldiv0 is (see above).

So instead of including the code for __aeabi_ldiv0 from libgcc.a, it will call the copy from libfoo.

This wouldn’t be so much of a problem if __aeabi_ldiv0 wasn’t a weak symbol.

Enters faulty.lib. In the real world case, libfoo is loaded by the system dynamic linker, and libbar by faulty.lib. When resolving symbols for libbar, faulty.lib has to resolve libfoo symbols with the system linker, using dlsym(). On Android, dlsym() returns NULL for weak (defined) symbols, so faulty.lib can’t resolve __aeabi_ldiv0.

The real world case wasn’t a problem with GCC 4.4.3 from the vanilla Android NDK because in that GCC version, __aeabi_ldivmod doesn’t call __aeabi_ldiv0.

This wouldn’t happen if shared libraries wouldn’t expose random platform ABI specific bits depending on what they use and depending on other symbols that happen to be in the same object files.

A similar issue happened a little while ago on Debian powerpc because a shared library was exporting ABI specific bits. Even worse, the toolchain was assuming the symbols would come from libgcc.a and generated wrong relocations for these symbols.

Update: Interestingly, the __aeabi_* symbols are hidden, in libgcc.a as provided on the Debian armel port.

2012-03-06 17:19:34+0100

faulty.lib, p.d.o, p.m.o | 5 Comments »

Introducing faulty.lib

TL;DR link: faulty.lib is on github.

Android applications are essentially zip archives, with the APK extension instead of ZIP. While most of Android is java, and java classes are usually loaded from a ZIP archive (usually with the JAR extension), Android applications using native code need to have native libraries on the file system. These native libraries are found under /data/data/$appid/lib, where $appid is the package name, as defined in the AndroidManifest.xml file.

So, when Android installs an application, it puts that APK file under /data/app. Then, if the APK contains native libraries under a lib/$ABI subdirectory (where $ABI is armeabi, armeabi-v7a or x86), it also decompresses the files and places them under /data/data/$appid/lib. This means native libraries are actually stored twice on internal flash: once compressed and once decompressed.

This is why Chrome for Android takes almost 50MB of internal flash space after installation.

Firefox for Android used to have that problem, and we decided we should stop doing that. Michael Wu thus implemented a custom dynamic linker, which would load most of Firefox libraries directly off the APK. This involves decompressing the zipped data somewhere in memory, and doing ld.so‘s job to make the library usable (please note that on Android, ld.so is actually named linker). There were initially circumstances under which we would decompress into a file and reuse it the next time Firefox starts, but we subsequently removed that possibility (except for debugging purpose) because it ended up being slower than decompressing each time (thanks to internal flash being so slow).

Anyways, in order to do ld.so‘s job, our custom linker was directly derived from Android’s system linker, with many tweaks. This custom linker has done its job quite well for some time, now, but has been recently replaced, see further below.

Considering Firefox can’t do anything useful involving Gecko until its libraries are loaded, in practice, this means Firefox can’t display a web page faster than completely decompressing the libraries. Or can it?

Just don’t sit down ’cause I’ve moved your chair

We know that a lot of code and data is not used during Firefox startup. Based on that knowledge, we started working on only loading the necessary bits. The core of the idea is, when a library is requested to be loaded, to reserve anonymous memory for its decompressed size, and… that’s all. That memory is protected such that any access to it triggers a segmentation fault. When a segmentation fault happens, the required bits are decompressed, and execution is resumed where it was before the segmentation fault.

The original prototype was decompressing from a normal zip deflated stream, which means it was impossible to seek in it. So, if an access was made at the end of the library, it was necessary to decompress the whole library. With some nasty binary reordering, and some difficulty, it was possible to avoid accessing the end of the library, but the approach is very much fragile. It only takes an unexpected code path to make things much slower than they should be.

Consequently, for the past months, I’ve been working on improving the original idea and, with some assistance from Julian Seward, implemented the scheme with seekable compressed streams. Instead of letting the zip archive creation tool deflate libraries, we store specially crafted files. Essentially, files are cut in small chunks, and each chunk is compressed individually. This means a less efficient compression, but it also means random access to chunks is possible.

However, instead of stacking on top of our existing custom linker, I started over, from the ground up. First, because it needed a serious clean up (a good part of linker.c is leftovers from the Android linker that we don’t use, and APKOpen.cpp is a messy mix of JNI stubs, library decompression handling (which in itself was also a mess) and Gecko initialization code) and most importantly, because it relied on some Android system linker internals and thus required binary compatibility with the system linker. Which, according to Google engineers that contacted us a few months ago, was going to break in what we now know will be called Android Jelly Bean.

The benefit of the clean slate approach is that the new code is not tied to Gecko at all and was designed to work on Android as well as on desktop Linux systems (which made debugging much much easier). We’re thus releasing the code as a separate project: faulty.lib. It is licensed under the Mozilla Public License version 2.0. Please feel free to try, contribute, and/or fork it.

This dynamic linker is not meant to completely follow standard ELF rules (most notably for symbol resolution), and as a result does some assumptions. It’s also still work in progress, with some obvious optimizations pending (like, avoiding to resolve the same symbols again and again during relocations), and some features missing (for example, symbol versioning).

The next blog post will give some information about how to build Firefox for Android to benefit from on-demand decompression. I will also detail a few of the tricks involved in this dynamic linker in subsequent blog posts.

2012-03-01 17:10:11+0100

faulty.lib, p.d.o, p.m.o | 5 Comments »