May 30th, 2013

日本へ引っ越し

Today, May the 30th, was my last day as a Mozilla employee. In a couple weeks, my wife, my cat and I will be on board of a flight heading about ten thousand kilometers east, and most of our stuff will be in some container on a boat. We’re moving to Japan. As adventurous as this may sound, I’m not venturing into unknown territory. My wife is Japanese, and I’ve lived there for close to 15 months. A long time ago, arguably.

I’m not actually leaving Mozilla. I’ll be back as a contractor, hopefully around the 25th of June. So as far as my fellow coworkers are concerned, I’ll be going on a long-ish vacation and changing timezone (but I’ll probably be around in the meanwhile on irc or bugmail, with high latency).

Jump-starting in a different country is not something really easy to pull off, and working for Mozilla as a remotee has been a key element in being able to do so. Although I’ve made it clear when I joined Mozilla that this would eventually happen, I’m thankful I can now actually do it.

2013-05-30 19:52:08+0200

me, p.d.o, p.m.o | 5 Comments »

May 27th, 2013

signal() doubly considered harmful

When you want to set signal handlers on UNIX systems, the typical choice is to use signal (specified in C89, C99 and POSIX.1-2001) or sigaction (specified in POSIX.1-2001 and System V r4).

Quoting the signal manual page:

The only portable use of signal() is to set a signal’s disposition to SIG_DFL or SIG_IGN. The semantics when using signal() to establish a signal handler vary across systems (and POSIX.1 explicitly permits this variation); do not use it for this purpose.

POSIX.1 solved the portability mess by specifying sigaction(2), which provides explicit control of the semantics when a signal handler is invoked; use that interface instead of signal().

Then it goes on about the UNIX vs BSD semantics, and how they affect signal delivery, which essentially is the main reason why one would want to stop using signal and use sigaction instead, with specifically chosen flags.

But this is not really what I wanted to talk about here.

One of the uses of signal or sigaction is to temporarily set a signal handler and restore the old signal handler once the job is done. Notwithstanding the fact that it’s a pretty horrible thing to do in a multi-threaded program, it’s also a horrible thing to do at all with signal if sigaction is used.

The core of the problem is the following: the information you get from signal() about the old signal handler is missing all the important pieces about it if it was originally set with sigaction(), namely, flags, masks and restorer.

So if you do use signal() to temporarily set a signal handler and then restore the previous signal handler, you risk resetting flags, masks and restorer. The first awful thing this means is the previous signal handler might be expecting three arguments, only one of which will be valid when it’s invoked. Unexpected things can also happen with the lack of expected flags or masks. This is why you’ll see horrible workarounds like this or that.

In short, if you do use signal() to temporarily set a signal handler and then restore the previous signal handler, you’re doing it wrong. And if you do that in a system library or driver, thank you for screwing things up. I’m looking at you libsc-a3xx.so.

2013-05-27 17:15:13+0200

p.d.o, p.m.o | 2 Comments »

March 14th, 2013

Google Reader death, or how the cloud model can fail you

If you’re a Google Reader user, you probably read in one of your subscriptions that Google is pulling the plug on Google Reader. It is yet another demonstration of why putting data in the cloud isn’t so much of a nice idea: the service you rely on may well disappear some day, with all the data it contains.

Sure Google, in its extreme goodness, allows you to “take out” the Google Reader data. Or does it?
These are what you’ll get from Google Takeout for Reader:

  • followers.json, following.json: both files contain similar data, that I suspect correspond to Buzz subscriptions (yet another dead service). Each friends item contains some information about your “friend”, and a stream identifier for their activity (I guess), as well as a few websites urls. For instance Tim Bray’s stream is “user/05198174665841271019/state/com.google/broadcast“. What the hell do I do with that? Fortunately, he has websites, but not all my “friends” have. Thankfully, I haven’t really been using this feature, so there’s almost nothing in these files.
  • liked.json, starred.json, shared.json, shared-by-followers.json: all have the same structure, and contain items you liked, starred, shared, or that the people you follow shared (yeah, that file is badly named). Each item contains an url (or so I hope), and the corresponding content (yay). shared-by-followers.json however doesn’t contain more than the items the people you follow actively shared: it doesn’t contain their feeds (and I’m pretty sure I read more from Tim Bray than the two links he shared)
  • subscriptions.xml: Essentially, a list of RSS feed urls, with no content ; nothing from Tim Bray here, but now that I think about it, I think I was only following his Buzz feed, so that went away with Buzz without me noticing.

Interestingly, while looking into shared-by-followers.json, I found urls that would correspond to friend streams. For instance, Tim Bray’s is http://www.google.com/reader/public/atom/user/05198174665841271019/state/com.google/broadcast. But it’s useless: all it displays is “permission denied”.

As for subscriptions, one of the strengths of Google Reader is that it allowed to search though past items, which means a big part of the interesting data is the archived items. But that’s not part of the “take out”. Sure, you have the feed urls, but most RSS feeds contain a limited amount of items, not the entire history of items for the given feed. So, history is more or less lost. Except if I star, like or share all items in all my subscriptions and “take out” again.

So much goodness.

It could have been worse, though.

2013-03-14 08:35:45+0200

p.d.o, p.m.o | 14 Comments »

February 19th, 2013

Ten years

Ten years ago, this very day, my first Debian package entered the Debian unstable repository. It was an addon for Mozilla Composer, Daniel Glazman’s Cascades.

On the same day, my second Debian package entered the Debian unstable repository as well. It was an addon for Mozilla Browser, Checky.

A few days later, my third Debian package entered Debian unstable. It was an addon for Mozilla Browser, Diggler.

Do you see a pattern? They are now abandoned software, although I made Checky and Diggler live a little longer (and I’m actually considering reviving Diggler) but they had their importance in my journey, and are part of the reason why I am where I am now.

My journey on the web started with NCSA Mosaic on VAX/VMS, then continued with Netscape Navigator, Netscape Communicator and Mozilla Suite on Linux.

That’s where I was ten years ago, sailing between Galeon (a browser using the Mozilla engine) and Mozilla Suite, and filing some layout bugs.

Ten years ago, there was a new kid on the block. It used to be called Phoenix, it had just changed its name to Firebird. Eventually, it changed again for Firefox. You may have heard about it. Because Firebird was so much nicer than the browser in the Mozilla Suite, I started using its Debian package, and wanted my packaged addons to work with it. So I contacted Eric Dorland, Phoenix/Firebird package maintainer at the time, and got the addons working. I then ended up fixing a bunch of packaging issues.

This is how I got involved in Firefox packaging for Debian, and what eventually led me to work for Mozilla.

2013-02-19 22:45:30+0200

firefox, p.m.o | No Comments »

December 29th, 2012

Firefox in Debian?

Got your attention? Don’t hold your breath, we’re not there yet, but we’re a step closer: it’s now possible to build Firefox from the Iceweasel package, since version 17.0.1-2 in experimental as of writing, 18.0~b6-1 from the iceweasel-beta repository, or 19.0~a2+20121228042015-1 from the iceweasel-aurora repository.

Before letting you know how you can get yourself a packaged Firefox based on the Iceweasel source, I’ll remind you that redistribution of Firefox packages requires a trademark license from Mozilla, so please keep the packages you build for yourself for now.

That being said, now it’s clear that such Firefox packages are not official, you can still test them for yourself. First download the Iceweasel source version of your liking, and extract it, then rename all source files from iceweasel_* to firefox_* (rename s/iceweasel/firefox/ iceweasel_* should do it). Edit debian/changelog so that the first line reads:

firefox (x.y.z-r) distribution; urgency=low

instead of:

iceweasel (x.y.z-r) distribution; urgency=low

and run the following command:

$ debian/rules debian/control

Now you’re all set. You can build the package the usual way.

Note there are a few differences between the xulrunner packages you get from building Iceweasel vs. from building Firefox that need to be addressed, and a few other details to sort out.

2012-12-29 11:00:21+0200

firefox | 1 Comment »

December 4th, 2012

Using distcc and clang on Linux to build Firefox faster on MacOSX

Apple makes nice hardware, until you look into the details. As it turns out, my Macbook Pro has issues with heat. So much that when building Firefox, the CPU decides it’s too hot and throttles its frequency to emit less heat. The result is that building Firefox takes much more time than it would if the CPU could stay at its highest frequency all the time. On this machine, it takes more than an hour to build Firefox.

Most of the time, this MBP runs Linux. The heat problem is the same, but I also have a much beefier build server, running Linux too, that I use for builds. I used to use distcc from the MBP, using both the MBP and the build server to build everything. This was convenient, because the source tree could stay on the MBP. It however turned out to be slower than doing a local build on the build server. So I now push my changes to the build server, which then does the build alone. It takes about 18 minutes for a clobber build (without ccache), this way.

Anyways, I don’t build often on Windows or Mac, but when I need to, I’m essentially doing a clobber build, and as a result, the MBP spends an awful lot of time doing so. So, since I already had a distcc daemon running, with clang available, and since I was in need for a Mac build, I figured I’d try something seemingly crazy: use the Linux build server.

It turns out Linux clang is able to emit Mach-O objects (the binary format used on MacOSX). So all I needed to do was to add something like the following to my .mozconfig:

export CC="distcc clang -ccc-host-triple x86_64-apple-darwin"
export CXX="distcc clang++ -ccc-host-triple x86_64-apple-darwin"

(After fixing bug 818092)

Now I can enjoy builds taking slightly less than 30 minutes. Which is more than twice as fast.

Update: in case this wasn’t very clear, the build is initiated from OSX, where preprocessing is handled (distcc pump mode is not enabled), which allows the Linux build server to do the right thing with clang.

2012-12-04 19:30:15+0200

p.m.o | 1 Comment »

November 27th, 2012

Hooking the memory allocator in Firefox

Supplanting the system memory allocator usually involves some tricks. In a cross-platform software like Firefox, this involves different tricks on different platforms. Firefox uses such tricks to implant jemalloc. Sadly, this makes replacing jemalloc itself even trickier.

For instance, trace-malloc, our leak detection tool, used on debug builds, requires that jemalloc is disabled.

Work is under way to make supplanting jemalloc much easier. It is not yet clear if this will be enabled by default on release builds, but it would make sense to enable the feature at least on nightlies.

What does the feature provide? A way to hook or replace jemalloc in Firefox at startup time (as opposed to build time, like trace-malloc). The idea is to build a specialized library (more on that further below) and make Firefox use it instead, or on top of jemalloc, with some weak linking tricks. To enable the feature, pass --enable-replace-malloc to configure or add ac_add_options --enable-replace-malloc to your mozconfig (provided you applied the patches or got a tree where the patches are landed).

With the feature built, you can start Firefox with a malloc replacement library easily:

  • On GNU/Linux:

    $ LD_PRELOAD=/path/to/library.so firefox
  • On OSX:
    $ DYLD_INSERT_LIBRARIES=/path/to/library.dylib firefox
  • On Windows:
    $ MOZ_REPLACE_MALLOC_LIB=drive:\path\to\library.dll firefox
  • On Android:

    $ am start -a android.activity.MAIN -n org.mozilla.fennec/.App --es env0 MOZ_REPLACE_MALLOC_LIB=/path/to/library.so

As I happen to have built Firefox with the feature enabled for all platforms on try, to validate that it works, you can toy around with these builds.

A replacement library is expected to provide the following functions, or any subset:

  • void replace_init(const malloc_table_t *table)
  • void *replace_malloc(size_t size)
  • int replace_posix_memalign(void **ptr, size_t alignment, size_t size)
  • void *replace_aligned_alloc(size_t alignment, size_t size)
  • void *replace_calloc(size_t num, size_t size)
  • void *replace_realloc(void *ptr, size_t size)
  • void replace_free(void *ptr)
  • void *replace_memalign(size_t alignment, size_t size)
  • void *replace_valloc(size_t size)
  • size_t replace_malloc_usable_size(usable_ptr_t ptr)
  • size_t replace_malloc_good_size(size_t size)
  • void replace_jemalloc_stats(jemalloc_stats_t *stats)
  • void replace_jemalloc_purge_freed_pages()
  • void replace_jemalloc_free_dirty_pages()

The first function, replace_init is the first function from the library that will be called (if it exists), before the first call to any other. It is passed a pointer to a function table containing pointers to the corresponding jemalloc functions from Firefox.

The last three functions are specific to jemalloc. jemalloc_stats is only important to replace if you want about:memory to still be accurate according to anything you’ve done in other functions, and jemalloc_purge_freed_pages and jemalloc_free_dirty_pages are used to force the allocator to return some unused memory to the system.

The other functions are the usual suspects, picked from C89, POSIX, C11, or OSX (malloc_good_size). They should however all be considered cross-platform (especially malloc_good_size).

All these functions, when they exist, are called instead of the corresponding jemalloc functions, which makes it the responsibility of the replacing functions to call back the corresponding jemalloc function if necessary.
This allows, for example, to:

  • Replace jemalloc entirely. The third patch bug 804303 does that to allow to replace the (currently default) old fork of jemalloc with a fresh jemalloc. Something similar could be done to test other allocators, like tcmalloc.
  • Make memory allocation functions randomly return NULL as in Out of Memory conditions, aka fuzzing.
  • Make all allocations bigger to add tracing data.
  • Log allocations.
  • etc.

A small implementation example

Consider the following question: how many times does realloc end up copying data? Stated differently, how many times does realloc not return the pointer it was given?

Create the memory/replace/realloc/realloc.c file with the following content:

// This header will declare all the replacement functions, such that you don't need
// to worry about exporting them with the right idiom (dllexport, visibility...)
#include "replace_malloc.h"
#include <stdlib.h>
#include <stdio.h>

static const malloc_table_t *funcs = NULL;
static unsigned int total = 0, copies = 0;

void print_stats()
{
  printf("%d reallocs, %d copies\n", total, copies);
}

void replace_init(const malloc_table_t *table)
{
  funcs = table;
  atexit(print_stats);
}

void *replace_realloc(void *ptr, size_t size)
{
  void *newptr = funcs->realloc(ptr, size);
  // Not thread-safe, but it's only an example.
  total++;
  // We don't want to count deallocations as copies.
  if (newptr && newptr != ptr)
    copies++;
  return newptr;
}

Add a memory/replace/realloc/Makefile.in file:

DEPTH           = @DEPTH@
topsrcdir       = @top_srcdir@
srcdir          = @srcdir@
VPATH           = @srcdir@

include $(DEPTH)/config/autoconf.mk

LIBRARY_NAME = replace_realloc
FORCE_SHARED_LIB = 1
NO_DIST_INSTALL = 1

CSRCS = realloc.c

MOZ_GLUE_LDFLAGS = # Don't link against mozglue
WRAP_LDFLAGS = # Never wrap malloc function calls with -Wl,--wrap

include $(topsrcdir)/config/rules.mk

Add the following to memory/replace/Makefile.in:

DIRS += realloc

Finally, build objdir/memory/replace. You’ll get a library in objdir/memory/replace/realloc that you can use as described at the beginning of this post.

On my system, after starting and quitting Firefox without doing much, it prints:

41078 reallocs, 37197 copies

It sure is a simple example, that can actually be fulfilled with other tools (like dtrace), but it’s now up to you, developers, to come up with more useful uses. The blocked bugs already show some. Note this facility still has the advantage of being more cross-platform than tools like dtrace, and to work happily on top of jemalloc (valgrind, for instance, doesn’t support that gracefully), which can be important when looking at some particular aspects of memory allocation. The above example, while simple, is a typical case where the underlying memory allocation library has an impact on the result: other memory allocation libraries have different size classes, which modifies how often realloc will need to actually reallocate, as opposed to grow the existing allocation in-place.

2012-11-27 13:49:15+0200

p.m.o | 7 Comments »

November 18th, 2012

Debian EFI mode boot on a Macbook Pro, without rEFIt

Diego’s post got me to switch from grub-pc to grub-efi to boot Debian on my Macbook Pro. But I wanted to go further: getting rid of rEFIt.

rEFIt is a pretty useful piece of software, but it’s essentially dead. There is the rEFInd fork, which keeps it up-to-date, but it doesn’t really help with FileVault. Moreover, the boot sequence for a Linux distro with rEFIt/rEFInd looks like: Apple EFI firmware → rEFIt/rEFInd → GRUB → Linux kernel. Each intermediate step adding its own timeout, so rEFIt/rEFInd can be seen as not-so-useful intermediate step.

Thankfully, Matthew Garrett did all the research to allow to directly boot GRUB from the Apple EFI firmware. Unfortunately, his blog post didn’t have much actual detail on how to do it.

So here it is, for a Debian system:

  • Install a few packages you’ll need in this process:
    # apt-get install hfsprogs icnsutils
  • Create a small HFS+ partition. I have a 9MB one, but it’s only filled by about 500K, so even smaller should work too. If, like me, you were previously using grub-pc, you probably have a GRUB partition, you can repurpose it. In gdisk, it looks like this:
    Number  Start (sector)    End (sector)  Size       Code  Name
       5       235284480       235302943   9.0 MiB     AF00  Apple HFS/HFS+
    
    Partition GUID code: 48465300-0000-11AA-AA11-00306543ECAC (Apple HFS/HFS+)
    Partition unique GUID: AD1F5465-B777-4178-AC4D-1DE8B2EB1B4B
    First sector: 235284480 (at 112.2 GiB)
    Last sector: 235302943 (at 112.2 GiB)
    Partition size: 18464 sectors (9.0 MiB)
    Attribute flags: 0000000000000000
    Partition name: 'Apple HFS/HFS+'
    
  • Create a HFS+ filesystem on that partition:

    # mkfs.hfsplus /dev/sda5 -v Debian

    (replace /dev/sda5 with whatever your partition is)

  • Add a fstab entry for that filesystem:
    # echo $(blkid -o export -s UUID /dev/sda5) /boot/efi auto defaults 0 0 >> /etc/fstab
  • Mount the filesystem:
    # mkdir /boot/efi
    # mount /boot/efi
    
  • Edit /usr/sbin/grub-install, look for « xfat », and remove the block of code that looks like:
    if test "x$efi_fs" = xfat; then :; else
        echo "${efidir} doesn't look like an EFI partition." 1>&2
        efidir=
    fi
    
  • Run grub-install. At this point, there should be a /boot/efi/EFI/debian/grubx64.efi file (if using grub-efi-amd64).
  • Create a /boot/efi/System/Library/CoreServices directory:
    # mkdir -p /boot/efi/System/Library/CoreServices
  • Create a hard link:
    # ln /boot/efi/EFI/debian/grubx64.efi /boot/efi/System/Library/CoreServices/boot.efi
  • Create a dummy mach_kernel file:
    # echo "This file is required for booting" > /boot/efi/mach_kernel
  • Grab the mactel-boot source code, unpack and build it:
    # wget http://www.codon.org.uk/~mjg59/mactel-boot/mactel-boot-0.9.tar.bz2
    # tar -jxf mactel-boot-0.9.tar.bz2
    # cd mactel-boot-0.9
    # make PRODUCTVERSION=Debian
    
  • Copy the SystemVersion.plist file:
    # cp SystemVersion.plist /boot/efi/System/Library/CoreServices/
  • Bless the boot file:
    # ./hfs-bless /boot/efi/System/Library/CoreServices/boot.efi
  • (optional) Add an icon:
    # rsvg-convert -w 128 -h 128 -o /tmp/debian.png /usr/share/reportbug/debian-swirl.svg
    # png2icns /boot/efi/.VolumeIcon.icns /tmp/debian.png
    # rm /tmp/debian.png
    

Now, the Apple Boot Manager, shown when holding down the option key when booting the Macbook Pro, looks like this:

And the Startup disk preferences dialog under OSX, like this:

2012-11-18 11:18:14+0200

debian, p.m.o | 29 Comments »

November 5th, 2012

Fun with weak dynamic linking

Dynamic linkers, at least in the UNIX world, usually allow to load libraries in a process address space at startup. On Linux systems, you load such a library with LD_PRELOAD. On OSX, with DYLD_INSERT_LIBRARIES.

On Linux systems, when using LD_PRELOAD, the dynamic linker will also use symbols from the (pre)loaded library instead of system libraries. For example, if a program calls the write function and a library exporting a write symbol is loaded with LD_PRELOAD, the write function from the loaded library will be used instead of that of the libc (even when the symbol version doesn’t match).

On OSX, symbol resolution is usually done with a “two-level namespace”: symbols are associated with library names, and when resolving symbols, both are used. More than that, the library name is used to find the library in the dyld search path. Libraries loaded with DYLD_INSERT_LIBRARIES won’t be used with two-level namespace symbol resolution, which makes it less useful than LD_PRELOAD. Fortunately, it is also possible to use a “flat” namespace, in which case only the symbol name is considered during symbol resolution, and is searched in all loaded libraries, in the order in which they were loaded. Flat namespace can be triggered by setting the DYLD_FORCE_FLAT_NAMESPACE environment variable, linking the main program with -force_flat_namespace, or linking programs and libraries with the -flat_namespace argument. Note the latter only affects the programs and libraries built with that argument, while the former two force to use a flat namespace for all libraries, including those which, like system ones, were built with a two-level namespace. There are also cases where a single symbol may be resolved with the flat namespace, while others in the program or library are using two-level namespace.

Weak dynamic linking is another feature that can be used to tell the dynamic linker to ignore missing symbols. Consider the following source code:

extern void foo() __attribute__((weak)); // weak_import is preferred on OSX.
int main() {
  if (foo)
    foo();
  return 0;
}

On Linux systems, this just works. Compile the program (you’ll need to build it with -fPIC, though), start it, and it will do nothing, since foo is defined nowhere.

Combined with shared library (pre)loading, this can be used to provide simple hooks in your application. For example, with the following source code built as a shared library:

#include <stdio.h>
void foo() {
  printf("foo\n");
}

Running the program again with LD_PRELOAD set to load that shared library (note LD_PRELOAD=foo.so won’t work, a path is needed, like LD_PRELOAD=./foo.so), it will print foo because the dynamic linker will have resolved the foo symbol to that of the shared library.

On OSX, unfortunately, things are not as easy. First, building the test program above fails:

Undefined symbols for architecture x86_64:
  "_foo", referenced from:
      _main in test-LIeVtB.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

There are linker options to force undefined symbols to be resolved at runtime:

  • -undefined dynamic_lookup, which will mark all undefined symbols as having to be looked up at runtime,
  • -Wl,-U,symbol_name, which only does so for the given symbol (note: you have to prepend an underscore to the symbol name)

With one of these options, the program works as on Linux: it does nothing when run alone, and prints foo when loading the library with DYLD_INSERT_LIBRARIES… if you build with XCode 4.5. Running the program built with XCode 4.3 fails when not loading the shared library, with the following error:

dyld: Symbol not found: _foo
  Referenced from: ./test
  Expected in: flat namespace
 in ./test
Trace/BPT trap: 5

And if at build time, you target OSX 10.5 (with MACOSX_DEPLOYMENT_TARGET or -mmacosx-version-min), the error is slightly different:

dyld: Symbol not found: _foo
  Referenced from: ./test
  Expected in: dynamic lookup

Trace/BPT trap: 5

Each error is due to a different bug:

  • Since OSX 10.6, the link edition rules in the __LINKEDIT segment are in a new, compressed, format: DYLD_INFO. The linker in Xcode < 4.5 forgets to flag weak imports as weak in the DYLD_INFO data. Compare the output for dyldinfo with a binary built with Xcode 4.5 vs. the output for a binary built with Xcode 4.3:
    $ dyldinfo -bind test-xcode4.5 | sed -n '2p;/foo/p'
    segment section          address        type    addend dylib            symbol
    __DATA  __got            0x100001038    pointer      0 flat-namespace   _foo (weak import)
    
    $ dyldinfo -bind test-xcode4.3 | sed -n '2p;/foo/p'
    segment section          address        type    addend dylib            symbol
    __DATA  __got            0x100001038    pointer      0 flat-namespace   _foo
    

    Notice the missing weak import. Manually setting the flag with a hexadecimal editor fixes it (in both bind and lazy_bind tables).

  • In older OSX releases, the link edition rules use a “classic” format. In that format, the symbol table is used for flags such as N_WEAK_REF (weak reference). In the new DYLD_INFO format, the N_WEAK_REF flag isn’t used, which partly explains the previous bug. When using the old format and two-level namespaces, missing weak symbols are errors. With flat namespace, they aren’t. This means the error we get with a program built for a 10.5 target goes away if building with -flat_namespace. Note this doesn’t work at the symbol level: if the library is built for two-level namespace (and is marked as such in the Mach-O header), and the weak symbol is without a corresponding library name, making it resolved with a flat namespace, it doesn’t work.

At this point, one could think weak dynamic linking is pretty much useless on OSX, at least, that it was before Xcode 4.5 was released. As it turns out, there are other use cases where it actually works. Consider the following code:

#include <malloc/malloc.h>
// The following is defined in malloc/malloc.h:
// extern size_t malloc_zone_pressure_relief(malloc_zone_t *zone, size_t goal) __attribute__((weak_import));
int main() {
  if (malloc_zone_pressure_relief)
    malloc_zone_pressure_relief(NULL, 0);
  return 0;
}

This program in itself is useless, but the point is, the malloc_zone_pressure_relief function is only available since OSX 10.7. Without the weak_import, the program would fail to start with an undefined symbol on OSX 10.6 and below. Without the if, it would crash because the symbol would resolve to NULL, and the program would jump there. But the program itself has to be built on OSX 10.7 at least, for the malloc_zone_pressure_relief function to be there when building.

And in that use-case, we end up with the right flags in DYLD_INFO:

$ dyldinfo -bind test | sed -n '2p;/malloc/p'
segment section          address        type    addend dylib            symbol
__DATA  __got            0x100001038    pointer      0 libSystem        _malloc_zone_pressure_relief (weak import)

This actually gives us a hint for a way out of our misery for our foo function on Xcode < 4.5: linking against a dummy library implementing the weak symbol. When doing so, we get the proper flag in DYLD_INFO, like with malloc_zone_pressure_relief:

$ dyldinfo -bind test | sed -n '2p;/foo/p'
segment section          address        type    addend dylib            symbol
__DATA  __got            0x100001038    pointer      0 libfoo           _foo (weak import)

And the linker additionally does something nice: when all symbols needed from a library are weak references, it marks the library import itself as weak:

$ otool -l test | grep -B 2 libfoo
          cmd LC_LOAD_WEAK_DYLIB
      cmdsize 40
         name libfoo.dylib (offset 24)

What this means is that even if the libfoo.dylib is missing, it will still work. The downside is that we are now using two-level namespace, which, as mentioned above, makes DYLD_INSERT_LIBRARIES useless. The program thus needs to be built with a flat namespace.

Update: It turns out some versions of XCode don’t conveniently mark libraries with only weak imports as weakly linked, so the -Wl,-weak_library,libraryname.dylib flag is required instead of -Llibraryname.

In summary, if you want to add optional hooks that a library loaded through LD_PRELOAD/ DYLD_INSERT_LIBRARIES can implement:

  • on OSX < 10.6, use weak symbols and build with -flat_namespace,
  • on OSX >= 10.6, when building with Xcode < 4.5, use weak symbols, build with -flat_namespace and link against a dummy library implementing the hook functions with -Wl,-weak_library,libraryname.dylib,
  • on Linux systems and on OSX >= 10.6, when building with Xcode 4.5, simply use weak symbols,
  • on Windows, to the best of my knowledge, weak dynamic linking is not supported.

Stay tuned for the next post, which will describe what this will be used for in Firefox.

2012-11-05 21:49:26+0200

p.m.o | 8 Comments »

August 23rd, 2012

Android Debug Bridge for Firefox

I uploaded an experimental add-on on AMO that adds some kind of support for the Android Debug Bridge to Firefox. It requires the adb/adb.exe executable from the Android SDK. Disclaimer: the add-on is experimental and is known to leak small amounts of memory and have various problems handling errors. It’s also on github, if you want to give a hand.

The add-on provides three features.

The first one is access to Android and Firefox OS devices filesystem through the adb:// protocol. Without the serial number for any device, a list of connected devices is shown, like in the screenshot above. With a device serial number in the url (adb://serial/), it shows filesystem contents, as below:

Please note the add-on doesn’t work well with symbolic links at the moment.

Another feature is the ability to grab the framebuffer on a device and display it as a PNG image. This feature is available through adbview:// urls, and like adb://, a list of connected devices will be shown if no device serial is given.

It can show a screen capture of Android devices:

As well as Firefox OS devices:

For Firefox OS devices, it requires a fork of the screencap tool. It should be included in recent Firefox OS builds.

Finally, the add-on improves the Remote Debugging feature included in Firefox. As described on Mark Finkle’s blog, you would normally type an adb command to forward a tcp port first. When you have several devices, that requires forwarding different local ports to port 6000 on each device, and remembering which is which.

The addon allows to skip that step, and to connect through the Android Debug Bridge directly, without local port forwarding.

Check out Mark’s post for instructions to enable Remote Debugging on both the device and desktop.

The prompt shown when opening the Remote Debugger is augmented with a list of devices. local is the desktop machine running Firefox. The other items in the list are the connected Android or Firefox OS devices. This relies on monkey-patching Firefox Remote Debugger code, so this is not completely reliable, currently only works with a recent Firefox 17 nightly, and may break at any time in the future.

When you select a device, you shouldn’t need to change the host:port field, but you could theoretically change it to access even more remote devices through the device’s network.

Once you selected a device in the Remote Debugger prompt, you get the usual prompt on the device itself:

(By the way, the screenshot above was taken with adbview://)

That brings up scripts running in the Firefox instance on the device:

Thanks to the mobile team for having brought Remote Debugging.

2012-08-23 20:35:51+0200

p.m.o | 3 Comments »