# November 23rd, 2013

## Don’t trust python’s os.execv

Python is nice and all, but its low-level functions have real disruptive discrepancies between platforms.

Case at point:

import os
os.execvp("sh", ["sh", "-c", "exit 1"])


As a UNIXy person, I’d expect running the above script to return an error code of 1. And I would be perfectly right… on UNIX systems.

On Windows, it returns 0.

You’d think such a difference in behavior would be documented? It’s not.

Thank you python.

2013-11-23 01:24:26+0100

# August 8th, 2013

## Faster builds for Firefox front-end developers with make

There’s a trend of blog posts about faster builds, so here’s another one.

There is a trick that allows to skip most compilation from the tree, but it only works if the only changes you make are under browser/ (which should be the case for a lot of front-end development).

• Get a xulrunner SDK nightly for your platform.
• Unpack it with tar or unzip depending on the file you got.
• Add the following to your mozconfig:
ac_add_options --with-libxul-sdk=/path/to/the/unpacked/xulrunner-sdk
• Build with your favorite command.
• Enjoy finishing your clobber build under 30 seconds (20 seconds here, including a 10 seconds configure).

2013-08-08 09:54:32+0100

# May 30th, 2013

## 日本へ引っ越し

Today, May the 30th, was my last day as a Mozilla employee. In a couple weeks, my wife, my cat and I will be on board of a flight heading about ten thousand kilometers east, and most of our stuff will be in some container on a boat. We’re moving to Japan. As adventurous as this may sound, I’m not venturing into unknown territory. My wife is Japanese, and I’ve lived there for close to 15 months. A long time ago, arguably.

I’m not actually leaving Mozilla. I’ll be back as a contractor, hopefully around the 25th of June. So as far as my fellow coworkers are concerned, I’ll be going on a long-ish vacation and changing timezone (but I’ll probably be around in the meanwhile on irc or bugmail, with high latency).

Jump-starting in a different country is not something really easy to pull off, and working for Mozilla as a remotee has been a key element in being able to do so. Although I’ve made it clear when I joined Mozilla that this would eventually happen, I’m thankful I can now actually do it.

2013-05-30 19:52:08+0100

# May 27th, 2013

## signal() doubly considered harmful

When you want to set signal handlers on UNIX systems, the typical choice is to use signal (specified in C89, C99 and POSIX.1-2001) or sigaction (specified in POSIX.1-2001 and System V r4).

Quoting the signal manual page:

The only portable use of signal() is to set a signal’s disposition to SIG_DFL or SIG_IGN. The semantics when using signal() to establish a signal handler vary across systems (and POSIX.1 explicitly permits this variation); do not use it for this purpose.

POSIX.1 solved the portability mess by specifying sigaction(2), which provides explicit control of the semantics when a signal handler is invoked; use that interface instead of signal().

Then it goes on about the UNIX vs BSD semantics, and how they affect signal delivery, which essentially is the main reason why one would want to stop using signal and use sigaction instead, with specifically chosen flags.

But this is not really what I wanted to talk about here.

One of the uses of signal or sigaction is to temporarily set a signal handler and restore the old signal handler once the job is done. Notwithstanding the fact that it’s a pretty horrible thing to do in a multi-threaded program, it’s also a horrible thing to do at all with signal if sigaction is used.

The core of the problem is the following: the information you get from signal() about the old signal handler is missing all the important pieces about it if it was originally set with sigaction(), namely, flags, masks and restorer.

So if you do use signal() to temporarily set a signal handler and then restore the previous signal handler, you risk resetting flags, masks and restorer. The first awful thing this means is the previous signal handler might be expecting three arguments, only one of which will be valid when it’s invoked. Unexpected things can also happen with the lack of expected flags or masks. This is why you’ll see horrible workarounds like this or that.

In short, if you do use signal() to temporarily set a signal handler and then restore the previous signal handler, you’re doing it wrong. And if you do that in a system library or driver, thank you for screwing things up. I’m looking at you libsc-a3xx.so.

2013-05-27 17:15:13+0100

# March 14th, 2013

## Google Reader death, or how the cloud model can fail you

If you’re a Google Reader user, you probably read in one of your subscriptions that Google is pulling the plug on Google Reader. It is yet another demonstration of why putting data in the cloud isn’t so much of a nice idea: the service you rely on may well disappear some day, with all the data it contains.

Sure Google, in its extreme goodness, allows you to “take out” the Google Reader data. Or does it?

• followers.json, following.json: both files contain similar data, that I suspect correspond to Buzz subscriptions (yet another dead service). Each friends item contains some information about your “friend”, and a stream identifier for their activity (I guess), as well as a few websites urls. For instance Tim Bray’s stream is “user/05198174665841271019/state/com.google/broadcast“. What the hell do I do with that? Fortunately, he has websites, but not all my “friends” have. Thankfully, I haven’t really been using this feature, so there’s almost nothing in these files.
• liked.json, starred.json, shared.json, shared-by-followers.json: all have the same structure, and contain items you liked, starred, shared, or that the people you follow shared (yeah, that file is badly named). Each item contains an url (or so I hope), and the corresponding content (yay). shared-by-followers.json however doesn’t contain more than the items the people you follow actively shared: it doesn’t contain their feeds (and I’m pretty sure I read more from Tim Bray than the two links he shared)
• subscriptions.xml: Essentially, a list of RSS feed urls, with no content ; nothing from Tim Bray here, but now that I think about it, I think I was only following his Buzz feed, so that went away with Buzz without me noticing.

Interestingly, while looking into shared-by-followers.json, I found urls that would correspond to friend streams. For instance, Tim Bray’s is http://www.google.com/reader/public/atom/user/05198174665841271019/state/com.google/broadcast. But it’s useless: all it displays is “permission denied”.

As for subscriptions, one of the strengths of Google Reader is that it allowed to search though past items, which means a big part of the interesting data is the archived items. But that’s not part of the “take out”. Sure, you have the feed urls, but most RSS feeds contain a limited amount of items, not the entire history of items for the given feed. So, history is more or less lost. Except if I star, like or share all items in all my subscriptions and “take out” again.

So much goodness.

It could have been worse, though.

2013-03-14 08:35:45+0100

# February 19th, 2013

## Ten years

Ten years ago, this very day, my first Debian package entered the Debian unstable repository. It was an addon for Mozilla Composer, Daniel Glazman’s Cascades.

On the same day, my second Debian package entered the Debian unstable repository as well. It was an addon for Mozilla Browser, Checky.

A few days later, my third Debian package entered Debian unstable. It was an addon for Mozilla Browser, Diggler.

Do you see a pattern? They are now abandoned software, although I made Checky and Diggler live a little longer (and I’m actually considering reviving Diggler) but they had their importance in my journey, and are part of the reason why I am where I am now.

My journey on the web started with NCSA Mosaic on VAX/VMS, then continued with Netscape Navigator, Netscape Communicator and Mozilla Suite on Linux.

That’s where I was ten years ago, sailing between Galeon (a browser using the Mozilla engine) and Mozilla Suite, and filing some layout bugs.

Ten years ago, there was a new kid on the block. It used to be called Phoenix, it had just changed its name to Firebird. Eventually, it changed again for Firefox. You may have heard about it. Because Firebird was so much nicer than the browser in the Mozilla Suite, I started using its Debian package, and wanted my packaged addons to work with it. So I contacted Eric Dorland, Phoenix/Firebird package maintainer at the time, and got the addons working. I then ended up fixing a bunch of packaging issues.

This is how I got involved in Firefox packaging for Debian, and what eventually led me to work for Mozilla.

2013-02-19 22:45:30+0100

# December 29th, 2012

## Firefox in Debian?

Got your attention? Don’t hold your breath, we’re not there yet, but we’re a step closer: it’s now possible to build Firefox from the Iceweasel package, since version 17.0.1-2 in experimental as of writing, 18.0~b6-1 from the iceweasel-beta repository, or 19.0~a2+20121228042015-1 from the iceweasel-aurora repository.

Before letting you know how you can get yourself a packaged Firefox based on the Iceweasel source, I’ll remind you that redistribution of Firefox packages requires a trademark license from Mozilla, so please keep the packages you build for yourself for now.

That being said, now it’s clear that such Firefox packages are not official, you can still test them for yourself. First download the Iceweasel source version of your liking, and extract it, then rename all source files from iceweasel_* to firefox_* (rename s/iceweasel/firefox/ iceweasel_* should do it). Edit debian/changelog so that the first line reads:

firefox (x.y.z-r) distribution; urgency=low

iceweasel (x.y.z-r) distribution; urgency=low

and run the following command:

$debian/rules debian/control Now you’re all set. You can build the package the usual way. Note there are a few differences between the xulrunner packages you get from building Iceweasel vs. from building Firefox that need to be addressed, and a few other details to sort out. 2012-12-29 11:00:21+0100 # December 4th, 2012 ## Using distcc and clang on Linux to build Firefox faster on MacOSX Apple makes nice hardware, until you look into the details. As it turns out, my Macbook Pro has issues with heat. So much that when building Firefox, the CPU decides it’s too hot and throttles its frequency to emit less heat. The result is that building Firefox takes much more time than it would if the CPU could stay at its highest frequency all the time. On this machine, it takes more than an hour to build Firefox. Most of the time, this MBP runs Linux. The heat problem is the same, but I also have a much beefier build server, running Linux too, that I use for builds. I used to use distcc from the MBP, using both the MBP and the build server to build everything. This was convenient, because the source tree could stay on the MBP. It however turned out to be slower than doing a local build on the build server. So I now push my changes to the build server, which then does the build alone. It takes about 18 minutes for a clobber build (without ccache), this way. Anyways, I don’t build often on Windows or Mac, but when I need to, I’m essentially doing a clobber build, and as a result, the MBP spends an awful lot of time doing so. So, since I already had a distcc daemon running, with clang available, and since I was in need for a Mac build, I figured I’d try something seemingly crazy: use the Linux build server. It turns out Linux clang is able to emit Mach-O objects (the binary format used on MacOSX). So all I needed to do was to add something like the following to my .mozconfig: export CC="distcc clang -ccc-host-triple x86_64-apple-darwin" export CXX="distcc clang++ -ccc-host-triple x86_64-apple-darwin"  (After fixing bug 818092) Now I can enjoy builds taking slightly less than 30 minutes. Which is more than twice as fast. Update: in case this wasn’t very clear, the build is initiated from OSX, where preprocessing is handled (distcc pump mode is not enabled), which allows the Linux build server to do the right thing with clang. 2012-12-04 19:30:15+0100 # November 27th, 2012 ## Hooking the memory allocator in Firefox Supplanting the system memory allocator usually involves some tricks. In a cross-platform software like Firefox, this involves different tricks on different platforms. Firefox uses such tricks to implant jemalloc. Sadly, this makes replacing jemalloc itself even trickier. For instance, trace-malloc, our leak detection tool, used on debug builds, requires that jemalloc is disabled. Work is under way to make supplanting jemalloc much easier. It is not yet clear if this will be enabled by default on release builds, but it would make sense to enable the feature at least on nightlies. What does the feature provide? A way to hook or replace jemalloc in Firefox at startup time (as opposed to build time, like trace-malloc). The idea is to build a specialized library (more on that further below) and make Firefox use it instead, or on top of jemalloc, with some weak linking tricks. To enable the feature, pass --enable-replace-malloc to configure or add ac_add_options --enable-replace-malloc to your mozconfig (provided you applied the patches or got a tree where the patches are landed). With the feature built, you can start Firefox with a malloc replacement library easily: • On GNU/Linux: $ LD_PRELOAD=/path/to/library.so firefox
• On OSX:
$DYLD_INSERT_LIBRARIES=/path/to/library.dylib firefox • On Windows: $ MOZ_REPLACE_MALLOC_LIB=drive:\path\to\library.dll firefox
• On Android:

am start -a android.activity.MAIN -n org.mozilla.fennec/.App --es env0 MOZ_REPLACE_MALLOC_LIB=/path/to/library.so As I happen to have built Firefox with the feature enabled for all platforms on try, to validate that it works, you can toy around with these builds. A replacement library is expected to provide the following functions, or any subset: • void replace_init(const malloc_table_t *table) • void *replace_malloc(size_t size) • int replace_posix_memalign(void **ptr, size_t alignment, size_t size) • void *replace_aligned_alloc(size_t alignment, size_t size) • void *replace_calloc(size_t num, size_t size) • void *replace_realloc(void *ptr, size_t size) • void replace_free(void *ptr) • void *replace_memalign(size_t alignment, size_t size) • void *replace_valloc(size_t size) • size_t replace_malloc_usable_size(usable_ptr_t ptr) • size_t replace_malloc_good_size(size_t size) • void replace_jemalloc_stats(jemalloc_stats_t *stats) • void replace_jemalloc_purge_freed_pages() • void replace_jemalloc_free_dirty_pages() The first function, replace_init is the first function from the library that will be called (if it exists), before the first call to any other. It is passed a pointer to a function table containing pointers to the corresponding jemalloc functions from Firefox. The last three functions are specific to jemalloc. jemalloc_stats is only important to replace if you want about:memory to still be accurate according to anything you’ve done in other functions, and jemalloc_purge_freed_pages and jemalloc_free_dirty_pages are used to force the allocator to return some unused memory to the system. The other functions are the usual suspects, picked from C89, POSIX, C11, or OSX (malloc_good_size). They should however all be considered cross-platform (especially malloc_good_size). All these functions, when they exist, are called instead of the corresponding jemalloc functions, which makes it the responsibility of the replacing functions to call back the corresponding jemalloc function if necessary. This allows, for example, to: • Replace jemalloc entirely. The third patch bug 804303 does that to allow to replace the (currently default) old fork of jemalloc with a fresh jemalloc. Something similar could be done to test other allocators, like tcmalloc. • Make memory allocation functions randomly return NULL as in Out of Memory conditions, aka fuzzing. • Make all allocations bigger to add tracing data. • Log allocations. • etc. ## A small implementation example Consider the following question: how many times does realloc end up copying data? Stated differently, how many times does realloc not return the pointer it was given? Create the memory/replace/realloc/realloc.c file with the following content: // This header will declare all the replacement functions, such that you don't need // to worry about exporting them with the right idiom (dllexport, visibility...) #include "replace_malloc.h" #include <stdlib.h> #include <stdio.h> static const malloc_table_t *funcs = NULL; static unsigned int total = 0, copies = 0; void print_stats() { printf("%d reallocs, %d copies\n", total, copies); } void replace_init(const malloc_table_t *table) { funcs = table; atexit(print_stats); } void *replace_realloc(void *ptr, size_t size) { void *newptr = funcs->realloc(ptr, size); // Not thread-safe, but it's only an example. total++; // We don't want to count deallocations as copies. if (newptr && newptr != ptr) copies++; return newptr; }  Add a memory/replace/realloc/Makefile.in file: DEPTH = @DEPTH@ topsrcdir = @top_srcdir@ srcdir = @srcdir@ VPATH = @srcdir@ include(DEPTH)/config/autoconf.mk

LIBRARY_NAME = replace_realloc
FORCE_SHARED_LIB = 1
NO_DIST_INSTALL = 1

CSRCS = realloc.c

MOZ_GLUE_LDFLAGS = # Don't link against mozglue
WRAP_LDFLAGS = # Never wrap malloc function calls with -Wl,--wrap

include $(topsrcdir)/config/rules.mk  Add the following to memory/replace/Makefile.in: DIRS += realloc Finally, build objdir/memory/replace. You’ll get a library in objdir/memory/replace/realloc that you can use as described at the beginning of this post. On my system, after starting and quitting Firefox without doing much, it prints: 41078 reallocs, 37197 copies It sure is a simple example, that can actually be fulfilled with other tools (like dtrace), but it’s now up to you, developers, to come up with more useful uses. The blocked bugs already show some. Note this facility still has the advantage of being more cross-platform than tools like dtrace, and to work happily on top of jemalloc (valgrind, for instance, doesn’t support that gracefully), which can be important when looking at some particular aspects of memory allocation. The above example, while simple, is a typical case where the underlying memory allocation library has an impact on the result: other memory allocation libraries have different size classes, which modifies how often realloc will need to actually reallocate, as opposed to grow the existing allocation in-place. 2012-11-27 13:49:15+0100 # November 18th, 2012 ## Debian EFI mode boot on a Macbook Pro, without rEFIt Diego’s post got me to switch from grub-pc to grub-efi to boot Debian on my Macbook Pro. But I wanted to go further: getting rid of rEFIt. rEFIt is a pretty useful piece of software, but it’s essentially dead. There is the rEFInd fork, which keeps it up-to-date, but it doesn’t really help with FileVault. Moreover, the boot sequence for a Linux distro with rEFIt/rEFInd looks like: Apple EFI firmware → rEFIt/rEFInd → GRUB → Linux kernel. Each intermediate step adding its own timeout, so rEFIt/rEFInd can be seen as not-so-useful intermediate step. Thankfully, Matthew Garrett did all the research to allow to directly boot GRUB from the Apple EFI firmware. Unfortunately, his blog post didn’t have much actual detail on how to do it. So here it is, for a Debian system: • Install a few packages you’ll need in this process: # apt-get install hfsprogs icnsutils • Create a small HFS+ partition. I have a 9MB one, but it’s only filled by about 500K, so even smaller should work too. If, like me, you were previously using grub-pc, you probably have a GRUB partition, you can repurpose it. In gdisk, it looks like this: Number Start (sector) End (sector) Size Code Name 5 235284480 235302943 9.0 MiB AF00 Apple HFS/HFS+ Partition GUID code: 48465300-0000-11AA-AA11-00306543ECAC (Apple HFS/HFS+) Partition unique GUID: AD1F5465-B777-4178-AC4D-1DE8B2EB1B4B First sector: 235284480 (at 112.2 GiB) Last sector: 235302943 (at 112.2 GiB) Partition size: 18464 sectors (9.0 MiB) Attribute flags: 0000000000000000 Partition name: 'Apple HFS/HFS+'  • Create a HFS+ filesystem on that partition: # mkfs.hfsplus /dev/sda5 -v Debian (replace /dev/sda5 with whatever your partition is) • Add a fstab entry for that filesystem: # echo$(blkid -o export -s UUID /dev/sda5) /boot/efi auto defaults 0 0 >> /etc/fstab
• Mount the filesystem:
# mkdir /boot/efi
# mount /boot/efi

• Edit /usr/sbin/grub-install, look for « xfat », and remove the block of code that looks like:
if test "x$efi_fs" = xfat; then :; else echo "${efidir} doesn't look like an EFI partition." 1>&2
efidir=
fi

• Run grub-install. At this point, there should be a /boot/efi/EFI/debian/grubx64.efi file (if using grub-efi-amd64).
• Create a /boot/efi/System/Library/CoreServices directory:
# mkdir -p /boot/efi/System/Library/CoreServices
# ln /boot/efi/EFI/debian/grubx64.efi /boot/efi/System/Library/CoreServices/boot.efi
• Create a dummy mach_kernel file:
# echo "This file is required for booting" > /boot/efi/mach_kernel
• Grab the mactel-boot source code, unpack and build it:
# wget http://www.codon.org.uk/~mjg59/mactel-boot/mactel-boot-0.9.tar.bz2
# tar -jxf mactel-boot-0.9.tar.bz2
# cd mactel-boot-0.9
# make PRODUCTVERSION=Debian

• Copy the SystemVersion.plist file:
# cp SystemVersion.plist /boot/efi/System/Library/CoreServices/
• Bless the boot file:
# ./hfs-bless /boot/efi/System/Library/CoreServices/boot.efi
# rsvg-convert -w 128 -h 128 -o /tmp/debian.png /usr/share/reportbug/debian-swirl.svg