{"id":2436,"date":"2012-03-01T17:10:11","date_gmt":"2012-03-01T16:10:11","guid":{"rendered":"http:\/\/glandium.org\/blog\/?p=2436"},"modified":"2012-03-03T13:55:27","modified_gmt":"2012-03-03T12:55:27","slug":"introducing-faulty-lib","status":"publish","type":"post","link":"https:\/\/glandium.org\/blog\/?p=2436","title":{"rendered":"Introducing faulty.lib"},"content":{"rendered":"

TL;DR link: faulty.lib is on github<\/a>.<\/p>\n

Android applications are essentially zip archives, with the APK extension instead of ZIP. While most of Android is java, and java classes are usually loaded from a ZIP archive (usually with the JAR extension), Android applications using native code need to have native libraries on the file system. These native libraries are found under \/data\/data\/$appid\/lib<\/code>, where $appid<\/code> is the package name, as defined in the AndroidManifest.xml<\/code> file.<\/p>\n

So, when Android installs an application, it puts that APK file under \/data\/app<\/code>. Then, if the APK contains native libraries under a lib\/$ABI<\/code> subdirectory (where $ABI<\/code> is armeabi, armeabi-v7a or x86), it also decompresses the files and places them under \/data\/data\/$appid\/lib<\/code>. This means native libraries are actually stored twice on internal flash: once compressed and once decompressed.<\/p>\n

This is why Chrome for Android<\/a> takes almost 50MB of internal flash space after installation.<\/p>\n

Firefox for Android used to have that problem, and we decided we should stop doing that. Michael Wu thus implemented a custom dynamic linker<\/a>, which would load most of Firefox libraries directly off the APK. This involves decompressing the zipped data somewhere in memory, and doing ld.so<\/code>'s job to make the library usable (please note that on Android, ld.so<\/code> is actually named linker<\/code>). There were initially circumstances under which we would decompress into a file and reuse it the next time Firefox starts, but we subsequently removed that possibility (except for debugging purpose) because it ended up being slower than decompressing each time (thanks to internal flash being so slow).<\/p>\n

Anyways, in order to do ld.so<\/code>'s job, our custom linker was directly derived from Android's system linker, with many tweaks. This custom linker has done its job quite well for some time, now, but has been recently replaced, see further below.<\/p>\n

Considering Firefox can't do anything useful involving Gecko until its libraries are loaded, in practice, this means Firefox can't display a web page faster than completely decompressing the libraries. Or can it?<\/p>\n

Just don\u00e2\u20ac\u2122t sit down 'cause I\u00e2\u20ac\u2122ve moved your chair<\/h2>\n

We know that a lot of code and data is not used during Firefox startup<\/a>. Based on that knowledge, we started working on only loading the necessary bits. The core of the idea is, when a library is requested to be loaded, to reserve anonymous memory for its decompressed size, and... that's all. That memory is protected such that any access to it triggers a segmentation fault. When a segmentation fault happens, the required bits are decompressed, and execution is resumed where it was before the segmentation fault.<\/p>\n

The original prototype was decompressing from a normal zip deflated stream, which means it was impossible to seek in it. So, if an access was made at the end of the library, it was necessary to decompress the whole library. With some nasty binary reordering, and some difficulty<\/a>, it was possible to avoid accessing the end of the library, but the approach is very much fragile. It only takes an unexpected code path to make things much slower than they should be.<\/p>\n

Consequently, for the past months, I've been working on improving the original idea and, with some assistance from Julian Seward, implemented the scheme with seekable compressed streams. Instead of letting the zip archive creation tool deflate libraries, we store specially crafted files. Essentially, files are cut in small chunks, and each chunk is compressed individually. This means a less efficient compression, but it also means random access to chunks is possible.<\/p>\n

However, instead of stacking on top of our existing custom linker, I started over, from the ground up. First, because it needed a serious clean up (a good part of linker.c<\/code> is leftovers from the Android linker that we don't use, and APKOpen.cpp<\/code> is a messy mix of JNI stubs, library decompression handling (which in itself was also a mess) and Gecko initialization code) and most importantly, because it relied on some Android system linker internals and thus required binary compatibility with the system linker. Which, according to Google engineers that contacted us a few months ago, was going to break in what we now know will be called Android Jelly Bean.<\/p>\n

The benefit of the clean slate approach is that the new code is not tied to Gecko at all and was designed to work on Android as well as on desktop Linux systems (which made debugging much much easier). We're thus releasing the code as a separate project: faulty.lib<\/b><\/a>. It is licensed under the Mozilla Public License version 2.0<\/a>. Please feel free to try, contribute, and\/or fork it.<\/p>\n

This dynamic linker is not meant to completely follow standard ELF rules (most notably for symbol resolution), and as a result does some assumptions. It's also still work in progress, with some obvious optimizations pending (like, avoiding to resolve the same symbols again and again during relocations), and some features missing (for example, symbol versioning).<\/p>\n

The next blog post will give some information about how to build Firefox for Android to benefit from on-demand decompression. I will also detail a few of the tricks involved in this dynamic linker in subsequent blog posts.<\/p>\n","protected":false},"excerpt":{"rendered":"

TL;DR link: faulty.lib is on github. Android applications are essentially zip archives, with the APK extension instead of ZIP. While most of Android is java, and java classes are usually loaded from a ZIP archive (usually with the JAR extension), Android applications using native code need to have native libraries on the file system. These […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[28,5,25],"tags":[23],"class_list":["post-2436","post","type-post","status-publish","format-standard","hentry","category-faulty-lib","category-pdo","category-planet-mozilla","tag-en"],"_links":{"self":[{"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/2436","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2436"}],"version-history":[{"count":29,"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/2436\/revisions"}],"predecessor-version":[{"id":2508,"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/2436\/revisions\/2508"}],"wp:attachment":[{"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2436"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2436"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2436"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}