{"id":2764,"date":"2012-11-05T21:49:26","date_gmt":"2012-11-05T20:49:26","guid":{"rendered":"http:\/\/glandium.org\/blog\/?p=2764"},"modified":"2019-09-03T15:30:34","modified_gmt":"2019-09-03T06:30:34","slug":"fun-with-weak-dynamic-linking","status":"publish","type":"post","link":"https:\/\/glandium.org\/blog\/?p=2764","title":{"rendered":"Fun with weak dynamic linking"},"content":{"rendered":"

Dynamic linkers, at least in the UNIX world, usually allow to load libraries in a process address space at startup. On Linux systems, you load such a library with LD_PRELOAD<\/code>. On OSX, with DYLD_INSERT_LIBRARIES<\/code>.<\/p>\n

On Linux systems, when using LD_PRELOAD<\/code>, the dynamic linker will also use symbols from the (pre)loaded library instead of system libraries. For example, if a program calls the write<\/code> function and a library exporting a write<\/code> symbol is loaded with LD_PRELOAD<\/code>, the write<\/code> function from the loaded library will be used instead of that of the libc (even when the symbol version doesn't match).<\/p>\n

On OSX, symbol resolution is usually done with a \"two-level namespace\": symbols are associated with library names, and when resolving symbols, both are used. More than that, the library name is used to find the library in the dyld search path. Libraries loaded with DYLD_INSERT_LIBRARIES<\/code> won't be used with two-level namespace symbol resolution, which makes it less useful than LD_PRELOAD<\/code>. Fortunately, it is also possible to use a \"flat\" namespace, in which case only the symbol name is considered during symbol resolution, and is searched in all loaded libraries, in the order in which they were loaded. Flat namespace can be triggered by setting the DYLD_FORCE_FLAT_NAMESPACE<\/code> environment variable, linking the main program with -force_flat_namespace<\/code>, or linking programs and libraries with the -flat_namespace<\/code> argument. Note the latter only affects the programs and libraries built with that argument, while the former two force to use a flat namespace for all libraries, including those which, like system ones, were built with a two-level namespace. There are also cases where a single symbol may be resolved with the flat namespace, while others in the program or library are using two-level namespace.<\/p>\n

Weak dynamic linking is another feature that can be used to tell the dynamic linker to ignore missing symbols. Consider the following source code:<\/p>\n

\n
extern void foo() __attribute__((weak)); \/\/ weak_import is preferred on OSX.\r\nint main() {\r\n  if (foo)\r\n    foo();\r\n  return 0;\r\n}\r\n<\/pre>\n<\/blockquote>\n
On Linux systems, this just works. Compile the program (you'll need to build it with -fPIC<\/code>, though), start it, and it will do nothing, since foo<\/code> is defined nowhere.<\/p>\n
Combined with shared library (pre)loading, this can be used to provide simple hooks in your application. For example, with the following source code built as a shared library:<\/p>\n
\n
#include <stdio.h>\r\nvoid foo() {\r\n  printf(\"foo\\n\");\r\n}\r\n<\/pre>\n<\/blockquote>\n
Running the program again with LD_PRELOAD<\/code> set to load that shared library (note LD_PRELOAD=foo.so<\/code> won't work, a path is needed, like LD_PRELOAD=.\/foo.so<\/code>), it will print foo<\/code> because the dynamic linker will have resolved the foo<\/code> symbol to that of the shared library.<\/p>\n
On OSX, unfortunately, things are not as easy. First, building the test program above fails:<\/p>\n
\n
Undefined symbols for architecture x86_64:\r\n  \"_foo\", referenced from:\r\n      _main in test-LIeVtB.o\r\nld: symbol(s) not found for architecture x86_64\r\nclang: error: linker command failed with exit code 1 (use -v to see invocation)\r\n<\/pre>\n<\/blockquote>\n
There are linker options to force undefined symbols to be resolved at runtime:<\/p>\n
    \n
  • -undefined dynamic_lookup<\/code>, which will mark all undefined symbols as having to be looked up at runtime,<\/li>\n
  • -Wl,-U,symbol_name<\/i><\/code>, which only does so for the given symbol (note: you have to prepend an underscore to the symbol name)<\/li>\n<\/ul>\n
    With one of these options, the program works as on Linux: it does nothing when run alone, and prints foo<\/code> when loading the library with DYLD_INSERT_LIBRARIES<\/code>... if you build with XCode 4.5. Running the program built with XCode 4.3 fails when not<\/em> loading the shared library, with the following error:<\/p>\n
    \n
    dyld: Symbol not found: _foo\r\n  Referenced from: .\/test\r\n  Expected in: flat namespace\r\n in .\/test\r\nTrace\/BPT trap: 5\r\n<\/pre>\n<\/blockquote>\n
    And if at build time, you target OSX 10.5 (with MACOSX_DEPLOYMENT_TARGET<\/code> or -mmacosx-version-min<\/code>), the error is slightly different:<\/p>\n
    \n
    dyld: Symbol not found: _foo\r\n  Referenced from: .\/test\r\n  Expected in: dynamic lookup\r\n\r\nTrace\/BPT trap: 5\r\n<\/pre>\n<\/blockquote>\n
    Each error is due to a different bug:<\/p>\n
      \n
    • Since OSX 10.6, the link edition rules in the __LINKEDIT<\/code> segment are in a new, compressed, format: DYLD_INFO. The linker in Xcode < 4.5 forgets to flag weak imports as weak in the DYLD_INFO data. Compare the output for dyldinfo<\/code> with a binary built with Xcode 4.5 vs. the output for a binary built with Xcode 4.3:
      \n
      \n
      $ dyldinfo -bind test-xcode4.5 | sed -n '2p;\/foo\/p'\r\nsegment section          address        type    addend dylib            symbol\r\n__DATA  __got            0x100001038    pointer      0 flat-namespace   _foo (weak import)\r\n\r\n$ dyldinfo -bind test-xcode4.3 | sed -n '2p;\/foo\/p'\r\nsegment section          address        type    addend dylib            symbol\r\n__DATA  __got            0x100001038    pointer      0 flat-namespace   _foo\r\n<\/pre>\n<\/blockquote>\n
      Notice the missing weak import<\/code>. Manually setting the flag with a hexadecimal editor fixes it (in both bind and lazy_bind tables).\n<\/li>\n
    • In older OSX releases, the link edition rules use a \"classic\" format. In that format, the symbol table is used for flags such as N_WEAK_REF<\/code> (weak reference). In the new DYLD_INFO<\/code> format, the N_WEAK_REF<\/code> flag isn't used, which partly explains the previous bug. When using the old format and two-level namespaces, missing weak symbols are errors. With flat namespace, they aren't. This means the error we get with a program built for a 10.5 target goes away if building with -flat_namespace<\/code>. Note this doesn't work at the symbol level: if the library is built for two-level namespace (and is marked as such in the Mach-O header), and the weak symbol is without a corresponding library name, making it resolved with a flat namespace, it doesn't work.<\/li>\n<\/ul>\n
      At this point, one could think weak dynamic linking is pretty much useless on OSX, at least, that it was before Xcode 4.5 was released. As it turns out, there are other use cases where it actually works. Consider the following code:<\/p>\n
      \n
      #include <malloc\/malloc.h>\r\n\/\/ The following is defined in malloc\/malloc.h:\r\n\/\/ extern size_t malloc_zone_pressure_relief(malloc_zone_t *zone, size_t goal) __attribute__((weak_import));\r\nint main() {\r\n  if (malloc_zone_pressure_relief)\r\n    malloc_zone_pressure_relief(NULL, 0);\r\n  return 0;\r\n}\r\n<\/pre>\n<\/blockquote>\n
      This program in itself is useless, but the point is, the malloc_zone_pressure_relief<\/code> function is only available since OSX 10.7. Without the weak_import<\/code>, the program would fail to start with an undefined symbol on OSX 10.6 and below. Without the if<\/code>, it would crash because the symbol would resolve to NULL, and the program would jump there. But the program itself has to be built on OSX 10.7 at least, for the malloc_zone_pressure_relief<\/code> function to be there when building.<\/p>\n
      And in that use-case, we end up with the right flags in DYLD_INFO<\/code>:<\/p>\n
      \n
      $ dyldinfo -bind test | sed -n '2p;\/malloc\/p'\r\nsegment section          address        type    addend dylib            symbol\r\n__DATA  __got            0x100001038    pointer      0 libSystem        _malloc_zone_pressure_relief (weak import)\r\n<\/pre>\n<\/blockquote>\n
      This actually gives us a hint for a way out of our misery for our foo<\/code> function on Xcode < 4.5: linking against a dummy library implementing the weak symbol. When doing so, we get the proper flag in DYLD_INFO<\/code>, like with malloc_zone_pressure_relief<\/code>:<\/p>\n
      \n
      $ dyldinfo -bind test | sed -n '2p;\/foo\/p'\r\nsegment section          address        type    addend dylib            symbol\r\n__DATA  __got            0x100001038    pointer      0 libfoo           _foo (weak import)\r\n<\/pre>\n<\/blockquote>\n
      And the linker additionally does something nice: when all symbols needed from a library are weak references, it marks the library import itself as weak:<\/p>\n
      \n
      $ otool -l test | grep -B 2 libfoo\r\n          cmd LC_LOAD_WEAK_DYLIB\r\n      cmdsize 40\r\n         name libfoo.dylib (offset 24)\r\n<\/pre>\n<\/blockquote>\n
      What this means is that even if the libfoo.dylib<\/code> is missing, it will still work. The downside is that we are now using two-level namespace, which, as mentioned above, makes DYLD_INSERT_LIBRARIES<\/code> useless. The program thus needs to be built with a flat namespace.<\/p>\n
      Update:<\/b> It turns out some versions of XCode don't conveniently mark libraries with only weak imports as weakly linked, so the -Wl,-weak_library,libraryname<\/em>.dylib<\/code> flag is required instead of -Llibraryname<\/em><\/code>.<\/p>\n
      In summary, if you want to add optional hooks that a library loaded through LD_PRELOAD<\/code>\/ DYLD_INSERT_LIBRARIES<\/code> can implement:<\/p>\n
        \n
      • on OSX < 10.6, use weak symbols and build with -flat_namespace<\/code>,<\/li>\n
      • on OSX >= 10.6, when building with Xcode < 4.5, use weak symbols, build with -flat_namespace<\/code> and link against a dummy library implementing the hook functions with -Wl,-weak_library,libraryname<\/em>.dylib<\/code>,<\/li>\n
      • on Linux systems and on OSX >= 10.6, when building with Xcode 4.5, simply use weak symbols,<\/li>\n
      • on Windows, to the best of my knowledge, weak dynamic linking is not supported.<\/li>\n<\/ul>\n
        Stay tuned for the next post, which will describe what this will be used for in Firefox.<\/p>\n","protected":false},"excerpt":{"rendered":"
        Dynamic linkers, at least in the UNIX world, usually allow to load libraries in a process address space at startup. On Linux systems, you load such a library with LD_PRELOAD. On OSX, with DYLD_INSERT_LIBRARIES. On Linux systems, when using LD_PRELOAD, the dynamic linker will also use symbols from the (pre)loaded library instead of system libraries. […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[25],"tags":[23],"class_list":["post-2764","post","type-post","status-publish","format-standard","hentry","category-planet-mozilla","tag-en"],"_links":{"self":[{"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/2764","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2764"}],"version-history":[{"count":67,"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/2764\/revisions"}],"predecessor-version":[{"id":3944,"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/2764\/revisions\/3944"}],"wp:attachment":[{"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2764"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2764"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/glandium.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2764"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}