Archive for February, 2010

Is there a conspiracy to prevent me from working on Iceweasel 3.6 ?

After realizing Iceweasel wasn’t working properly on various Debian architectures, and spending time fixing that, this week a new cairo version triggered a longstanding bug which would kill Iceweasel when closing a tab. I think I got it fixed now, but what next ?

2010-02-25 18:09:54+0200

firefox | 7 Comments »

Mind blowing news

The Firefox logo, the very reason why Iceweasel exists, is now free as in speech. Its use is still limited by trademarks, but it is free.

The news is not entirely official, but apparently the change has occurred a year ago, and the license text for the branding hasn’t been changed accordingly.

Waw.

Update: Now official.

2010-02-23 23:38:03+0200

firefox | 12 Comments »

Green is beautiful

buildd status for xulrunner mostly green

This is it! xulrunner now builds and successfully runs the checks and xpcshell-tests on all release architectures, including arm, mips, powerpc and sparc, that were all failing 2 weeks ago. 2 patch pulled from upstream, 5 patches written and sent upstream, and several more as not quite successful attempts.

What puzzles me is that e.g. the arm issue in the test suite has been known upstream for a year, yet Firefox was released on maemo (arm) without it being fixed. I mean, it only took me a few hours (if you don’t count the time lost waiting for the builds to end), starting with no specific knowledge of arm, and even less of jit compilers.

Anyways, this all means work on 1.9.2 can go on.

2010-02-23 08:29:36+0200

xulrunner | 3 Comments »

Massive FAIL #2

Massive iceape buildd failure

This is what would have happened if I had waited before uploading 2.0.3-2. Courtesy of dh_install --fail-missing and its non-handling of arch:any only builds.

2010-02-18 14:33:41+0200

xulrunner | No Comments »

Massive FAIL

Massive xulrunner buildd failure

In there, a lot are due to dh_auto_configure not doing the right thing for autoconf 2.13 generated configure files, others are due to make setting MAKEFLAGS to “w”, which prevents make -s to do what I expected it to do, armel has yet another crasher in the test suite (*sigh*), and I broke mipsel with the mips patch.

1.9.1.8-2 should fix the first two, and allow me to debug the armel issue (I disabled the test suite to be able to have the buildds generate the binaries, as they are much faster than the porterbox). Mipsel will have to wait for 1.9.1.8-3.

This also means 1.9.2 is delayed. Again. (and upstream releasing Firefox 3.5.8 one day later than scheduled didn’t help either)

Update: I stand corrected, the one responsible for MAKEFLAGS setting to “w” seems to be make itself, not dh.

2010-02-18 10:29:08+0200

xulrunner | 2 Comments »

RC_buggy_arches = 0

(maybe)

After the two updates in the previous post, I got further and eventually fixed the root of the remaining problems on ARM.

It appears I got on the wrong track at the very beginning of the investigation. My first test was to disable the JIT compiler during the tests, and it didn’t affect the result. I assumed from there that the JIT compiler was doing fine and wasn’t involved. Which led me to believe the alignment of the stack at 32-bits word boundary instead of 64-bits was a standard behaviour. I turns out my first test itself was wrong, because I hadn’t disabled the JIT compiler at all. You need to modify the runxpcshelltests.py script to disable the JIT, and not fiddle with the javascript.options.jit.chrome preference.

The JIT compiler was misaligning the stack, and disassembling the generated code leading to broken behaviour revealed that it was doing so when setting up arguments on the stack for function calls. While I was afraid this would be difficult to fix without changing a lot of code, and first only disabled the JIT compiler, it turns out the way nanojit works by compiling the last instructions first did make it straightforward to fix.

But ARM was not the only RC buggy architecture left after sparc and powerpc were fixed. MIPS remained. Alpha fails, too, but it is not a release architecture. Update: It appears hurd fails too, but like Alpha, it is not a release architecture.

So, while I didn’t want to wait 50 hours for a build, I managed to execute a failing test with the last xulrunner-1.9.1 packages that got built (before I enabled the test suite) under qemu. And I was able to track down the origin of the bug: the xpcom layer that translate function calls between languages doesn’t fill arguments correctly when they are smaller than 32 bits. While the code is adapted to the mipsel case, which is little endian, it doesn’t work on mips, which is big endian. I came up with a patch that should work, and which I hope won’t break mipsel as a consequence. All that is left to do is to build and test the fix. I hope someone else will be able to do that for me.

All in all except a 1.9.1.8 release this week with a lot less red on the buildd status page.

2010-02-15 09:36:49+0200

xulrunner | 1 Comment »

When will Iceweasel 3.6 come ?

That is a question I hear more and more, and I felt it had to be answered publicly. Short answer: when it’s ready.

The current target for Squeeze is 3.5, for various reasons, but the main is that it shares the same Gecko version as Iceape (SeaMonkey) and Icedove (Thunderbird), which means that while the security team (or myself) will have to update these three for security updates, they will share patches. So 3.6 is less a priority than having 3.5 in shape for the release.

Last week, I wrote on a mailing list post that I was expecting 3.6 to come this week. Well, that was before I discovered the sad reality about the state of the current version.

But 3.6 is still on the radar. I already went through all the patches we apply on 1.9.1, and rebasing should now be straightforward.

Most of the 3.5 failures have now also been addressed: sparc and powerpc were fixed first, and I am now (re-re-re…-re)building on arm for, I hope, the last time. Unfortunately, mips is still missing, and without a porterbox, I can’t do much. I tried, some time ago, in qemu, but there is no way I’m going to wait 50 hours for a build again ; it was already painful on a real arm box (where it takes 90 minutes just to link libxul.so). Anyways, 2 patches stolen from upstream and 5 patches sent (or almost sent for the one currently under validation) later, we should be down to one release architecture failing instead of four. Which is already good.

On the other hand, a new 3.5 upstream release, with security fixes, is planned for next tuesday, which means some time will have to be spent for stable updates, too. So, in the next few days, expect that I’ll be working on stable security updates, then unstable updates, including the three architectures fix, and then work on 3.6 will continue.

Which means we can hope for a 3.6 release in experimental by the end of next week, or the following one, in the worst case.

Update: *sigh* 4 patches on arm is not enough :(. At least this time I think I’m done with xptcinvoke and xptcstubs.

Update:In the end, 2 patches and disabling the JIT compiler did the trick on arm. The 2 other patches were actually addressing bugs that only existed because the JIT is doing something wrong with the stack alignment.

2010-02-13 09:34:00+0200

firefox | 15 Comments »

Iceweasel upgrade homepage, a week later

When users upgrade Firefox, they are shown a special web page after restart. As this was obviously very Firefox oriented, this was disabled in Iceweasel. Until a week ago.

A week ago, I created a (quite raw) web page that users of Iceweasel 3.5.6-2 would see after they install or upgrade. The idea behind this page was suggested by Moritz Muehlenhoff during my BoF at DebConf 9, and consisted of displaying a call for help message in a splash screen. What most looked like this in Iceweasel is the Firefox feature mentioned above. I unfortunately wasn’t able to implement it until now.

So, a week after these pages were put online and Iceweasel 3.5.6-2 was uploaded, even though I can’t say it’s a huge success, it at least led to some movement: More than 12000 users (still growing) saw the pages, out of which almost 1800 (roughly 15%) followed the link for a new logo. While there was no contribution for a new logo yet, there has already been several translations contributions for the web pages, in german, spanish and italian. Next iceweasel upload will consequently send users to a localized version of the web page when it exists.

The server logs also allow to have some figures on the usage of localized versions of Iceweasel, but as this is based on user agent strings, it is not totally accurate, though somehow interesting:

  1. english: 58.3%
  2. german: 15.2%
  3. french: 7.9%
  4. spanish: 3.8%
  5. italian: 2.7%
  6. polish: 2.5%
  7. japanese: 2.2%
  8. russian: 2.1%
  9. portuguese: 1%
  10. czech: 0.6%

Another interesting figure is that apparently 2.4% of users haven’t upgraded xulrunner-1.9.1 and are not using version 1.9.1.6 with their Iceweasel 3.5.6. I’m afraid of what this can mean for stable security updates, although, as being security updates, they may be followed better.

Somewhere around 7% of users have, at some point, changed their user agent string to include Firefox in it, because web sites suck. This is a very interesting figure because despite that being quite significant, we haven’t got that many complaints (I’m not saying there weren’t, but I would have expected more, considering the amount of users involved). Fortunately, this was changed recently, so all these users could reset their user agent string.

And last but not least, except amd64 and x86 users, we saw 14 powerpc users and 7 arm users.

2010-02-10 13:00:44+0200

firefox | 5 Comments »

broken_arch -= 2

The xulrunner test suite now passes on sparc, involving a fix for an existing bug, and a new bug report.

Next target is powerpc, where, interestingly, the test suite only failed 3 tests on the porterbox, where the buildd failed 58… And contrary to sparc, the failures don’t involve crashes.

Update: powerpc is now fixed, too, with a little help from luck and good timing: the 3 failing tests had all the same origin, and it was fixed a few months ago in upstream trunk.

2010-02-05 19:40:25+0200

xulrunner | No Comments »

xulrunner 1.9.1.6-2 diff stats

As part of preparing xulrunner 1.9.2 (which would lead to Iceweasel 3.6), I have gone through all the patches currently applied to the latest xulrunner release, to keep track of what should be upstreamed, what needed feedback, what patches I needed to update in upstream bugzilla, etc.

As I have talked about our differences with upstream on a few occasions already, and as I was doing this grunt work anyway, I thought it would be nice to make the results public.

First, a few reference numbers (excluding changes to generated configure files):

  • Changes between upstream 1.9.0.1 and Debian release 1.9.0.1-1: 85 files changed, 1163 insertions(+), 644 deletions(-)
  • Changes between upstream 1.9.0.16 and Debian release 1.9.0.16-1: 86 files changed, 1177 insertions(+), 634 deletions(-)
  • Changes between upstream 1.9.1 and Debian release 1.9.1-1: 70 files changed, 770 insertions(+), 363 deletions(-)
  • Changes between upstream 1.9.1.6 and Debian release 1.9.1.6-1: 76 files changed, 796 insertions(+), 380 deletions(-)
  • Changes between upstream 1.9.1.5 and upstream 1.9.1.6: 165 files changed, 3094 insertions(+), 1387 deletions(-)
  • Changes between upstream 1.9.1 and upstream 1.9.1.6: 1263 files changed, 64322 insertions(+), 67376 deletions(-) (though more than half of that relates to upstream bump of sqlite3 and the vorbis stack)

A few observations on the above data:

  • There are less changes between upstream and Debian than between two consecutive upstream versions.
  • Debian changes only account for changes to the xulrunner package, i.e. it excludes changes to Iceweasel itself, or to nspr or nss, but these are far less important in number.
  • The number of changes applied decreases when switching to a new upstream branch. This is mostly due to patches getting applied upstream.
  • Debian patches on a given branch are pretty stable. In other words, most of the work is done on the first release of the branch.

The latter used to be true, but is not anymore: xulrunner 1.9.1.6-2 added a lot of changes and is now up to: 98 files changed, 1107 insertions(+), 517 deletions(-).

What happened ? will you ask me. A lot happened: the test suite, attempts at correctly installing C++ headers and IDL files, bug fixes. Work has also been done such that make clean/distclean doesn’t leave stuff around and that xulrunner can be properly built twice in a row.

Differences between 1.9.1.6-1 and 1.9.1.6-2 look like this: 31 files changed, 341 insertions(+), 167 deletions(-).

Some were stolen from bugzilla (13 files changed, 162 insertions(+), 70 deletions(-)):

Many others were sent:

A few others were filed without a patch, as I felt a proper upstreamable patch needed discussion (3 files changed, 16 insertions(+), 3 deletions(-)):

A few changes are related to the way we build and how we run the test suite (5 files changed, 35 insertions(+), 18 deletions(-)):

  • Disable python-xpcom tests for now.
  • Disable javaxpcom tests at build time when DEB_NO_JAR is unset.
  • Synchronize config/rules.mk and js/src/config/rules.mk for check-sync-dirs.py (we used to localize the changes required in each config/rules.mk, but as we now run make check they need to be synchronized)
  • Workaround for https://bugzilla.mozilla.org/show_bug.cgi?id=455238

Finally, a few were related to the Debian changes history (2 files changed, 1 insertions(+), 26 deletions(-)):

  • Remove –enable-system-lcms check, which somehow resisted merges with upstream
  • Properly clean xulrunner/installer/*.system.conf

Going through the rest of the changes between upstream and the latest Debian release, we can split in 5 categories of changes:
Those that were picked from upstream, or sent and eventually applied in 1.9.2 or trunk (12 files changed, 68 insertions(+), 41 deletions(-)):

Those that were picked from bugzilla or sent there, but are not applied upstream for various reasons (29 files changed, 138 insertions(+), 209 deletions(-)):

Those that were reported but for which I wanted feedback for a proper fix, yet a patch was applied in Debian because this was necessary (6 files changed, 22 insertions(+), 13 deletions(-)):

Those that are yet to be sent to bugzilla (15 files changed, 111 insertions(+), 54 deletions(-)):

  • xulrunner-config: Give more appropriate cflags and libs
  • Install applications in /usr/local/lib instead of /usr/lib
  • Unhide release notes link in about: but only if app.releaseNotesURL is defined
  • In about:, Don’t put an about:blank link when there is no vendorURL defined
  • Don’t use static strings when setting environment variables
  • Exec instead of uselessly forking in xulrunner launcher script
  • Disable optimization on alpha for the url-classifier component (this will probably wait for the test suite failures on various architectures to be fixed)
  • Build js shell and xpcshell against system libreadline
  • Don’t export js_SetTraceableNativeFailed, which is only used internally
  • Increase stability and performance on mips and mipsel

And finally, the changes that are either Debian specific or not deemed upstreamable (30 files changed, 460 insertions(+), 77 deletions(-) ; this can sound huge, but see the details below):

  • Feature changes:
    • Disable APNG support when system libpng doesn’t support it (6 files changed, 71 insertions(+), 10 deletions(-) ; mostly #ifdef’ing parts of the code related to animation)
    • Use application.ini from where the xulrunner-stub lies (1 files changed, 14 insertions(+), 3 deletions(-) ; This one may be dropped and replaced by something totally different, possibly upstreamed)
    • Don’t register plugins if the MOZILLA_DISABLE_PLUGINS environment variable is set (1 files changed, 25 insertions(+), 0 deletions(-) ; very useful to isolate plugin-related crashes ; I don’t think it’s worth upstreaming considering the upcoming plugin-in-separate-process work)
    • Remove (un|)registering system (1 files changed, 0 insertions(+), 43 deletions(-) ; xulrunner versions that are installed by a package manager don’t need to be registered, since the package does it already ; a variant that conditionally removes it may be upstreamable)
  • Preferences changes:
    • Set javascript.options.showInConsole (1 files changed, 1 insertions(+), 0 deletions(-))
    • Set DPI to system settings (1 files changed, 1 insertions(+), 1 deletions(-))
  • Cosmetic changes:
    • Install loading_16_grey.gif in classic.jar, and add an override for loading16.png (1 files changed, 2 insertions(+), 0 deletions(-) ; Since we don’t support APNG, we replace the most used one by its (ugly) GIF equivalent)
    • Add links for about:bugs and about:README.Debian in about: (3 files changed, 19 insertions(+), 0 deletions(-) ; I’d like to make that something living in debian/extra-stuff)
  • Integration changes:
    • Add another preferences directory for applications: defaults/syspref (1 files changed, 2 insertions(+), 0 deletions(-))
    • Add an rpath to libpyloader.so and _xpcom.so (2 files changed, 2 insertions(+), 0 deletions(-) ; allows python xpcom to work out of the box)
    • Forward-port nsIBadCertListener from 1.8 (4 files changed, 266 insertions(+), 1 deletions(-) ; This is necessary for at least galeon and kazehakase, because the new interface is useless for embedding applications)
  • Build-related changes:
    • Allow to override the PYTHON_SO variable (1 files changed, 1 insertions(+), 1 deletions(-))
    • Put the crmf library before the NSS libraries (1 files changed, 1 insertions(+), 1 deletions(-))
    • Allow to build java jar files in 2 pass (2 files changed, 8 insertions(+), 2 deletions(-))
    • Avoid libxpcom being excluded from linked libraries by -Wl,–as-needed (1 files changed, 2 insertions(+), 0 deletions(-))
    • Ignore system libjpeg, libpng and zlib version checking (1 files changed, 3 insertions(+), 3 deletions(-))
    • Force to not use -fshort-wchar (1 files changed, 1 insertions(+), 1 deletions(-))
    • Add soversion to libmozjs (2 files changed, 19 insertions(+), 2 deletions(-) ; useful for couchdb, libjavascript-perl and others)
    • Don’t build example component (1 files changed, 0 insertions(+), 1 deletions(-))
    • Don’t install system profile (1 files changed, 0 insertions(+), 2 deletions(-))

Note, the numbers in the various diff stats obviously don’t add up, since some changes overlap.

All that information doesn’t account the few additional components added in debian/filemonitor and debian/extra-stuff, that account for nearly 600 lines (excluding licensing boilerplate), but I’ll talk about these in another post.

This analysis also helped spotting another harmless merge error (the first one was spotted a few weeks ago and just left the –enable-system-lcms around, without any effect): a variable replacement in a Makefile that doesn’t have any effect, since the 2 variables have the same value. Only a pointless 1 line change.

2010-02-04 12:52:14+0200

xulrunner | No Comments »