Firefox is now built with clang LTO on all* platforms

You might have read that Mozilla recently switched Windows builds to clang-cl. More recently, those Windows builds have seen both PGO (Profile-Guided Optimization) and LTO (Link-Time Optimization) enabled.

As of next nightly (as of writing, obviously), all tier-1 platforms are now built with clang with LTO enabled. Yes, this means Linux, Mac and Android arm, aarch64 and x86. Linux builds also have PGO enabled.

Mac and Android builds were already using clang, so the only difference is LTO being enabled, which brought some performance improvements.

The most impressive difference, though, was on Linux, where we’re getting more than 5% performance improvements on most Talos tests (up to 18% (!) on some tests) compared to GCC 6.4 with PGO. I must say I wasn’t expecting switching from GCC to clang would make such a difference. And that is with clang 6. A quick test with upcoming clang 7 suggests we’d additionally get between 2 and 5% performance improvement from an upgrade, but our static analysis plugin doesn’t like it.

This doesn’t mean GCC is being unsupported. As a matter of fact, we still have automated jobs using GCC for some static analysis, and we also have jobs ensuring everything still builds with a baseline of GCC 6.x.

You might wonder if we tried LTO with GCC, or tried upgrading to GCC 8.x. As a matter of fact, I did. Enabling LTO turned up linker errors, and upgrading to GCC 7.x turned up breaking binary compatibility with older systems, and if I remember correctly had some problems with our test suite. GCC 8.1 was barely out when I was looking into this, and we all know to stay away from any new major GCC version until one or two minor updates. Considering the expected future advantages from using clang (cross-language inlining with Rust, consistency between platforms), it seemed a better deal to switch to clang than to try to address those issues.

Update: As there’s been some interest on reddit and HN, and I failed to mention it originally, it’s worth noting that comparing GCC+PGO vs. clang+LTO or GCC+PGO vs. clang+PGO was a win for clang overall in both cases, although GCC was winning on a few benchmarks. If I remember correctly, clang without PGO/LTO was also winning against GCC without PGO.

Anyways, what led me on this quest was a casual conversation at our last All Hands, where we were discussing possibly turning on LTO on Mac, and how that should roughly just be about turning a switch.

Famous last words.

At least, that’s a somehow reasonable assumption. But when you have a codebase the size of Firefox, you’re up for “interesting” discoveries.

This involved compiler bugs, linker bugs (with a special mention for a bug in ld64 that Apple has apparently fixed in Xcode 9 but hasn’t released the source of), build system problems, elfhack issues, crash report problems, clang plugin problems (would you have guessed that __attribute__((annotate("foo"))) can affect the generated machine code?), sccache issues, inline assembly bugs (getting inputs, outputs and clobbers correctly is hard), binutils bugs, and more.

I won’t bother you with all the details, but here we are, 3 months later with it all, finally, mostly done. Counting only the bugs assigned to me, there are 77 bugs on bugzilla (so, leaving out anything in other bug trackers, like LLVM’s). Some of them relied on work from other people (most notably, Nathan Froyd’s work to switch to clang and then non-NDK clang on Android). This spread over about 150 commits on mozilla-central, 20 of which were backouts. Not everything went according to plan, obviously, although some of those backouts were on purpose as a taskcluster trick.

Hopefully, this sticks, and Firefox 64 will ship built with clang with LTO on all tier-1 platforms as well as PGO on some. Downstreams are encouraged to do the same if they can. The build system will soon choose clang by default on all builds, but won’t enable PGO/LTO.

As a bonus, as of a few days ago, Linux builds are also finally using Position Independent Executables, which improves Address Space Layout Randomization for the few things that are in the executables instead of some library (most notably, mozglue and the allocator). This was actually necessary for LTO, because clang doesn’t build position independent code in executables that are not PIE (but GCC does), and that causes other problems.

Work is not entirely over, though, as more inline assembly bugs might be remaining only not causing visible problems by sheer luck, so I’m now working on a systematic analysis of inline assembly blocks with our clang plugin.

2018-09-12 17:10:49+0900

p.m.o

You can leave a response, or trackback from your own site.

38 Responses to “Firefox is now built with clang LTO on all* platforms”

  1. GCC Says:

    I know most of the users don’t care but for me GCC is the preferred choice. Therefore sad to hear Firefox using Clang now by default.

  2. jason_s Says:

    I know most of the users don’t care but for me GCC is the preferred choice

    why?

  3. Mark Says:

    @jason_s,

    OpenMP workloads. Native compiles that run faster. And many other small cases where clang is just not there yet.

  4. BatmanAoD Says:

    @Mark This article claims notable performance improvements on all platforms from switching to Clang. So what do you mean about “native compiles that run faster”?

  5. osajdf Says:

    And what the hell is a PGO and LTO? You tech nerds do nothing to help anyone understand what is going on.

  6. glandium Says:

    @osajdf: Updated with the acronym meanings.

  7. Jan Hubicka Says:

    I am building Fireofx with LTO+PGO regularly using GCC 7 and 8 and it works. There are also talos tests of it made half a year ago at https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-central&newProject=try&newRevision=7e5bd52e36fcc1703ced01fe87e831a716677295&framework=1&selectedTimeRange=172800 How those compare to clang?

  8. Jan Hubicka Says:

    Hmm, my reading is that the link I posted above actually compares the LTO+PGO build to last two-days of official builds. It seems to show no regressions. So does it mean that GCC is still consistently faster if LTO+PGO is enabled?

  9. glandium Says:

    @Jan: Can’t compare results from 6 months ago to results from now, see https://elvis314.wordpress.com/2018/09/12/looking-at-firefox-performance-57-vs-63/. Also, not a lot of samples on mozilla-central are with clang PGO+LTO yet (it’s not even been 2 days).

  10. Jan Hubicka Says:

    OK, Would it be possible to re-run the GCC LTO tests and/or have link to the page comparing clang builds with previous GCC one? I saved comparison with trunk at the time GCC LTO builds was fresh, so I could check if all the performance improvements seems to be due to LTO or there are other reasons.

    I would say most of speedups are for LTO. Also note that GCC builds with LTO are smaller than clangs. I collected some data for GNU Cauldron.

    https://gcc.gnu.org/wiki/cauldron2018?action=AttachFile&do=view&target=Hubicka_+LTO_IPA+bof.pdf

  11. GCC Says:

    A question. If it only comes down to doing a benchmark. Why wouldn’t i just use Windows and Edge or Chrome? I choose to use GCC and Firefox and Linux … Benchmarks have little to do with that.

    That is why i said i am sad to see this being considered as a progress. In my opinion this insn’t progress. But OK we are all different and have different opinion on such things.

  12. Jan Hubicka Says:

    I have finally found correct way to build Firefox with LTO+PGO and did comparsion of GCC 8 builds with official and my own Clang 7 build. Seems GCC wins in benchmarks I tried plus binary is significantly smaller.

    http://hubicka.blogspot.com/2018/12/firefox-64-built-with-gcc-and-clang.html

    What seem to play issue is Firefoxes watchdog killing the training run before it streams PGO data to disk.

  13. How to recover data from crashed garmin device Says:

    where we are going to discuss one of the most frequent issues that are encountered by the users regarding Recover data from crashed Garmin device. http://www.gpsupdatesupport.com/garmin-update/how-to-recover-data-from-crashed-garmin-device/

  14. nick jonas Says:

    Great post thanks for sharing keep sharing.

  15. mark wilson Says:

    Yahoo Support Service Available 24/7 Yahoo Support service is 24/7 accessible for the customers who want to resolve their Yahoo problem. Yahoo Support assures you to solve your problem within no time but it also depends on the type of Yahoo issue. Our Yahoo Support 1-855-436-1666 team is reliable and valuable as it doesn’t share any of the personal information of the customer with anyone. Read more- https://www.contact-customerservice.net/yahoo-customer-services/

  16. mark wilson Says:

    https://www.contact-customerservice.net/ https://www.contact-customerservice.net/bt-support/ https://www.contact-customerservice.net/setting-bt-email-win-10/ https://www.contact-customerservice.net/set-up-email-on-iphone/ https://www.contact-customerservice.net/yahoo-mail-password-recovery/ https://www.contact-customerservice.net/yahoo-customer-services/ https://www.contact-customerservice.net/yahoo-phone-number/ https://www.contact-customerservice.net/change-gmail-password/ https://www.contact-customerservice.net/gmail-customer-care/ https://www.contact-customerservice.net/gmail-customer-service/ https://www.contact-customerservice.net/gmail-help-centre/ https://www.contact-customerservice.net/gmail-support/ https://www.contact-customerservice.net/gmail-recover-deleted-emails/ https://www.contact-customerservice.net/gmail-sign-up/ https://www.contact-customerservice.net/gmail-phone-number/ https://www.contact-customerservice.net/contact-facebook-help-center/ https://www.contact-customerservice.net/facebook-support/ https://www.contact-customerservice.net/sage-support/ https://www.contact-customerservice.net/youtube-customer-service-uk/ https://www.contact-customerservice.net/avg-antivirus-support-uk/ https://www.contact-customerservice.net/norton-antivirus-support-uk/ https://www.contact-customerservice.net/facebook-phone-number/ https://www.contact-customerservice.net/facebook-marketplace/

  17. Reset HP Officejet 3830 Printer Says:

    Thank you so much for posting this kind of content, your content delivery is awesome, would like to bookmark your site on my browser for reading out your new posts. Keep posting these kind of posts.

  18. Garmin RV 780/785 Update Says:

    Garmin RV 780/785 GPS devices have always helped the users in obtaining the best and the proper navigation results. These devices are compatible, portable, and affordable and are easy to use. To know more about the features, visit our website.

  19. Connect Dell V313W All-in-One Inkjet Printer to WiFi Says:

    Pretty great post. I simply stumbled upon your blog and wanted to say that I have really enjoyed surfing around your blog posts. In any case I’ll be subscribing in your feed and I am hoping you write once more very soon!

  20. Samuel Beckett Says:

    Remove all the problem of Yahoo account with Yahoo customer service number Sometimes, users might find themselves in some technical glitches that cannot be handled by nontechnical hands. So, to get immediate help, you can try our <a href=”Yahoo”>https://customer-carenumber-usa.com/yahoo-customer-service/“>Yahoo customer service number1-833-775-7700. We are backed with knowledgeable technicians who’ll guide you at every step over the phone call. Call us right now. Read More:- <a href=”https://customer-carenumber-usa.com/yahoo-customer-service/“>https://customer-carenumber-usa.com/yahoo-customer-service/

  21. Samuel Beckett Says:

    Yahoo customer service number offers the right solution

    One of the common issues of Yahoo that is faced by Yahoo users is a security threat. You can know more about it through our reliable Yahoo customer service number 1-833-775-7700. Users will get immediate assistance from a team of technical engineers. So, whatever your technical error is, contact our team for quick and hassle-free removal. Read More:- https://customer-carenumber-usa.com/yahoo-customer-service/

  22. Samuel Beckett Says:

    Consider getting connected with professionals for fixing Yahoo password issue

    These days, Yahoo has become the most trusted and used email service providers because of its multiple features and benefits that are related to it. This email allows the users to compose emails, make live video calls, send emails and make attachment and many more. All these benefits can be availed with just a Yahoo email account. But, sometimes, forgetting or losing password can be a huge loss for you as you can’t access your account. Fixing the yahoo password issues of Yahoo account can be solved in a few minutes with the help of technical support service. For all sorts of technical issues, users can just make a call to the experts. By following these simple steps, password of Yahoo account can also be changed in trouble-free manner: • Firstly, you’re needed to sign into the yahoo mail sign up new account. After this, click on the gear icon which is present on the upper-right corner. In a mobile browser, you can just tap the Menu option. • Now, the next step is to click on the “yahoo email account” which is available at the bottom of the menu page • Check for the “Account Security” option and then tap on it. After this, you’re needed to enter the current password of the account. • Now, change the password and then enter the new password two times to confirm the password. Make sure that the password that you’re entering is strong and has special characters and numbers. That’s it. Now, your password will be changed by following these simple steps. On getting confused on any step, users can easily contact the experts. A team of technical representative helps the users in handling any sort of password problems related to Yahoo account. With a phone call, you’ll be able to contact the professionals and share your technical issues. https://email-tips.net/fix-yahoo-password-issue/

  23. mark wilson Says:

    Reset your outlook password easily: Outlook Password Reset

    Are you looking for Outlook Password Reset 1-833-293-8333? Whether you are a new outlook user or an existing one, we will help every one of you. We have a team of dedicated professionals who are working 24/7 to overcome the password issues within a short span of time. Our outlook experts are round the clock available to answer your queries related to outlook password. You can take the help of our outlook experts and seek reliable assistance on the problems. More read- https://www.customercare-email.net/outlook-customer-service/

  24. Samuel Beckett Says:

    Have A Word With Yahoo Customer Service Experts Right Now We will surely help you to have a word with the reliable customer care executive who will direct you the right resolution way through which you will effectively resolve almost all sorts of problems and hurdles from the root. Here, we will help over the phone call at <a href=”Yahoo”>https://www.emailcontacthelp.com/yahoo-customer-support-services.html“>Yahoo Customer Service 1-833-891-2999 and we will be at your disposal. Read More :- <a href=”https://www.emailcontacthelp.com/yahoo-customer-support-services.html“>https://www.emailcontacthelp.com/yahoo-customer-support-services.html“>https://www.emailcontacthelp.com/yahoo-customer-support-services.html

  25. mark wilson Says:

    Confronting Issue while Doing Gmail Sign Up If you are facing a problem while doing Gmail Sign Up 1-833-293-8333 then check whether you have entered the correct information or not . If you have entered correct sign details then check link that whether you are using official Gmail link or not. Sometimes because of browser cache and cookies, Gmail Sign Up problem also occur so for resolving this delete the junk files, cache, and cookies from your browser. https://www.customercare-email.net/sign-up-gmail/

  26. Samuel Beckett Says:

    Make Google Homepage for Enjoying Faster Service Make <a href=”Google”>http://googlehome-page.com/“>Google Homepage 1-833-554-3444 for enjoying the faster and other features of Google. By making Google as your homepage you can easily add an extension and can easily bookmark the links which you use again and again. If you need any guidance regarding making Google as your homepage then just avail Gmail Help as they will assist and will also resolve your query. <a href=”http://googlehome-page.com/“>http://googlehome-page.com/“>http://googlehome-page.com/

  27. Annahere Says:

    This is really helpful blog, thank you for sharing information Firefox Not Opening

  28. Samuel Beckett Says:

    Facing Yahoo Mail Not Sending Issue then just avail Yahoo Support If you are unable to send or receive Yahoo Mail then first check your internet connection whether it is working properly or not. If everything is alright from your side then avail <a href=”Yahoo”>https://www.emailcontacthelp.com/yahoo-technical-support.html“>Yahoo Support 1-833-891-2999 for reporting Yahoo Mail Not Sending Issue. Yahoo Support team will surely solve your query within no time and will also make your account error-free. <a href=”https://www.emailcontacthelp.com/yahoo-technical-support.html“>https://www.emailcontacthelp.com/yahoo-technical-support.html“>https://www.emailcontacthelp.com/yahoo-technical-support.html

  29. Samuel Beckett Says:

    Unable to send or receive Mails on Gmail then take Gmail Help Facing problem while sending and receiving emails than first check your internet connection and then after that sign out your Gmail account and then again Sign in your Gmail account. If after performing the above-mentioned thing you are not able to send or receive emails then take <a href=”Gmail”>https://www.emailcontacthelp.com/gmail-help-phone-number.html“>Gmail Help. You can take Gmail help by calling at 1-833-891-2999. Read more: – <a href=”https://www.emailcontacthelp.com/gmail-help-phone-number.html“>https://www.emailcontacthelp.com/gmail-help-phone-number.html“>https://www.emailcontacthelp.com/gmail-help-phone-number.html

  30. Samuel Beckett Says:

    Reset Sprint Password by Availing Sprint Customer Service On should always reset the Sprint account to maintain account safety as by doing so your account hacking issue will get depreciated. While resting your Sprint Password must create a strong password and a strong password can be created by using the alphabet, numbers, and special characters. For maintaining privacy and security do not share your Sprint Account Password with anyone. If you are not able to change your password then just avail <a href=”Sprint”>https://sprint-customerservice.com/“>Sprint Customer Servcice 1-833-293-6333. Read more:- <a href=”https://sprint-customerservice.com/“>https://sprint-customerservice.com/“>https://sprint-customerservice.com/

  31. Epson Printer Printing Blank Pages Says:

    Great Post I found on Different Printing Types, I enjoyed the article while searching for more updates on Epson. You may also Visit:<a href=”Epson”>https://www.epsonprintersupport247.com/how-to-fix-epson-printer-printing-blank-pages/“>Epson Printer Printing Blank Pages

  32. mark wilson Says:

    Unable to Recover Verizon Password then take help of Verizon customer service Forgotten Verizon Password and now unable to recover it? Then for guidance read the blog and article related to recover a password if then too you are not able to recover it then for help contact Verizon Customer Service and for contacting us just make a call or drop a message at 1-855-431-5111. Feel free to contact us as we are 24/7 accessible to help our customers. https://www.customerservicephone-number.com/verizon-wireless-customer-service-phone-number/

  33. mark wilson Says:

    Unable to do video conference Skype call then call at Skype Support Phone Number

    As we all know that nowadays the new features of connecting 50 people in one video Skype call is on trends if in case you are unable to use this feature then first update your Skype and permit the Skype for making a video call. If after doing this you are unable to do a video conference call on Skype then just call on Skype Support Phone Number i.e. 1-833-293-8333 for resolving your problem. https://www.customercare-email.net/skype-customer-service/

  34. Apple support Says:

    Hello! My name is Martinlutharz . I have been working with the team available at Apple Support Phone Number for long years ago. I have much experienced in dealing with the Apple device related problems. No matter the issue is nasty or complicated; once the users call on this helpline number let me know their hurdle they get instant handy solution in an effective manner. In case I am busy on another call, one of our proficient team members will lend your hand and provide you fruitful solution within a pinch.

    Read more :- https://www.applesupportphonenumber.co/

  35. Dell Support Says:

    As a highly versatile Dell Printer engineer with years of understanding and victorious history of solving queries associated with Dell printer I Robertjonz is just one phone call away from you. You can easily resolve your entire printer related issue by just making a call at Dell Printer Support number.

    https://www.dellsupport.co/

  36. Outlook Support Phone Number Says:

    Does your office 365 software is surrounded with a number of issues that you can’t resolved by yourself? Just don’t be anxious put a call on Office 365 Technical Support number and join hands with me. As an experienced technician I will certainly provide you the optimum solution in one go. So, if you wish to overcome your printer related issues just grab our support. Visit here: https://outlookcustomerservicee.blogspot.com/2019/08/with-outlook-customer-support-with.html

  37. aliza decruz Says:

    We are a top rated and professional SEO company in USA, offering best Local SEO services for various types of businesses. We offer on page and off page SEO services, which are first rated to give you more results. Our SEO professionals are very smart to improve your website’s ranking easily.

    https://bestfirstrankseoservicesusa.blogspot.com

  38. belkin router Says:

    While conducting the entire procedure of Belkin wireless Router setup you might encounter some errors. To be able to rectify those errors, read our blog and get solutions as well.

Leave a Reply