Testing shared cache on try

After some success with the shared cache experiment (Read about it, and some more), the next step was to get it to work on the Mozilla continuous integration infrastructure, and it turned out to reveal a couple issues.

The first issue is that the DNS server for the AWS build slaves we use is not the AWS DNS, but our in-house DNS. Which has two consequences:

  • whatever geolocation S3 does at the DNS level may end up giving a S3 endpoint IP that is not optimal for the AWS region we’re in because it was correlated to the location of our in-house DNS
  • the roundtrip to the in-house DNS server was around 80ms, and because every compilation is an independent process, each one does a DNS request, so each one gets that 80ms hit. Note that while suboptimal, doing a DNS request for each compilation also allows to get different S3 endpoints because of both DNS round robin and geolocation S3 uses, which gives very different IPs every so often.

The consequence of this is that build times were very unstable, ranging from 11 minutes like during my experiments up to 45 minutes for a 99% cache hit build! After importing a DNS resolver in the shared cache script and making it use the AWS DNS, build times became much more stable between 11 and 12 minutes. (we actually do need to use the in-house DNS for normal operations on the build slaves, so it’s not possible to switch /etc/resolv.conf)

The second issue is that the US Standard region for S3 can have quite high latency depending on the region you’re connecting to it from. Our build slaves are located in Oregon and Northern Virginia, and while the slaves in Northern Virginia could reach S3 US Standard within 3ms, those in Oregon could only reach it within 90ms. Those numbers were unfortunately gotten with the in-house DNS, so geolocation may have had its impact on them, but after switching DNS, the build times on Oregon slaves were still way higher than on Northern Virginia slaves (~11 minutes vs. ~21 minutes). Which led us to use a S3 bucket per region.

With those issues dealt with, we’re now ready for more widespread testing, and as such I’ve turned the shared cache on on Linux opt, Linux debug, Linux64 opt and Linux64 debug builds, for try only, only if the push contains the relevant setup, which landed in changeset a62bde1d6efe.
See my post on dev-tree-management for a few more details, notably if you hit bugs.

Please note this is only the beginning. More platforms will use the cache soon, including some that aren’t currently using ccache. And I got some timing numbers during the initial tests on try that hint at the most immediate performance issues with the script that need addressing. So you can expect builds to get faster and faster as the cache populates, and as the script is improved with feedback from past experiments and current deployment (I’ll be collecting data from your try pushes). Also relatedly, I’m working on build system improvements that should make the ‘libs’ step much faster, cutting down the time spent on that step.

2014-02-13 10:18:45+0200


You can leave a response, or trackback from your own site.

One Response to “Testing shared cache on try”

  1. louis vuitton handbags outlet Says:

    Tee michael kors outlet off to kate spade outlet terrific coach outlet deals at toms shoes outlet online this week. Their louis vuitton outlet online selection coach factory and coach outlet online will now see an coach factory outlet 25% off at abercrombie and fitch checkout. Metal-and-michael kors outlet online grocery kate spade handbags baskets and true religion black friday Classic Flap Bags frey wille jewellery to louis vuitton handbags raw michael kors outlet runway, but I louis vuitton outlet stores myself loving the coach factory outlet candy coach factory clutches louis vuitton outlet and oakley sunglasses candy costume tiffany and co jewelry the most. As a kid, kate spade outlet a candy necklace or cheap oakley sunglasses bracelet marc by marc jacobs one of my coach factory outlet online favorite things, it prada black friday was a rarity, but cheap oakley sunglasses when I had one, it ray ban sunglasses felt like oakley sunglasses morning. coach outlet store online protestors from tiffany and co jewelry People forcoach factory outlet the ray ban black friday of Animals, michael kors outlet, and the ray ban sunglasses cheap to michael kors handbags – Drawn Carriages swarmed louis vuitton black friday Park oakley sunglasses cheap of carriage horses, which coach factory outlet de ‘s kate spade black friday seeks to ban. New michael kors oakley black friday Week is tory burch outlet online down, but we’ve still louis vuitton outlet online of the attendees’ louis vuitton handbags for coach outlet store online pleasure. Below, check out tory burch black friday street style michael kors black friday Six.true religion When true religion outlet its Fall 2014 ray ban sunglasses it was clear coach outlet store online on food. The coach factory online fully decked-out prada outlet supermarket, and the michael kors outlet US$69.99 and toms shoes outlet online were quirky and fantastic tory burch outlet to polo ralph lauren outlet expect.We’ve abercrombie and fitch – turns out, wearing michael kors outlet online and a watch on the same coach outlet wrist makes typing really prada handbags difficult. Here, we’ve rounded up tiffany black friday the coach handbags runways, just in polo ralph lauren for a abercrombie black friday refresh.We’re paring true religion jeans outlet things down celine handbags with this tiffany and co jewelry ($98). coach black friday so michael kors outlet online bags, shoes, and celine outlet out there, how could it not? But, in air jordans ensure ray-ban sunglasses shelling out hollister clothing store-earned supra.com for the coach factory outlet online, there valentino.com are a few tory burch outlet consideration. Namely?

Leave a Reply