Starting with Firefox 68, the Linux builds shipped by Mozilla should be reproducible (it is not currently automatically validated that it definitely is, but 68.0 is). These builds are optimized with Profile Guided Optimization, and the profile data was not kept and published until recently, which is why they weren't reproducible until now.

The following instructions require running Docker on a Linux host (this may or may not work on a non-Linux host, I don't know what e.g. Docker for Mac does, and if the docker support in the mach command works with it). I'll try to make them generic enough that they may apply to any subsequent release of Firefox.

• Clone either the mozilla-unified or mozilla-release repository. You can use Mercurial or Git (with git-cinnabar), it doesn't matter.
• Checkout the FIREFOX_68_0_RELEASE tag and find out what its Mercurial changeset id is (it is 353628fec415324ca6aa333ab6c47d447ecc128e).
• Open the Taskcluster index tool in a browser tab.
• In the input field type or copy/paste gecko.v2.mozilla-release.shippable.revision.353628fec415324ca6aa333ab6c47d447ecc128e.firefox.linux64-opt and press the Enter key. (replace 353628fec415324ca6aa333ab6c47d447ecc128e with the right revision if you're trying for another release)
• This will fill the "Indexed Task" pane, where you will find a TaskId. Follow the link there, it will bring you to the corresponding Task Run Logs
• Switch to the Task Details
• Scroll down to the "Dependencies" list, and check the task name that begins with "build-docker-image". For the Firefox 68 build task, it is build-docker-image-debian7-amd64-build.
• Take that name, remove the "build-docker-image-" prefix, and run the following command, from inside the repository, to download the corresponding docker image:

  $ ./mach taskcluster-load-image debian7-amd64-build

  Obviously, replace debian7-amd64-build with whatever you found in the task dependencies. The image can also be built from the source tree, but this is out of scope for this post.

• The command output will give you a docker run -ti ... command to try. Run it. It will open a shell in the docker image.
• From the docker shell, run the following commands:

  $ echo no-api-key > /builds/mozilla-desktop-geoloc-api.key
  $ echo no-api-key > /builds/sb-gapi.data
  $ echo no-api-key > /builds/gls-gapi.data
  

  Or replace no-api-key with the actual keys if you have them.

• Back to the Task Details, check the env part of the "Payload". You'll need to export all these variables with the corresponding values. e.g.

  $ export EXTRA_MOZHARNESS_CONFIG='{"update_channel": "release", "mozconfig_variant": "release"}'
  $ export GECKO_BASE_REPOSITORY='https://hg.mozilla.org/mozilla-unified'
  $ export GECKO_HEAD_REPOSITORY='https://hg.mozilla.org/releases/mozilla-release'
  ...

• Set the missing TASKCLUSTER_ROOT_URL environment variable:

  $ export TASKCLUSTER_ROOT_URL='https://taskcluster.net'

• Change the value of MOZHARNESS_ACTIONS to:

  $ export MOZHARNESS_ACTIONS='build'

  The original value contains get-secrets, which will try to download from http://taskcluster/, which will fail with a DNS error, and check-test, which runs make check, which is not necessary to get a working Firefox.

• Take command part of the "Payload", and run that in the docker shell:

  $ /builds/worker/bin/run-task --gecko-checkout /builds/worker/workspace/build/src -- /builds/worker/workspace/build/src/taskcluster/scripts/builder/build-linux.sh

• Once the build is finished, in another terminal, check what the container id of your running docker container is, and extract the build artifact from there:

  $ docker ps
  CONTAINER ID        IMAGE                                                                                  COMMAND             CREATED             STATUS              PORTS               NAMES
  d234383ba9c7        debian7-amd64-build:be96d1b734e1a152a861ce786861fca6e70bcb996bf67347f5af4f146db157ec   "bash"              2 hours ago         Up 2 hours                              nifty_hermann
  $ docker cp d234383ba9c7:/builds/worker/artifacts/target.tar.bz2 .

  (replace d234383ba9c7 with your container id)

• Now you can exit the docker shell. That will remove the container.

After all the above, you can finally compare your target.tar.bz2 to the Linux64 Firefox 68 release. You will find a few inevitable differences:

• The .chk files will be different, because they are self-signatures for FIPS mode that are generated with one-time throw-away keys.
• The Firefox 68 release contains .sig files that your build won't contain. They are signature files, which aren't reproducible outside Mozilla automation for obvious reasons.
• Consequently, the precomplete file contains instructions for the .sig files in the Firefox 68 release that won't be in your build.
• The omni.ja files are different. If you extract them (they are uncompressed zip files with a few tweaks to the format), you'll see the only difference is in modules/AppConstants.jsm, for the three API keys you created a file for earlier.

Everything else is identical bit for bit.

All the above is a rather long list of manual steps. Ideally, most of it would be automated. We're not there yet. We only recently got to the point where the profile data is available to make it possible at all. In other words, this is a starting point. It's valuable to know it does work but requires manual steps and what those are.

It is also worth noting that while the above downloads and uses pre-built compilers and other tools, it is also possible to rebuild those, although they likely won't be bit-for-bit identical. But differences in those shouldn't incur differences in Firefox. Replacing the pre-built ones with ones you'd build yourself unfortunately currently requires some more manual work.

As for Windows and Mac builds, long story short, they are not reproducible as of writing. Mac builds are not optimized with PGO, but Windows builds are, and their profile data won't be available until Firefox 69. Both platforms require SDKs that Mozilla can't redistribute per their license (but are otherwise available for download from Microsoft or Apple, respectively), which makes the setup more complex. And in all likeliness, for both platforms, the toolchains are not deterministic yet (that's at least true for Mac). Also, binary signatures would need to be tripped off the executables and libraries before any comparison.

2019-07-11 11:31:22+0900

p.m.o

Responses are currently closed, but you can trackback from your own site.