vault

Build Reproducibility

12 min read

Build Reproducibility

A build is reproducible when independent runs of it produce bit-for-bit identical output artifacts. The canonical definition from the Reproducible Builds project: “a build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts” (reproducible-builds.org, Definition). The emphasis on bit-by-bit is deliberate — reproducibility means the bytes hash the same, not merely that the programs behave the same. This note owns the build-system mechanism of determinism: the concrete sources of non-determinism that make two builds of identical source differ, and the concrete fixes for each. It is the second half of the story begun by Hermetic and Reproducible Builds: hermeticity seals the build against the host, and reproducibility eliminates the non-determinism the build generates internally.

The relationship is worth stating sharply because it is routinely confused. Hermeticity is about inputs; reproducibility is about outputs. A hermetic build has fully-declared, host-isolated inputs — but it can still be non-reproducible, because the build process itself can inject variance that has nothing to do with the host: it can stamp the current time into a header, iterate a directory in filesystem order, embed its absolute working-directory path into debug symbols, or seed a data structure with a random value. Sealing the box (hermeticity) is necessary but not sufficient; you must also ensure that what happens inside the box is deterministic. That is what this note is about.

Uncertain

Verify: the anti-tamper / independent-rebuild verification angle — using reproducibility to detect a compromised build (rebuild independently, compare hashes, and if they match, trust that the binary corresponds to the source) — is deliberately not taught here. That security use is owned by the future DevSecOps note Reproducible Builds (a ghost link at time of writing). This note owns only the build-system mechanism of achieving determinism. Do not conflate the two; forward-link, do not pre-teach. #uncertain

Mental Model — Determinism as a Pure Function

The ideal is to make the build a pure function: artifact = f(source, environment, instructions), where f has no hidden dependence on when it runs, where it runs, or what order the filesystem happens to return files in. Every source of non-reproducibility is a hidden argument to f that you failed to control — usually the clock, the current directory, or the filesystem’s iteration order. Fixing reproducibility is the systematic work of finding each hidden argument and either removing it or pinning it to a fixed value.

flowchart TD
    SRC["source code (fixed)"] --> F
    ENV["build environment (declared)"] --> F
    INSTR["build instructions (fixed)"] --> F
    F["build process f()"] --> ART["artifact"]

    HIDDEN["HIDDEN ARGUMENTS<br/>(the enemies of reproducibility)"]
    CLOCK["wall clock → timestamps"] --> HIDDEN
    PATH["build path → embedded paths"] --> HIDDEN
    ORDER["readdir order → file ordering"] --> HIDDEN
    RAND["RNG / hash seed → nondeterminism"] --> HIDDEN
    HIDDEN -.->|leak into| F

    ART --> H1["hash A"]
    ART2["artifact (rebuilt)"] --> H2["hash B"]
    H1 --> CMP{"A == B ?"}
    H2 --> CMP
    CMP -->|yes| REPRO["reproducible ✓"]
    CMP -->|no| DIFF["diff the artifacts → find<br/>the leaked hidden argument"]

What it shows and the insight to take: the build should be a pure function of the three declared inputs on the left, but four hidden arguments (clock, path, ordering, randomness) leak in and make f impure. Reproducibility work is the loop on the right: rebuild, compare hashes, and when they differ, diff the two artifacts to identify which hidden argument leaked — then eliminate it. The insight: non-reproducibility is always a specific, findable leak, not vague flakiness; the diff points straight at it.

The Sources of Non-Determinism and Their Fixes

The Reproducible Builds project documents a well-enumerated catalogue of variance sources (reproducible-builds.org, Docs). The four dominant ones account for the vast majority of real-world irreproducibility.

flowchart LR
    subgraph SOURCES["Sources of non-determinism"]
        T["Timestamps"]
        P["Build path"]
        O["File / input ordering"]
        R["Randomness & hash seeds"]
        L["Locale / timezone"]
        A["Archive metadata (uid/gid/perms)"]
    end
    T --> FT["SOURCE_DATE_EPOCH<br/>gzip -n · strip-nondeterminism"]
    P --> FP["-ffile-prefix-map<br/>BUILD_PATH_PREFIX_MAP"]
    O --> FO["sort inputs<br/>LC_ALL=C sort · --sort=name"]
    R --> FR["fix/seed the RNG<br/>PYTHONHASHSEED=0"]
    L --> FL["LC_ALL=C · TZ=UTC"]
    A --> FA["--owner=0 --group=0<br/>clamp mtime, normalize perms"]

Caption: each variance source on the left has a concrete, well-known fix on the right. The insight: reproducibility is not a mysterious property — it is a checklist of a half-dozen leaks, each with a standard countermeasure.

1. Timestamps — the biggest offender

Timestamps are “the biggest source of reproducibility issues” because “many build tools record the current date and time. The filesystem does, and most archive formats will happily record modification times” (reproducible-builds.org, Timestamps). They leak in at many points: modification times embedded in tar, zip, ar, and gzip archives; the __DATE__/__TIME__ C preprocessor macros; documentation generators stamping “built on …”; and timestamps written into the software itself.

The standard fix is SOURCE_DATE_EPOCH, an environment variable that “specifies the last modification of something, usually the source code, measured in the number [of] seconds since the Unix epoch” (reproducible-builds.org, SOURCE_DATE_EPOCH). The rule is that a tool “will use its value … instead of the current date and time (when set)” (reproducible-builds.org, Timestamps). The guiding philosophy: “use a date that is relevant to the source code instead of the build: old software can always be built later” (reproducible-builds.org, Timestamps) — so the value is typically the commit date of the source, not the moment of compilation.

The spec is precise about tool behavior: read the variable and use it if present; fall back to the current time if unset; apply UTC-based conversions consistently; and respect the value exactly (reproducible-builds.org, SOURCE_DATE_EPOCH). There is one notable clamping rule: for zip files, implementations use max(315532800, SOURCE_DATE_EPOCH) because the ZIP format cannot represent timestamps before 1980 (315532800 is 1980-01-01 UTC in Unix seconds) (reproducible-builds.org, SOURCE_DATE_EPOCH). Support is broad — CMake, Docker Buildx, GCC, Maven, Sphinx and many others honor it (reproducible-builds.org, SOURCE_DATE_EPOCH).

# Derive the epoch from the last git commit, then build with it pinned.
export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)   # commit time, Unix seconds
#                          └─ %ct = committer date as a Unix timestamp
 
# gzip embeds an mtime in its header by default; -n suppresses it.
gzip -n data.txt                # -n: do NOT store the original name/timestamp
 
# tar records each member's mtime; clamp it to the epoch and normalize ownership.
tar --sort=name \               # deterministic member order (see §3)
    --mtime="@${SOURCE_DATE_EPOCH}" \  # every member gets the pinned time
    --owner=0 --group=0 --numeric-owner \  # normalize uid/gid (see §5)
    -cf archive.tar ./out

Line-by-line: %ct extracts the committer timestamp as raw Unix seconds — the “relevant to the source” date. gzip -n is essential because gzip otherwise stamps the current time into its header, making the compressed output differ every run even from identical input. In tar, --mtime="@N" overrides every member’s modification time with the pinned epoch (the @ prefix means “Unix seconds”), and --owner=0 --group=0 --numeric-owner strips the building user’s identity out of the archive. Where a tool cannot honor SOURCE_DATE_EPOCH, the fallback is post-processing normalization with a tool like strip-nondeterminism, which will “remove the timestamps entirely or normalize them to a predetermined date” (reproducible-builds.org, Timestamps). A cruder fallback, libfaketime (intercepting time calls via LD_PRELOAD), is explicitly warned against because it “will go wrong” when a build relies on time differences (reproducible-builds.org, Timestamps).

2. Build path — the absolute directory leaks into binaries

Compilers “write the path of the source in the debug information in order to locate the associated source files” (reproducible-builds.org, Build path), so a binary built in /home/alice/project differs from one built in /build/project even with identical source. The two leak points are DWARF debug symbols (which capture source file paths) and the __FILE__ macro (which expands to the absolute path, commonly inside assert), and some packages additionally “save the compile options in the build output” (reproducible-builds.org, Build path).

The fix is compiler prefix-mapping (GCC 8+/Clang 10+): -fdebug-prefix-map=OLD=NEW “strip[s] directory prefixes from debug info,” -fmacro-prefix-map=OLD=NEW addresses the __FILE__ leak, and -ffile-prefix-map=OLD=NEW “is an alias for both” (reproducible-builds.org, Build path). Debian ships this as the fixfilepath build flag (reproducible-builds.org, Build path). A cross-tool alternative under discussion is BUILD_PATH_PREFIX_MAP (reproducible-builds.org, Docs). The docs note that post-processing debug symbols is problematic, “making compiler-level solutions preferable” (reproducible-builds.org, Build path).

gcc -ffile-prefix-map=$(pwd)=. -g -c foo.c -o foo.o
#      └─ rewrites the current build dir to "." in BOTH
#         debug info (DWARF) and __FILE__ macro expansions

3. File and input ordering — filesystem order is not stable

“Most filesystems do not guarantee that listing files in a directory always results in the same order” (reproducible-builds.org, Stable inputs). So any build step that globs a directory and processes files in readdir order — linking *.o, adding files to an archive, concatenating sources — can order its output differently on two machines. Worse, sorting can be locale-dependent: GNU Make’s $(wildcard *.c) “sorts according to the current locale,” and “different locales have different orders of e.g. uppercase characters relative to lowercase characters” (reproducible-builds.org, Stable inputs).

The fix is explicit, locale-independent sorting: either list inputs manually, or sort with a fixed locale. Use the sort function that “does not take locale into account,” or force LC_ALL=C sort for command-line tools, and pass ordering flags to archivers (reproducible-builds.org, Stable inputs).

# Non-deterministic: filesystem order, locale-dependent
tar -cf out.tar $(find . -name '*.txt')
 
# Deterministic: fixed locale + explicit sort + tar's own sort
find . -name '*.txt' | LC_ALL=C sort | tar --sort=name -cf out.tar -T -
#                       └─ byte-order sort, locale-independent
#                                              └─ tar orders members by name too

4. Randomness, locale, and archive metadata

Remaining leaks are smaller but real. Randomness: any RNG, hash-map iteration order (e.g. Python dict/set ordering under hash randomization — pin PYTHONHASHSEED=0), or temporary filenames with random suffixes must be seeded or removed — “eliminate non-deterministic operations from build processes” (reproducible-builds.org, Docs). Locale/timezone: “standardize locale and timezone settings during builds” — conventionally LC_ALL=C and TZ=UTC — since locale changes collation and message text, and timezone changes any formatted date. Archive metadata: beyond mtimes, archives record uid/gid, permissions, and sometimes device numbers; normalize them (--owner=0 --group=0, fixed umask). Nix illustrates the endgame of metadata normalization: it “sets the last-modified timestamp on all files in the build result to 1 (00:00:01 1/1/1970 UTC)” and normalizes permissions (nix.dev, Derivations) — every file in a Nix store path has the same fixed timestamp, sidestepping the whole timestamp problem by fiat.

Verifying Reproducibility

Reproducibility is claimed by declaring the environment and proven by rebuilding and comparing. Verification “occurs through bit-by-bit comparison using cryptographically secure hash functions,” ensuring “exact matching — not functional equivalence” (reproducible-builds.org, Definition). Note the scope: artifacts to compare are “executables and packages but exclude ancillary outputs like build logs” (reproducible-builds.org, Definition).

sequenceDiagram
    participant S as Source @ rev + declared env
    participant B1 as Builder 1
    participant B2 as Builder 2 (independent)
    participant D as diffoscope

    S->>B1: build (SOURCE_DATE_EPOCH, LC_ALL=C, prefix-map)
    S->>B2: build (same declared env, different machine/time)
    B1-->>D: artifact_1  (sha256 = X)
    B2-->>D: artifact_2  (sha256 = Y)
    alt X == Y
        D-->>S: REPRODUCIBLE ✓
    else X != Y
        D-->>S: diffoscope drills into archives/binaries<br/>→ pinpoints the leaked hidden argument
    end

Caption: the workflow that turns reproducibility from an aspiration into a checked property. When hashes differ, diffoscope recursively unpacks both artifacts (archives within archives, ELF sections, PE resources) and shows the exact differing bytes, which almost always reveals a timestamp, an embedded path, or a reordered member. The insight: reproducibility is falsifiable and debuggable — a hash mismatch is a lead, and diffoscope follows it to the root cause.

The Reproducible Builds workflow requires declaring “relevant build environment attributes, build instructions, source code specifications, and expected reproducible artifacts,” while preferring to “reduce this set of attributes” (reproducible-builds.org, Definition) — the fewer environment attributes the output depends on, the more robustly reproducible it is.

Why Reproducibility Underpins Verifiable Provenance

Reproducibility is what makes a provenance claim meaningful. A provenance attestation says “this artifact was built from this source with this toolchain.” That statement is only checkable if rebuilding from the stated source and toolchain yields the same bytes — otherwise there is no way to confirm the artifact actually corresponds to the claimed inputs. Bit-for-bit reproducibility is therefore the mechanism that lets provenance be independently confirmed rather than merely asserted. This note owns that build-mechanism connection; the strategy of using independent rebuilds to detect tampering, and the supply-chain frameworks (SLSA build levels, in-toto attestations) built on top, are owned by DevSecOps and Supply Chain Security MOC and the forthcoming Reproducible Builds security note — cross-linked, not re-taught here.

Common Misunderstandings

  • “Hermetic implies reproducible.” No — hermeticity controls inputs, but a sealed build can still stamp the clock or iterate a directory in filesystem order. Reproducibility is a separate discipline layered on top (Hermetic and Reproducible Builds).
  • “Reproducible means the same behavior.” No — it means bit-identical bytes. Two functionally-equivalent binaries with different embedded timestamps are not reproducible (reproducible-builds.org, Definition).
  • “Set the clock and you’re done.” Timestamps are the biggest single source, but path, ordering, randomness, locale, and archive metadata each independently break bit-identity.
  • “libfaketime fixes timestamps.” It is an unreliable last resort that “will go wrong” when the build depends on elapsed time (reproducible-builds.org, Timestamps); prefer SOURCE_DATE_EPOCH or strip-nondeterminism.

See Also

Graph View

Backlinks