Build Caching Local and Remote

A build cache is the memoization layer of a build system: every unit of work — compile a source file, link a binary, run a test — is treated as a pure function whose output depends only on its declared inputs, so the system can hash those inputs into a key, look the key up, and copy a previously-computed result instead of re-running the work. When the cache lives on the local disk it turns a rebuild of an unchanged tree into a few-millisecond no-op; when the cache is shared over the network, the result one engineer (or one CI runner) computed becomes an instant hit for everyone else on the team — “the outputs from one machine can be safely reused on another machine, which can make builds significantly faster” (Bazel, Remote caching). The entire scheme rests on one precondition — hermeticity, fully-declared inputs — because a cache key can only be trusted if it captures everything the action actually reads. This note is about the build system’s own action cache (Bazel, Gradle, Buck2, the Go toolchain), the layer beneath the CI platform.

Layer boundary — read this first

This is the build-system cache, keyed by a hash of an individual action’s inputs (source bytes + compiler + flags + dependency outputs). It is a different cache at a different layer from Pipeline Caching and Parallelism, which is the CI-platform dependency cache — a save/restore of a whole directory (node_modules, ~/.m2) keyed by a lockfile hash between otherwise-clean runner jobs. The CI cache restores one coarse blob at job start; the build cache resolves thousands of fine-grained actions during the build, each keyed independently. When a build tool with its own remote cache is present, it is finer-grained and more correct than the CI-level directory cache and often makes it redundant. The Go Build Cache is a concrete, single-machine instance of the mechanism taught here.

Mental Model — Memoize a Pure Function Over the Build Graph

The build system decomposes a build into a directed acyclic graph of actions. Each action — “compile package fmt”, “link the executable” — is, in principle, a deterministic function: the same source bytes, the same compiler binary, the same flags, and the same dependency outputs always produce byte-identical output. Memoizing a pure function requires a key derived from all of its inputs, and that key is the action key (Bazel calls the identifier of the cached result an action digest; the Go toolchain calls it an action ID). Caching is then just: compute the key, probe the cache, and on a hit skip the work entirely.

flowchart TD
    subgraph INPUTS["Declared inputs of ONE action"]
        SRC["source file<br/>content hashes"]
        CMD["command line<br/>+ arguments"]
        ENV["environment variables<br/>(only declared ones)"]
        TOOL["compiler / tool<br/>binary hash"]
        DEPS["upstream dependency<br/>output digests"]
    end
    INPUTS --> HASH["SHA-256 over the<br/>serialized action"]
    HASH --> KEY["Action key<br/>(action digest / action ID)"]
    KEY --> PROBE{"probe cache:<br/>does this key exist?"}
    PROBE -->|"HIT"| COPY["copy stored output<br/>— skip the work"]
    PROBE -->|"MISS"| RUN["execute the action"]
    RUN --> OUT["output bytes"]
    OUT --> STORE["store: key → output<br/>(and upload the bytes)"]
    COPY --> NEXT["output feeds<br/>downstream actions"]
    STORE --> NEXT

What it shows and the insight to take: the cache key is a hash of the whole action, not just the source file. Change the compiler version, add a flag, or change an upstream dependency’s output, and the key changes, and the stale entry is simply never consulted — correctness is by construction, not by an invalidation timer. The one thing that can break this is an input the system didn’t hash because it wasn’t declared. That is why hermeticity is the precondition: a build is hermetic when it “always returns the same output by isolating the build from changes to the host system,” treating tools as versioned inputs and refusing to read undeclared files (Bazel, Hermeticity). Hermeticity is exactly what makes an action cacheable — if the declared inputs are the only inputs, the key is trustworthy.

Two Stores — The Action Cache and the Content-Addressable Store

Mature build caches split into two logically distinct maps, cleanest in Bazel’s Remote Execution API (REAPI) which most tools now speak. Bazel’s own docs name them:

• The Action Cache (AC) is “a map of action hashes to action result metadata” (Bazel, Remote caching). Given an action key it returns an ActionResult — the manifest of what the action produced (which output files, their digests, exit code, and the digests of captured stdout/stderr), not the bytes themselves.
• The Content-Addressable Store (CAS) stores “the actual output files” (Bazel, Remote caching), each blob addressed purely by the hash of its content.

The separation is what makes the cache deduplicating. In REAPI a Digest is a {hash, size_bytes} pair — a lowercase-hex content hash plus the byte length (remote_execution.proto). Because a blob’s address is its content hash, two different actions that happen to emit identical bytes (the same header compiled by two projects with identical settings) resolve to the same CAS entry and share one physical copy. The AC is the thin index (key → manifest); the CAS is the fat, deduplicated content pool. The Go build cache implements the same two-level idea locally: an action ID (the key) maps to an output ID (the hash of the result), and the bytes are stored under the output ID so identical outputs dedup ([The Go Build Cache]] mirrors this design; cmd/go).

flowchart LR
    K["Action key<br/>sha256:9f2c…"] --> AC["Action Cache (AC)<br/>key → ActionResult manifest"]
    AC --> AR["ActionResult:<br/>output_files[]<br/>exit_code<br/>stdout/stderr digests"]
    AR -->|"each output referenced<br/>by content digest"| CAS["Content-Addressable Store (CAS)<br/>digest → raw bytes"]
    CAS --> B1["blob sha256:aa… (foo.o)"]
    CAS --> B2["blob sha256:bb… (bar.o)"]
    AR -. "two actions,<br/>identical output" .-> B1

What it shows and the insight to take: a cache hit is a two-hop resolution — probe the AC with the action key to get the manifest, then fetch each referenced output from the CAS by its content digest. The indirection is the whole point: the manifest is tiny, the bytes are shared, and the content-addressing means the CAS can never hold a wrong blob under a digest (the address is a checksum of the content). This is why people say the cache is “content-addressed all the way down.”

How the Key Is Actually Computed — Merkle Trees Over the Inputs

The subtle part is hashing the inputs, because one input is an entire directory tree of source and dependency files. REAPI encodes that tree as a Merkle tree. An action’s Action message carries a command_digest and an input_root_digest; the input root digest is “the digest of the root Directory for the input files,” and a Directory recursively “contains zero or more children FileNodes, DirectoryNodes” where each FileNode holds “the digest of the file’s content” and each DirectoryNode holds “the digest of the Directory object represented” (remote_execution.proto). Hashing bubbles up: change one leaf file’s bytes and its FileNode digest changes, which changes its parent Directory’s digest, which changes the grandparent’s, all the way to the input_root_digest. A single content edit at any depth propagates to the root hash.

The action key itself is then the digest of the serialized Action proto — which references the command_digest and input_root_digest. To make that stable across machines, REAPI mandates canonical serialization: “clients and servers MUST ensure that they serialize messages according to the following rules, even if there are alternate valid encodings,” and output paths in the Command “MUST be deduplicated and sorted lexicographically by code point” (proto). Determinism of the key encoding is as important as determinism of the build: if two machines serialized the same logical action differently, they would compute different keys and never share a hit.

A crucial correctness subtlety lives here too: the Action includes fields like timeout and do_not_cache, and “two Actions with different timeouts are different, even if they are otherwise identical” (proto) — so a result cached under a generous timeout is not silently served to a request that demanded a stricter one.

flowchart TD
    F1["src/a.c<br/>FileNode digest = H(bytes)"] --> D1["Directory 'src'<br/>digest = H(children digests)"]
    F2["src/b.h<br/>FileNode digest = H(bytes)"] --> D1
    D1 --> ROOT["input root Directory<br/>input_root_digest"]
    LIB["deps/libfoo<br/>DirectoryNode digest"] --> ROOT
    ROOT --> ACT["Action proto<br/>{command_digest,<br/>input_root_digest, timeout}"]
    CMDD["Command proto<br/>args, env, output_paths"] --> ACT
    ACT --> AK["action_digest = H(serialized Action)<br/>= the cache key"]

What it shows and the insight to take: the key is not a hash of a flat list of files — it is the hash of a tree of hashes. This buys two things at once: any change anywhere flips the root (correctness), and unchanged subtrees keep their digests so a remote executor can ask “which of these blobs do you already have?” and skip re-uploading the world (efficiency, via REAPI’s FindMissingBlobs). The Merkle structure is simultaneously the invalidation mechanism and the transfer-minimization mechanism.

Local Cache vs. Remote Cache — Restore-vs-Miss Walk-through

The local cache is a directory on the developer’s own machine (Bazel’s ~/.cache/bazel, Go’s $GOCACHE). It makes an engineer’s own incremental rebuilds instant — recompile after a one-line edit and only the actions whose keys changed re-run. But it is private: a teammate, or a fresh CI runner with an empty disk, gets nothing from it.

The remote cache is a shared network service (bazel-remote, BuildBuddy, Google Cloud Storage, an S3 bucket, or an nginx WebDAV server) that many machines read and write (Bazel, Remote caching). Now the first machine to build an action populates the shared AC+CAS, and every subsequent machine — other developers, every CI job — gets a hit. The lookup order is: check the local cache first, then the remote cache; on a remote hit download the output, and on a full miss execute locally and upload the new result (Bazel, Remote caching).

sequenceDiagram
    participant B as Build client
    participant L as Local cache (disk)
    participant AC as Remote Action Cache
    participant CAS as Remote CAS
    Note over B: compute action key = H(serialized Action)
    B->>L: probe local cache(key)
    alt Local HIT
        L-->>B: outputs on disk — done (fastest)
    else Local MISS
        B->>AC: GetActionResult(key)
        alt Remote HIT
            AC-->>B: ActionResult (manifest of output digests)
            B->>CAS: BatchReadBlobs(output digests)
            CAS-->>B: output bytes
            Note over B: materialize outputs, also seed local cache
        else Remote MISS
            AC-->>B: NOT_FOUND
            B->>B: execute the action locally
            B->>CAS: BatchUpdateBlobs(new output bytes)
            B->>AC: UpdateActionResult(key → manifest)
        end
    end

What it shows and the insight to take: a cold CI runner with no local state still skips almost all work if the remote cache is warm — a full clean build becomes a flurry of cheap GetActionResult probes and blob downloads rather than actual compilation. This is the headline win: cache results are shared across the whole team and the whole fleet, so redundant compilation happens once globally instead of once per machine. The UpdateActionResult / BatchUpdateBlobs on the miss path is also the exact point where write-access matters for security (below). Note the RPC names — GetActionResult, UpdateActionResult on the ActionCache service; BatchReadBlobs, BatchUpdateBlobs, FindMissingBlobs on the ContentAddressableStorage service — are the REAPI surface every compatible cache implements (proto).

Configuration — Wiring Up a Remote Cache

A minimal Bazel remote-cache setup is a handful of flags, usually in .bazelrc:

# .bazelrc — point Bazel at a shared remote cache
build --remote_cache=grpcs://cache.internal.example.com   # AC+CAS endpoint (gRPC/HTTP)
build --remote_upload_local_results=true   # write results computed locally back to the shared cache
build --experimental_guard_against_concurrent_changes  # detect inputs mutated mid-build (see below)
build --remote_timeout=60s                 # give up on a slow cache rather than stalling the build

Line by line: --remote_cache names the shared AC+CAS service (the same endpoint serves both maps). --remote_upload_local_results=true makes this machine a writer — it publishes results it computed so others hit them; setting it false makes a machine read-only (the common CI-agent / untrusted-worker posture). --experimental_guard_against_concurrent_changes addresses a real correctness hazard Bazel documents: “when an input file is modified during a build, Bazel might upload invalid results to the remote cache” (Bazel, Remote caching) — the flag makes Bazel re-verify input digests so a torn read is not immortalized in the shared cache. --remote_timeout bounds how long a cache round-trip may take so a degraded cache falls back to local execution instead of hanging.

The single most important operational knob is who may write. Bazel’s guidance is blunt: “Take care in who has the ability to write to the remote cache. You may want only your CI system to be able to write” (Bazel, Remote caching). The typical split: CI runs on trusted, pinned toolchains and has write access; developer laptops have read-only access. That asymmetry is the primary defense against cache poisoning.

Cache Invalidation and the Poisoning Risk

Invalidation is implicit and automatic — this is the feature people find counter-intuitive. There is no “clear the cache when X changes” rule, because a stale entry is never matched in the first place: if a relevant input changed, the recomputed key differs and the old entry is inert (it just ages out under the cache’s garbage collection). The entire correctness argument reduces to one sentence: the key must capture every input. Everything that can go wrong is a violation of that sentence.

The dangerous failure is cache poisoning: a wrong output stored under a valid key, so every machine that computes that key downloads corrupted results and trusts them. It has two root causes:

1. Non-hermeticity — an undeclared input. If an action secretly reads a file, an environment variable, the wall clock, or the network, that hidden input is not in the key. Two builds with different hidden inputs collide on the same key, and whichever ran first poisons the entry for the other. Bazel’s canonical example: because “only environment variables explicitly whitelisted via --action_env are included in an action definition,” two machines “with different $PATH variables” can share a cache key while the underlying tools differ (Bazel, Remote caching). This is why the fix for cache bugs is almost always more hermeticity (strict sandboxing to make undeclared reads fail), not cache-clearing.
2. A malicious or buggy writer. Anyone with write access can push arbitrary bytes under any key. A compromised CI job, or a developer laptop with mistaken write access running a patched compiler, can inject a backdoored .o that the whole fleet then trusts. Hence the least-privilege writer model above; the supply-chain framing of this exact attack — trusting build outputs — is owned by DevSecOps and Supply Chain Security MOC, cross-linked, not re-taught here.

flowchart TD
    A["Action reads an UNDECLARED input<br/>(host $PATH, clock, network file)"] --> B["input not in the action key"]
    B --> C{"two builds,<br/>different hidden input,<br/>SAME key"}
    C --> D["first build writes output X"]
    C --> E["second build should get Y<br/>but gets cached X"]
    E --> POISON["POISONED: every machine<br/>downloads wrong result<br/>under a valid-looking key"]
    F["Fix: strict sandboxing<br/>→ undeclared read FAILS the build<br/>→ input must be declared<br/>→ key now captures it"] --> B

What it shows and the insight to take: poisoning is not a cache bug — it is a hermeticity bug that the cache faithfully amplifies. A non-hermetic build is merely flaky on one machine; the moment you share its cache, that flakiness becomes a fleet-wide correctness failure served at the speed of a hit. The remote cache does not create the problem; it broadcasts it. This is the deepest reason the Continuous Integration and Delivery MOC treats hermeticity as the enabler of everything downstream.

Alternatives and Adjacent Layers

Cache layer	Key	Granularity	Shared?	Owned by
Build-system action cache (this note)	hash of one action’s declared inputs	per-action (thousands/build)	local + remote	Bazel/Go/Gradle/Buck2
CI dependency cache	lockfile hash	one directory blob per job	across CI jobs	Pipeline Caching and Parallelism
Container layer cache	Dockerfile instruction + context	per-image-layer	registry-shared	Container Image Layers and Copy-on-Write
Compiler-level cache (ccache, sccache)	preprocessed source + flags hash	per-compilation-unit	local (or remote for sccache)	standalone tools

When to choose which: if your build tool has a native content-addressed remote cache (Bazel, Buck2, Gradle build cache, Turborepo), prefer it — it is the finest-grained and most correct layer, and it often makes the coarse CI directory cache redundant. For polyglot or script-driven pipelines with no graph-aware build tool, the CI dependency cache is the pragmatic fallback. For a C/C++ codebase not on Bazel, ccache/sccache retrofit the same idea at the compiler-invocation level. These layers stack: a Bazel build inside a container inside a CI job can hit all three, each catching what the others miss.

Production Notes

The remote build cache is the load-bearing reason large monorepos build in minutes rather than hours: at Google-scale, the first engineer to touch a target pays the compile cost and every subsequent build across thousands of engineers and CI machines gets a hit, so the effective cost of a clean build collapses toward the cost of downloading results (Bazel, Remote caching). The two operational lessons teams learn the hard way both trace back to hermeticity. First, cache hit rate is a hermeticity metric in disguise — a mysteriously low hit rate almost always means some action embeds a non-deterministic input (a timestamp, an absolute path, a build ID) that flips its key every run; the fix is to make the action reproducible (see Build Reproducibility), not to enlarge the cache. Second, a poisoned shared cache is a production incident: because a bad entry is served fleet-wide at hit speed, the blast radius is every machine, and recovery means both purging the entry and fixing the hermeticity hole that let it in — clearing alone just re-poisons on the next run. The single defensive discipline that prevents both is strict sandboxing that turns an undeclared read into a hard build failure, so non-hermeticity is caught at author time instead of discovered as corruption in production (Bazel, Hermeticity). The natural next step once a shared cache exists is to move the execution itself onto the shared fleet — Remote Build Execution.

See Also

• Pipeline Caching and Parallelism — the CI-platform dependency cache (lockfile-hash-keyed directory restore); the other cache layer, coarser and job-scoped, distinct from this action cache
• Remote Build Execution — the natural extension: once results are shared over a cache, distribute the actions across a worker fleet; shares the same CAS + REAPI substrate
• Hermetic and Reproducible Builds — the precondition that makes an action’s key trustworthy; non-hermeticity is what causes cache poisoning
• Build Reproducibility — deterministic outputs keep cache keys stable; low hit rate is usually a reproducibility failure
• The Go Build Cache — a concrete single-machine instance of this exact mechanism (action ID → output ID, content-addressed)
• Container Image Layers and Copy-on-Write — the layer cache, a sibling content-addressed caching layer
• DevSecOps and Supply Chain Security MOC — owns the supply-chain threat model of trusting build outputs (the poisoning attack, verified provenance)
• Continuous Integration and Delivery MOC — parent MOC (§3 Build Systems)