vault

Building on Every Commit

16 min read

Building on Every Commit

Continuous Integration has one mechanical heart: a server watches the shared mainline, and every change that lands triggers an automated checkout, build, and test run whose result is reported back. Fowler states the rule directly: “Every time the mainline receives a commit, the CI service checks out the head of the mainline into an integration environment and performs a full build” (Fowler, Continuous Integration). This is what makes integration continuous rather than a periodic event — the build is not a nightly ritual or a pre-release gate but a reflex fired on each change, so that a defect is bound tightly to the small commit that introduced it. The consequence is an invariant the whole team relies on: the mainline is always green — always in a known-good, buildable, tested state — and the instant a commit turns it red, “nobody has a higher priority task than fixing the build.” This note traces that loop end to end: what the CI server does on each commit, how per-commit and per-pull-request triggers differ, why the always-green invariant is load-bearing, and what “stop the line” enforces.

Mental Model — A Reflex, Not a Ceremony

The mental shift that defines CI is temporal. In a pre-CI world, “integration” is a phase — a distinct, dreaded period late in a project where everyone’s long-diverged branches are merged and the resulting mess is stabilized. CI dissolves the phase into a reflex: because every commit is integrated and verified immediately, there is never a large accumulation of un-integrated work to reconcile. DORA frames the payoff as eliminating “long integration and stabilization phases … by integrating small batches of code frequently” (DORA, Trunk-Based Development).

flowchart LR
    subgraph BEFORE["Without CI — integration is a phase"]
        B1["Branch<br/>weeks"] --> B2["Branch<br/>weeks"] --> BIG["Big-bang<br/>merge + stabilize<br/>(painful, risky)"]
    end
    subgraph AFTER["With CI — integration is a reflex"]
        A1["commit"] --> AB1["build+test"]
        A2["commit"] --> AB2["build+test"]
        A3["commit"] --> AB3["build+test"]
        AB1 --> GREEN["Mainline<br/>always green"]
        AB2 --> GREEN
        AB3 --> GREEN
    end

What it shows and the insight to take: the top row is the failure mode CI was invented to kill — work diverges for weeks, then collides in one expensive merge whose bugs are impossible to attribute. The bottom row replaces that with a stream of small commits, each individually built and tested, converging on a mainline that never leaves a known-good state. The defining property is attribution: when a small commit breaks the build, you know exactly which change is responsible, because it is the only thing that changed. Building on every commit is what buys you that attribution — the smaller and more frequent the commit, the sharper the diagnosis.

The Core Loop: Checkout → Build → Test → Report

Strip a CI system down and the same four-step loop is running underneath every product — GitHub Actions, GitLab CI, Jenkins, Tekton. A commit lands; the server checks out exactly that commit; it builds; it runs the automated test suite; it reports a pass/fail status back to the place developers will see it.

sequenceDiagram
    participant Dev as Developer
    participant VCS as Git server (mainline)
    participant CI as CI server
    participant Run as Runner / agent
    participant UI as Status surface (PR, commit, chat)

    Dev->>VCS: push commit (SHA abc123)
    VCS-->>CI: webhook event (push / PR update)
    CI->>Run: dispatch job for SHA abc123
    Run->>VCS: checkout exact commit abc123
    Run->>Run: build (compile / package)
    Run->>Run: run self-testing suite
    alt all green
        Run-->>CI: exit 0
        CI-->>UI: commit status = success ✅
        CI-->>VCS: mark SHA abc123 mergeable
    else any red
        Run-->>CI: non-zero exit
        CI-->>UI: commit status = failure ❌
        CI-->>Dev: notify — build is broken
    end

What it shows and the insight to take: the loop is entirely automated and pinned to a specific commit SHA — this is the detail that makes the result meaningful. The build tests abc123, not “roughly the current state of things,” so a green status is a claim about an exact, reproducible snapshot of the code. Two properties are non-negotiable for the report to be worth anything: the build must be triggered automatically (a human deciding when to build reintroduces the delay and forgetfulness CI removes), and the suite must be self-testing — Fowler: “you aren’t really doing continuous integration unless you have self-testing code,” code you can run tests against and “be confident that, should the tests pass, your code is free of any substantial defects” (Fowler, Self-Testing Code). A green light from a suite nobody trusts reports nothing.

Walking the four steps concretely:

  1. Checkout the exact commit. The runner fetches the precise SHA that triggered the run into a clean integration environment. Cleanliness matters: leftover state from a previous build (“it works on the dirty workspace”) is how a build passes CI yet fails elsewhere. Fowler’s phrasing — “checks out the head of the mainline into an integration environment” — emphasizes integration environment, i.e. not the developer’s machine, so the build is validated somewhere neutral and reproducible.
  2. Build. Compile, package, produce the binaries. In a staged pipeline this is the commit build — “the build that’s needed when someone pushes commits to the mainline … must be done quickly” (Fowler, Continuous Integration) — and its output binaries are what later stages consume, per Build Once Promote the Artifact: “Usually the first stage of a deployment pipeline will do any compilation and provide binaries for later stages” (Fowler, Deployment Pipeline).
  3. Run the test suite. The self-testing suite runs against the freshly built artifact. In the commit build this is the fast tier (compile + unit tests); slower integration and end-to-end tiers run in secondary stages so the developer-facing loop stays short — the Fast Feedback and Build Times concern.
  4. Report status. The pass/fail is published where it will be acted on: as a commit status, a check on the pull request, a red bar on the dashboard, a message in chat. A result nobody sees is a build nobody trusts. This status is the atom that Pull Request Gates and Required Checks later turns into an enforced merge gate.

Per-Commit versus Per-Pull-Request Triggers

“Build on every commit” is the principle; which commits and which ref get built is a real design decision with security and correctness consequences. The two dominant trigger models — build-the-branch-push versus build-the-merge-of-a-pull-request — answer different questions.

A push trigger builds the commit as it landed on its branch. In GitHub Actions the push event “Runs your workflow when you push a commit or tag” and “is the event for building every commit automatically”; it “Runs on the actual branch being pushed to” (GitHub, Events that trigger workflows). This is the literal reading of “build on every commit” and is exactly right for the mainline: every push to trunk must be built to keep the always-green invariant honest.

A pull-request trigger answers a subtly different and more useful pre-merge question: would the mainline still be green if I merged this? GitHub’s pull_request event runs by default on activity types “opened, synchronize, or reopened” (synchronize fires on every new push to the PR’s head branch), and crucially it “Runs on a ‘merge branch’ (refs/pull/PULL_REQUEST_NUMBER/merge) simulating a merged state, allowing CI tests against the merged result” (GitHub docs). So a PR build does not test your branch in isolation; it tests your branch merged into the current target, which is what you actually care about before landing.

GitLab draws the same distinction with different names. A branch pipeline runs on a plain push; a merge request pipeline is triggered when you “Create a new merge request from a source branch that has one or more commits” or “Push a new commit to the source branch for a merge request,” but by default it runs “on the contents of the source branch only and ignore[s] the content of the target branch.” To test the actual merge, GitLab offers merged results pipelines — “a pipeline that tests the result of merging the source and target branches together” — and merge trains for serializing multiple merges (GitLab, Merge request pipelines).

AspectPer-commit (push) buildPer-PR buildMerge-result / merge-train build
Question answered“Does this commit build in isolation?”“Does my change build?” (GitLab MR: source-only)“Does trunk stay green after merge?”
Ref builtThe pushed branch tipGH: simulated merge ref; GL MR: source branch onlyActual merge of source + latest target
GitHub namepushpull_request (opened/synchronize/reopened)merge queue re-tests against latest target
GitLab namebranch pipelinemerge request pipelinemerged results pipeline / merge train
Primary useKeeping the mainline greenFast per-change feedback on a PRThe trustworthy pre-merge gate
Skew riskn/aCan pass in isolation, break after merge (semantic conflict)Closes the skew window — tests the real merge

Uncertain

Verify: whether GitHub’s pull_request event always builds the simulated merge ref vs the head ref, and the exact conditions under which the merge ref is unavailable (e.g. when the PR has merge conflicts, GitHub cannot compute refs/pull/N/merge and behavior differs). Reason: the fetched docs state the default merge-ref behavior but the edge cases (conflicting PRs, pull_request_target running on the base context) are nuanced and version-sensitive. To resolve: consult the current GitHub Actions events reference and the pull_request_target security guidance directly at authoring/use time. The core distinction (push builds the branch; PR builds the merge) is well-supported by the primary source. #uncertain

The security dimension is real and worth flagging: building code from a forked pull request means running untrusted code on your runner. GitHub’s pull_request_target variant “executes in the default branch context rather than merge commit context … Useful for privileged operations on fork submissions, though it carries security risks.” The threat model and mitigations belong to Least-Privilege Pipeline Runners and Secret Injection in Pipelines; the point here is only that “build on every commit” quietly includes “build on every stranger’s commit” when your repo is public.

The Mainline-Always-Green Invariant

The reason to build on every commit is to sustain one invariant: the mainline is always in a known-good, releasable state. Every developer who checks out trunk gets code that compiles and passes its tests; every commit builds on a solid base rather than an unknown one. This is what makes “keeping the software always releasable” — continuous delivery — even possible: you cannot release from a mainline whose health is unknown.

The invariant depends on frequency. DORA’s data ties always-green directly to how often people integrate: high performers “Merge branches to trunk at least once a day” and keep “three or fewer active branches” (DORA, Trunk-Based Development), and Fowler is categorical: “Every developer should commit to the mainline every day. In practice, those experienced with Continuous Integration integrate more frequently than that” (Fowler, Continuous Integration). The logic is mechanical: the longer a branch lives un-integrated, the more it diverges, the larger and riskier its eventual merge, and the harder it is to keep trunk green through that merge. Long-lived branches “require bigger and more complex merge events” and “frequently introduce bugs or regressions” (DORA). Building on every commit only protects the mainline if commits to the mainline are small and frequent — which is why this note and Trunk-Based Development are two halves of one practice.

Modern teams enforce the invariant rather than trusting it to discipline, by gating the merge itself. Branch protection with required status checks means “Required status checks must have a successful, skipped, or neutral status before collaborators can make changes to a protected branch” (GitHub, About protected branches). The full mechanics of gating are owned by Pull Request Gates and Required Checks; here note only why the gate exists: it turns “mainline should be green” from a hope into an enforced precondition of merging.

The Skew Problem and Merge Queues

There is a subtle gap the naive gate leaves open. A PR can be tested green against trunk-as-it-was-when-the-PR-opened, but trunk moves on underneath it as other PRs merge. Two individually-green PRs can break trunk when both land — a semantic conflict no textual merge conflict warns you about. GitHub’s “strict” required-status-check mode addresses this by demanding “The branch must be up to date with the base branch before merging,” at the cost of forcing a re-test (and re-push) whenever trunk advances (GitHub docs).

Merge queues automate the fix at scale: they ensure “the pull request’s changes pass all required status checks when applied to the latest version of the target branch and any pull requests already in the queue,” serializing merges and re-testing against accumulated changes before landing (GitHub docs) — the same role GitLab’s merge trains play. This is “build on every commit” taken to its logical end: build on every commit as it will actually land on trunk, in order, so the always-green invariant survives concurrent merges. The queue mechanics live in Pull Request Gates and Required Checks and Merge and Branching Strategies.

A Red Build Stops the Line

The invariant is only as strong as the team’s response to breaking it. Building on every commit detects a broken mainline; the discipline that repairs it is stop-the-line. Fowler is emphatic: “Should the integration build fail, then it needs to be fixed right away,” quoting Kent Beck — “nobody has a higher priority task than fixing the build” — and giving the default remedy: “Usually the best way to fix the build is to revert the latest commit from the mainline, taking the system back to the last-known good build” (Fowler, Continuous Integration).

stateDiagram-v2
    [*] --> Green
    Green --> Red: commit breaks the build
    Red --> Red: further commits blocked / discouraged
    Red --> Green: revert offending commit (fast path)
    Red --> Green: roll-forward fix (if trivial & quick)
    Green --> [*]
    note right of Red
        Top team priority.
        Do not pile new commits
        on a broken base.
    end note

What it shows and the insight to take: the mainline is a two-state machine, and the only healthy resting state is Green. Red is an alarm, not a status — it is transient by policy, cleared as fast as possible, ideally by reverting to the last-known-good build rather than debugging live on a broken trunk. The reason “stop the line” (the Andon-cord metaphor from the Toyota Production System) matters is compounding: every commit stacked on a red base builds on unknown foundations, muddies attribution (was it my change or the pre-existing breakage?), and grows the eventual cleanup. Reverting first, diagnosing after, keeps the loss bounded to one commit. The full discipline — blocking further merges, the Andon-cord culture, roll-forward vs revert — is owned by Stop the Line and the Broken Build Rule; the connection here is that building on every commit is the detector and stop-the-line is the actuator, and neither works without the other. Detection with no response is noise; response with no detection is impossible.

Common Misunderstandings

  • “CI means we run a build server.” No — CI is the practice of integrating and verifying every change on a shared mainline frequently. A build server that runs nightly, or that builds long-lived feature branches which never merge, is CI theater: it has the tool without the practice. If work isn’t integrated to trunk daily, you are not doing CI regardless of what runs (DORA, Continuous Integration).
  • “Green PR checks mean trunk stays green.” Only if the checks tested the merge, not the branch in isolation, and only if trunk hasn’t advanced since. Without strict checks or a merge queue, two green PRs can still break trunk via semantic conflict — the skew problem above.
  • “Build on every commit means test everything on every commit.” No — the commit build runs the fast tier and must stay quick; the full slow suite runs in secondary stages. Trying to run every end-to-end test on every commit is how the loop gets slow enough that people stop committing frequently, which breaks the whole practice (Fast Feedback and Build Times).
  • “A red build is one person’s problem.” It is the team’s problem, because a red mainline blocks everyone — nobody can safely build on it. That is the whole force of “nobody has a higher priority task than fixing the build.”

Production Notes

  • Revert-first is the humane default. Under time pressure, the instinct is to “just fix it forward.” Fowler’s guidance — revert to last-known-good — is usually right because it restores the invariant immediately and lets the author debug their reverted change at leisure, off the critical path. Roll-forward is fine only when the fix is trivial and faster than a revert.
  • Attribution decays with batch size. The diagnostic power of building on every commit is highest when commits are small. A build that goes red after a 2,000-line squash-merge tells you far less than one that goes red after a 20-line commit. This is the concrete, day-to-day reason Trunk-Based Development and small batches are inseparable from CI.
  • The status surface is part of the system. Where the result is reported determines whether it is acted on. A failing build that only turns a row red on a dashboard nobody watches is functionally undetected. Route failures to where the author is (PR check, chat mention) — the report step is not an afterthought.
  • Boundary reminder. This note owns the CI machinery — the per-commit loop, triggers, the green invariant. What happens after a green artifact exists (promotion, canary, rollout strategy, DORA delivery metrics) belongs to The Deployment Pipeline, Site Reliability Engineering MOC, and the delivery sections of the parent MOC. Build-on-commit ends at “we have a trustworthy green artifact for this SHA.”

See Also

Graph View