vault

Pipeline Reliability and Flakiness

17 min read

Pipeline Reliability and Flakiness

A Continuous Integration / Continuous Delivery (CI/CD) pipeline is not a script that runs “off to the side” — it is a production system in its own right, with an availability target, a latency budget, upstream dependencies, and its own failure modes. When it is slow or unreliable, engineers route around it: they merge on a re-run, they disable a “flaky” gate, they stop trusting the green checkmark. This note is about the reliability of the pipeline and its infrastructure — the runners, the caches, the registries, the network, the third-party actions — and the discipline of making pipeline stages safe to retry. It is deliberately distinct from Flaky Test Management, which is about non-deterministic tests inside a build; here the flakiness lives one layer down, in the machinery that runs those tests. The two are cross-linked because they present the same symptom — an intermittent red that a re-run turns green — but the root causes, the owners, and the fixes are different.

Boundary with Flaky Test Management

A flaky test fails because the test code or the system under test is non-deterministic (an unmocked clock, a race, a shared fixture). A flaky pipeline fails because the execution environment is non-deterministic (a runner ran out of disk, a registry pull timed out, a spot instance was reclaimed mid-job, a transitive apt mirror 503’d). Same red X, different layer. This note owns the second layer. When a re-run turns a build green, the very first triage question is which layer failed — because “quarantine the test” and “add disk to the runner” are opposite fixes.

Mental Model — The Pipeline Is a Production System

The single most useful reframing is to stop thinking of the pipeline as a build script and start thinking of it as a distributed service with a Service Level Objective (SLO). It has requests (pipeline runs triggered by commits), a success rate (what fraction go green on the first attempt), a latency distribution (P50/P95 wall-clock from push to green), and a dependency graph of things that can fail underneath it. Every property you would demand of a user-facing service — observability, retries with backoff, idempotency, capacity planning, dependency isolation — applies to the pipeline, because a change cannot reach production any faster or more reliably than the pipeline that carries it.

flowchart TD
    TRIGGER["Commit / PR push<br/>(the 'request')"] --> ORCH["Orchestrator<br/>schedules jobs onto runners"]
    ORCH --> RUNNER["Runner / Agent / Executor<br/>(the compute)"]

    RUNNER --> DEP1["Package registries<br/>npm · PyPI · apt · Maven"]
    RUNNER --> DEP2["Container/artifact registry<br/>OCI pull/push"]
    RUNNER --> DEP3["Build/test caches<br/>restore + save"]
    RUNNER --> DEP4["Third-party actions/plugins<br/>external code"]
    RUNNER --> DEP5["Cloud APIs / secrets<br/>OIDC · deploy targets"]

    DEP1 & DEP2 & DEP3 & DEP4 & DEP5 -. "any one can<br/>fail transiently" .-> RED["Red build<br/>NOT a code defect"]

    RUNNER --> GREEN["Green build<br/>trustworthy signal"]

    style RED fill:#e74c3c,color:#fff
    style GREEN fill:#2ecc71,color:#fff

What it shows and the insight to take: a pipeline run is a fan-out of network calls to systems the pipeline does not own. The green path is a straight line, but every dashed edge is an independent point of failure whose combined availability multiplies. If a single run touches five dependencies each at 99.9% availability, the run’s infrastructure ceiling is roughly 0.999⁵ ≈ 99.5% — meaning about 1 in 200 runs fails for reasons that have nothing to do with the code being tested. That is the floor of pipeline flakiness before a single flaky test enters the picture, and it is why “just re-run it” becomes a culture: statistically, re-running often works, which is exactly what makes it corrosive.

The Two Layers of Flakiness — Infrastructure vs. Test

The word “flaky” is overloaded, and conflating the two layers is the most common diagnostic mistake. A precise taxonomy is the prerequisite to a fix.

flowchart TD
    FLAKE["Intermittent red<br/>(re-run turns it green)"] --> Q{"Where did it fail?"}

    Q -->|"Inside the test process"| TEST["TEST FLAKINESS<br/>owned by Flaky Test Management"]
    Q -->|"In the machinery around it"| INFRA["PIPELINE FLAKINESS<br/>owned by THIS note"]

    TEST --> T1["race conditions · timing"]
    TEST --> T2["shared/order-dependent state"]
    TEST --> T3["unmocked time/network/random"]

    INFRA --> I1["runner OOM / disk-full / evicted"]
    INFRA --> I2["registry / mirror timeout · 5xx"]
    INFRA --> I3["cache corruption / miss storm"]
    INFRA --> I4["network partition · DNS · TLS"]
    INFRA --> I5["orchestrator throttle / rate limit"]
    INFRA --> I6["spot/preemptible reclamation"]

    style TEST fill:#3498db,color:#fff
    style INFRA fill:#e67e22,color:#fff

What it shows and the insight to take: the branch point is the boundary of the test process. Everything to the left is a determinism defect in code and belongs to Flaky Test Management — the fix is deflaking, quarantine, and flakiness scoring. Everything to the right is an environment defect and belongs here — the fix is capacity, isolation, retries, and dependency hardening. The insight is that you cannot fix what you have not attributed: a mature pipeline captures enough signal (exit codes, structured logs, resource metrics) to route a red build to the correct layer automatically, rather than dumping every intermittent failure into one undifferentiated “flaky” bucket that trains everyone to smash the re-run button.

Infrastructure failure modes in detail

  • Runner resource exhaustion. A job that fills the disk (Docker layer bloat, uncleaned artifacts), exhausts memory (an OOM-killed test process reports as a mysterious exit code 137), or saturates CPU on a shared runner. On GitHub-hosted runners a job is hard-capped at 6 hours; on self-hosted runners at 5 days, after which it is killed regardless of progress (GitHub Actions limits). A job that hangs on a lock and hits the ceiling looks identical to a real timeout.
  • Registry and mirror flakiness. Every npm install, pip install, apt-get, or docker pull is a live dependency on infrastructure you do not control. A public mirror returning a transient 503, or a rate-limited registry, fails the job with no code change involved.
  • Cache corruption and cache-miss storms. A partially-written or key-collided cache can poison a build (stale compiled objects, mismatched lockfile state), while a mass cache invalidation (a lockfile bump) can suddenly send every job to the slow cold-build path at once, spiking latency and load.
  • Orchestrator throttling. The control plane itself has limits: GitHub caps trigger events at 1,500 per 10 seconds per repository and queued runs at 500 per 10 seconds, and self-hosted jobs that sit unclaimed for 24 hours are auto-cancelled (GitHub Actions limits). Hit these during a merge storm and jobs fail to even start.
  • Preemption. Spot / preemptible instances and autoscaled ephemeral runners can be reclaimed mid-job, terminating the work with no fault of the code.

Retry and Idempotency of Pipeline Stages

If the infrastructure layer has an irreducible transient failure rate, the correct engineering response is not to pretend it does not exist — it is to make stages safe to retry and then retry them selectively. The two ideas are inseparable: a retry is only safe if the stage is idempotent, and blind retries of the wrong stage are how a transient failure becomes a duplicated deploy or a corrupted artifact.

Selective, condition-scoped retries

Blanket “retry the whole pipeline 3×” is an anti-pattern — it masks real failures and wastes compute. Mature CI systems let you scope retries to the failure class. GitLab CI is the clearest example: retry:max sets the attempt count (0–2), and retry:when restricts which failures are eligible (GitLab retry). The distinction between an infrastructure failure and a script failure is encoded directly in these values.

test-job:
  script: ./run-tests.sh
  retry:
    max: 2                        # up to 2 extra attempts (3 total)
    when:                         # ONLY retry these failure classes
      - runner_system_failure     # the runner itself broke — infra, safe to retry
      - stuck_or_timeout_failure  # job hung / timed out — often infra
      - api_failure               # GitLab API hiccup — infra
      - scheduler_failure         # scheduler couldn't assign a runner — infra
    # NOTE: script_failure is deliberately OMITTED — a real test failure
    # must NOT be silently retried into a green, or the gate is worthless.

Line-by-line: max: 2 bounds the blast radius of retrying; when: is the load-bearing part — it lists only infrastructure failure classes (runner_system_failure, stuck_or_timeout_failure, api_failure, scheduler_failure). The critical omission is script_failure: retrying a genuine assertion failure is exactly the “re-run culture” that lets a real defect slip through. GitLab additionally supports retry:exit_codes to retry on specific process exit codes (GitLab retry). The principle generalizes: retry the environment, never the verdict.

Uncertain

Verify: the exact, current set of retry:when values (this note lists runner_system_failure, stuck_or_timeout_failure, api_failure, scheduler_failure, script_failure). Reason: the fetched GitLab reference page was truncated and returned only three values explicitly (runner_system_failure, stuck_or_timeout_failure, script_failure); the others are asserted from prior knowledge of the keyword. To resolve: re-fetch the full retry:when table at authoring time and confirm the complete enum (GitLab has added values like unknown_failure, job_execution_timeout, unmet_prerequisites over time). #uncertain

Idempotency is the precondition for a safe retry

A stage is idempotent if running it twice produces the same end state as running it once. Read-mostly stages (compile, test) are naturally close to idempotent. The dangerous ones are stages with external side effects — pushing an artifact, cutting a release tag, running a database migration, deploying. Retrying those without idempotency guarantees produces duplicate publishes, doubled deploys, or half-applied migrations.

stateDiagram-v2
    [*] --> Running
    Running --> Succeeded: exit 0
    Running --> InfraFailed: runner/registry/network error
    Running --> ScriptFailed: real assertion/build failure

    InfraFailed --> Retryable: is the stage idempotent?
    Retryable --> Running: yes → retry with backoff
    Retryable --> Halt: no → fail closed, alert human

    ScriptFailed --> Halt: never auto-retry a verdict

    Succeeded --> [*]
    Halt --> [*]

    note right of Retryable
      Idempotency gates the retry:
      push-by-digest = safe (same bytes)
      re-tag mutable = unsafe
      migration = needs idempotent DDL
      deploy = needs converge-to-state
    end note

What it shows and the insight to take: the state machine branches a failure into two fundamentally different classes. An InfraFailed transition is eligible for a retry, but only actually retried if the stage is idempotent; a ScriptFailed transition must always halt. The note on Retryable is the crux: build-once-promote (see Build Once Promote the Artifact) makes delivery naturally idempotent because you push and promote an immutable digest — pushing the same sha256:… twice is a no-op, whereas re-tagging a mutable tag is not. GitOps pull-delivery is idempotent by construction: a reconciler converges the cluster to a declared state, so running it N times lands on the same state (OpenGitOps principles). Push-based imperative deploys must be engineered to that same standard.

StageNaturally idempotent?How to make retry safe
Compile / unit testYes (read-mostly)Just retry on infra failure
Restore cacheYes (fallback to cold build)Treat corrupt cache as miss
Push image by digestYes (content-addressed)Same bytes → same digest → no-op
Re-tag a mutable tagNoAvoid; prefer immutable/digest refs
DB migrationNo by defaultIdempotent DDL (IF NOT EXISTS), versioned migrations
Deploy (push/imperative)No by defaultMake it converge-to-desired-state, not “apply a delta”
Deploy (GitOps/pull)Yes (reconciliation)Reconciler re-runs converge to same state

Self-Hosted Runner Reliability

The runner (also called an agent or executor — see Runners Agents and Executors) is the single most common source of pipeline flakiness once you leave vendor-hosted compute, because now you own its reliability. The decisive design choice is ephemeral vs. persistent runners.

A persistent runner reuses the same machine across many jobs. It is faster (warm caches, pre-pulled images) but accumulates state — leftover files, filled disks, mutated global config, secrets from a prior job — so job N can be broken or poisoned by job N−1. An ephemeral runner is created fresh, runs exactly one job, and is destroyed. It trades cold-start cost for a clean, isolated, reproducible environment and is the strongly preferred model for both reliability and security (a compromised job cannot persist).

GitHub’s Actions Runner Controller (ARC) is the reference implementation of ephemeral autoscaling: a Kubernetes operator that scales self-hosted runners as container pods on demand (ARC docs).

sequenceDiagram
    participant GH as GitHub Actions Service
    participant L as Runner ScaleSet Listener
    participant C as EphemeralRunner Controller
    participant P as Runner Pod (one job, then gone)

    L->>GH: HTTPS long-poll (hold ~open)
    GH-->>L: job available (matching labels)
    L->>C: patch RunnerSet desired replicas +1
    C->>GH: request Just-in-Time (JIT) config token
    C->>P: create pod (retry up to 5× on failure)
    P->>GH: register with JIT token, claim the one job
    P->>P: run steps
    P-->>GH: report status
    Note over P: pod destroyed — no state survives
    Note over GH: unaccepted job unassigned after 24h

What it shows and the insight to take: ARC never exposes an inbound port — the Listener holds an outbound long-poll to GitHub, so no firewall hole is needed. When a job appears, the controller mints a Just-in-Time (JIT) registration token and creates a pod that runs exactly one job and is then discarded; pod creation itself retries up to 5 attempts (ARC docs). The reliability wins are structural: no cross-job contamination (each pod is fresh), no leaked long-lived registration credential (JIT tokens are single-use), and autoscaling absorbs load spikes instead of queueing behind a fixed pool. Buildkite’s agent model reaches the same outbound-poll design from a different direction — its agents poll the Buildkite API over HTTPS with “no need to forward ports or provide incoming firewall access,” and route jobs to the agent that most recently completed one, exploiting warm caches (Buildkite agent).

The reliability checklist for self-hosted runners falls out of this model:

  • Prefer ephemeral, single-use runners. Eliminates the entire class of “job poisoned by predecessor” flakiness.
  • Autoscale on queue depth. A fixed pool either wastes money idle or queues (and hits the 24-hour unclaimed-job timeout) under load.
  • Isolate jobs. One pod/VM per job; never let two untrusted jobs share a kernel namespace or a working directory.
  • Cap and reclaim resources. Enforce disk/memory limits and clean up between (or destroy after) jobs, so an OOM in one job does not cascade.
  • Pre-warm images/caches on the node to cut cold-start latency without sacrificing job-level isolation.

The Cost of a Slow or Unreliable Pipeline as a Delivery Risk

The reason any of this matters is economic and behavioral, not aesthetic. A slow or unreliable pipeline is a direct delivery risk because it changes how humans behave.

Slowness compounds. CI value comes from a short feedback loop (see Fast Feedback and Build Times). When a pipeline is slow, developers batch changes to amortize the wait, which produces bigger, riskier merges and makes failures harder to bisect — the exact opposite of Continuous Integration’s small-batch premise. Context-switching cost is real: a 40-minute pipeline means the developer has moved on and must re-page-in the change when it finally fails.

Unreliability destroys the signal. The most insidious cost is not the wasted minutes but the erosion of trust. Once “red sometimes means nothing,” engineers learn to re-run reflexively, then to merge past a red gate “because it’s probably just flaky,” and eventually a real failure rides through on that assumption. A gate nobody trusts is worse than no gate, because it still costs time while providing no safety.

This connects directly to delivery-performance measurement, owned by Site Reliability Engineering MOC. The DORA program frames delivery health as throughput (deployment frequency, change lead time) balanced against stability (change fail rate, failed deployment recovery time), and its current model adds a fifth metric, deployment rework rate — the share of deploys that are unplanned reactions to a production incident (DORA four keys). A flaky pipeline degrades every one of these: it lengthens lead time (re-runs and blocked merges), depresses deployment frequency (people ship less often to avoid the pain), and — when a masked flake was actually a real defect — inflates change fail rate and rework. Pipeline reliability is thus a leading indicator of delivery performance, not a separate concern.

Boundary — this note vs. SRE

The measurement of delivery performance (the DORA metrics themselves, error budgets, MTTR) is owned by DORA Metrics and Delivery Performance and Site Reliability Engineering MOC. This note owns the pipeline-machinery causes that move those numbers. Teach the reliability mechanism here; link the metric there.

Failure Modes and How to Diagnose Them

SymptomLikely layerFirst diagnosticFix direction
Exit code 137, job killedInfra (OOM)Runner memory metricsBigger runner / cap memory / fix leak
“No space left on device”Infra (disk)Runner disk usage over timeClean between jobs / ephemeral runners
Times out at exactly the capInfra or hangIs it a lock/wait? vs. genuinely slowFix hang / raise timeout / shard work
npm/pip/docker pull 5xxInfra (registry)Is the failure in a fetch step?Retry-with-backoff / mirror / vendored deps
Passes on re-run, no code changeInfra or testAttribute by failure class/exit codeRoute to correct layer (do NOT just re-run)
Fails only under load / merge stormInfra (throttle/queue)Orchestrator rate-limit logsAutoscale runners / respect API limits
Green locally, red in CI onlyInfra (env drift)Diff CI env vs. localHermetic build (see Hermetic and Reproducible Builds)

The meta-diagnostic discipline: every “just re-run it” is an unlogged incident. A pipeline that silently absorbs transient failures gives no data to fix them. Instrument retries — count them, tag them by failure class, alert when the infra retry rate crosses a threshold — so that flakiness becomes a measured, budgeted quantity rather than folklore. This is the same “toil is a signal, not a fact of life” stance SRE takes toward operational load.

Production Notes

  • Attribute before you retry. The highest-leverage investment is failure classification — capturing exit codes, resource metrics, and structured logs so a red build is automatically routed to “infra” or “test.” Without it, both a real bug and a reclaimed spot instance land in the same “flaky” bucket and get the same (wrong) reflexive re-run.
  • Ephemeral runners are the default answer. Nearly every “works on the second try” runner problem — leftover state, filled disk, leaked secret, poisoned cache — is designed out by one-job-then-destroy runners. ARC and Buildkite’s agent-stack-for-Kubernetes both exist to make this cheap (ARC docs; Buildkite agent).
  • Budget pipeline availability like a service. Set an explicit target (e.g., “≥98% of runs must be green-or-red for code reasons, not infra”) and track the infra-failure rate against it. When it slips, that is a reliability incident with an owner, not a background annoyance.
  • Make the deploy stage idempotent first. Before you enable any retry on a stage with side effects, prove it is idempotent — push-by-digest, converge-to-state, idempotent migrations. GitOps pull-reconciliation gives this for free (OpenGitOps); imperative push pipelines must engineer it.
  • Respect the platform’s limits. Merge storms hit orchestrator rate limits (GitHub: 1,500 trigger events / 10s, 500 queued runs / 10s, 24-hour unclaimed-job cancel) — design fan-in and autoscaling to stay under them rather than discovering them during an incident (GitHub Actions limits).

See Also

Graph View

Backlinks