Buildkite

Buildkite is a CI/CD platform built around a distinctive hybrid model: Buildkite runs the control plane as a hosted SaaS — the scheduler, the web UI, the pipeline orchestration and visualization — while you run the agents (the machines that actually execute builds) on your own infrastructure (Buildkite, Architecture). The single most important consequence, and Buildkite’s core selling point, is where your data lives: because builds run on your compute, “sensitive data, such as source code and secrets, remain within your environment and are not seen by the Buildkite Platform” (Buildkite, Architecture). You get the managed-SaaS experience — no orchestration server to run, patch, or scale, a polished UI, hosted job history — without shipping your source code and credentials to a third party. This threads the needle between the two ends of Self-Hosted versus Cloud Runners: fully-managed cloud CI (convenient, but your code runs on the vendor’s machines) and fully self-hosted CI like Jenkins (total control, but you run everything including the fragile control plane).

This note teaches Buildkite’s model — the split between hosted control plane and self-hosted agents, how agents poll and claim work, how pipelines are defined and dynamically generated at runtime — and cross-links the general machinery (Self-Hosted versus Cloud Runners, CI CD Platform Models Compared) rather than re-teaching it. In the taxonomy of CI CD Platform Models Compared, Buildkite is the canonical hybrid SaaS platform: standalone (works with any Git host), hosted-orchestration, self-hosted-execution.

Mental Model — Split the Plane, Keep Your Data

The organizing idea is a clean split of responsibilities across a trust boundary. Buildkite-the-company operates the control plane: it decides what should run and when, stores pipeline definitions and build history, and renders the UI. Your infrastructure operates the data plane: agents that receive a job assignment, check out your source, run your build with your secrets, and report back only status, logs, and whatever artifacts you choose to upload. Source code and secrets never traverse the boundary into Buildkite’s platform.

flowchart TB
    subgraph BK["Buildkite SaaS — the CONTROL PLANE (their infra)"]
        SCHED["Scheduler /<br/>orchestration"]
        UI["Web UI +<br/>build history"]
        API["Agent API<br/>(HTTPS endpoint)"]
    end

    subgraph YOU["YOUR infrastructure — the DATA PLANE"]
        direction LR
        AG1["Agent<br/>(polls out)"]
        AG2["Agent<br/>(polls out)"]
        SRC["Source code<br/>+ secrets<br/>(never leave)"]
        AG1 --- SRC
        AG2 --- SRC
    end

    AG1 -.->|"HTTPS long-poll<br/>outbound only"| API
    AG2 -.->|"HTTPS long-poll<br/>outbound only"| API
    API -->|"hands job to<br/>a waiting agent"| AG1
    SCHED --- API
    SCHED --- UI

    style SRC fill:#4a3,color:#fff

What it shows and the insight to take: the dashed arrows point outbound from your agents to Buildkite — agents initiate every connection. Nothing from Buildkite connects into your network, and the green node — your source and secrets — sits entirely on your side of the boundary. The insight: the “hybrid” in Buildkite’s model is not a vague marketing word; it is a precise architectural split where orchestration is centralized and hosted but execution and data are decentralized and yours. This is what lets a security-conscious enterprise adopt a SaaS CI tool without the source code ever touching the vendor’s servers.

The Agent — A Small Outbound-Polling Runner

The Buildkite Agent is “a small, reliable and cross-platform build runner” — a single Go binary you install on any machine you want to run builds (Buildkite, Agent v3). Its connection model is the key to the whole security story: the agent communicates with Buildkite over HTTPS by polling outbound, so it needs no inbound firewall rules, no port forwarding, no public IP. After registering, the agent “periodically polls the Buildkite platform, looking for new work, waiting to accept an available job” (Buildkite, Agent v3). This is a pull model — the agent asks for work rather than the platform pushing to it — which is exactly why an agent can live deep inside a private VPC or an on-prem datacenter and still participate.

The agent lifecycle is a simple loop:

sequenceDiagram
    participant A as Buildkite Agent<br/>(your infra)
    participant BK as Buildkite API<br/>(control plane)
    A->>BK: register (agent token)
    loop until shutdown
        A->>BK: poll — any work? (HTTPS)
        BK-->>A: here is a job (or: nothing)
        Note over A: accept job
        A->>A: checkout source · run commands · run hooks
        A-->>BK: stream logs + report exit status
        A-->>BK: upload artifacts (optional)
    end

What it shows and the insight to take: every arrow originates at the agent; the platform only ever responds. The agent registers once with an agent token, then loops: poll, accept, execute, report. The insight: because the agent is stateless between jobs and initiates all traffic, agents are trivially horizontally scalable and disposable — spin up more to go faster, tear them down when idle, and none of it requires opening your network. When several agents are free, Buildkite orders them “by how recently these agents successfully completed a job,” preferring recently-active agents so warm caches get reused.

Ephemeral vs persistent agents. Agents can be persistent (long-lived hosts that run job after job — common self-hosted) or ephemeral (destroyed after each job — the default for Buildkite-hosted agents and the recommended pattern for untrusted or isolation-sensitive work). Ephemeral agents give the clean-environment-per-build hygiene that Self-Hosted versus Cloud Runners describes. Hooks — shell scripts that fire at defined lifecycle points (environment, checkout, command, pre-exit) at either the agent-machine level or the pipeline level — are the extension mechanism, used for secrets injection, environment setup, and overriding defaults.

Clusters and Queues — How Work Is Targeted

Buildkite organizes agents with two nested concepts (Buildkite, Clusters). A cluster is an isolated grouping of agents and pipelines within your organization — “isolated sets of agents and pipelines within the one Buildkite organization” — enforcing a hard boundary: pipelines in one cluster “cannot trigger or access artifacts from pipelines associated with another cluster” unless you explicitly allow it. Clusters are the coarse isolation unit (e.g. separate production and development clusters, or per-team clusters). Within a cluster, a queue is the finer subdivision, typically by infrastructure shape — architecture (x86, arm64, Apple silicon) or machine type (linux, macos, windows, gpu).

Agents register to a cluster using an agent token and advertise a queue; pipeline steps then target a queue so a job only runs on the right hardware:

steps:
  - label: "Build (arm64)"
    command: scripts/build.sh
    agents:
      queue: "arm64-linux"   # only agents in this queue pick it up

flowchart TD
    ORG["Buildkite Organization"]
    ORG --> C1["Cluster: production<br/>(isolated)"]
    ORG --> C2["Cluster: development<br/>(isolated)"]
    C1 --> Q1["Queue: linux-x86"]
    C1 --> Q2["Queue: arm64"]
    Q1 --> AG1["agents…"]
    Q2 --> AG2["agents…"]
    C2 --> Q3["Queue: default"]
    Q3 --> AG3["agents…"]

What it shows and the insight to take: clusters are hard walls (no cross-cluster artifact access by default), queues are routing lanes within a wall. The insight: this two-level scheme is how one Buildkite organization safely mixes production and non-production compute, and how a monorepo routes its ARM build to ARM agents and its GPU tests to GPU agents — all from a one-line agents: { queue: … } selector on a step. Queues can be self-hosted (your agents) or, if you opt into it, Buildkite hosted agents, where Buildkite provides the compute too — a fully-managed escape hatch for teams without the resources to run their own agents.

Pipelines and Steps

A Buildkite pipeline is defined in YAML — either in the web UI or, preferably, in a pipeline.yml committed to the repo (typically under .buildkite/), which the docs note “gives you access to more configuration options and environment variables than the web interface” (Buildkite, Defining steps). This is Buildkite’s Pipeline as Code. There are a handful of step types:

• command — run a script or shell command (the workhorse; runs on an agent).
• wait — a synchronization barrier: everything before it must finish before anything after it starts (fan-in).
• block — pause for manual approval before continuing (a human gate — Buildkite’s Approval Gates and Manual Deployment).
• input — collect user-provided data mid-build.
• trigger — kick off a build in another pipeline (compose pipelines).
• group — visually bundle related steps in the UI.

steps:
  - label: ":test_tube: Tests"
    command: scripts/tests.sh
    agents:
      queue: "linux"
  - wait                        # barrier: tests must all pass first
  - block: ":rocket: Deploy?"   # human approval gate
  - label: ":ship: Deploy"
    command: scripts/deploy.sh
    agents:
      queue: "deploy"

The wait step is Buildkite’s fan-in barrier (Fan-Out and Fan-In); combined with parallel command steps before it, it expresses fan-out/fan-in cleanly. Build states are terminal conditions like passed, failed, blocked, or canceled.

Dynamic Pipeline Generation — The Standout Feature

Buildkite’s most distinctive capability is dynamic pipelines: instead of a static pipeline.yml, a script generates the pipeline steps at runtime and pipes them to buildkite-agent pipeline upload, which injects them into the running build (Buildkite, Dynamic pipelines). The mechanism: a build starts with a small bootstrap step that runs a generator script; that script emits YAML (or JSON) on stdout; pipeline upload reads it and the emitted steps “appear during the build” as if they had been there all along.

flowchart TD
    START["Build starts"] --> BOOT["Bootstrap step:<br/>run generator script"]
    BOOT --> GEN["Script emits YAML/JSON<br/>(bash/python/ruby/go/…)"]
    GEN --> UP["buildkite-agent<br/>pipeline upload"]
    UP --> INJ["Steps injected into<br/>the running build"]
    INJ --> RUN["Generated steps execute"]

What it shows and the insight to take: the pipeline is not fixed at commit time — it is computed when the build runs, by ordinary code in any language (Bash, Python, Ruby, Node, Go, and more). The insight: this collapses the usual ceiling on pipeline expressiveness. Where a static YAML pipeline forces you to encode all conditionals in a limited templating dialect, Buildkite lets you write a real program to decide what runs. Concrete use cases from the docs:

• Monorepo path-filtering — inspect which files changed and generate build/test steps only for the affected services.
• Matrix from data — “a test step for each subdirectory,” generated by iterating over a directory listing.
• Conditional steps — branch, actor, or file-change logic in real code instead of contorted YAML if expressions.
• Infrastructure routing — pick target queues at generation time, or retry on alternate infrastructure.
• Policy enforcement — a central generator script that stamps every pipeline with org-wide rules.

A minimal generator in Bash:

#!/bin/bash
set -euo pipefail                       # MANDATORY: a failed generation must fail the build
 
echo "steps:"
for svc in $(git diff --name-only origin/main | cut -d/ -f1 | sort -u); do
  if [ -d "services/$svc" ]; then       # only for services that actually changed
    cat <<EOF
  - label: "test $svc"
    command: "make -C services/$svc test"
    key: "test-$svc"                    # explicit key → safe on retries
EOF
  fi
done

The generator is invoked and uploaded by the bootstrap step:

steps:
  - label: ":pipeline: generate"
    command: ".buildkite/generate.sh | buildkite-agent pipeline upload"

The critical practices, straight from the docs: always set -euo pipefail so a broken generator fails the build instead of silently uploading nothing; set an explicit key on generated steps so retries don’t duplicate them; and note insertion order — generated steps appear after the calling step, and multiple uploads insert in reverse order. Environment variables are interpolated by the agent at upload time; use $$ to defer a variable to runtime.

Why the Hybrid Model Appeals

Concern	Fully-managed cloud CI	Buildkite (hybrid)	Fully self-hosted (Jenkins)
Where builds run	Vendor’s machines	Your infrastructure	Your infrastructure
Where source/secrets go	To the vendor	Stay in your network	Stay in your network
Who runs the orchestrator/UI	Vendor	Vendor	You (patch, scale, HA it yourself)
Inbound firewall needed	n/a	None (agents poll out)	Depends
Scaling compute	Vendor’s limits/pricing	Your capacity, your cost	Your capacity, your cost
Ops burden	Lowest	Low–medium (agents only)	Highest (whole stack)

What the comparison shows: Buildkite occupies the middle column deliberately. Against fully-managed cloud CI, it wins on data residency and compute control — your code and secrets never leave, and you can run builds on beefy or specialized hardware (GPUs, huge memory, Apple silicon) that a shared cloud runner may not offer, at your own cost basis. Against fully self-hosted Jenkins, it wins on operational burden — you run only the stateless, disposable agents, not the fragile stateful control plane with its plugin sprawl and single-point-of-failure controller. The appeal is precisely for organizations that must keep data in-house (regulatory, security, IP sensitivity) but don’t want to operate a CI control plane. The cost is a per-seat SaaS dependency and trusting Buildkite with build metadata and orchestration.

Failure Modes and Trade-offs

• You still run the agents. “Only the agents” is less than Jenkins, but it is not nothing: agent hosts must be provisioned, secured, patched, and autoscaled. A neglected persistent agent accumulates state and drift; the ephemeral-agent pattern (fresh host per job) is the fix but requires autoscaling plumbing.
• Secrets are your responsibility. Because secrets never go to Buildkite, Buildkite cannot manage them for you — you wire secret retrieval into agent hooks (fetching from Vault, cloud secret managers, etc.). This is more secure but more work than a SaaS secret store.
• Control-plane dependency. If Buildkite’s hosted control plane has an outage, scheduling stops even though your agents are healthy — you do not own the orchestrator. This is the flip side of not having to run it.
• Dynamic-pipeline footguns. A generator script without set -euo pipefail can silently upload an empty pipeline (build “passes” having done nothing); generated steps without stable keys duplicate on retry. The power of arbitrary-code generation comes with arbitrary-code failure modes.

Alternatives and When to Choose Buildkite

Choose Buildkite when you want a managed CI experience — clean UI, no orchestrator to operate — but have a hard requirement that source and secrets stay on your infrastructure, or you need to run builds on your own specialized/large compute at your own cost. Its dynamic-pipeline model also makes it a strong fit for large monorepos where the set of steps genuinely must be computed per-commit. Prefer GitHub Actions or GitLab CI CD when your code already lives on that forge and you are comfortable with hosted runners — the tighter repo integration and zero-agent setup lower friction. Prefer Jenkins when you need a fully air-gapped, no-external-dependency system and are willing to operate the whole stack. Prefer Tekton when you are all-in on Kubernetes and want CI expressed as cluster-native CRDs. The distinguishing axis, per CI CD Platform Models Compared, is hosted-vs-self-hosted execution: Buildkite is the platform that splits it — hosted control, self-hosted execution.

Production Notes

The standard Buildkite production pattern is autoscaling ephemeral agents: run agents in an autoscaling group (or as Kubernetes pods) that scales on the depth of the Buildkite job queue, so agents spin up under load and scale to zero when idle — Buildkite publishes agent-scaling tooling for exactly this (the Elastic CI Stack for AWS and the Agent Stack for Kubernetes are the common deployment shapes). Combine ephemeral agents (clean environment per job) with agent hooks that fetch short-lived secrets at job start (never long-lived keys baked into the AMI), and cluster/queue isolation to keep production compute separated from PR compute. For large monorepos, dynamic pipelines that path-filter on the changed file set are the difference between a 40-minute build-everything pipeline and a 3-minute build-only-what-changed one — which is the whole Fast Feedback and Build Times game.

See Also

• Self-Hosted versus Cloud Runners — the general trade-off Buildkite’s hybrid model threads
• CI CD Platform Models Compared — where hybrid SaaS sits among hosted/self-hosted/k8s-native models
• Jenkins — the fully self-hosted alternative (you run the control plane too)
• GitHub Actions, GitLab CI CD, Tekton — sibling §8 tool notes
• Pipeline as Code — Buildkite’s pipeline.yml, extended by runtime generation
• Runners Agents and Executors — the general runner model; the Buildkite agent is one instance
• Fan-Out and Fan-In — the wait barrier as fan-in
• Approval Gates and Manual Deployment — the block step as a human gate
• Continuous Integration and Delivery MOC — parent map