CircleCI

CircleCI is a hosted, cloud-first Continuous Integration and Continuous Delivery (CI/CD) platform delivered as Software-as-a-Service (SaaS). You describe your pipeline as a single version-controlled file — .circleci/config.yml at the repository root — and CircleCI’s cloud provisions clean, ephemeral execution environments on every push, runs your jobs in them, and orchestrates those jobs into workflows. Its two signature features are orbs — reusable, versioned packages of YAML configuration published to a public registry — and its flexible executors (Docker, Linux VM, macOS, Windows, GPU, Arm), which let one config target radically different environments. CircleCI is the archetypal cloud-first, repo-integrated CI service: minimal infrastructure to run, config-as-code in the repo, and a hosted control plane that manages the compute for you.

CircleCI belongs to §8 of the Continuous Integration and Delivery MOC as a concrete engine. It implements the Pipeline as Code principle (the whole pipeline is a repo file), leans on the Self-Hosted versus Cloud Runners trade-off (default cloud runners, optional self-hosted), and lands on the “hosted SaaS, repo-integrated” corner of CI CD Platform Models Compared. This note teaches CircleCI’s machinery — how its config, executors, jobs, workflows, and orbs actually fit together — and cross-links the general principles rather than restating them.

Mental Model — Config in the Repo, Compute in the Cloud

The core loop: a developer pushes a commit; CircleCI reads .circleci/config.yml from that exact commit; it materializes a pipeline (one triggered run of the config); the pipeline runs one or more workflows (orchestrations of jobs); each job runs in its own fresh executor (a container or VM); each job is a list of steps (commands or reusable actions). Nothing persists between runs unless you explicitly cache it or pass it through a workspace.

flowchart TD
    PUSH["git push / PR / cron / API"] --> PIPE["Pipeline<br/>(one run of .circleci/config.yml)"]
    PIPE --> WF["Workflow(s)<br/>orchestrate job order"]
    WF --> J1["Job: build<br/>(runs in an executor)"]
    WF --> J2["Job: test<br/>requires: build"]
    WF --> J3["Job: deploy<br/>requires: test"]
    J1 --> S1["steps: checkout · run · save_cache"]
    J2 --> S2["steps: restore_cache · run tests"]
    subgraph EXEC["Executor choices per job"]
        D["docker"]
        M["machine (Linux VM)"]
        MAC["macos"]
        W["windows"]
    end
    J1 -.->|"picks one"| EXEC

What it shows and the insight to take: the hierarchy is strict — pipeline → workflow → job → step — and the executor is chosen per job, so one workflow can build in a lightweight Docker container and run integration tests in a full Linux VM. Because every job gets a fresh, ephemeral environment, CircleCI pipelines are reproducible by construction; anything you want to survive between jobs must be an explicit cache or workspace.

config.yml, Pipelines, Jobs, and Steps

Everything starts from .circleci/config.yml. The first meaningful line is usually version: 2.1 — config version 2.1 unlocks the modern feature set (orbs, reusable commands/executors/jobs, parameters, matrix jobs, and dynamic configuration); the older 2.0 lacks these. Below that you declare jobs and workflows.

A job is a unit of work that runs in a single executor and contains an ordered list of steps. Steps are either built-in (checkout to clone the repo, run to execute a shell command, save_cache/restore_cache, store_artifacts, store_test_results, persist_to_workspace/attach_workspace) or invoked from an orb.

version: 2.1                      # 2.1 unlocks orbs, reusable config, matrix, dynamic config
 
jobs:
  build:                          # a job = steps run inside one executor
    docker:                       # executor type: Docker
      - image: cimg/node:20.11    # CircleCI "convenience image"
    steps:
      - checkout                  # clone the repo at this commit
      - restore_cache:            # try to reuse node_modules from a prior run
          keys:
            - deps-{{ checksum "package-lock.json" }}
      - run: npm ci               # install dependencies
      - save_cache:               # cache keyed by the lockfile's hash
          key: deps-{{ checksum "package-lock.json" }}
          paths: [ node_modules ]
      - run: npm run build
      - persist_to_workspace:     # hand build output to later jobs
          root: .
          paths: [ dist ]
 
  test:
    docker:
      - image: cimg/node:20.11
    parallelism: 4                # split this job across 4 identical containers
    steps:
      - checkout
      - attach_workspace:         # receive build output from the build job
          at: .
      - run: circleci tests glob "test/**/*.spec.js" | circleci tests split --split-by=timings > /tmp/tests
      - run: npm test -- $(cat /tmp/tests)
      - store_test_results:       # feed timing data back for smarter splitting
          path: test-results

Line-by-line, the load-bearing ideas: restore_cache/save_cache keyed by {{ checksum "package-lock.json" }} mean the cache is reused only while the lockfile is unchanged — the standard Pipeline Caching and Parallelism pattern. persist_to_workspace and attach_workspace move build outputs (not dependencies) forward through the workflow; this is distinct from caching. parallelism: 4 spins up four identical containers for one job, and circleci tests split --split-by=timings distributes the test files across them using timing data saved by store_test_results on previous runs — CircleCI’s native test splitting, which balances slow and fast test files so no single container becomes the bottleneck (CircleCI, A guide to test splitting; CircleCI, split tests).

Execution Environments (Executors)

Each job runs in exactly one execution environment, and CircleCI offers several, selected by the executor key in the job (CircleCI, Execution environments overview):

• docker — the job runs inside a Docker container from an image you name (often a CircleCI convenience image like cimg/node, cimg/python). Fastest to spin up and cheapest, because it is a container, not a full operating system — “generally makes building your software more efficient.” Best for most build/test jobs. You cannot build Docker images with the plain Docker executor unless you add remote Docker or use machine.
• machine — a complete Linux virtual machine (Ubuntu). Heavier but gives you full control, a real kernel, and the ability to build/run Docker images natively. The legacy machine: true shorthand is deprecated; cloud users must name an image.
• macos — a macOS VM with a chosen Xcode version, for iOS/macOS builds.
• windows — a Windows environment, used via the Windows orb (config 2.1 cloud) or as a machine executor with a Windows image on self-hosted Server; supports PowerShell.
• GPU and Arm — CUDA-accelerated Linux/Windows GPU environments, and ARM-architecture VMs (arm.medium, arm.large).

Whatever executor you pick, you also set a resource class — the CPU/RAM size of the environment (e.g. medium, large, xlarge). CircleCI meters usage in credits partly by resource class, so choosing the smallest class that keeps the job fast is a real cost lever.

flowchart TD
    JOB["A job needs an environment"] --> Q1{"Need a full OS<br/>or to build Docker images?"}
    Q1 -->|No| DOCKER["docker executor<br/>(container, fastest, cheapest)"]
    Q1 -->|Yes, Linux| MACHINE["machine executor<br/>(Ubuntu VM)"]
    Q1 -->|iOS / macOS build| MACOS["macos executor<br/>(Xcode VM)"]
    Q1 -->|Windows build| WIN["windows executor / orb"]
    JOB --> RC["+ resource class<br/>(CPU/RAM size → credit cost)"]

What it shows and the insight to take: the executor decision is a speed/capability/cost trade-off made per job. Reach for docker by default; escalate to machine only when you need a real VM (building images, kernel features, privileged operations). For jobs whose slow setup dominates (macOS/Windows dependency installs), CircleCI’s self-hosted runner variants keep state between runs and avoid re-installing dependencies every time — a bridge to the Self-Hosted versus Cloud Runners model when the cloud’s ephemeral environments are too slow or too locked-down (private-network access, special hardware).

Workflows — Orchestrating Jobs

A workflow is “a set of rules for defining a collection of jobs and their run order” (CircleCI, Workflow orchestration). Without dependencies, jobs run in parallel; the requires key makes a job wait for named upstream jobs, which is how you build sequential chains and fan-out/fan-in graphs — one build fans out to many parallel test jobs, and a deploy fans them back in by requiring all of them.

workflows:
  build-test-deploy:
    jobs:
      - build
      - lint:                     # runs in parallel with unit
          requires: [ build ]
      - unit:
          requires: [ build ]
      - hold:                     # a manual approval GATE
          type: approval
          requires: [ lint, unit ]
      - deploy:
          requires: [ hold ]      # only after a human approves
          context: prod-secrets   # inject org-level secrets
          filters:
            branches:
              only: main          # deploy only from main

Key mechanics (CircleCI, Workflow orchestration):

• Approval jobs — a job with type: approval is a manual gate: the workflow pauses at it (ON_HOLD) until a human approves in the web app or via API. Downstream jobs require it. Approvals are valid for up to 90 days. This is the Approval Gates and Manual Deployment pattern expressed as a graph node.
• Contexts — the context: key injects organization-level secrets/environment variables with access controls (security-group, project, and expression restrictions), instead of per-project variables. This is CircleCI’s Secret Injection in Pipelines mechanism.
• Filters — filters.branches / filters.tags gate a job on branch or tag (with regex); tags require explicit tag filters or the workflow won’t run for them.
• Scheduled runs — the (legacy) triggers.schedule key with POSIX cron runs a workflow on a timer; note step-syntax (*/20) is unsupported and runs may be delayed up to 15 minutes.
• Conditional workflows — when / unless with pipeline-parameter expressions skip or run whole workflows.
• Status-based requires — a job can require an upstream job’s specific terminal status (success, failed, canceled, terminal), enabling e.g. a rollback job that only runs when release failed.

flowchart LR
    B["build"] --> L["lint"]
    B --> U["unit"]
    L --> H{{"hold<br/>type: approval"}}
    U --> H
    H -->|human approves| D["deploy<br/>(context: prod-secrets)"]
    style H fill:#fd6

What it shows and the insight to take: a workflow is a directed acyclic graph (DAG) of jobs joined by requires, and the approval node makes human sign-off a first-class part of that graph. Fan-out/fan-in is nothing more than several jobs sharing an upstream requires and a downstream job requiring all of them — the same Fan-Out and Fan-In shape every DAG-based CI engine implements, here spelled in requires.

Orbs — Reusable, Versioned Config Packages

Orbs are CircleCI’s answer to configuration duplication: “a reusable package of YAML configuration that condenses repeated pieces of config into a single line of code” (CircleCI, Orbs concepts). An orb bundles three kinds of reusable elements (CircleCI, Orb intro):

• Commands — parameterized sequences of steps you can drop into any job.
• Jobs — complete, parameterized jobs (steps + an executor) usable directly in a workflow.
• Executors — reusable environment definitions (image + resource class) shared across jobs.

An orb is addressed by a slug of the form namespace/orb-name, where the namespace identifies the owning individual/company/organization and the orb name describes what it does. You import it with the orbs: key and pin a version, which follows semantic versioning (SemVer) — major.minor.patch, where major = breaking change, minor = new feature, patch = bug fix. You may pin loosely (aws-cli@4 or aws-cli@4.1) to float forward on compatible patches/minors, or tightly (aws-cli@4.1.2) for reproducibility.

version: 2.1
orbs:
  node: circleci/node@5.2.0       # slug = namespace/orb @ SemVer version
  aws-cli: circleci/aws-cli@4.1.3
workflows:
  deploy:
    jobs:
      - node/test                 # a JOB provided by the node orb
      - aws-cli/setup:            # a COMMAND/job from the aws-cli orb
          requires: [ node/test ]

Orbs live in the public Orb Registry, and CircleCI distinguishes provenance tiers (CircleCI, Orbs concepts): certified orbs are authored and maintained by CircleCI itself; partner orbs are built by CircleCI technology partners (and are not “certified”); community orbs are published by the broader community. Namespaces default to appearing as “community” in the registry. Beyond public orbs, private orbs restrict an orb to a specific organization — the way a company shares internal, proprietary pipeline building-blocks without publishing them.

flowchart TD
    subgraph ORB["An orb (namespace/name @ SemVer)"]
        CMD["commands<br/>(reusable step sequences)"]
        JOBS["jobs<br/>(reusable full jobs)"]
        EXE["executors<br/>(reusable environments)"]
    end
    REG["Orb Registry"] --> ORB
    TIER["Tiers: certified (CircleCI) ·<br/>partner · community · private (org-only)"] -.-> REG
    ORB -->|"imported via orbs: key"| CFG["your config.yml"]

What it shows and the insight to take: orbs are to CircleCI what a package manager is to code — versioned, semver-pinned, registry-distributed config reuse. The single-line import of a certified orb (circleci/aws-cli@4.1.3) replaces dozens of lines of hand-written setup, and private orbs let a platform team ship a paved-road pipeline as a dependency. This is exactly the “reusable/templated pipeline” ambition of Pipeline as Code, realized as a distribution mechanism.

Dynamic Configuration and the Cloud-First Model

Version 2.1 also enables dynamic configuration: a small setup workflow runs first, computes configuration at runtime (e.g. which jobs to run based on which paths changed, or a parallelism value from a JSON file), and hands a generated config to the continuation orb, which continues the pipeline with it. This lets a monorepo build only the affected components without hand-maintaining a giant static config.

CircleCI is fundamentally cloud-first: the default and headline experience is fully hosted — CircleCI provisions the compute, you never run a control plane. Self-hosted runners exist (Container Runner on Kubernetes, Machine Runner on Linux/Windows/macOS) for the minority of cases that need private-network access, specialized hardware, or compliance isolation — but that is an opt-in escape hatch, not the primary model, which is exactly what distinguishes it from a self-hosted-first tool like Jenkins. See Self-Hosted versus Cloud Runners for the general trade-off.

Failure Modes and Common Misunderstandings

• Cache vs workspace confusion. save_cache/restore_cache is for reusable dependencies keyed by a lockfile hash across runs; persist_to_workspace/attach_workspace is for build outputs passed between jobs within one workflow. Using the wrong one (e.g. caching build artifacts) causes stale or missing files.
• Stale cache keys never invalidate. A cache key is immutable once written; if you forget to include the lockfile checksum in the key, you get a permanently stale cache. Use {{ checksum "…" }} in the key.
• Docker executor can’t build images. The plain docker executor has no Docker daemon; building images needs setup_remote_docker or the machine executor. A frequent first-pipeline surprise.
• Secrets on forked-PR builds. Contexts and project secrets are withheld from fork pull requests by default — a security feature that surprises contributors whose deploy steps “mysteriously” fail on forks.
• Config version 2.0 vs 2.1. Orbs, reusable commands, matrix jobs, and dynamic config require 2.1; copying a 2.0 example and adding orbs: fails until you bump the version.
• Credit/cost blow-ups from resource classes. Over-large resource classes and high parallelism multiply credit consumption; the cost is real and easy to overlook.

Alternatives and When to Choose Them

CircleCI’s identity is hosted SaaS, repo-integrated (VCS-agnostic across GitHub/GitLab/Bitbucket), config-as-code, cloud-first with optional self-hosted runners. Compare:

Tool	Model	Choose when
CircleCI	Hosted SaaS, cloud-first, orbs for reuse, multi-VCS	You want a managed CI with minimal ops, strong caching/parallelism, and reusable orbs; not tied to one VCS
GitHub Actions	Hosted + self-hosted, tightly GitHub-integrated, Marketplace actions	Your code is on GitHub and you want deepest native integration and a huge action ecosystem
GitLab CI CD	Integrated into GitLab, needs: DAG, hosted or self-hosted	You use GitLab and want CI built into the same product as repo/registry
Jenkins	Self-hosted-first, plugin-driven	You need maximum control/on-prem and already run Jenkins
Buildkite	Hybrid: hosted control plane, your own agents	You want a managed UI but must run builds on your own compute at scale

CircleCI’s sweet spot is teams that want a polished, low-ops hosted CI with best-in-class caching/test-splitting and the orb ecosystem for config reuse, and who are not locked to a single source-control vendor. Where the whole shop lives on GitHub, GitHub Actions often wins on integration depth; where builds must run on private/owned compute at scale, Buildkite’s hybrid model or self-hosted runners fit better. The full axis comparison is in CI CD Platform Models Compared.

Production Notes

CircleCI popularized several practices now standard across CI: config-as-code in the repo, first-class test splitting by timing data to cut wall-clock test time, and orbs as versioned, registry-distributed config reuse (a genuine novelty when introduced — most CI tools still copy-paste config). The parallelism + circleci tests split --split-by=timings combination is the canonical way to take a 30-minute test suite down to a few minutes by sharding across identical containers, and because timing data is fed back via store_test_results, the split self-tunes over time. On the security side, CircleCI’s own January 2023 breach (in which an engineer’s laptop malware led to exfiltration of customer secrets and a mass-rotation advisory) is a widely-cited reminder that a CI platform holds production credentials and is itself a high-value target — the Least-Privilege Pipeline Runners and OIDC and Secretless Pipeline Authentication disciplines exist precisely because of incidents like this. CircleCI now supports OIDC token issuance so jobs can obtain short-lived cloud credentials instead of storing long-lived keys.

Uncertain

Verify: the exact scope and dates of the CircleCI January 2023 security incident and the current state of its OIDC support. Reason: recounted here from general knowledge and the platform’s own advisories, not re-fetched from a primary post-mortem during this write-up; the OIDC feature set evolves. To resolve: read CircleCI’s official security advisory and current OIDC docs before relying on specifics. #uncertain

See Also

• Continuous Integration and Delivery MOC — parent; CircleCI is a §8 tool implementing the pipeline machinery
• Pipeline as Code — the config-in-repo principle CircleCI embodies via .circleci/config.yml and orbs
• Self-Hosted versus Cloud Runners — CircleCI is cloud-first with an optional self-hosted-runner escape hatch
• CI CD Platform Models Compared — where CircleCI sits (hosted SaaS, repo-integrated, cloud-first)
• Pipeline Caching and Parallelism — the cache-by-lockfile-hash and test-splitting patterns above
• Matrix Builds — CircleCI’s matrix jobs (config 2.1) for OS/version fan-out
• Approval Gates and Manual Deployment — the type: approval workflow gate
• Secret Injection in Pipelines — CircleCI contexts and forked-PR secret withholding
• Fan-Out and Fan-In — the requires-based DAG shape of CircleCI workflows
• GitHub Actions, GitLab CI CD, Jenkins, Buildkite — sibling CI/CD platforms