vault

controller-runtime

15 min read

controller-runtime

sigs.k8s.io/controller-runtime is the Go library that supplies everything a Kubernetes controller needs except the business logic — the shared caches, the split client, leader election, the metrics and health-probe servers, the webhook scaffolding, and the event-routing machinery that turns watch streams into reconcile calls. It is a Kubernetes SIG-API-Machinery project (github.com/kubernetes-sigs/controller-runtime), and it is the foundation that both Kubebuilder and the Operator SDK generate code on top of: nearly every modern Go-based operator in the ecosystem — cert-manager, the Prometheus Operator, CloudNativePG, ArgoCD’s controllers, Cluster API — is a controller-runtime program. The library exists because the Controller Pattern has a large, subtle, identical plumbing layer in every controller — list-watch, an informer cache, a deduplicating workqueue, leader election, graceful shutdown — and hand-writing that plumbing correctly is hard and pointless to repeat. controller-runtime writes it once, exposes a small surface (most importantly the Reconciler interface and the Manager), and lets the controller author write only the reconcile function. The Kubebuilder Book calls the result the controller’s “architecture”: a Manager holding shared infrastructure, controllers built declaratively with a builder, and a Reconcile method that is “the heart” of each one (book.kubebuilder.io — Architecture). This note is the implementation-level companion to Controller Pattern: where that note describes the shape a controller must have, this one describes the library that materializes that shape.

Mental Model

flowchart TB
    subgraph MGR["Manager — owns all shared infrastructure"]
        CACHE["Cache<br/>shared informers,<br/>backs the read client"]
        CLIENT["Client (split)<br/>reads ← cache<br/>writes → API server"]
        LE["Leader election<br/>(Lease)"]
        METRICS["Metrics + health<br/>+ webhook servers"]
        subgraph C1["Controller A"]
            B1["builder: For/Owns/Watches<br/>+ Predicates"]
            R1["your Reconciler.Reconcile()"]
        end
        subgraph C2["Controller B"]
            B2["builder ..."]
            R2["your Reconciler ..."]
        end
    end
    APISERVER[(kube-apiserver)]
    APISERVER -- watch --> CACHE
    CACHE --> B1 & B2
    B1 -- "Request{ns/name}" --> R1
    B2 -- "Request{ns/name}" --> R2
    R1 & R2 -- read --> CACHE
    R1 & R2 -- write --> APISERVER
    R1 & R2 -- "Result{RequeueAfter} / error" --> B1

What this diagram shows. The Manager is the top-level object that owns everything shared: one Cache (one set of informers, even if ten controllers run inside), one split Client, leader election, and the metrics/health/webhook servers. Each controller registered with the Manager is constructed by a fluent builder (For/Owns/Watches, optionally filtered by Predicates) and delegates to a user-supplied Reconciler. The insight to extract: controller-runtime’s value is the box labelled “Manager.” Everything inside it except the two Reconcile() methods is library code you did not write and do not maintain. You bring the business logic; controller-runtime brings the caches, the client, the leader election, the graceful shutdown, the queueing, and the event routing. That division — your reconcile function, their everything-else — is the whole reason the library exists.

Mechanical Walk-through

The Manager — pkg/manager

The Manager is the entry point. ctrl.NewManager(config, options) constructs it; it then:

  • Builds and owns the shared Cache — so N controllers in one binary share one set of informers and one watch connection per resource type, rather than N.
  • Builds the Client — the split read/write client (below).
  • Runs leader election — if enabled, the Manager contends for a Lease and only runs the controllers when it holds leadership; this is how you run multiple replicas of a controller for HA while preserving the single-writer discipline.
  • Serves the metrics endpoint — Prometheus metrics for reconcile counts, latency, queue depth, errors.
  • Serves health and readiness probes — /healthz, /readyz endpoints you register checks against.
  • Serves the webhook server — admission and conversion webhooks register here.
  • Handles graceful shutdown — on SIGTERM, it stops feeding the workqueues, lets in-flight reconciles finish, and exits cleanly.

mgr.Start(ctx) blocks and runs everything. One Manager typically hosts every controller and webhook in an operator binary.

The Cache — pkg/cache

The Cache is a set of shared informers — the list-watch machinery from Watch and Informers, wrapped so it presents as a read-only client. It auto-populates by watching the API server and stays fresh as events arrive. Two consequences:

  • Reads are cheap and local. A Get or List through the cache hits process memory, not the API server — essential for the controller-pattern rule “read current state on every reconcile.” Without a cache, reconciling would hammer the API server.
  • The cache supports field indexing. FieldIndexer lets you build indexes (e.g. “all Pods on node X”) for fast lookups inside reconcile.

One caveat the docs flag: the cache is eventually consistent with the API server — a Create followed immediately by a Get through the cache may not yet see the new object, because the watch event has not propagated. Reconcilers must tolerate this (and the level-triggered design already does — the next reconcile will see it).

The Client — pkg/client

controller-runtime’s default Client is a split client: reads come from the Cache, writes go straight to the API server. This is the right default — reads are frequent (every reconcile reads current state) and benefit hugely from caching; writes are less frequent and must be durable and immediately consistent. A controller author can also construct a client that talks only to the API server (bypassing the cache) for the rare cases that need strict read-after-write — but the split client is what you want almost always.

The Reconciler — pkg/reconcile

The one interface the controller author must implement. As of controller-runtime v0.19 the Reconciler is a type alias for the generic TypedReconciler[Request] (pkg.go.dev — reconcile):

type Reconciler = TypedReconciler[Request]
 
type TypedReconciler[request comparable] interface {
    Reconcile(ctx context.Context, req request) (Result, error)
}

For almost every controller you implement the un-typed alias, which has the familiar signature:

Reconcile(ctx context.Context, req Request) (Result, error)

Request carries only {Namespace, Name} — no event payload, enforcing the level-triggered discipline (see Controller Pattern). The generic TypedReconciler exists for advanced cases (priority-queue reconcilers, custom request types) that pre-generics controller-runtime could not express type-safely. The return value (Result, error) tells the framework what to do next:

ReturnEffect
(Result{}, nil)Done — do not requeue until the next watch event.
(Result{RequeueAfter: d}, nil)Requeue after delay d — for periodic polling (cert renewal windows, external-resource readiness checks). The modern idiom for any deliberate requeue.
(Result{}, err)Requeue with exponential backoff — the standard transient-error retry.
(Result{}, reconcile.TerminalError(err))Treat as permanent — do not requeue (the desired state is impossible). Added in v0.15.
(Result{Requeue: true}, nil)Deprecated. The Result.Requeue boolean still exists but the GoDoc explicitly says “this setting is deprecated as it causes confusion and there is no good reason to use it” — return Result{RequeueAfter: d} with a small d (e.g. time.Second) instead.

The Result{RequeueAfter: d} pattern is the idiom for any work that cannot complete in one pass: never block inside Reconcile waiting for an external resource — return RequeueAfter and re-check on the next pass. This keeps reconcile fast and the worker goroutines free. v0.24 also added Result.Priority (an *int) used only by the optional priority-queue source — irrelevant to the standard workqueue.

The builder — pkg/builder

The builder is a fluent API for declaring what triggers a controller without touching the underlying informer/handler wiring:

  • For(&MyKind{}) — the primary resource. The controller watches it; any change enqueues that object’s key.
  • Owns(&childType{}) — owned children. The controller watches the child type and, when a child changes, enqueues the owner (resolved via ownerReferences). This is how a reconcile fires when a Pod the controller created crashes — without the author writing any event-mapping code.
  • Watches(&otherType{}, handler) — an arbitrary other resource, with a custom handler mapping its events to reconcile requests. For relationships not expressible as ownership.

.Complete(reconciler) finishes the build and registers the controller with the Manager. Three lines of builder calls replace dozens of lines of hand-wired sources and handlers.

Predicates — pkg/predicate

A Predicate filters watch events before they reach the workqueue — so a reconcile is not even scheduled for events the controller does not care about. The most important one is GenerationChangedPredicate: it drops events where only .status changed (metadata.generation is unchanged), which breaks the self-reconcile storm — the failure mode where a controller’s own status write triggers another reconcile, which writes status, ad infinitum (see Controller Pattern §Failure Modes). Other predicates filter by label, by annotation, or by arbitrary logic. Predicates are pure performance/correctness hygiene: fewer pointless reconciles.

Finalizer helpers

controller-runtime’s controllerutil package provides AddFinalizer, RemoveFinalizer, and ContainsFinalizer — the small helpers for the Finalizers dance: add a finalizer on first reconcile so deletion is blocked; on a delete request, run external cleanup, then remove the finalizer so the API server can finally delete the object. Hand-managing the finalizer slice is error-prone; the helpers make it a few correct lines.

The webhook scaffolding — pkg/webhook

The Manager also runs an HTTPS webhook server. controller-runtime provides builders for admission webhooks — implement a Validator/Defaulter (or a raw admission.Handler) and register it — and for conversion webhooks (translating a CRD between versions — see Conversion Webhooks). The same Manager that runs your controllers serves the webhooks, so an operator is a single binary doing both.

Configuration / API Surface

A complete minimal operator main plus a controller registration, annotated:

func main() {
    // 1. Construct the Manager. It builds the shared Cache and Client,
    //    and is configured for leader election and the metrics/probe servers.
    mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
        Scheme:                 scheme,                 // Go-type ↔ GVK registry
        LeaderElection:         true,                   // HA: one active replica
        LeaderElectionID:       "widget-operator.example.com",
        Metrics:                server.Options{BindAddress: ":8080"},
        HealthProbeBindAddress: ":8081",
    })
    if err != nil { os.Exit(1) }
 
    // 2. Register a health and a readiness check on the Manager's probe server.
    _ = mgr.AddHealthzCheck("ping", healthz.Ping)
    _ = mgr.AddReadyzCheck("ping", healthz.Ping)
 
    // 3. Build and register the controller via the fluent builder.
    err = ctrl.NewControllerManagedBy(mgr).
        For(&appv1.Widget{}).                            // primary resource — watch it
        Owns(&appsv1.Deployment{}).                      // owned child — watch, enqueue owner
        WithEventFilter(predicate.GenerationChangedPredicate{}). // drop status-only events
        Complete(&WidgetReconciler{                      // your business logic
            Client: mgr.GetClient(),                     // the split read/write client
            Scheme: mgr.GetScheme(),
        })
    if err != nil { os.Exit(1) }
 
    // 4. Start everything. Blocks until SIGTERM; then shuts down gracefully.
    if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
        os.Exit(1)
    }
}
 
// The Reconciler — the ONLY part that is genuinely yours.
type WidgetReconciler struct {
    client.Client
    Scheme *runtime.Scheme
}
 
func (r *WidgetReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var widget appv1.Widget
    if err := r.Get(ctx, req.NamespacedName, &widget); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err) // gone → nothing to do
    }
    // ... ensure child Deployment exists / matches (idempotent) ...
    // ... update widget.Status via r.Status().Update(ctx, &widget) ...
    return ctrl.Result{RequeueAfter: time.Minute}, nil   // periodic re-check
}

Line-by-line, the points worth keeping:

  • (1) The Manager is configured declaratively with options — leader election, metrics, probes — and builds the shared Cache and split Client for you. You never write informer or workqueue code.
  • (2) Health and readiness endpoints come from the Manager; you only supply the check functions.
  • (3) The builder chain is the controller’s wiring: For declares the watched primary resource, Owns declares children (changes to a child enqueue its owner — automatically), WithEventFilter attaches a predicate to suppress status-only self-reconciles. .Complete(reconciler) plugs in your logic.
  • (4) mgr.Start runs every registered controller and webhook, handles SIGTERM, and shuts down cleanly.
  • The Reconciler is the only non-boilerplate code in the file. Everything above it is library configuration. This is the division of labor controller-runtime exists to provide.

Failure Modes

  • Using the non-cached client for reads. Constructing an API-server-only client and reading through it in Reconcile produces a hot loop of GET/LIST calls against the API server. Symptom: API-server CPU spikes when the operator runs. Use mgr.GetClient() (the split client) for reads.
  • Expecting read-after-write through the cache. A Create followed by an immediate cached Get may not see the object — the cache is eventually consistent. Symptom: a reconcile “loses” an object it just created. The level-triggered design tolerates this; do not work around it with a non-cached read unless you truly need strict consistency.
  • Mutating a cached object. Objects returned by the cache are shared; mutating one corrupts other goroutines’ views. Always DeepCopy() before modifying. Symptom: heisenbugs where status flips unpredictably under concurrency.
  • No GenerationChangedPredicate on a status-writing controller. Every status write triggers a fresh reconcile → another status write → a CPU-pegged self-reconcile storm. Symptom: 100% CPU on the operator Pod. Fix: filter status-only events with the predicate.
  • Blocking inside Reconcile. A time.Sleep or a long synchronous external call holds a worker goroutine; the workqueue backs up. Symptom: reconcile latency for other objects climbs. Fix: return Result{RequeueAfter: d} and re-check next pass.
  • Leader election disabled with multiple replicas. Two operator replicas both reconcile every object — two writers, oscillation. Symptom: conflicting updates, doubled API load. Fix: LeaderElection: true.
  • controller-runtime / client-go / Kubernetes version skew. Each controller-runtime minor version is tested against a specific client-go minor and the matching Kubernetes minor; mismatched versions cause subtle API incompatibilities. Per the project’s VERSIONING.md, controller-runtime publishes “a minor version for each Kubernetes minor release,” and the compatible k8s.io/*, client-go, and minimum Go version for any given release are pinned in that release’s go.mod. controller-runtime is still pre-1.0 (v0.x) and does make breaking changes between minor versions — pin it and read the release notes on upgrade. As of May 2026 the latest release is v0.24.1 (12 May 2026), targeting k8s.io/* v1.36 (release notes); the previous minors were v0.24.0 (30 Apr 2026), v0.23.0 (19 Jan 2026), and v0.22.5 (19 Jan 2026). Any specific compatibility claim below the minor level should be checked against the actual go.mod of the version you pin.

Alternatives and When to Choose Them

  • Raw client-go + the tools/cache informers + a hand-built workqueue. This is what controller-runtime is built on top of, and what controllers were written with before it existed. Choose it only if you need control controller-runtime abstracts away (the built-in kube-controller-manager controllers use client-go directly, for historical and tight-integration reasons). For a new operator, hand-rolling this plumbing is wasted effort and a source of bugs.
  • Kubebuilder. Not an alternative — a scaffolding layer on top of controller-runtime. It generates the project layout, CRD YAML from Go struct tags, RBAC manifests, and the main.go/Reconciler skeletons shown above. Choose Kubebuilder to start a controller-runtime project; controller-runtime is what the generated code runs on.
  • Operator SDK. Red Hat’s scaffolding tool; for Go operators it wraps Kubebuilder (hence also controller-runtime), and additionally supports Ansible- and Helm-based operators that need no Go at all. Choose it for the Ansible/Helm operator styles, or when targeting the Operator Lifecycle Manager / OperatorHub ecosystem.
  • Non-Go frameworks — Kopf (Python), kube-rs (Rust), the Java Operator SDK, Metacontroller (webhook-based, language-agnostic). Choose these when the team’s language is not Go. They re-implement the same Manager/cache/reconcile ideas; controller-runtime is the Go-specific, and by far the most widely used, implementation.

The decision rule: if you are writing a controller or operator in Go, you are using controller-runtime — directly, or (more commonly) via Kubebuilder or the Operator SDK which generate code onto it. The only real choice is whether to scaffold with Kubebuilder/Operator SDK or wire up controller-runtime by hand; scaffolding is almost always the right call.

Production Notes

  • The whole modern Go operator ecosystem sits on controller-runtime. cert-manager, the Prometheus Operator, CloudNativePG, Cluster API, ArgoCD’s controllers, Crossplane — all are controller-runtime programs. Reading any of their codebases is reading controller-runtime usage at production quality.
  • The Manager-hosts-everything pattern is universal. A real operator binary registers many controllers (one per CRD it manages) and its webhooks with a single Manager, so they share one cache and one watch connection per resource type — a significant efficiency win over per-controller informers.
  • envtest ships with controller-runtime (pkg/envtest) — it spins up a real local kube-apiserver and etcd for integration tests, so a reconciler can be tested against a genuine API server without a full cluster. This is the standard operator testing approach and a major reason the framework is trusted.
  • Metrics come for free. The Manager’s metrics endpoint exposes per-controller reconcile counts, latencies, error rates, and workqueue depth. Alerting on rising workqueue depth or reconcile-error rate is the standard way to detect a misbehaving or overloaded operator in production.
  • Pin the version and read the changelog. Because controller-runtime is v0.x with breaking changes between minor versions, an unpinned dependency or a careless upgrade can break an operator’s build or behavior. Treat a controller-runtime bump as a deliberate, tested change. As of May 2026 the current minor is v0.24 (targeting Kubernetes 1.36); every Kubernetes minor gets a corresponding controller-runtime minor.

See Also

Graph View

Backlinks