Reconcile Function Patterns

The reconcile function is the one piece of a controller the author actually writes (everything else is supplied by controller-runtime) — and writing it correctly is a craft with a small, well-established set of patterns. A Reconcile(ctx, req) call is handed only an object’s {namespace, name} — never an event, never a diff — and must drive that object from whatever its current state happens to be toward its declared desired state. The Kubebuilder Book states the central rule plainly: “the controller’s reconciliation loop needs to be idempotent” (Kubebuilder — Good Practices). Because the control loop is level-triggered, reconcile runs many times for the same object — on startup, on every relevant change, on requeue, on periodic resync — and every run must produce the same correct result. The ideal a reconcile function chases is to be a pure function of cluster state: read desired and observed state, compute the difference, apply it, and return — with no hidden memory between calls, no assumptions about why it was invoked, and no side effects that are not safe to repeat. This note collects the patterns that make that ideal achievable. It is the practical companion to Controller Pattern (the conceptual shape) and controller-runtime (the library).

Mental Model

flowchart TD
    START["Reconcile(ctx, req{ns,name})"]
    GET["Get the primary object"]
    NF{"NotFound?"}
    DONE_NF["return Result{}, nil<br/>(object gone — nothing to do)"]
    DEL{"DeletionTimestamp set?"}
    FIN["run external cleanup<br/>(idempotent),<br/>RemoveFinalizer"]
    ADDFIN["ensure finalizer present"]
    DIFF["read OBSERVED child/external state<br/>compute diff vs DESIRED (spec)"]
    APPLY["apply diff:<br/>create / update children (idempotent)"]
    STATUS["update .status subresource<br/>set observedGeneration"]
    RET{"work complete?"}
    OK["return Result{}, nil"]
    REQ["return Result{RequeueAfter: d}, nil<br/>or Result{}, err (backoff)"]
    START --> GET --> NF
    NF -- yes --> DONE_NF
    NF -- no --> DEL
    DEL -- yes --> FIN --> OK
    DEL -- no --> ADDFIN --> DIFF --> APPLY --> STATUS --> RET
    RET -- yes --> OK
    RET -- no --> REQ

What this diagram shows. Every well-formed reconcile function has the same skeleton: fetch the object, handle the “it’s gone” and “it’s being deleted” cases first, otherwise ensure the finalizer, diff desired against observed, apply, write status, and decide whether to requeue. The insight to extract: reconcile is not an event handler with a branch per event type — it is a single straight-line convergence function that is correct regardless of what triggered it. There is no if event == created anywhere; the same code path handles “object just created,” “child crashed,” “periodic resync,” and “operator just restarted.” That uniformity is what idempotency buys.

Mechanical Walk-through

Pattern 1 — Idempotency: every action safe to repeat

Reconcile will run again. On operator restart it runs for every object at once; on a transient error it runs again after backoff; controller-runtime issues a periodic resync regardless of changes. Therefore every action must be a no-op if already done. Don’t “create the Deployment” — ensure the Deployment exists and matches: Get it, and Create only on NotFound, Update only on drift. Don’t increment a counter; set a value. Don’t append to a list; reconcile the list to a target. The Operator SDK phrases the design rule as “have idempotent subroutines, where each subroutine performs one of the actions” — compose reconcile from ensureX() helpers, each individually re-runnable.

Pattern 2 — Level-triggered: react to state, not to events

The Request carries only {namespace, name} — deliberately. A controller is level-triggered: it observes the current level of the system, not the edges (events) that got it there. If three events fire for one object while a reconcile is in flight, controller-runtime’s workqueue coalesces them into one pending key — the next reconcile sees the final state and handles all three at once. Reconcile must therefore never depend on having seen every intermediate state; missing an event is normal and harmless because the next run reads the truth from the API. This is why “use watches, not event payloads” is the controller mantra.

Pattern 3 — Read desired + observed, compute diff, apply

The body of a healthy reconcile is three steps. Read desired state: the primary object’s .spec. Read observed state: the actual children and external resources — via controller-runtime’s cached client, so it is cheap. Compute and apply the diff: for each thing that should exist, create or update it to match. This is the Desired State vs Observed State dichotomy made into code. The reconcile does not “remember” what it did last time; it re-derives everything from the cluster, which is what makes it a pure function of cluster state.

Pattern 4 — One object per reconcile; one change per pass

Reconcile is invoked per named object — never reconcile a whole list, and never batch. Equally important, the Operator SDK guidance is to make at most one meaningful mutation per pass: “don’t perform multiple changes to your custom resource or dependent actions in a single run of Reconcile. Whenever you update the custom resource you’re watching, just exit the reconcile loop, and let the next run continue.” Updating the object you watch triggers another reconcile anyway; do one step, return, and let the next pass — where the earlier idempotent steps now no-op — carry on. This keeps each pass small, observable, and free of intra-pass race conditions.

Pattern 5 — Error-then-requeue and RequeueAfter

The return value (Result, error) is how reconcile asks for another pass:

Return	Meaning
Result{}, nil	Done. No requeue until the next watch event or resync.
Result{}, err	Transient failure. Requeue with exponential backoff — controller-runtime’s per-item rate limiter uses baseDelay·2^failures, growing from ~5 ms to ~16 min, per object.
Result{RequeueAfter: d}, nil	Success, but re-check after d — for time-based polling (a cert renewal window, an external resource that is still provisioning). Not an error.
Result{}, reconcile.TerminalError(err)	Permanent failure — the desired state is impossible; do not requeue.

The discipline: return an error for things that should retry fast with backoff; return RequeueAfter for things that are progressing normally but not yet finished. Never time.Sleep inside Reconcile waiting for an external resource — that pins a worker goroutine and backs up the queue for every other object. Return and re-check.

Pattern 6 — Status as a subresource; observedGeneration

A controller writes the object’s .status to report observed reality. Status must be a subresource (subresources: { status: {} } on the CRD). Two reasons. First, writing .status via the status subresource does not bump metadata.generation — generation increments only on .spec changes. Second, controller-runtime’s default GenerationChangedPredicate then filters status-only updates out of the workqueue, which is the standard defense against the hotloop (Pattern 7). To let clients tell “has the controller seen my latest spec?”, copy metadata.generation into status.observedGeneration at the end of a successful reconcile: when status.observedGeneration == metadata.generation, the status reflects the current spec. Status conditions (type/status/reason/message/lastTransitionTime, observedGeneration per condition) are the recommended way to express richer state.

Pattern 7 — Avoid hotloops (the self-reconcile storm)

The classic operator bug: reconcile writes .status, the controller watches its own object, the status write enqueues a fresh reconcile, which writes .status again — a CPU-pegged infinite loop. Two guards: (1) status is a subresource so the write does not change generation; (2) the GenerationChangedPredicate on the controller drops events where generation is unchanged. With both in place a status write does not re-trigger reconcile. The general rule: never let a controller’s own writes feed back into its own trigger. A subtler variant is returning Result{Requeue: true} unconditionally — that requeues forever; requeue only when there is genuinely pending work.

Pattern 8 — Finalizers: blocking deletion for cleanup

When a controller manages external state (a cloud load balancer, a DNS record, a database) that Kubernetes garbage collection cannot clean up, it must use a finalizer. The reconcile pattern, straight from the Kubebuilder finalizer reference:

1. Not being deleted (obj.ObjectMeta.DeletionTimestamp.IsZero()): ensure the finalizer is present — if !controllerutil.ContainsFinalizer(obj, name) { controllerutil.AddFinalizer(obj, name); r.Update(ctx, obj) }.
2. Being deleted (DeletionTimestamp is set): if the finalizer is still in the list, run the external cleanup, then controllerutil.RemoveFinalizer(obj, name) and r.Update(...). The apiserver completes deletion only once the last finalizer is gone.
3. The cleanup must be idempotent — deletion may reconcile several times; “delete the cloud LB” must succeed (or no-op) whether the LB exists or was already removed.

Pattern 9 — Owner references for garbage collection

For children that live inside Kubernetes (a Deployment created for a custom resource), do not use a finalizer — set an owner reference on the child pointing at the primary object (controllerutil.SetControllerReference(owner, child, scheme)). When the owner is deleted, the apiserver’s garbage collector cascade-deletes the children automatically. Owner references also power controller-runtime’s Owns() builder: a change to a child enqueues the owner for reconcile. Use finalizers for external resources, owner references for in-cluster children — they solve cleanup for the two different cases.

Configuration / API Surface

A reconcile function exhibiting the patterns, annotated:

func (r *WidgetReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var widget appv1.Widget
    if err := r.Get(ctx, req.NamespacedName, &widget); err != nil {
        // Pattern 1/2: object may be gone — that is a valid, no-op outcome.
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
 
    finalizer := "widget.example.com/cleanup"
    if !widget.ObjectMeta.DeletionTimestamp.IsZero() {
        // Pattern 8: being deleted — run idempotent external cleanup, drop finalizer.
        if controllerutil.ContainsFinalizer(&widget, finalizer) {
            if err := r.deleteExternalResources(ctx, &widget); err != nil {
                return ctrl.Result{}, err            // Pattern 5: retry with backoff
            }
            controllerutil.RemoveFinalizer(&widget, finalizer)
            if err := r.Update(ctx, &widget); err != nil {
                return ctrl.Result{}, err
            }
        }
        return ctrl.Result{}, nil                    // Pattern 4: one change, then return
    }
    if !controllerutil.ContainsFinalizer(&widget, finalizer) {
        controllerutil.AddFinalizer(&widget, finalizer)
        return ctrl.Result{}, r.Update(ctx, &widget) // wrote spec metadata — exit, re-run
    }
 
    // Pattern 3: desired = widget.Spec; observed = the actual child Deployment.
    var dep appsv1.Deployment
    err := r.Get(ctx, depKey(&widget), &dep)
    switch {
    case apierrors.IsNotFound(err):
        dep = buildDeployment(&widget)               // idempotent: ensure-create
        _ = controllerutil.SetControllerReference(&widget, &dep, r.Scheme) // Pattern 9
        if err := r.Create(ctx, &dep); err != nil {
            return ctrl.Result{}, err
        }
    case err != nil:
        return ctrl.Result{}, err
    default:
        if needsUpdate(&dep, &widget) {              // idempotent: ensure-match
            if err := r.Update(ctx, &dep); err != nil {
                return ctrl.Result{}, err
            }
        }
    }
 
    // Pattern 6: report observed reality via the status SUBRESOURCE.
    widget.Status.ReadyReplicas = dep.Status.ReadyReplicas
    widget.Status.ObservedGeneration = widget.Generation
    if err := r.Status().Update(ctx, &widget); err != nil {
        return ctrl.Result{}, err
    }
    // Pattern 5: success, but re-check periodically — NOT an error.
    return ctrl.Result{RequeueAfter: 5 * time.Minute}, nil
}

Points worth keeping: every mutating call is Create-on-NotFound or Update-on-drift (idempotent); r.Status().Update targets the status subresource so it does not bump generation; SetControllerReference wires GC; the function returns after any .spec/metadata write rather than continuing; and RequeueAfter expresses “all good, check again later” without faking an error.

Failure Modes

• Non-idempotent action. Reconcile appends to a slice or creates without checking existence; a second pass duplicates work. Symptom: two Deployments, a doubled count, “already exists” errors. Fix: every action becomes ensureX — read first, act on the gap.
• Hotloop / self-reconcile storm. A status write re-triggers reconcile, which writes status, forever. Symptom: 100% CPU on the operator Pod, status fields flickering. Fix: status as subresource + GenerationChangedPredicate.
• Blocking inside Reconcile. A time.Sleep or long synchronous call holds a worker goroutine; the queue backs up for all objects. Symptom: reconcile latency climbs cluster-wide. Fix: RequeueAfter.
• error vs RequeueAfter confusion. Returning an error for a normal in-progress state floods logs with fake errors and burns through the backoff schedule; returning RequeueAfter for a genuine failure hides the problem. Match the return to the situation.
• Finalizer never removed. A bug in the cleanup path means RemoveFinalizer is never reached; the object is stuck Terminating forever. Symptom: kubectl delete hangs. Fix: make cleanup tolerant of partial state; consider a TerminalError escape hatch for genuinely impossible cleanups.
• Mutating a cached object. Objects from controller-runtime’s cache are shared; mutating one without DeepCopy() corrupts other goroutines’ views. Symptom: heisenbugs under concurrency.
• Multiple writers. Two operator replicas without leader election both reconcile every object — oscillation and conflicting resourceVersion updates. Fix: leader election (see controller-runtime).

Alternatives and When to Choose Them

• Imperative, event-driven handlers — a callback per event type (onCreate, onUpdate, onDelete) with mutable in-memory state. This is exactly the anti-pattern the level-triggered model rejects: it loses correctness on missed events, operator restarts, and out-of-order delivery. The Operator SDK explicitly warns that “writing reconciliation logic according to specific events breaks the operator pattern.” Use the level-triggered reconcile.
• A one-shot script / Job — for a one-time setup with no ongoing desired state, a Job is simpler than a controller; there is nothing to reconcile. Use a controller only when state must be continuously maintained against drift.
• ValidatingAdmissionPolicy / CEL — for enforcing invariants at admission time, in-process CEL policy is cheaper than a controller. A controller converges state over time; admission policy rejects bad state up front. They are complementary, not substitutes.
• Server-Side Apply inside reconcile — rather than hand-coding Create-vs-Update, a reconcile can Patch with client.Apply and a field manager, letting the apiserver compute the diff via field ownership. This is increasingly the recommended way to make the “ensure-match” step both idempotent and conflict-free; see Server-Side Apply.

Production Notes

• Every modern Go operator follows this skeleton. cert-manager, CloudNativePG, the Prometheus Operator, Cluster API — read any of their Reconcile methods and the get-handle-deletion-ensure-children-update-status shape is immediately recognizable. It is not a style choice; it is the shape correctness forces.
• observedGeneration is what makes operators debuggable. When an operator looks stuck, the first check is status.observedGeneration vs metadata.generation: equal means “the controller has acted on the current spec, the problem is downstream”; unequal means “the controller has not caught up.” Surfacing it is cheap and pays for itself the first incident.
• Alert on reconcile error rate and workqueue depth. controller-runtime’s Manager exposes per-controller reconcile counts, error rates, and queue depth as Prometheus metrics for free. A rising error rate or a queue depth that does not drain is the canonical signal of a misbehaving reconcile — often a hotloop or a blocking call.
• Test with envtest. controller-runtime’s envtest runs a real kube-apiserver and etcd locally, so a reconcile function can be exercised against a genuine API server — including the idempotency property (run reconcile twice, assert the second is a no-op) and the finalizer path. This is the standard operator test harness.
• Red Hat’s SRE guidance distills the same patterns: idempotent reconciliation, status conditions, finalizers for external cleanup, owner references for in-cluster GC, and one change per pass — the convergent set every production operator team rediscovers.

See Also

• Controller Pattern — the conceptual shape; this note is its practical companion
• controller-runtime — the library that supplies the Reconciler interface, Result, the rate limiter, and controllerutil helpers
• Kubernetes Control Loop Pattern — the level-triggered model these patterns express
• Operator Pattern — the kind of controller most reconcile-function craft is invested in
• Finalizers — the deletion-blocking mechanism Pattern 8 drives
• Owner References and Garbage Collection — the in-cluster cleanup mechanism of Pattern 9
• Desired State vs Observed State — the dichotomy Pattern 3 turns into code
• Server-Side Apply — the conflict-free way to implement the “ensure-match” step
• Custom Resource Definition — where subresources.status and the schema are declared
• Kubernetes MOC §13 — Extensibility