vault

Just-One-More-Controller Anti-Pattern

11 min read

Just-One-More-Controller Anti-Pattern

The just-one-more-controller anti-pattern is the unbounded accumulation of CRDs, operators, and controllers in a cluster — every problem gets “an operator for that,” each installed with a one-line helm install, until the cluster runs dozens of third-party controllers that nobody owns, nobody version-tracks, and nobody can confidently remove. Each individual operator solves a real problem, and that is exactly the trap: the decision to add the next one is always locally justified, while the cumulative cost — overlapping admission webhooks, conflicting reconcile loops, version skew against the Kubernetes minor, apiserver watch and CPU load, and a control plane straining under it all — is borne by no single decision and tracked by no one. This is the sidecar-sprawl failure mode lifted from the Pod level to the cluster level: locally rational, globally corrosive. The fix is not “never install operators” — operators are how Kubernetes grew its ecosystem — but to treat each one as a long-term operational liability with a named owner, and to keep an “operator budget.”

Mental Model

The danger is not any single operator; it is the interaction surface that grows quadratically as operators are added. Each new controller can collide with every existing one.

flowchart TB
    subgraph CLUSTER["A cluster after a year of 'just one more'"]
        O1[cert-manager]
        O2[external-dns]
        O3[Prometheus operator]
        O4[a CNPG / DB operator]
        O5[an autoscaler operator]
        O6[Kyverno + OPA Gatekeeper]
        O7[...20 more, owner unknown]
    end
    API[(kube-apiserver + etcd)]
    O1 & O2 & O3 & O4 & O5 & O6 & O7 -->|"each: watch streams,<br/>reconcile loops, CRDs,<br/>often a webhook"| API
    API --> P1["webhook wedge<br/>failurePolicy: Fail + backend down"]
    API --> P2["two controllers fight<br/>over the same field"]
    API --> P3["unmaintained operator<br/>blocks the K8s upgrade"]
    API --> P4["watch/CPU load<br/>strains the control plane"]

What this diagram shows. Every operator added to the cluster is another set of long-lived watch streams, another reconcile loop, one or more CRDs, and — very often — an admission webhook, all converging on the single kube-apiserver. The failure modes on the right are not exotic: they are the predictable consequences of crowding that one shared surface. The insight to extract: operators do not compose for free. The cost of operator N is not “one more operator” — it is one more set of potential conflicts with the N−1 already installed, and one more thing that can block the next cluster upgrade.

Mechanical Walk-through

Why it is tempting

  • Each operator solves a real problem. Certificate lifecycle → cert-manager. DNS records → external-dns. Postgres → CNPG. The need is genuine; the operator is the obvious answer.
  • Installation is a one-liner. helm install or an OLM subscription, and it is running in seconds. The install cost is trivial, so the install decision is never scrutinized.
  • The “platform” framing makes adding feel like progress. Every new operator is a new capability the platform team can advertise. Subtraction — removing an operator — feels like regression and is rarely anyone’s job.
  • The cost is deferred and diffuse. The webhook conflict, the upgrade block, the watch-load creep — none of these bites at install time. They bite months later, during an incident or an upgrade, by which point the operator’s original owner may have left the team.

What goes wrong — concrete failure modes

  1. Webhook-induced cluster wedge. An operator’s ValidatingAdmissionWebhook (or MutatingAdmissionWebhook) configured with failurePolicy: Fail will reject every matching API write when its backing Pods are unavailable. In the notorious circular case, the webhook’s own Pods cannot be (re)created because the webhook that must admit them is down — and with failurePolicy: Fail, the whole cluster freezes. The more operators, the more webhooks, the higher the odds one of them is configured this way. (See Common Kubernetes Failures Catalog, category 2.)

  2. Two controllers fighting over the same resource. Operator A sets a label or a spec.replicas value; operator B’s reconcile loop removes or overrides it; A reconciles again; B again — an infinite mutation war. Server-Side Apply field-ownership conflicts are the polite version of this; an outright reconcile fight is the impolite version. The Kubernetes admission-webhook good-practices documentation has a dedicated section, “Prevent loops caused by competing controllers,” warning that “if your webhook adds a label that a different controller removes, your webhook gets called again. This leads to a loop” — and recommends detecting it via an audit policy at level: RequestResponse on the patch verb, watching for the same field being updated and reverted repeatedly (admission webhook good practices).

  3. An unmaintained operator blocks a Kubernetes upgrade. Operators have a version-skew relationship with the Kubernetes minor: they use APIs that get deprecated and removed, and they may not tolerate a newer apiserver. An abandoned operator pinned to a removed API version cannot be upgraded — and so the cluster cannot be upgraded until someone rips it out, mid-migration, under time pressure.

  4. CRD-version conflicts and conversion-webhook fragility. Two operators may ship CRDs in the same or colliding API groups (one reason the docs insist on a domain you control for the group name). An operator that changes its CRD’s storage version needs a working conversion webhook; if that webhook is down, every read of the affected custom resources fails.

  5. Apiserver watch and CPU load. Every controller maintains long-lived LIST/WATCH streams (Watch and Informers); every CRD adds objects to etcd and watch fan-out to the apiserver. Dozens of operators, each watching broadly, is a measurable load on the control plane — and a poorly-written operator that re-LISTs without pagination can OOM the apiserver outright (see Common Kubernetes Failures Catalog, category 1).

  6. “Who owns this controller?” The meta-failure. A cluster has 30 controllers; an incident implicates one of them; nobody on the current team installed it, knows its config, or knows whether removing it is safe. Operational knowledge has not scaled with operator count.

Configuration / API Surface — the webhook-wedge mechanism

# Each operator typically ships a ValidatingWebhookConfiguration like this.
# Stack a dozen of these and the interaction surface explodes.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata: { name: some-operator-webhook }
webhooks:
  - name: validate.some-operator.example.com
    failurePolicy: Fail                  # ← webhook down ⇒ DENY every matching write
    namespaceSelector: {}                # ← empty selector: intercepts EVERYTHING, incl. kube-system
    rules:
      - apiGroups: ["*"]                 # ← broad rule: matches far more than this operator needs
        apiVersions: ["*"]
        resources: ["*"]
        operations: ["CREATE", "UPDATE"]
    clientConfig:
      service: { name: some-operator, namespace: ops }   # backed by Pods that can themselves fail

Line-by-line, every line is a sprawl hazard. failurePolicy: Fail means an unavailable backend freezes writes — survivable for one well-run operator, a cluster-wide outage waiting to happen across a dozen. An empty namespaceSelector intercepts kube-system, so the webhook can block the very controllers that would recover it. The wildcard rules make the webhook a chokepoint for all API traffic, not just this operator’s resources. And the backing service is itself Pods that can crash. One such webhook is a managed risk; twenty, owned by no one, is a structural fragility. The mitigation pattern — failurePolicy: Ignore, tight namespaceSelector excluding system namespaces, narrow rules, bounded timeoutSeconds — is detailed in Common Kubernetes Failures Catalog and the admission webhook good-practices docs.

The Correct Alternative

Operators are not the problem; ungoverned operators are. Five prescriptive moves:

  1. Treat every operator as a long-term liability with a named owner. An operator is not a feature you install and forget; it is a piece of always-running infrastructure that must be patched, version-tracked against the Kubernetes minor, and on someone’s runbook. If no individual or team will own it for its lifetime, do not install it. This single rule prevents most sprawl.

  2. Inventory and prune. Periodically enumerate every CRD, every operator, and every webhook in the cluster, and ask of each: is this still used, who owns it, does it tolerate our next K8s version? Anything unowned or unused is removed. Sprawl is reversible only if someone deliberately reverses it.

  3. Prefer built-in mechanisms and CEL-based policy over yet another webhook controller. Before adding a policy operator, ask whether the need is met by a built-in admission controller, Pod Security Admission, or — critically — an in-process ValidatingAdmissionPolicy (CEL-based, Kubernetes v1.30 [stable] per the validating admission policy docs) and its now-mature sibling MutatingAdmissionPolicy (KEP-3962: alpha in v1.32, beta in v1.34, and Kubernetes v1.36 [stable], enabled by default, per the mutating admission policy docs, as of May 2026). CEL policies run inside the apiserver: no external webhook backend, no Pods to crash, no failurePolicy wedge, no extra controller to own. With both the validating and mutating halves now stable, the entire common case of admission policy — reject-on-condition and default-injection — can be expressed in-process, removing the most common reason to install a policy operator at all. Every webhook not added is a controller not owned.

  4. Use Operator Lifecycle Manager for managed install and upgrade. Where operators are genuinely needed, OLM provides versioned subscriptions, dependency resolution, and controlled upgrades — turning a pile of ad-hoc helm installs into a managed inventory. It does not reduce the count, but it makes the count governable.

  5. Run an “operator budget.” Make the total a number the platform team explicitly owns and reviews. Adding operator N+1 requires justifying it against the budget — and, ideally, retiring something. A budget converts “just one more” from a reflex into a decision.

The governing principle mirrors the sidecar-sprawl rule one level up: the cost of a controller is not its install command; it is its entire operational lifetime and its interaction with every other controller already present. Price it accordingly.

Production Notes

  • The Operator SDK’s own best-practices guidance pushes against sprawl from the design side: an operator should not manage more than one custom-resource kind, and an operator with too many concerns should be split — discipline that, applied cluster-wide, also argues for fewer, well-scoped operators rather than many overlapping ones (sdk.operatorframework.io).
  • The Common Kubernetes Failures Catalog records the canonical incident shape: “a simple admission webhook led to a cluster outage” — a ValidatingWebhookConfiguration interacting badly with node auto-repair to lose nodes and block control-plane upgrades. Every operator that ships a webhook is a chance to reproduce it.
  • The ValidatingAdmissionPolicy / MutatingAdmissionPolicy / CEL movement is the platform-engineering community’s structural answer to webhook sprawl: pull policy back into the apiserver so it stops being yet-another-controller. With both halves now stable (validating since v1.30, mutating since v1.36), the correct default for new policy needs is “no new controller” — reach for a webhook-based policy operator only when a need genuinely exceeds CEL’s reach.
  • Sprawl is an organizational failure as much as a technical one. The decisive question at install time is not “does this operator work?” but “who will still own this operator, and track its version against ours, in eighteen months?” If that question has no answer, the operator is a future incident with a deferred trigger date.

Uncertain

Verify: that CEL admission policy can replace a given webhook. ValidatingAdmissionPolicy (stable v1.30) and MutatingAdmissionPolicy (stable v1.36) cover the common reject-and-default cases, but CEL runs in-process with no network egress and limited cross-object visibility — policies that must call an external service, validate against data outside the admission request, or perform complex multi-object correlation may still require a webhook-based controller. Reason: feature scope of in-process CEL is evolving release-to-release; do not assume a specific webhook is replaceable without checking the CEL admission feature set for your apiserver minor. To resolve: prototype the policy as a ValidatingAdmissionPolicy/MutatingAdmissionPolicy against your target cluster version before retiring the webhook. The “dozens of controllers” framing remains a qualitative observation of mature production clusters, not a measured statistic. uncertain

See Also

Graph View