vault

MutatingAdmissionWebhook

9 min read

MutatingAdmissionWebhook

A mutating admission webhook extends the API server’s mutating-admission phase by calling out, over HTTPS, to an external service that may rewrite the object being admitted before it is persisted (Kubernetes — Dynamic Admission Control). The compiled-in MutatingAdmissionWebhook plugin (one of the Admission Controllers) reads MutatingWebhookConfiguration objects; for each request matching a configuration’s rules it POSTs an AdmissionReview, and the webhook’s response carries a JSONPatch that the apiserver applies to the object. Unlike a ValidatingAdmissionWebhook, a mutating webhook changes the object — its canonical use is automatic sidecar injection (Istio, Linkerd, the Vault Agent injector). The webhook configuration API admissionregistration.k8s.io/v1 reached GA in Kubernetes 1.16.

Mental Model

A mutating webhook is the same dynamic escape hatch as a validating webhook, but it runs in the earlier phase and is permitted to modify the object rather than only veto it. Where validating webhooks answer “may this object exist?”, mutating webhooks answer “what should this object actually be?” — defaulting fields, adding labels, and most importantly injecting containers and volumes that the author never wrote.

The mental shift for newcomers: the Pod that lands in etcd is not the Pod in your YAML. When Istio is installed, a Pod manifest with one container ends up with three (your container, the istio-proxy sidecar, an init container) — none of which appear in your file. They were spliced in by a mutating webhook at admission time. This is why kubectl get pod -o yaml shows containers you did not write.

sequenceDiagram
    autonumber
    participant U as kubectl
    participant API as kube-apiserver<br/>(MutatingAdmissionWebhook plugin)
    participant M1 as Istio sidecar injector
    participant M2 as Vault agent injector
    U->>API: CREATE Pod (1 container)
    API->>M1: AdmissionReview{request}
    M1-->>API: response{patch: add istio-proxy + initContainer}
    Note over API: object now changed
    API->>M2: AdmissionReview{request} (mutated object)
    M2-->>API: response{patch: add vault-agent}
    Note over API: M2's patch changed the object again
    API->>M1: REINVOKE (reinvocationPolicy: IfNeeded)
    M1-->>API: response{no further change}
    API->>API: schema validation -> validating admission -> persist
    API-->>U: 201 Created (3+ containers)

Two mutating webhooks plus reinvocation. The insight: webhook order is not guaranteed, and because one webhook’s patch can invalidate another’s assumptions, the apiserver re-invokes webhooks (reinvocationPolicy: IfNeeded) until the object stops changing. A webhook may be called more than once and must therefore be idempotent.

Mechanical Walk-through

  1. An administrator creates a MutatingWebhookConfiguration. Its shape is identical to a ValidatingWebhookConfiguration (rules, clientConfig, failurePolicy, matchPolicy, namespaceSelector, objectSelector, timeoutSeconds, sideEffects, admissionReviewVersions) plus one extra field: reinvocationPolicy.
  2. On a matching create/update, the apiserver POSTs an AdmissionReview to the webhook. The webhook computes the desired changes and returns a response with allowed: true, a patchType: JSONPatch, and a base64-encoded JSON Patch (RFC 6902) describing the edits (a list of add/replace/remove operations on JSON pointers).
  3. The apiserver applies the patch to the object, then proceeds to the next mutating webhook with the already-mutated object.
  4. Mutating webhooks are evaluated sequentially (not in parallel — unlike validating webhooks) precisely because each one sees the cumulative result of the previous. The apiserver evaluates them lexicographically by webhook name, but this is explicitly not a correctness contract: webhooks using reinvocationPolicy: IfNeeded may be reordered to cut down on extra invocations, and you cannot force “injector A before injector B.” Note also that this whole webhook pass runs after the in-process mutating admission policies have already mutated the object (see the ordering note below).
  5. Reinvocation. After the full mutating pass, if any mutating step — another webhook or a mutating admission policy — changed the object, every webhook whose reinvocationPolicy is IfNeeded is called again with the final object. This catches the case where webhook B’s mutation should have made webhook A run differently. Reinvocation happens at most one extra round: a webhook is invoked at most twice. A webhook with reinvocationPolicy: Never (the default) runs only once. The reinvocation budget is shared with mutating admission policies, which is why a webhook must be idempotent even if it is the only webhook in the cluster.
  6. After mutating admission settles, the object goes through schema validation and validating admission (see ValidatingAdmissionWebhook) and is then persisted.

Configuration / API Surface

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: istio-sidecar-injector
webhooks:
- name: sidecar-injector.istio.io
  rules:
  - apiGroups:   [""]
    apiVersions: ["v1"]
    operations:  ["CREATE"]            # injection only makes sense at Pod create
    resources:   ["pods"]
    scope:       "Namespaced"
  clientConfig:
    service:
      name: istiod
      namespace: istio-system
      path: "/inject"
      port: 443
    caBundle: <base64 PEM>
  admissionReviewVersions: ["v1"]
  sideEffects: None
  timeoutSeconds: 10                   # 1..30; default 10
  failurePolicy: Fail                  # Fail | Ignore
  matchPolicy: Equivalent
  reinvocationPolicy: IfNeeded         # IfNeeded | Never -- re-run if a later webhook mutated the object
  namespaceSelector:
    matchLabels:
      istio-injection: enabled         # only inject in opted-in namespaces
  objectSelector:
    matchExpressions:
    - key: sidecar.istio.io/inject     # let an individual Pod opt out via label
      operator: NotIn
      values: ["false"]

The fields shared with ValidatingWebhookConfiguration behave identically (see ValidatingAdmissionWebhook); the points specific to mutation:

  • reinvocationPolicy — IfNeeded re-runs this webhook if a later mutating webhook changed the object after it ran; Never (default) runs it exactly once. Use IfNeeded when correctness depends on the final object (e.g. a label-injector that must label the sidecar containers another webhook added). The webhook must be idempotent because it may run twice.
  • namespaceSelector + objectSelector — the standard opt-in pattern for injectors: a namespace label (istio-injection: enabled) enables injection cluster-team-wide, and a per-Pod label (sidecar.istio.io/inject: "false") lets an individual workload opt back out.
  • operations: ["CREATE"] — sidecar injection runs at Pod creation only; you cannot meaningfully inject a container into an already-running Pod (Pods are immutable in their container set), so injectors do not match UPDATE.

There is no reorderPolicy field on MutatingWebhookConfiguration in admissionregistration.k8s.io/v1 (verified against the v1 API as of Kubernetes 1.36) — mutating webhook ordering is not user-controllable, and reinvocationPolicy is the only mutation-specific knob. What ordering does exist is fixed by the apiserver: across the three mutating mechanisms it is built-in mutating controllers → mutating admission policies → mutating webhooks (KEP 3962 README), so by the time your webhook runs, all CEL policies have already mutated the object. Among webhooks themselves the apiserver evaluates lexicographically by webhook name, but webhooks using reinvocationPolicy: IfNeeded “may be reordered to minimize the number of additional invocations” (Dynamic Admission Control) — so name order is not a contract you may depend on for correctness.

Failure Modes

  • Cluster wedge. Identical to the ValidatingAdmissionWebhook failure mode: failurePolicy: Fail + broad rules + an in-cluster webhook whose Pods die = the cluster cannot create the Pods that would heal the webhook. Mitigations are the same — exclude kube-system, scope selectors tightly, run redundantly, monitor.
  • Non-idempotent injection → double sidecars. Because reinvocation can call a webhook twice, an injector that naively appends a sidecar without checking whether it already added one will inject two copies. Real injectors guard with a matchConditions or an annotation (sidecar.istio.io/status) marking the Pod as already-injected.
  • Ordering surprises. Two injectors that both add init containers, or both touch the same volume, can produce different results depending on the (unguaranteed) order the apiserver happens to call them. This is a genuine, hard-to-debug footgun — there is no supported way to force “injector A before injector B.” The defence is to keep mutating webhooks independent and non-overlapping in what they touch.
  • JSON Patch fragility. A JSON Patch encodes operations against specific JSON pointers (/spec/containers/0/...). If an earlier webhook reindexed an array, a later webhook’s hard-coded index can patch the wrong element. Webhooks that emit JSON Patch must compute pointers against the current object, not the original.
  • Mutation invisible to the author. A developer debugging “why does my Pod have a container I never declared” must know to look at MutatingWebhookConfiguration objects. kubectl get mutatingwebhookconfigurations is the diagnostic.

Alternatives and When to Choose Them

  • MutatingAdmissionPolicy (in-process CEL). Stable since Kubernetes 1.36 (admissionregistration.k8s.io/v1, on by default). For defaulting and labelling that can be expressed as a CEL ApplyConfiguration or JSONPatch, prefer it: no network hop, no certificate management, no wedge risk, and it runs earlier in the mutating pass (before any webhook). The honest dividing line: a CEL policy can only compute its mutation from data already present in the admission request (the object, oldObject, params, namespaceObject, the requesting user) because CEL has no network and no clock-driven side effects; a webhook is the only option when the mutation needs arbitrary code or external state — a value fetched from an external API, a freshly generated secret, a TLS cert minted on the fly, a decision that depends on querying another system. Classic sidecar injection sits right on the boundary: a static, known sidecar is now expressible as a CEL ApplyConfiguration, but an injector that templates the sidecar from cluster-side configuration it must fetch still needs the webhook. Expect the simple cases to migrate to policies and the genuinely-needs-external-data cases to stay on webhooks.
  • Built-in mutators. ServiceAccount, DefaultStorageClass, Priority, DefaultTolerationSeconds are compiled-in mutating plugins; if one already does what you need, no webhook is required.
  • vs ValidatingAdmissionWebhook. Same protocol, same configuration shape minus reinvocationPolicy. Choose mutating when you can fix the object (inject, default, label); choose validating when you can only reject an invariant violation. Many policy engines register both.

Production Notes

  • Sidecar injection is the canonical use. Istio’s istiod serves /inject to add the istio-proxy Envoy sidecar and an istio-init init container; Linkerd’s linkerd-proxy-injector adds linkerd-proxy; HashiCorp’s Vault Agent Injector adds a vault-agent sidecar (and init container) that fetches secrets and writes them to a shared emptyDir. All three are MutatingWebhookConfiguration objects, all three gate on a namespace label, all three are idempotent. See Sidecar Containers and HashiCorp Vault on Kubernetes.
  • The native sidecar container feature (init container with restartPolicy: Always, GA 1.33) changes how injected sidecars are expressed but not that a mutating webhook does the injecting — modern injectors emit native sidecar init containers via the webhook patch.
  • Good practice mirrors validating webhooks: scope rules to CREATE on pods only, exclude control-plane namespaces, set a short timeoutSeconds, run ≥2 replicas with a PodDisruptionBudget, and treat the injector’s certificate rotation as a first-class operational concern.

See Also

Graph View

Backlinks