vault

Conversion Webhooks

14 min read

Conversion Webhooks

A conversion webhook is the mechanism by which a CustomResourceDefinition with multiple API versions translates an object from one version to another on the fly (Kubernetes — CRD Versioning). A CRD may serve v1beta1, v1, and v2 simultaneously, but every object is stored in exactly one version — the storage version — in etcd. When a client requests an object at a version other than the storage version, the kube-apiserver must convert it. The CRD’s spec.conversion field selects how: strategy: None (the versions are structurally identical and the apiserver merely relabels apiVersion) or strategy: Webhook (the apiserver calls an external HTTPS endpoint that performs the conversion). Conversion is what lets an operator evolve its API across versions without breaking existing clients or stored data — old clients keep speaking v1beta1, new clients speak v2, and the same etcd bytes serve both. The conversion contract is the ConversionReview request/response object, and the recommended design is the hub-and-spoke pattern: pick one version as the hub and convert every other version to and from it.

Mental Model

A CRD’s version list is a set of lenses onto a single stored fact. Storage holds one canonical representation; every served version is a projection of it. Conversion is the function that maps between projections.

flowchart TB
    C1["Client A<br/>GET .../v1beta1/widgets/foo"]
    C2["Client B<br/>GET .../v2/widgets/foo"]
    API["kube-apiserver"]
    ETCD[("etcd<br/>stored as v1 — the storage version")]
    WH["Conversion webhook<br/>(HTTPS endpoint)"]
    C1 --> API
    C2 --> API
    API -- "read raw bytes" --> ETCD
    API -- "ConversionReview<br/>objects=[v1], desiredAPIVersion=example.com/v1beta1" --> WH
    WH -- "convertedObjects=[v1beta1]" --> API
    API -- "v1beta1 object" --> C1
    API -- "ConversionReview desiredAPIVersion=v2" --> WH
    WH -- "convertedObjects=[v2]" --> API
    API -- "v2 object" --> C2

What this diagram shows. Both clients read the same object, stored once in etcd as the storage version (v1 here). The apiserver, not the client, drives conversion: it reads the raw stored bytes, sends them to the webhook in a ConversionReview tagged with the desiredAPIVersion, and returns the webhook’s output. The insight to extract: the storage version is an implementation detail invisible to clients — a client that only ever uses v1beta1 never knows the object is stored as v1. The webhook is the only place version-specific transformation logic lives; the apiserver itself knows nothing about your field semantics. This is also why the webhook is on the read path of every request for non-storage versions — it must be fast and highly available.

Mechanical Walk-through

Why conversion is needed at all

A CRD evolves. Version v1beta1 had a field cronSpec; the team renamed it to schedule in v1. Three facts are now simultaneously true: (1) objects created months ago are stored in etcd in whatever was the storage version then; (2) a controller written against v1 expects schedule; (3) a legacy script still POSTs v1beta1 objects with cronSpec. Without conversion, the apiserver would have to either reject one of these or silently lose data. Conversion makes all three consistent: every read and write passes through a function that losslessly round-trips the object between the requested version and the storage version.

The “lossless” requirement is the hard part. If v1 adds a field that v1beta1 lacks, converting v1 → v1beta1 → v1 must not drop it. The canonical trick: when down-converting, stash the unrepresentable data in an annotation so the up-conversion can restore it. The Kubebuilder multiversion tutorial demonstrates exactly this pattern. Concretely, suppose v1 adds spec.timeoutSeconds that v1beta1 cannot express. The v1 → v1beta1 direction writes metadata.annotations["example.com/timeoutSeconds"] = strconv.Itoa(src.Spec.TimeoutSeconds) and leaves the field out of the projected v1beta1 object; the v1beta1 → v1 direction reads that annotation back, parses it, and deletes it from the converted object. A client at v1beta1 sees the annotation (which is fine — annotations are free-form), and a client at v1 sees the typed field round-tripped intact. The pattern is mechanical, but it is also the only place where data-loss bugs originate, so every spoke’s ConvertTo and ConvertFrom should have round-trip unit tests that prove v1 == FromHub(ToHub(v1)) for a representative corpus.

When conversion actually fires

Conversion is not invoked on every request — only on those where the wire form needs to be in a version different from the stored form. The precise rules (apiextensions-apiserver conversion code):

  1. On a GET/LIST/WATCH at version V, the apiserver reads the raw bytes from etcd, parses them as the storage version S, and if V ≠ S calls the conversion webhook with desiredAPIVersion = V. If V = S, no webhook call.
  2. On a CREATE/UPDATE/PATCH posted at version V, the apiserver first runs admission against the object at version V, then converts to S before writing to etcd. If V ≠ S, that initial conversion is a webhook call; the response path may then need another conversion back to V for the client.
  3. On a strategic-merge or JSON PATCH against version V, the apiserver may need to convert the stored object to V, apply the patch, then convert back to S to write — potentially two webhook calls per patch request, in opposite directions.
  4. Webhooks for the same ConversionReview are batched per LIST page: a LIST returning 500 stored objects produces one webhook call carrying 500 entries in request.objects, not 500 separate calls. Watch events, by contrast, convert one object at a time.

The practical takeaway: any time served includes versions other than the storage version, the conversion webhook sits on the critical path of nearly every operation against that resource — including the operator’s own reconcile reads. The simplest way to eliminate that risk during a transition is to bump the storage version to the new served version and run a storage-version migration as soon as feasible, after which the now-only-served v1beta1 (kept for legacy clients) is the cheap direction and the new version is webhook-free.

The two strategies

strategy: None — the apiserver does the conversion itself, and that conversion is only relabeling the apiVersion string. This is correct only if every served version has a structurally identical schema (no field renamed, no type changed, no field added or removed in a way that matters). It is the right choice for a “version bump with no schema change” — e.g. promoting v1beta1 to v1 verbatim. Pick None whenever you can; it has no extra moving parts.

strategy: Webhook — the apiserver calls an external HTTPS service to perform conversion. Required the moment any two served versions differ structurally. The webhook is configured under spec.conversion.webhook.

The ConversionReview exchange

When conversion is needed, the apiserver POSTs a ConversionReview object (apiVersion: apiextensions.k8s.io/v1) to the webhook. The request carries:

  • request.uid — a unique identifier for this call (the webhook must echo it).
  • request.desiredAPIVersion — the target version, e.g. example.com/v1.
  • request.objects — the list of objects to convert, each in its current version.

The webhook converts every object and returns a ConversionReview whose response carries:

  • response.uid — the echoed request UID.
  • response.convertedObjects — the objects, now in desiredAPIVersion, in the same order as request.objects.
  • response.result — a metav1.Status with status: Success or status: Failed (plus a message on failure).

A single ConversionReview may contain many objects — a LIST of 500 stored objects at a non-storage version produces one webhook call with 500 entries, not 500 calls.

The hub-and-spoke pattern

With N versions, naïvely you would need N×(N−1) conversion functions (every pair, both directions). The hub-and-spoke pattern collapses this to 2×(N−1): designate one version as the hub (conventionally the storage version), and write only spoke ↔ hub conversions. To convert v1beta1 → v2, the webhook runs v1beta1 → hub → v2. controller-runtime’s pkg/conversion package formalizes this with two interfaces. The hub type implements Hub — a marker interface whose only requirement is a single Hub() method with no parameters or return value, distinguishing it from any other runtime.Object. Each spoke type implements Convertible:

type Convertible interface {
    runtime.Object
    ConvertTo(dst Hub) error      // me → hub
    ConvertFrom(src Hub) error    // hub → me
}

You write exactly two methods per spoke; the framework’s webhook handler in pkg/webhook/conversion decodes each ConversionReview object, looks up which type is the hub, and composes the two-step path automatically. The hard work — the field-by-field mapping inside ConvertTo/ConvertFrom — remains entirely yours.

Configuration / API Surface

A CRD with two structurally different versions and a webhook converter:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crontabs.example.com
spec:
  group: example.com
  scope: Namespaced
  names:
    plural: crontabs
    singular: crontab
    kind: CronTab
  versions:
    - name: v1beta1
      served: true            # clients may use this version
      storage: false          # NOT the storage version
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                cronSpec: { type: string }   # old field name
    - name: v1
      served: true
      storage: true            # exactly ONE version has storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                schedule: { type: string }   # renamed from cronSpec
  conversion:
    strategy: Webhook          # None would be illegal — schemas differ
    webhook:
      clientConfig:
        service:               # in-cluster service reference
          namespace: default
          name: conversion-webhook
          port: 443
          path: /convert       # optional; defaults to "/"
        caBundle: LS0tLS1CRUdJTi...   # base64 CA cert to verify the TLS endpoint
      conversionReviewVersions:        # ConversionReview API versions the webhook accepts
        - v1
        - v1beta1

Line-by-line, the load-bearing points:

  • served — whether clients may use this version at all. A version can be served: false (kept only so old etcd data can still be decoded) while no longer accepting traffic.
  • storage: true — exactly one version must set this. It is the version written to etcd. Changing it requires migrating existing objects (see Failure Modes).
  • conversion.strategy — None is only correct when all served schemas are structurally identical for the fields the user touches. The apiserver does not validate that the schemas agree before accepting the CRD: it simply changes the apiVersion string and prunes unknown fields per the destination version’s schema (CRD Versioning — None strategy). If you mark strategy: None while versions actually differ, the documented consequence is data corruption (“this is unlikely to lead to good results if the schemas differ between the storage and requested version”), not a refusal at admission. The correctness burden is entirely on the CRD author.
  • webhook.clientConfig — either an in-cluster service reference (apiserver builds https://conversion-webhook.default.svc:443/convert) or a raw url. The caBundle lets the apiserver verify the webhook’s TLS certificate; conversion webhooks are always HTTPS.
  • conversionReviewVersions — the apiserver picks the first version in this list that it also supports. List v1 first; v1beta1 is legacy.
  • No configurable timeout. Unlike ValidatingWebhookConfiguration and MutatingWebhookConfiguration, which carry a per-webhook timeoutSeconds field, the WebhookConversion struct in apiextensions.k8s.io/v1 has only two fields: clientConfig and conversionReviewVersions — no timeoutSeconds (apiextensions-apiserver types.go). The API server calls every conversion webhook with a hard-coded 30-second timeout (kubernetes/kubernetes#91338 — closed without resolution); this is observable in API-server logs as ?timeout=30s appended to the webhook URL. The rationale is that conversion blocks the request and has no failurePolicy: Ignore escape (every served-but-not-stored read must convert), so allowing operators to set arbitrarily short timeouts would be a foot-gun. Operationally: assume 30s is the budget you have, and build the webhook to respond well under that — most real conversion calls complete in single-digit milliseconds.

Failure Modes

  • Webhook unavailable. Because conversion is on the read path, an unreachable conversion webhook makes every request for a non-storage version fail — including the controller’s own reconcile reads. Symptom: conversion webhook ... failed errors on kubectl get, operator reconcile stalls. The conversion webhook must be as available as the apiserver itself; run multiple replicas, and prefer strategy: None whenever the schemas permit it.
  • Lossy conversion. Converting v2 → v1beta1 → v2 drops a field that v1beta1 cannot represent. Symptom: an object silently loses data after a round-trip. Fix: stash unrepresentable fields in annotations during down-conversion and restore them on up-conversion (the Kubebuilder tutorial pattern).
  • Stale stored objects after a storage-version bump. Flipping storage: true from v1 to v2 does not rewrite existing etcd objects — they remain encoded in v1 and are converted on every read. The CRD’s status.storedVersions field lists every encoding still present in etcd; you cannot retire v1 (set served: false and drop it from spec.versions) until that list contains only v2. The fix is to force re-persistence: either the long-standing out-of-cluster kube-storage-version-migrator, the in-cluster Storage Version Migration API (storagemigration.k8s.io/v1beta1, available since Kubernetes 1.30 with the feature gate enabled), or a manual kubectl get widgets -A -o yaml | kubectl replace -f - sweep that touches every object.
  • caBundle mismatch. If the webhook’s serving certificate is not signed by the CA in caBundle, the apiserver refuses the TLS handshake. Symptom: x509: certificate signed by unknown authority. cert-manager’s CA-injector commonly automates keeping caBundle in sync.
  • UID or ordering mismatch. A webhook that fails to echo request.uid, or returns convertedObjects in a different order than request.objects, produces apiserver errors. The contract is strict: same UID, same order, same count.

Alternatives and When to Choose Them

  • strategy: None — not a different tool, the other strategy. Always prefer it when every served version is structurally identical; it removes an entire failure surface (no webhook to deploy, secure, and keep available). Most “we just want to call our API v1 now” version bumps need only None.
  • A single, never-versioned CRD — if the resource’s schema is genuinely stable, do not introduce multiple versions at all. Versioning is a cost; pay it only when the API actually evolves.
  • In-process conversion — there is no in-process CRD conversion analogous to ValidatingAdmissionPolicy for admission. CRD conversion is either None or an out-of-process webhook; CEL-based or otherwise in-process conversion of CRDs does not exist in upstream Kubernetes as of v1.36 (May 2026). This is a meaningful availability gap: a CEL-based in-process equivalent would remove the conversion webhook from the hot path the same way ValidatingAdmissionPolicy removed many validating webhooks.
  • Built-in API conversion — core Kubernetes types (apps/v1, batch/v1, …) convert between versions inside the apiserver via compiled-in Go conversion functions. Conversion webhooks are the CRD-author’s equivalent of that machinery, externalized because the apiserver cannot compile in code it has never seen.

Production Notes

  • cert-manager and the Prometheus Operator ship multi-version CRDs and rely on conversion webhooks (or careful None-compatible evolution) in production; their operator binaries serve the conversion endpoint from the same process that runs the controllers, via controller-runtime’s webhook server.
  • controller-runtime hosts the conversion webhook in the same Manager that runs the operator’s controllers (see controller-runtime §webhook scaffolding). One binary, one HTTPS server, both admission and conversion webhooks — so the conversion endpoint shares the operator’s availability story.
  • Kubebuilder’s multiversion tutorial is the canonical worked example: it scaffolds a hub type, ConvertTo/ConvertFrom methods on each spoke, and the annotation-stashing trick for lossless round-tripping. Anyone adding a second version to a real CRD should follow it rather than improvising.
  • Plan the storage-version migration before adding a version. The common operational mistake is to add v2, flip storage to v2, and assume the job is done — leaving old objects converted on every read indefinitely and blocking the eventual removal of v1. Run the storage-version migrator as part of the rollout.
  • Keep old versions served: true long after deprecation. Removing a served version breaks every client still using it. The deprecation should be announced, the client base migrated, and only then the version dropped — exactly the discipline core Kubernetes follows with its own API deprecation policy (see Kubernetes API Groups and Versions).

See Also

Graph View

Backlinks