vault

Custom Resource Definition

15 min read

Custom Resource Definition

A CustomResourceDefinition (CRD) is the Kubernetes mechanism for adding entirely new kinds of resources to the cluster’s API — extending the API surface without recompiling or restarting the API server. The Kubernetes documentation frames the distinction precisely: “A resource is an endpoint in the Kubernetes API that stores a collection of API objects of a certain kind … A custom resource is an extension of the Kubernetes API that is not necessarily available in a default Kubernetes installation. … The CustomResourceDefinition API resource allows you to define custom resources” (k8s.io — Custom Resources). Once you kubectl apply a CRD object, the API server immediately begins serving a full RESTful endpoint for the new kind — kubectl get, list, watch, create, update, patch, delete all work, the resource participates in Kubernetes RBAC, it is stored in etcd alongside built-in resources, and it gets validation, defaulting, and a /status subresource if you ask for one. The crucial conceptual point, and the reason this note is paired with Operator Pattern: a CRD is pure data — it defines a type and stores instances, but it does nothing. A Certificate CRD does not issue certificates; a PostgresCluster CRD does not provision databases. A CRD becomes behavior only when a controller — see Controller Pattern — watches instances of it and acts. The CRD is the API; the controller is the implementation. This note covers the API half: how you declare a new type, what the API server gives you for free, and where the CRD approach reaches its limits.

Mental Model

flowchart TB
    AUTHOR[Cluster admin or<br/>operator vendor] -->|"kubectl apply -f crd.yaml"| CRDOBJ["CustomResourceDefinition object<br/>(group, versions[], scope, names)"]
    CRDOBJ -->|registers a new type| APISERVER[kube-apiserver]
    APISERVER -->|"now serves a REST endpoint:<br/>/apis/GROUP/VERSION/.../KIND"| ENDPOINT[New API endpoint]
    USER[User] -->|"kubectl apply -f my-postgres.yaml"| ENDPOINT
    ENDPOINT -->|validate against<br/>OpenAPI v3 + CEL schema| VALIDATE{Schema valid?}
    VALIDATE -- no --> REJECT[422 rejected]
    VALIDATE -- yes --> STORE[(etcd<br/>stores the custom-resource instance)]
    STORE -.->|watch| CONTROLLER["Controller / Operator<br/>(separate process)"]
    CONTROLLER -->|"acts: provision StatefulSet,<br/>configure replication, ..."| WORLD[Real world]
    CONTROLLER -->|"writes .status"| STORE
    style CRDOBJ fill:#e8f0fe
    style CONTROLLER fill:#fde8e8

What this diagram shows. Two distinct lifecycles. The blue path is the CRD itself: applying a CustomResourceDefinition object teaches the API server a new type, after which the API server serves, validates, and stores instances of that type entirely on its own — no custom code runs. The red box is the controller, a separate program that watches the stored instances and makes the real world match them. The diagram’s load-bearing insight: everything in the blue path is free and built in; nothing in the red box is. A CRD with no controller is a perfectly functional, validated, queryable database table that no one reads. People who deploy a CRD and expect things to happen have confused the schema for the engine.

Mechanical Walk-through

What the API server gives you for free

When a CustomResourceDefinition is accepted, the API server immediately provides, with zero custom code:

  • A full REST endpoint under /apis/<group>/<version>/... supporting get / list / watch / create / update / patch / delete with the same semantics as built-in resources.
  • Watch streams — controllers and kubectl get -w can subscribe to changes (see Watch and Informers).
  • RBAC integration — the new kind is an ordinary resource for Kubernetes RBAC; you write Roles with apiGroups, resources, verbs exactly as for Pods.
  • Schema validation — the API server rejects instances that violate the CRD’s OpenAPI v3 schema before they reach etcd.
  • Defaulting — schema default: values are applied server-side on write.
  • The standard object shape — every custom-resource instance is {apiVersion, kind, metadata, spec, status}, integrating cleanly with Kubernetes Object Model, Owner References and Garbage Collection, Finalizers, labels, and annotations.
  • kubectl integration — kubectl get, describe, explain all work; additionalPrinterColumns customizes the get table output.
  • Audit logging — operations on the custom resource appear in the audit log like any other.
  • Field selectors over selectableFields — spec.versions[*].selectableFields declares JSONPaths (e.g. .spec.color) that the API server then accepts in --field-selector filters on LIST/WATCH/DELETECOLLECTION, indexed in etcd just like the built-in spec.nodeName selector on Pods. This is GA since Kubernetes 1.32 (KEP-4358; Field Selectors), having moved through alpha in 1.30 and beta in 1.31. Before this, every CRD effectively supported only metadata.name and metadata.namespace in field selectors, which forced controllers into full-list-and-filter patterns over potentially huge collections.

Anatomy of a CRD object

A CustomResourceDefinition has four substantive parts inside its spec:

  1. group — the API group the new kind belongs to, e.g. acme.example.com. Convention: a DNS domain you control, to avoid collisions.
  2. versions[] — a list of versions, because a CRD evolves over time. Each version entry carries:
    • name — e.g. v1alpha1, v1beta1, v1.
    • served (boolean) — whether the API server serves this version’s endpoint. You can keep an old version served: true for backward compatibility while clients migrate.
    • storage (boolean) — exactly one version must be the storage version; that is the form the object is persisted as in etcd. Other versions are converted to/from it on read/write.
    • schema.openAPIV3Schema — the OpenAPI v3 schema validating instances of this version (see below).
    • subresources — optionally enables the /status and /scale subresources.
    • additionalPrinterColumns — extra columns for kubectl get.
  3. scope — Namespaced (instances live in a namespace, like Pods) or Cluster (instances are cluster-global, like Nodes). Chosen once, immutable.
  4. names — kind (PascalCase, e.g. CronTab), plural (lowercase, the URL path segment, e.g. crontabs), singular, listKind, and optional shortNames (e.g. ct) and categories (so the kind appears under kubectl get all-style groupings).

Structural schemas (mandatory in apiextensions.k8s.io/v1)

Early CRDs (the apiextensions.k8s.io/v1beta1 API) could be schemaless — the API server stored arbitrary JSON. That is gone. The GA API, apiextensions.k8s.io/v1, generally available since Kubernetes 1.16 (k8s.io — CRDs), requires a structural schema. The structural-schema rules are precise (see the Future of CRDs: Structural Schemas blog and the CRD docs): every leaf field has a declared type; logical combinators (allOf, anyOf, oneOf, not) are restricted to value validation and may not introduce or alter type/shape (they cannot live at the top of the schema or at object roots in a way that obscures the property set); and the property tree must be statically knowable. Structural schemas are what unlock the API-server-side machinery the rest of this note depends on: pruning of unknown fields, server-side defaulting from default:, conversion between versions, server-side kubectl apply (SSA), and field selectors over selectableFields.

The legacy apiextensions.k8s.io/v1beta1 API was deprecated in Kubernetes 1.16 and removed in Kubernetes 1.22 (Deprecated API Migration Guide). Migration to v1 is mechanical but not field-identical: spec.scope is no longer defaulted to Namespaced and must be set; spec.version, spec.validation, spec.subresources, and spec.additionalPrinterColumns move under each entry of spec.versions[]; spec.conversion.webhookClientConfig moves to spec.conversion.webhook.clientConfig; the JSONPath field on printer columns is renamed jsonPath; spec.preserveUnknownFields: true is disallowed at the spec level (use x-kubernetes-preserve-unknown-fields: true inside schema nodes); and spec.versions[*].schema.openAPIV3Schema is mandatory. Any pre-2020 CRD shipped under v1beta1 will fail to apply on a current cluster.

Validation: OpenAPI v3 plus CEL rules

Two layers of validation:

  1. OpenAPI v3 structural validation — types, required, enum, minimum/maximum, minLength/maxLength, pattern (regex), format. The API server enforces these on every write.
  2. CEL validation rules (x-kubernetes-validations) — for cross-field and conditional constraints that OpenAPI cannot express (“spec.replicas must be odd”, “spec.maxSize must be ≥ spec.minSize”, “spec.tier cannot change once set”). These use the Common Expression Language; see CEL in Kubernetes. CEL rules were beta and on-by-default since Kubernetes 1.25, and reached GA in Kubernetes 1.29 (k8s.io — CRD validation rules beta; Google Open Source Blog — CRD validation using CEL). They are a partial replacement for validating admission webhooks — running in-process in the API server, with no extra network hop, no webhook to operate, and no availability risk.

Subresources: /status and /scale

A CRD can opt into two subresources via spec.versions[].subresources:

  • status: {} — splits the resource into a /status REST endpoint. This is what lets a controller write .status (observed state) without bumping metadata.generation (which then tracks only user-driven .spec changes — the basis of the “has the controller caught up?” check). Without the status subresource, a write to .status is just an ordinary update and the generation/observedGeneration discipline breaks. Real operators almost always enable it.
  • scale: {} — exposes a /scale endpoint, mapping configured JSONPaths for spec.replicas, status.replicas, and a label selector. This is what lets the Horizontal Pod Autoscaler scale a custom resource — kubectl scale and the HPA both target the /scale subresource generically, so any CRD that wires it up is autoscalable.

Conversion between versions

When a CRD has multiple versions[], the API server must convert objects between them — a client may read v1 while the storage version is v1beta1. Two strategies, set in spec.conversion.strategy:

  • None — the API server changes only the apiVersion field and otherwise passes the object through unchanged. Valid only if all versions are structurally identical (a pure rename). Most early CRDs use this.
  • Webhook — the API server calls an external conversion webhook that translates the object between versions. Required whenever versions differ in shape (a renamed field, a restructured nested object). See Conversion Webhooks for the request/response contract and the operational cost (the webhook is now on the critical path of every read of that resource).

Configuration / API Surface

A complete CRD plus a custom-resource instance, annotated:

apiVersion: apiextensions.k8s.io/v1          # the GA CRD API — GA since K8s 1.16
kind: CustomResourceDefinition
metadata:
  # name MUST be exactly <plural>.<group>
  name: crontabs.stable.example.com
spec:
  group: stable.example.com                  # the API group of the new kind
  scope: Namespaced                          # instances live in namespaces
  names:
    kind: CronTab                            # PascalCase; what `kind:` says in instances
    plural: crontabs                         # lowercase; the URL path segment
    singular: crontab
    shortNames: [ct]                         # `kubectl get ct`
    categories: [all]                        # appears in `kubectl get all`
  versions:
    - name: v1                               # this version's identifier
      served: true                           # API server serves /apis/.../v1/crontabs
      storage: true                          # persisted in etcd in THIS version's shape
      subresources:
        status: {}                           # enable the /status subresource
        scale:                               # enable the /scale subresource (HPA-able)
          specReplicasPath: .spec.replicas
          statusReplicasPath: .status.replicas
          labelSelectorPath: .status.selector
      additionalPrinterColumns:              # extra `kubectl get crontabs` columns
        - name: Schedule
          type: string
          jsonPath: .spec.cronSpec
        - name: Replicas
          type: integer
          jsonPath: .spec.replicas
      schema:
        openAPIV3Schema:                     # STRUCTURAL schema — mandatory in v1
          type: object
          properties:
            spec:
              type: object
              properties:
                cronSpec:
                  type: string
                  pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'  # 5-field cron
                image:
                  type: string
                replicas:
                  type: integer
                  minimum: 1
                  maximum: 10
                  default: 1                 # server-side defaulting
              required: [cronSpec, image]
              # CEL cross-field rule — GA since K8s 1.29
              x-kubernetes-validations:
                - rule: "self.replicas <= 5 || self.image.startsWith('ha-')"
                  message: "replicas > 5 requires an image whose name starts with 'ha-'"
            status:                          # the controller writes here, not the user
              type: object
              properties:
                replicas:
                  type: integer
                selector:
                  type: string
# An INSTANCE of the new kind — applied by an ordinary user.
apiVersion: stable.example.com/v1
kind: CronTab
metadata:
  name: nightly-report
  namespace: analytics
spec:
  cronSpec: "0 2 * * *"
  image: report-runner:v3
  replicas: 2
# .status is omitted — only the controller writes it.

Line-by-line, the points worth internalizing:

  • metadata.name must be <plural>.<group>. The API server enforces this naming; it is not arbitrary.
  • served vs storage. Exactly one version is storage: true. Multiple may be served: true — that is how you run v1beta1 and v1 simultaneously during a migration.
  • The schema is structural. Every field has a type. Unknown fields a user adds are pruned (silently dropped) by the API server unless the schema explicitly allows them via x-kubernetes-preserve-unknown-fields: true.
  • default: is server-side. A user who omits replicas gets 1 written into etcd by the API server — no controller involved.
  • x-kubernetes-validations runs CEL in the API server; the rejection happens at admission, before etcd, with no webhook.
  • The instance carries no .status. Users declare .spec (desired state); the controller owns .status (observed state). This is the Desired State vs Observed State split made concrete.

Failure Modes

  • CRD applied, nothing happens. The single most common confusion: a CRD without a controller is inert. kubectl get crontabs works, instances are stored — but no real-world action occurs because no operator is running. The CRD is the schema; you also need the engine.
  • v1beta1 CRD on a modern cluster. apiextensions.k8s.io/v1beta1 was removed in Kubernetes 1.22; legacy CRDs fail to apply. Migrate the manifest to apiextensions.k8s.io/v1.
  • Non-structural schema rejected. The v1 API rejects CRDs whose schema is not structural (missing type, disallowed oneOf/anyOf placement). The API server’s error message names the offending path.
  • Fields silently disappear. Server-side pruning drops any field not in the structural schema. A user adds spec.newField, applies successfully, then kubectl get shows it gone — because the schema did not declare it. Fix: declare the field, or set x-kubernetes-preserve-unknown-fields: true.
  • Two CRDs claim the same group+kind. Two operator vendors both define kind: Database in group db.io — the second apply conflicts. CRD groups should use DNS domains the author controls.
  • Conversion webhook down. With conversion.strategy: Webhook, if the webhook is unavailable, every read and write of that custom resource fails — the webhook is on the critical path. See Conversion Webhooks.
  • Storage-version change without re-encoding. Flipping which version is storage: true does not rewrite existing objects; they stay in the old storage encoding until next written. The on-cluster Storage Version Migration API (storagemigration.k8s.io/v1beta1, available when the API server runs Kubernetes 1.30 or later with the feature enabled) or a manual kubectl get … -o yaml | kubectl replace -f - sweep is needed before retiring the old version — status.storedVersions on the CRD lists every encoding still present in etcd, and the migration is not complete until only the new version remains in that list.
  • Deleting a CRD deletes all its instances. Removing a CustomResourceDefinition cascades: every custom-resource instance of that kind is garbage-collected. A fat-fingered kubectl delete crd can wipe an operator’s entire managed estate.

Alternatives and When to Choose Them

The main architectural decision is CRD vs. Aggregated API Server:

DimensionCRDAggregated API Server
Implementation effortApply a YAML objectBuild, deploy, and operate a separate API server
Storageetcd, automaticYour choice — etcd, a SQL DB, computed on the fly
Custom business logic on API callsLimited (validation/conversion webhooks)Arbitrary — full control of every verb
Custom subresources beyond status/scaleNoYes
Protobuf serialization, custom field selectorsLimitedFull
Operational costNear zeroA whole extra API server to run and secure

The Kubernetes documentation gives a long comparison table; the practical rule: use a CRD unless you specifically need something a CRD cannot do. Reasons to reach for an aggregated API server: you need to not store objects in etcd (e.g. metrics.k8s.io computes metrics on demand), you need arbitrary subresources, you need protobuf, or you need to run logic on read paths. For >95 % of extensions — including essentially every operator — a CRD is the right choice. Other alternatives:

  • ValidatingAdmissionPolicy (VAP) — if you only need to validate existing resource types with CEL, not define a new type, you do not need a CRD at all.
  • ConfigMap-as-data — if you just need to store structured config and a controller reads it, a ConfigMap works, but you lose typed schema validation, the /status subresource, RBAC granularity per kind, and kubectl ergonomics. A CRD is almost always better once a controller is involved.

Production Notes

  • Operators ship their CRDs. When you install cert-manager, the Prometheus Operator, ArgoCD, or CloudNativePG, step one is always applying a bundle of CRDs (Certificate, ServiceMonitor, Application, Cluster, …). The CRDs and the controller are versioned and released together — a controller expects a specific CRD schema version.
  • CRD upgrades are delicate. Adding an optional field is safe. Removing a field, tightening validation, or changing the storage version requires a migration plan and often a conversion webhook. The Operator Lifecycle Manager exists partly to sequence CRD upgrades safely.
  • Cluster-scoped vs namespaced is permanent. scope cannot be changed after creation; choosing wrong means delete-and-recreate the CRD (destroying all instances). Decide deliberately.
  • CEL validation has displaced many webhooks. Since the 1.29 GA of x-kubernetes-validations, new operators increasingly express their validation as CEL rules in the CRD schema rather than running a validating webhook — fewer moving parts, no availability risk, lower latency. Webhooks remain necessary for validation that needs to query other objects or external systems.
  • additionalPrinterColumns is underused. A well-chosen set of printer columns makes kubectl get <yourkind> genuinely useful for operators debugging the system; vendors that skip it produce CRDs that are painful to operate.

See Also

Graph View

Backlinks