vault

Database Operators

13 min read

Database Operators

A database operator is the canonical, most-mature application of the Operator Pattern: a domain-specific Kubernetes controller that runs a stateful database with real Day-2 automation — leader election, automated failover, backups and point-in-time recovery, safe rolling version upgrades, monitoring integration — the operational knowledge a skilled DBA would otherwise carry in their head. The category exists because the generic Kubernetes workload controllers are deliberately too generic for databases: a StatefulSet gives a workload stable identities, stable per-Pod storage, and ordered rollout, but it has no idea what a “primary” is, cannot promote the most-caught-up replica, cannot run pg_basebackup, and cannot decide that a crashed Pod means “fail over” rather than “just restart” — and that judgment is precisely where data is lost or saved. Database operators encode that judgment in a reconcile loop behind a clean CRD. This note surveys the canonical examples — CloudNativePG and the Zalando Postgres Operator for PostgreSQL, Strimzi for Apache Kafka, Vitess for sharded MySQL, the MongoDB operators, Elastic Cloud on Kubernetes (ECK) for Elasticsearch, and Redis operators — and the operator-maturity lens for telling a serious one from a glorified StatefulSet wrapper. It is the concrete companion to the abstract Operator Pattern note.

Mental Model

flowchart TB
    subgraph BARE["A bare StatefulSet gives you..."]
        B1[Stable Pod names<br/>db-0, db-1, db-2]
        B2[Stable per-Pod PVCs]
        B3[Ordered rollout]
        B4["Restart-on-crash<br/>(and nothing more)"]
    end
    subgraph OP["A database operator ADDS the DBA's judgment"]
        O1[Leader election +<br/>automated failover<br/>'promote the caught-up replica']
        O2[Backup to object storage +<br/>point-in-time recovery]
        O3[Safe minor & major<br/>version upgrades]
        O4[Connection pooling]
        O5[Monitoring / metrics /<br/>alerting integration]
    end
    CRD["CRD: kind: Cluster<br/>(declarative DB API)"]
    CTRL["Operator controller<br/>(codified DBA reconcile loop)"]
    USER[User] -->|"kubectl apply<br/>kind: Cluster"| CRD
    CRD --> CTRL
    CTRL --> O1 & O2 & O3 & O4 & O5
    CTRL -- "still builds on" --> B1 & B2 & B3 & B4

What this diagram shows. The left column is everything a StatefulSet gives you for free — and where it stops. Restart-on-crash is the entire recovery story a bare StatefulSet knows. The right column is what running a real database actually requires, and what no generic controller can supply because it is application-specific: deciding which replica to promote on primary failure (the most-caught-up one, never a stale one — promoting stale loses committed writes), taking and verifying backups, performing the version-specific upgrade dance (which for Postgres may mean pg_upgrade and replica-by-replica rollout, not a naive rolling restart). The operator’s reconcile loop is the DBA’s runbook turned into code. The insight to extract: a database operator is not “a StatefulSet plus YAML” — it is the failover, backup, and upgrade logic, and you should evaluate operators almost entirely on how well they do those three things.

Mechanical Walk-through

What a good database operator does that a StatefulSet cannot

  1. Leader election and automated failover. The operator continuously observes replication topology and Pod health. When the primary fails, it must pick the replica with the least replication lag, promote it, re-point the read-write Service at the new primary, and reconfigure the surviving replicas to follow it. Getting this wrong — promoting a stale replica — silently loses committed transactions. This is the hardest, least-exercised, highest-stakes code in any database operator.
  2. Backup and point-in-time recovery (PITR). Continuous WAL/binlog archiving to object storage (S3/GCS/Azure Blob) plus periodic base backups, so the database can be restored not just to “the last backup” but to any second before a bad DELETE. The operator schedules backups, tracks the last successful one in .status, and drives restore.
  3. Safe rolling version upgrades. Minor upgrades (patch releases) are usually a careful replica-by-replica rolling restart, primary last (or demote-then-replace). Major upgrades may need pg_upgrade-style logic, schema-migration ordering, and validation gates. A naive StatefulSet rolling update — reverse-ordinal, one at a time — ignores all of this and can break replication or quorum.
  4. Connection pooling. Databases tolerate a bounded number of connections; a fleet of app Pods can exhaust them. Operators integrate a pooler (PgBouncer for Postgres) as a managed component.
  5. Monitoring integration. Exporting Prometheus metrics, emitting Kubernetes Events, wiring up alerting — the operator maturity model’s “Deep Insights” level.

The operator maturity lens

The Operator Framework’s five-level capability maturity model (operatorframework.io) is the right rubric for evaluating a database operator:

LevelNameWhat it means for a database operator
IBasic InstallProvisions the database from the CRD; reports Ready.
IISeamless UpgradesUpgrades the operator and the managed database safely.
IIIFull LifecycleBackups, restore/PITR, automated failover, reconfiguration, member management.
IVDeep InsightsPrometheus metrics, Events, alerting, replication-lag visibility.
VAuto PilotAutonomous scaling, auto-tuning, anomaly detection.

A Level-I operator that installs a database and walks away has barely earned the name and is barely better than hand-written manifests — worse, it implies a safety it does not deliver. The value of a database operator lives at Levels III–V, where it handles the 3 a.m. failover and the quarterly major-version upgrade. Mature operators (CloudNativePG, Strimzi) sit at Level IV–V.

The survey

PostgreSQL — CloudNativePG (CNPG). The current gold-standard PostgreSQL operator, originally EDB’s proprietary product, open-sourced in 2022 and accepted into the CNCF Sandbox on January 21, 2025 (enterprisedb.com), with a request to advance to Incubating already in flight. CNCF reports it has had the more frequent releases and larger contributor base of the Postgres operators. CNPG manages instances, streaming replication, automated failover, continuous backup to object storage (via Barman), and PITR. A notable design choice: CNPG does not use a StatefulSet — it manages each instance with finer-grained controllers for tighter control over rolling updates and failover ordering than StatefulSet’s rigid reverse-ordinal model allows (blog.palark.com). This is a recurring theme in mature operators: StatefulSet semantics are too rigid for orderly failover, so the operator replicates the parts it needs in its own controllers. CNPG’s CRD is kind: Cluster (postgresql.cnpg.io/v1).

PostgreSQL — Zalando Postgres Operator. The other mature Postgres operator, built on Patroni (the HA/failover layer) and Spilo (the Postgres+Patroni container image). It ships built-in PgBouncer connection pooling (CNPG requires a separate pooler deployment) and a “team API” model suited to multi-tenant environments. As of 2026 its release cadence has slowed relative to CNPG (ithy.com comparison). Choose Zalando if you already run a Patroni-based stack or want its team-isolation model; CNPG is the default recommendation for new deployments.

Apache Kafka — Strimzi. The reference Kafka operator. Originally a Red Hat project, it entered the CNCF Sandbox in 2019 and became a CNCF Incubating project on February 8, 2024 (cncf.io). Strimzi manages a notoriously stateful system — Kafka brokers, topics, users, and rolling upgrades — via CRDs (Kafka, KafkaTopic, KafkaUser, etc.). Rolling a Kafka cluster safely (without dropping under-replicated partitions below their min-ISR) is exactly the application-specific judgment an operator must encode.

Sharded MySQL — Vitess. Vitess is more than an operator — it is a database clustering and horizontal-sharding system for MySQL, originally built at YouTube to scale MySQL beyond a single server, and itself a CNCF Graduated project. On Kubernetes, the Vitess Operator manages the Vitess components (vtgate, vttablet, vtctld) and the underlying MySQL instances. Choose Vitess when MySQL must shard horizontally; it is heavier than a single-cluster operator.

MongoDB. MongoDB Inc. ships the official MongoDB Community/Enterprise Kubernetes Operators and the MongoDB Atlas Operator (which manages Atlas-hosted clusters as Kubernetes objects — closer in spirit to Crossplane). They manage replica sets, sharded clusters, and upgrades.

Elasticsearch — Elastic Cloud on Kubernetes (ECK). Elastic’s official operator. Manages Elasticsearch cluster topology (master/data/ingest node roles), Kibana, rolling upgrades, and node scaling — handling the application-specific care Elasticsearch needs during rollouts (shard relocation, avoiding yellow/red cluster state).

Redis. Several operators exist (the Redis Operator from OT-CONTAINER-KIT, Spotahome’s Redis Operator, the commercial Redis Enterprise Operator) managing Redis replication, Sentinel-based failover, or Redis Cluster sharding.

Configuration / API Surface

The user-facing surface of a database operator is one declarative CRD. Using CloudNativePG as the example:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster                                  # (1) the CRD CNPG defines — the entire interface
metadata:
  name: payments-db
  namespace: payments
spec:
  instances: 3                                 # (2) 1 primary + 2 streaming replicas
  imageName: ghcr.io/cloudnative-pg/postgresql:16.4
  primaryUpdateStrategy: unsupervised          # (3) operator drives switchover during upgrades
 
  storage:
    size: 100Gi                                # (4) operator provisions the PVCs
    storageClass: fast-ssd
 
  postgresql:
    parameters:                                # (5) operator renders postgresql.conf
      max_connections: "200"
      shared_buffers: "512MB"
 
  bootstrap:
    initdb:                                    # (6) operator runs initdb on first reconcile
      database: payments
      owner: payments_app
 
  backup:
    barmanObjectStore:                         # (7) continuous WAL archiving + base backups
      destinationPath: s3://acme-db-backups/payments-db
      s3Credentials:
        accessKeyId:     { name: s3-creds, key: ACCESS_KEY_ID }
        secretAccessKey: { name: s3-creds, key: SECRET_ACCESS_KEY }
    retentionPolicy: "30d"                     # (8) keep 30 days of recoverability
 
  monitoring:
    enablePodMonitor: true                     # (9) wire up Prometheus scraping
# .status is written by the operator: currentPrimary, instances, phase,
#   firstRecoverabilityPoint, last successful backup — NOT set by the user.
  1. kind: Cluster is the operator’s CRD — the user never writes a StatefulSet, a Service, a PVC, a ConfigMap, or a backup CronJob.
  2. instances: 3 → the operator builds a primary plus two streaming replicas, wires up replication, and assigns the read-write and read-only Services.
  3. primaryUpdateStrategy: unsupervised lets the operator perform the primary switchover automatically during a rolling upgrade — the Level III “safe upgrade” behavior.
  4. storage — the operator creates the per-instance PVCs; it does not delegate to a volumeClaimTemplate it does not control.
  5. postgresql.parameters — the operator renders postgresql.conf; the user expresses intent, not file syntax.
  6. bootstrap.initdb runs once, on the first reconcile, to initialize the cluster.
  7. backup.barmanObjectStore configures continuous backup to S3-compatible object storage — this is what makes PITR possible.
  8. retentionPolicy bounds how far back recovery is possible.
  9. enablePodMonitor integrates with the Prometheus Operator — the Level IV “Deep Insights” behavior.

To restore to a point in time, the user creates a new Cluster with a bootstrap.recovery block pointing at the backup and a target timestamp — the operator drives the restore. PITR is a declarative operation, not a runbook.

Failure Modes

  1. Buggy failover promotes a stale replica. The failover path is the operator’s hardest, least-exercised code; a bug there can promote a replica missing committed transactions — the exact data-loss disaster the operator was supposed to prevent. This is why mature operators are conservative and heavily tested; it is also why you must evaluate an operator’s failover logic before trusting it with production data.
  2. The operator is down. A database operator is itself an ordinary Deployment. If it crashes, the database keeps serving traffic, but nothing reconciles — a failed primary is not failed over, backups stop, drift accumulates. The operator needs HA (leader election, a Pod Disruption Budget) and its own monitoring.
  3. Untested backups. An operator that configures backups is not the same as backups that restore. The classic incident: backups ran for a year, the restore was never tested, and recovery fails when it matters. Periodically restore into a scratch namespace and verify.
  4. CRD / operator version skew. The controller expects a specific CRD schema; upgrading one without the other causes misread fields or crashes. The Operator Lifecycle Manager exists to sequence these atomically.
  5. Storage misconfiguration underneath. Zonal block volumes (EBS, GCP PD) pin a Pod to a zone; a reschedule to another zone leaves the disk un-attachable. Use topology-aware provisioning (WaitForFirstConsumer) or regional volumes — see the StatefulSet failure-modes note.
  6. Treating an immature operator as a managed service. A Level-I operator that installs and walks away gives a false sense of safety. Without Level III–V behavior, you have an in-cluster database — a data-loss incident waiting to happen.

Alternatives and When to Choose Them

  • A managed cloud database (RDS, Aurora, Cloud SQL, MSK, Confluent Cloud, MongoDB Atlas, Elastic Cloud) — very often the right answer. The cloud provider operates the database: no operator to trust, no CRD to maintain, no failover code that might be buggy. The Kubernetes MOC decision framework explicitly recommends managed services for stateful workloads until operator and storage discipline are mature. Choose an in-cluster database operator over a managed service only for a specific reason: multi-cloud portability, regulatory data-locality, cost at very large scale, on-prem with no managed option, or a need to colocate the database with the workload.
  • A bare StatefulSet — adequate only for development, ephemeral test data, or a database whose loss is genuinely acceptable. For production, a StatefulSet alone has no failover, no backup automation, no safe-upgrade logic — it is the In-Cluster Database Anti-Pattern.
  • Helm chart for the database — a Helm chart installs a database but does not operate it: no reconcile loop, no failover, no backup, no event response. Many operators are themselves installed via Helm — but the operator, not the chart, is what runs the database.
  • A different operator for the same engine — for Postgres alone there are CNPG, Zalando, Crunchy Data PGO, StackGres, and more. Evaluate on maturity level, project health (release cadence, contributors), failover conservatism, and backup/PITR quality — not on which has the most features.

Production Notes

  • CNPG is the modern reference Postgres operator — strongest CNCF momentum, frequent releases, Level IV+ maturity, and the interesting architectural choice to avoid StatefulSet for finer rollout/failover control. If you must run Postgres on Kubernetes in 2026, CNPG is the default starting point.
  • Mature operators routinely abandon the StatefulSet. CNPG manages instances directly; this recurs across mature operators because StatefulSet’s reverse-ordinal, one-at-a-time rollout cannot express “demote the leader, then replace it” or “never update the primary while a backup runs.” When an operator reimplements parts of StatefulSet, that is a sign of maturity, not of reinventing the wheel.
  • OperatorHub.io catalogs the ecosystem — CNPG, Strimzi, the MongoDB operators, ECK, and hundreds more. The Operator Pattern is how Kubernetes’ apparent surface area exploded: vanilla Kubernetes has a handful of built-in kinds; the thousands of kubectl get <something> kinds people use come from CRDs and operators.
  • Treat any third-party database operator as a dependency you trust with your data. An operator with a buggy failover path can lose a database. Before adopting one for production: check its maturity level, project health, test coverage of the failover path, and how conservative its automatic actions are.
  • Backups are the operator’s job to configure and yours to verify. The most-cited stateful-workload incidents on k8s.af cluster around storage and backup misunderstandings. Schedule a recurring restore drill.

See Also

Graph View

Backlinks