vault

Resource Versioning and Optimistic Concurrency

18 min read

Resource Versioning and Optimistic Concurrency

Every Kubernetes object carries a metadata.resourceVersion — an opaque, monotonically advancing token that the kube-apiserver treats as the object’s optimistic-concurrency token (kubernetes.io — API Concepts). Under the hood it is derived from etcd’s mod_revision field, a 64-bit cluster-wide MVCC counter incremented on every write (etcd Data Model). When a client PUTs or PATCHes an object it includes the resourceVersion it observed; the API server rejects the write with HTTP 409 Conflict if the live object’s version has since advanced, leaving the client to re-read and retry. This is compare-and-swap at the REST layer: instead of acquiring a row-level lock (which scales poorly across a distributed cluster of controllers), every writer races optimistically, and the conflict surfaces only when two writers concurrently modify the same object. The companion mechanism — metadata.generation (incremented on spec change) versus status.observedGeneration (updated by the controller when it has reconciled) — is the canonical way for the rest of the system to ask: “is the controller current with the spec, or is it lagging?” These two protocols together — version-token concurrency and generation-tracked reconciliation lag — are the foundation that makes Kubernetes’ eventually-consistent control-loop model (Kubernetes Control Loop Pattern) safe under concurrent writes from arbitrarily many controllers.

Mental Model

flowchart LR
    subgraph "etcd (MVCC store)"
        ETCD[(etcd k/v<br/>mod_revision = N)]
    end
    subgraph "kube-apiserver"
        APIS[apiserver<br/>watch cache]
    end
    subgraph "Client A (HPA)"
        A1[GET pod p<br/>RV=42] --> A2[modify replicas]
        A2 --> A3[PUT pod p<br/>resourceVersion: 42]
    end
    subgraph "Client B (user kubectl)"
        B1[GET pod p<br/>RV=42] --> B2[modify replicas]
        B2 --> B3[PUT pod p<br/>resourceVersion: 42]
    end
    ETCD <-->|MVCC reads/writes| APIS
    A3 --> APIS
    B3 --> APIS
    APIS --> RES1[A wins: write→RV=43]
    APIS --> RES2[B loses: HTTP 409<br/>'object has been modified']
    RES2 --> B4[Client B re-reads<br/>RV=43, retries]

What this diagram shows. Two clients (the HPA and a user kubectl) read the same Pod at resourceVersion=42, both modify it locally, both attempt to PUT back with the version they saw. The API server forwards both writes to etcd’s MVCC layer, which atomically compares each write’s expected mod_revision against the current value: A’s PUT lands first (etcd advances to revision 43; the API server propagates the new resourceVersion), B’s PUT then arrives carrying the stale 42, and the comparison fails — etcd refuses the write, the API server translates the failure to HTTP 409 Conflict with the message “the object has been modified; please apply your changes to the newest version and try again”. The insight to extract is that resourceVersion is the cluster’s logical clock for an object: no two distinct versions ever coexist, and any client that wants to “modify this object without clobbering” must include the version it read in its write request. This is compare-and-swap, not lock-then-modify, and the whole controller ecosystem — schedulers, HPAs, garbage collectors, operators — is built to retry on 409 rather than coordinate locks.

Mechanical Walk-through

What resourceVersion actually is

metadata.resourceVersion is declared in the K8s API as a string (not an integer) that clients must treat as opaque (kubernetes.io — API Concepts). The documentation is explicit: “Clients should not assume that the resource version has any specific meaning, nor that it has any specific structure. Resource versions should not be assumed to be numeric, comparable, or larger than any other resource version.” In practice, on the default etcd-backed API server, the resourceVersion is a decimal-encoded etcd revision, and is numerically comparable within a single resource’s history — but the contract makes this an implementation detail. Alternative storage backends (the now-historical Cassandra prototype, the in-memory store used in kube-apiserver --etcd-servers= for tests) are free to use other encodings.

The underlying etcd model is MVCC (Multi-Version Concurrency Control). Every key in etcd has a mod_revision (the revision when it was last written), a create_revision (the revision when it was created), and a version (the per-key change counter). The cluster’s global revision counter increments on every write anywhere in the keyspace; this is what gives resourceVersion its monotonicity within a cluster (etcd’s Data Model docs explain this in depth).

The compare-and-swap update path

When a client PUTs an object (a full-object replace), the K8s API server retrieves the current object, checks that request.body.metadata.resourceVersion == current.metadata.resourceVersion, and if so commits the write. The mechanism is etcd’s transactional Txn operation: the API server submits Txn(If(mod_revision == N), Then(Put(key, newValue)), Else(()) — etcd atomically performs the put only if the revision matches. If the comparison fails, etcd returns “failed precondition” and the API server translates this into an HTTP 409 Conflict response with the canonical message (API Concepts):

{
  "kind": "Status",
  "apiVersion": "v1",
  "status": "Failure",
  "message": "Operation cannot be fulfilled on pods \"my-pod\": the object has been modified; please apply your changes to the newest version and try again",
  "reason": "Conflict",
  "code": 409
}

The client’s expected response is documented in the same page: “the client should retrieve the resource and try again.”

PATCH and the version-optional shortcut

PATCH requests fall into four flavors (kubernetes.io — API Concepts):

Patch typeContent-TyperesourceVersion behavior
Strategic mergeapplication/strategic-merge-patch+jsonOptional in body; if absent, server uses live object’s version
JSON merge (RFC 7396)application/merge-patch+jsonOptional in body
JSON Patch (RFC 6902)application/json-patch+jsonNo version check unless body includes a test op
Server-side applyapplication/apply-patch+yamlVersion-free; conflicts surface through Server-Side Apply’s field ownership ledger instead

The asymmetry is deliberate: a PATCH that touches only a narrow field set rarely conflicts with concurrent writes to other fields, so requiring resourceVersion on every PATCH would force unnecessary retries. Clients that want strict concurrency on a patch include metadata.resourceVersion in the body; the server then performs the same CAS check as for PUT.

resourceVersion in LIST and WATCH

The same token is used for reads, where its meaning is governed by a companion query parameter, resourceVersionMatch, supported on LIST since Kubernetes v1.19 (API Concepts; common parameters). The docs are emphatic: “You should always set the resourceVersionMatch parameter when setting resourceVersion on a list request.” If resourceVersion is set without resourceVersionMatch, the server falls back to legacy behavior, which is a frequent source of subtle staleness bugs.

resourceVersionMatch takes two values:

  • NotOlderThan — “return data at least as new as the provided resourceVersion. The newest available data is preferred, but any data not older than this resource version may be served.” This is the watch-cache-friendly mode: the watch cache simply waits until its own revision is at least as fresh as the request, then serves from memory — no quorum read against etcd. It is the recommended high-performance path. The watch cache only supports NotOlderThan semantics.
  • Exact — return data at exactly the requested resourceVersion. The server reads that specific historical snapshot from etcd; the client should verify the returned collection’s .metadata.resourceVersion equals what it asked for. Used for reproducible pagination (a multi-page LIST should pin every page to the same revision so items don’t shift between pages).

With those two modes in hand, the special resourceVersion values resolve as follows:

  • LIST with resourceVersion="0" (and no match mode, or with the historical default) asks the apiserver to serve from its watch cache — any cached data, possibly somewhat stale — without a quorum read against etcd. This is the cheapest possible list, used by informers on startup to warm their caches where slight staleness is harmless.
  • LIST with resourceVersion unset and resourceVersionMatch unset is a consistent read: the server determines etcd’s latest committed revision, waits for the serving layer to catch up to it, and returns strongly-consistent data. As of recent releases this can be served from the watch cache once it has synced to that revision (KEP-2340, “consistent reads from cache”), but it is still the most expensive read because it requires establishing the current revision; prefer NotOlderThan with a known resourceVersion unless you have a hard freshness requirement.
  • LIST with resourceVersion=<N> and resourceVersionMatch=Exact asks for a snapshot pinned at revision N.
  • WATCH with resourceVersion=<N> subscribes to all events after revision N. If N is older than the watch cache’s oldest retained event, the server returns HTTP 410 Gone (“too old resource version”) and the client must re-LIST to resync (List-Watch Semantics); all events between N and the resync point are considered lost and recovered by the re-list. The watch cache retains a bounded window of recent events per resource type, sized via --watch-cache-sizes. Watch bookmarks (type: BOOKMARK synthetic events) periodically hand the client a fresh resourceVersion mid-stream so a later reconnect can resume at a still-cached revision instead of falling off the back of the window into a 410.

The dual role — concurrency token on writes, cursor on reads — is why the field is named resourceVersion rather than version or revision: it tracks the object’s progress through the API server’s stream of writes from both producer and consumer angles.

metadata.generation vs status.observedGeneration

A separate but related concurrency mechanism tracks reconciliation lag:

  • metadata.generation is set by the API server and incremented only when spec changes. Changes to metadata (labels, annotations) or status do not bump generation. This is enforced by the API server’s storage layer; user clients cannot set generation.
  • status.observedGeneration is set by the resource’s owning controller (Deployment controller, StatefulSet controller, operator). After the controller reconciles spec generation N, it writes status.observedGeneration: N.

The pair lets any observer answer “has the controller seen the latest spec?” by comparing status.observedGeneration >= metadata.generation. The kubectl wait --for=condition=available deployment/foo command, the rolling-update progress logic, and most operator readiness checks consult this pair (alenkacz — implementing observedGeneration). A subtlety worth internalizing: status.observedGeneration answers only “did the controller see generation N,” not “did it finish converging to it.” A controller may set observedGeneration: N the instant it begins reconciling generation N, while the work (a rolling update, a volume provision) is still in flight. That is why readiness is gated on a condition (e.g. Available/Progressing), not on observedGeneration alone. Modern Kubernetes API conventions also let an individual entry in status.conditions[] carry its own observedGeneration, so a reader can tell per condition whether that condition’s truth value reflects the current spec or a stale one — e.g. generation: 12 but conditions[Available].observedGeneration: 9 means the Available condition is stale.

Uncertain

Verify: the precise point in each in-tree controller’s reconcile at which status.observedGeneration is written relative to the work completing — specifically whether the Deployment, StatefulSet, and DaemonSet controllers set it on observation of a new spec generation or only on completion of convergence. Reason: this is per-controller implementation detail not pinned to a single spec page, and the conventions deliberately separate “observed the generation” (observedGeneration) from “converged” (conditions), so a single rule does not hold across controllers. To resolve: read the relevant controller’s syncDeployment/syncStatefulSet status-update code in kubernetes/kubernetes for the target minor, and treat per-condition observedGeneration as the authoritative staleness signal for a given condition. uncertain

Why optimistic concurrency rather than locking

The classical alternative — distributed locking — has well-known problems at K8s scale:

  • Lock acquisition latency dominates write throughput when contention is low (the common case).
  • Lock holder failures require leases, lease renewals, and detection of expired leases — exactly the problems Raft and etcd already solve.
  • Cross-resource locks (a controller that needs to atomically update a Deployment and a ReplicaSet) explode in complexity.

The optimistic approach is the standard solution for high-read, low-conflict, multi-writer workloads (it is the same trade-off MVCC databases like PostgreSQL, CockroachDB, and Spanner make). The cost is paid by the retry path: clients must be prepared to re-read and retry. controller-runtime’s reconcile loop (Kubernetes Control Loop Pattern) makes this trivial — a 409 returns an error, the reconcile is requeued with exponential backoff, the next iteration re-reads from the informer cache and tries again. The pattern collapses gracefully under contention rather than blocking.

Configuration / API Surface

A 409 round-trip with curl

# 1. Read the current object — note the resourceVersion in the response.
$ kubectl get configmap/myconfig -o json | jq '.metadata.resourceVersion'
"4825612"
 
# 2. Modify it locally to bump a value.
$ kubectl get configmap/myconfig -o json > config.json
$ jq '.data.timeout = "30s"' config.json > config-new.json
 
# 3. (Meanwhile, another process modifies the same ConfigMap, bumping RV to 4825613.)
 
# 4. Attempt PUT with the stale resourceVersion.
$ curl -X PUT https://kubernetes.default/api/v1/namespaces/default/configmaps/myconfig \
       -H 'Content-Type: application/json' \
       --data-binary @config-new.json
{
  "kind": "Status",
  "status": "Failure",
  "message": "Operation cannot be fulfilled on configmaps \"myconfig\": the object has been modified; please apply your changes to the newest version and try again",
  "reason": "Conflict",
  "code": 409
}
 
# 5. Retry: re-GET, re-apply edits, re-PUT.
$ kubectl get configmap/myconfig -o json > config.json
$ jq '.data.timeout = "30s"' config.json > config-new.json
$ curl -X PUT ... @config-new.json   # succeeds; RV advances to 4825614.

The pattern — GET, modify, PUT, on 409 retry from GET — is the canonical optimistic-concurrency loop. Controllers wrap this in a RetryOnConflict helper (see below).

Go: retry.RetryOnConflict in client-go

import (
    "k8s.io/client-go/util/retry"
    apierrors "k8s.io/apimachinery/pkg/api/errors"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
 
err := retry.RetryOnConflict(retry.DefaultRetry, func() error {
    // 1. RE-READ from the API server every iteration. Reading the
    //    informer cache here is WRONG because the cache may be lagging.
    cm, err := clientset.CoreV1().ConfigMaps("default").
        Get(ctx, "myconfig", metav1.GetOptions{})
    if err != nil {
        return err
    }
 
    // 2. Modify the freshly-read object.
    if cm.Data == nil {
        cm.Data = map[string]string{}
    }
    cm.Data["timeout"] = "30s"
 
    // 3. PUT it back. If the live RV has advanced since (1), this returns 409.
    _, err = clientset.CoreV1().ConfigMaps("default").
        Update(ctx, cm, metav1.UpdateOptions{})
    return err   // RetryOnConflict re-runs the body on apierrors.IsConflict
})

Line-by-line:

  • retry.RetryOnConflict is a client-go utility that re-invokes the closure on apierrors.IsConflict(err) (HTTP 409) with exponential backoff, up to a configurable max attempts. The DefaultRetry is 5 attempts starting at 10 ms.
  • The closure re-reads the object on each iteration. Re-using a stale in-memory copy is the leading cause of “permanently stuck on 409” controllers.
  • The closure must be idempotent — it may run multiple times and must produce the same logical effect each time.

Reading metadata.generation in a reconcile loop

func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var obj myv1.MyResource
    if err := r.Get(ctx, req.NamespacedName, &obj); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
 
    // Skip work if we've already reconciled this generation.
    if obj.Status.ObservedGeneration == obj.Generation {
        return ctrl.Result{}, nil
    }
 
    // ... reconcile spec ...
 
    obj.Status.ObservedGeneration = obj.Generation
    if err := r.Status().Update(ctx, &obj); err != nil {
        return ctrl.Result{}, err     // 409 → requeue
    }
    return ctrl.Result{}, nil
}

The obj.Status.ObservedGeneration = obj.Generation write is the signal to the rest of the system that this reconciler has acted on spec generation N. kubectl wait, the rolling-update progress check in the Deployment controller, and operator-readiness probes all consult this field.

Failure Modes

  1. Reading from the informer cache, writing with that cache’s resourceVersion. The informer cache lags the apiserver by milliseconds-to-seconds (longer under load). A controller that reads from cache, modifies, and PUTs back the cached RV will receive 409s in a steady-state cluster where any other writer (the metrics agent, the kubelet status updater) is actively writing. The fix: re-read from the apiserver in the retry-loop body, or use PATCH where the version check is optional.
  2. Retry without re-reading. A controller that catches 409 and retries with the same in-memory object will loop forever, returning 409 each time. The retry must re-fetch.
  3. Setting metadata.generation from the client. The API server silently ignores any client-supplied value for generation (it is server-managed). Code that tries to drive generation from the outside has misunderstood the contract.
  4. Updating status through the main resource instead of the /status subresource. Controllers should write status via the /status subresource (r.Status().Update(...) in controller-runtime), not through a plain object update. The reason is not that a status write through the main path bumps generation — generation increments only on a spec change regardless of which path the write takes. The real reasons: (a) the /status subresource enforces the spec/status split, so a status write physically cannot clobber spec (and a spec write through the main path cannot clobber status), preventing a controller from accidentally reverting a user’s concurrent spec edit; and (b) for CustomResourceDefinitions the status subresource must be explicitly enabled, and only when it is does the apiserver guarantee that updates through it leave spec/generation untouched — write status through the main resource on a CRD without the subresource and you fight the user’s spec writes for the same object version. Using the subresource also narrows the conflict window (a status write only contends with other status writers).
  5. Using resourceVersion for “did this object change since last reconcile” checks. Tempting but wrong: resourceVersion bumps for any change including metadata annotations written by operators, garbage-collection finalizer updates, status writes by other controllers, etc. metadata.generation is the spec-only signal.
  6. Watch from resourceVersion=0 after a disconnect. A reconnecting watch should resume from the last RV it processed (ideally the RV from the most recent watch bookmark), not from 0. Contrary to a common misconception, watching from resourceVersion=0 does not replay all history — per the API Concepts, 0 (and unset) means “start at the most recent resource version” and streams only changes from that point forward. The real hazard is the opposite: a reconnect from 0 silently skips every event that happened between your last-processed RV and “now,” leaving the client’s cache permanently inconsistent with no error raised. This is why bookmarks exist — they keep handing the client a fresh, still-cached RV to resume from so it neither falls off the back of the window into a 410 Gone nor resorts to a lossy 0. controller-runtime’s reflector manages this resume-RV bookkeeping for you; hand-rolled watch loops must do it themselves.
  7. Assuming resourceVersion is monotonic across resources. RVs are monotonic within a resource type, but a Pod’s RV=100 and a Service’s RV=99 do not imply the Pod was written later than the Service in wall-clock time — they are global etcd revisions but reads/writes interleave. Cross-resource ordering requires explicit timestamps or watches.
  8. Hot conflict loops. If two controllers both want to be sole owner of a field and both retry on 409, they enter a livelock — each successful write triggers a 409 for the other, which re-reads and re-writes its preferred value. The fix: Server-Side Apply’s field ownership ledger surfaces this explicitly rather than oscillating.

Alternatives and When to Choose Them

  • Pessimistic locking (acquire-lock-then-modify). The relational-database alternative. Kubernetes does not provide row-level locks; the coordination.k8s.io/Lease API offers a separate lease primitive used for leader-election in controller-managers, not for object-level concurrency.
  • Server-Side Apply (Server-Side Apply). For multi-actor co-ownership, SSA’s managedFields ledger replaces ad-hoc resourceVersion-driven retry. The two compose: SSA still consults resourceVersion under the hood, but conflicts surface at the field-ownership layer (manager M owns field F) rather than the object-version layer (RV is stale).
  • Patch with field-selector preconditions. JSON Patch’s test operations let a client assert “field X must equal Y” as a precondition; the patch fails if not. Useful for narrow correctness invariants but unusual in mainstream controller code.
  • External coordination (etcd elections, Consul leases, ZooKeeper). Pre-cloud-native applications often layered explicit coordination on top. Kubernetes-native code should not — the API server is the authoritative concurrency surface.
  • Watch-only reconcile (no writes). Some controllers exist purely to observe state and emit Events or metrics; they avoid concurrency by never writing. This is the cheapest concurrency strategy when applicable.

Production Notes

  • High-write resources hit 409s frequently. The Node, Pod, and Lease resources in a busy cluster receive many writes per second (heartbeats, status updates). Any controller modifying Nodes or Pods must expect 409s as a normal mode of operation; treating them as errors floods logs and metrics.
  • timebertt’s controller-at-scale write-up (medium — Controllers at Scale: Clients, Caches, Conflicts) documents that high-RPS controllers should prefer PATCH over PUT (narrower conflict windows) and prefer Server-Side Apply over PATCH (per-field conflict surfaces rather than whole-object). This is the operationally-justified migration path many large clusters have taken.
  • etcd 1.5 MiB object size cap. The compare-and-swap mechanism does not protect against objects growing past etcd’s per-object size limit. Operators that accumulate fields in status (a common anti-pattern: storing per-pod reconciliation logs there) eventually hit a wall where PATCH succeeds but a subsequent re-read fails with etcdserver: request is too large.
  • etcd revision is not exposed as a separate stable API field. On the default etcd backend, an object’s resourceVersion is the etcd mod_revision decimal-encoded, so there is no separate “raw revision” field to read — the contract is that you treat resourceVersion as opaque. (An earlier draft of this note named an apiServerSourceRevision client-go field for this purpose; that field could not be verified against client-go and appears not to exist — removed rather than left as a confident-but-wrong claim.) Tooling that needs the numeric revision for cross-cluster comparison reads resourceVersion and knowingly relies on the etcd-backend implementation detail, accepting that it is non-portable.
  • Failure stories. k8s.af documents several operator outages traced to 409-retry storms where controllers re-read from the cache (lagging), re-PUT, hit 409, loop. The mitigation in every case was “re-read from the apiserver, not the cache.”

See Also

Graph View