vault

List-Watch Semantics

14 min read

List-Watch Semantics

List-Watch is the contract by which a client maintains an eventually-consistent local view of a Kubernetes resource without polling. The contract pivots on metadata.resourceVersion, an opaque token that, for core kube-apiserver storage, is the monotonically-increasing etcd revision at which an object last changed. The pattern is: LIST once to obtain a consistent snapshot and a resourceVersion rv, then WATCH from rv+ to receive an ordered stream of ADDED/MODIFIED/DELETED/BOOKMARK/ERROR events (Kubernetes — API Concepts). The complexity beneath the simple description — the apiserver’s in-memory watch cache, the 410 Gone “too old resource version” error, the staleness tradeoffs of resourceVersion=0, and the bookmark event type added by KEP-956 (GA in Kubernetes 1.17) — is what every controller author and every operator developer must understand to avoid the canonical “my controller stopped reconciling” bug.

Mental Model

ResourceVersion is etcd’s revision number, surfaced through the apiserver. Every successful write in etcd advances a cluster-wide monotonic counter; that counter becomes the new object’s metadata.resourceVersion. List-Watch reads can specify a resourceVersion to anchor consistency, and watches resume from a resourceVersion to recover from disconnects.


sequenceDiagram
    autonumber
    participant C as Client
    participant API as kube-apiserver<br/>(watch cache)
    participant E as etcd

    Note over E: rv=100 (current revision)
    C->>API: LIST /pods?resourceVersion=0
    API->>API: serve from watch cache snapshot
    API-->>C: items + resourceVersion=100
    C->>API: WATCH /pods?resourceVersion=100&allowWatchBookmarks=true
    Note over API: long-lived HTTP/2 stream
    E->>API: rv=101 (Pod foo created)
    API-->>C: {type:ADDED, object:foo@rv=101}
    E->>API: rv=102 (Pod bar updated)
    API-->>C: {type:MODIFIED, object:bar@rv=102}
    Note over API,E: no relevant changes for 60s
    API-->>C: {type:BOOKMARK, rv=180}
    Note over C: client persists rv=180
    C-xAPI: network drop
    Note over API,E: cache evicts events ≤ rv=200, current rv=300
    C->>API: WATCH /pods?resourceVersion=180
    API-->>C: {type:ERROR, code:410, "too old resource version"}
    C->>API: LIST /pods?resourceVersion=0
    API-->>C: items + resourceVersion=300
    C->>API: WATCH /pods?resourceVersion=300

The full resourceVersion lifecycle from a client’s perspective. The insight: the watch is a recovery-friendly protocol, but only if the client respects 410 Gone. The bookmark events (KEP-956) let the client checkpoint a resourceVersion even when nothing it cares about has changed — without bookmarks, the client’s last known rv stays at 100 while the cluster moves past, and the cache will inevitably drop rv=100 from its history.

Mechanical Walk-through

What resourceVersion actually is

Inside the apiserver, metadata.resourceVersion is the stringified etcd revision at which the object was most recently written. The string format is treated as opaque by the contract — clients must not parse, compare, or arithmetic — but mechanically it is the etcd revision integer. Several variants exist:

  • Per-object resourceVersion — the etcd revision of the most recent write to that key. Returned on Get and on each item in a List.
  • List resourceVersion — returned on metadata.resourceVersion of a list response. It is the etcd revision at which the LIST was consistent — every object’s individual rv is ≤ this value.
  • Watch from resourceVersion — the rv at which the watch should resume. The server sends events strictly newer than this.

The apiserver never increments rv on its own; rv increments only when etcd commits.

The LIST resourceVersion parameter — staleness tradeoff

A LIST request can specify resourceVersion:

  • Absent (no parameter): the apiserver issues a quorum read from etcd. Strong consistency, but expensive: this is the LIST that hammers etcd in large clusters.
  • resourceVersion=0: the apiserver serves from its watch cache — the most recent snapshot it knows about, which may lag etcd by milliseconds. Per the API contract docs, for LIST resourceVersion=0 means “any cached version is acceptable, return the most recent the cache has” and is treated like an unset version that does not require a quorum read (API Concepts). This is the recommended initial LIST for informers; it turns the most common API call into a cache hit. The companion optimization for the absent-rv (consistent) case is KEP-2340 ConsistentListFromCache, which serves even a strongly-consistent LIST from the watch cache by issuing an etcd “progress notification” request and waiting until the cache has caught up to etcd’s current revision — eliminating the expensive quorum LIST. ConsistentListFromCache was alpha in Kubernetes 1.28 and graduated to beta in 1.31 (it also requires etcd v3.4.31+/v3.5.13+ for the watch-progress fix); as of this note’s writing (Kubernetes ~1.36, May 2026) confirm its current graduation stage against the feature-gate reference before quoting it as GA.
  • resourceVersion=N (a specific value): the apiserver tries to serve a LIST at that revision. If N is beyond the watch cache’s history, it may return 410 Gone. Useful for paginated LISTs where the client wants every page at the same revision.
  • resourceVersion=N, resourceVersionMatch=NotOlderThan: serve from cache, but only if the cache is at rv ≥ N. Otherwise wait (briefly) or return error. The mechanism for “I just wrote rv=N, give me a LIST that reflects that.”

The WATCH resourceVersion parameter

A WATCH request specifies the rv from which to resume. The apiserver:

  • Looks up rv in its watch cache’s ring buffer. If present, replays events strictly newer than rv from the buffer, then transitions to the live event stream.
  • If rv is older than the oldest entry in the buffer, returns the ERROR watch event with code 410 (“too old resource version: rv (currentRv)”) and closes the stream.
  • If rv equals the current rv, just transitions to the live stream.
  • If rv is in the future (e.g., from a different cluster), waits — but the apiserver caps wait and ultimately errors.

The watch cache — apiserver’s in-memory event ring

The watch cache (kubernetes/kubernetes apiserver/pkg/storage/cacher) is a per-resource in-memory ring buffer of event history plus a snapshot of every object’s current state. Each kube-apiserver process has its own cache, populated by a watch the apiserver itself opens against etcd.

The buffer’s purpose is twofold:

  1. Serve watches: when a client sends WATCH ?resourceVersion=N, the apiserver replays the buffer’s entries with rv > N before flipping to the live tail. No client request hits etcd unless N is outside the buffer.
  2. Serve LISTs from cache: when a client sends LIST ?resourceVersion=0, the apiserver returns the cache’s snapshot of every object, plus the latest rv it has seen. No etcd round-trip.

The buffer is sized per resource via --watch-cache-sizes (default ~100 for most types, more for high-churn types like Endpoints and EndpointSlices). Buffer policy is: keep events for at least the most recent N changes and for at least a minimum time window, evicting older entries to make room. That time window has historically been a hard-coded 75 seconds in the cacher (eventFreshDuration in staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go); more recent kube-apiserver versions widened it to max(75s, a function of the request timeout) so that long-lived watches with a large timeout are not prematurely expired (kubernetes/kubernetes PR #129205, cacher source). A client that disconnects for longer than the buffer’s window will get 410 on reconnect — this is the contract.

Watch events

The watch stream emits typed events:

{ "type": "ADDED",    "object": { /* object */ } }
{ "type": "MODIFIED", "object": { /* object */ } }
{ "type": "DELETED",  "object": { /* tombstone */ } }
{ "type": "BOOKMARK", "object": { "metadata": { "resourceVersion": "..." } } }
{ "type": "ERROR",    "object": { /* Status object */ } }
  • ADDED is fired on creation and on the initial snapshot when the client requested resourceVersion=0 LIST followed by WATCH from that rv.
  • MODIFIED is fired on update or status update.
  • DELETED carries a “tombstone” object — usually the object’s last state — to let consumers reconcile deletion against their cached version.
  • BOOKMARK (see below) is a no-payload heartbeat.
  • ERROR is terminal — the stream closes after the error.

Watch Bookmarks — KEP-956

Problem: without bookmarks, the client’s tracked rv only advances when an event it actually cares about flows through. On a cluster with high overall churn but few changes to this client’s filtered resource, the apiserver moves past rv=N while the client’s last-known rv stays at N. The buffer eventually evicts N, and on any reconnect the client takes a 410.

KEP-956 introduced the BOOKMARK event type (KEP-956):

  • Alpha in Kubernetes 1.15.
  • Stable / GA in Kubernetes 1.17.
  • Opt-in via ?allowWatchBookmarks=true (client-go enables this by default).
  • The apiserver periodically sends a BOOKMARK event whose object.metadata.resourceVersion reflects the current server rv.
  • The client persists the bookmark rv. On reconnect, it WATCHes from the bookmark rv — much more likely to be inside the cache window than the last “real” event rv.

The frequency is implementation-defined; the apiserver targets one bookmark per minute when nothing else is flowing, and skips bookmarks when real events are flowing fast enough.

Watch List — KEP-3157

The traditional initial-sync flow (LIST then WATCH) is heavyweight for the apiserver: a LIST materializes every matching object into a single HTTP response (sometimes hundreds of megabytes), all allocated in memory at once. KEP-3157 WatchList (“Streaming Lists”) lets a client request WATCH ?resourceVersion=&sendInitialEvents=true&resourceVersionMatch=NotOlderThan and receive the snapshot as a stream of ADDED events followed by a special BOOKMARK (carrying the annotation k8s.io/initial-events-end) indicating the snapshot is complete, all on the watch HTTP/2 stream. The separate initial LIST is eliminated, and the apiserver streams objects rather than buffering the entire collection in memory (API Streaming blog). Note sendInitialEvents is rejected unless the WatchList feature gate is enabled on the server and resourceVersionMatch=NotOlderThan is set.

The timeline (verified against the feature-gate reference and the promotion PR): the server-side WatchList gate was alpha in Kubernetes 1.27 and graduated to beta, enabled by default, in 1.32; the client-side WatchListClient gate (used by client-go and kube-controller-manager) was enabled by default for kube-controller-manager in 1.32 as well (PR #128053, Feature Gates reference). client-go’s Reflector adopts WatchList when both the client gate is on and the server signals support; operators built on controller-runtime inherit this transparently.

Configuration / Code

Curl a watch

# Open a long-lived watch on Pods, starting from rv=0 (gets initial state via ADDED events)
curl --cacert ca.crt --cert client.crt --key client.key \
  "https://APISERVER:6443/api/v1/namespaces/default/pods?watch=true&resourceVersion=0&allowWatchBookmarks=true"

The connection stays open; events arrive as newline-delimited JSON. The sendInitialEvents=true parameter (server WatchList gate: alpha 1.27, beta-by-default 1.32) returns the initial snapshot as ADDED events on the same stream; it must be paired with resourceVersionMatch=NotOlderThan.

Inspect resourceVersion on objects

# Every object carries its rv
kubectl get pod nginx -o jsonpath='{.metadata.resourceVersion}'
# 12873492
 
# A list's rv is the cluster-wide consistent point at which the list was served
kubectl get pods -o jsonpath='{.metadata.resourceVersion}'
# 12873500

Force a re-list (the canonical 410 recovery in client-go)

// Reflector pseudocode for ListAndWatch
listResult, _ := lister.List(opts{ResourceVersion: "0"})
rv := listResult.ResourceVersion
populateCache(listResult.Items)
 
watcher, err := watcher.Watch(opts{
    ResourceVersion: rv,
    AllowWatchBookmarks: true,
})
for event := range watcher.ResultChan() {
    switch event.Type {
    case watch.Added, watch.Modified, watch.Deleted:
        rv = event.Object.GetResourceVersion()
        applyDelta(event)
    case watch.Bookmark:
        rv = event.Object.GetResourceVersion()  // persist, no other action
    case watch.Error:
        if isExpired(event) { // HTTP 410
            // discard cache, re-LIST from rv=0, re-WATCH from the new rv
            return restartWatch()
        }
    }
}

The rv = ... assignment on every event (including BOOKMARK) is the critical part; the cache stays warm across long quiet periods.

Tuning the apiserver watch cache

# kube-apiserver flags
--watch-cache=true                      # default
--default-watch-cache-size=100          # default; per-resource floor
--watch-cache-sizes=endpoints#1000,services#1000,pods#500   # per-resource overrides

On large clusters, increasing --watch-cache-sizes for high-churn resources (Endpoints, EndpointSlices, Pods, Leases) reduces 410-Gone rates and absorbs reconnect bursts. The cost is apiserver memory.

Failure Modes

  • HTTP 410 Gone after a client pause. A controller that stops processing events for longer than the cache window comes back to a 410. Properly written controllers (every client-go user with a Reflector) handle this with re-LIST + re-WATCH automatically. Ad-hoc curl --no-buffer | jq scripts do not, and silently stop receiving events.

  • resourceVersion=0 staleness gotcha. LIST with rv=0 serves from cache and may lag etcd by milliseconds to seconds. For controllers (level-triggered, re-read freshest state on each event) this is irrelevant. For “I just created Foo, immediately LIST and assert it exists” tests it is wrong — use resourceVersionMatch=NotOlderThan with the rv from the create, or omit rv and pay the quorum read.

  • Bookmark not enabled. Older clients without allowWatchBookmarks=true accumulate 410s in proportion to the cluster’s overall churn vs the watched resource’s churn. The fix is one parameter; the bug looks like “controllers periodically rebuild their world for no reason.” client-go enables bookmarks by default since 1.17; only hand-rolled clients miss it.

  • Per-watch latency on the apiserver. Long-lived HTTP/2 watches consume an apiserver goroutine and a portion of the watch-cache fan-out. On a cluster with 10k controllers each watching ~5 resources, the apiserver maintains ~50k watch goroutines. apiserver tuning (--max-requests-inflight, --max-mutating-requests-inflight, API Priority and Fairness) becomes load-bearing at this scale.

  • The “watch cache always-stale during apiserver restart” gotcha. When an apiserver restarts, its watch cache must be rebuilt by streaming events from etcd. During this window, clients hitting the restarting apiserver may see stale LIST results or 503s. In HA control planes with multiple apiserver replicas, load-balancer health checks and --shutdown-delay-duration smooth this out; on single-apiserver clusters there is no smoothing.

  • DELETED tombstone lacks the latest body. The DELETED event’s object is a “best-effort last known state.” For deletions that have been pending in finalizer state for a while, the body may reflect a stale spec. Controllers that need to act on the exact final state of a deleted object should record state pre-deletion.

  • Polling instead of watching. A controller author who isn’t familiar with informers may write a loop that LISTs every 5 seconds. On a large cluster this is the canonical apiserver-toxin: every LIST is a megabyte-scale response, and a hundred such controllers can saturate the apiserver. The “use watches, not polling” mantra is the most important operational rule. See Watch and Informers.

Alternatives and When to Choose Them

  • Raw watch (no cache) via client-go/tools/watch.UntilWithSync — useful for one-shot “wait for this Pod to be Ready” tooling. Not suitable for general controllers; you give up the local cache.

  • Polling: never the right answer for K8s objects, but is the correct answer for external resources a controller depends on (cloud-provider APIs that lack their own watch, customer-managed databases). For those, poll with backoff and write the polled state into a CRD’s status so K8s-native components can watch that.

  • etcd direct watches: technically possible (etcd’s gRPC watch API is the underlying mechanism the apiserver uses). Never appropriate from outside the control plane; you bypass authn/authz/admission/encryption-at-rest and tie yourself to etcd’s storage format, not the K8s API.

  • gRPC streaming on custom APIs: an API Aggregation Layer extension server can implement non-watch streaming semantics for use cases that the LIST-WATCH protocol doesn’t fit (e.g., logs, exec, port-forward — all of which are handled by special apiserver endpoints, not watches).

Production Notes

  • Large clusters live and die on the watch cache. Operators of very large clusters (10k+ nodes) routinely tune --watch-cache-sizes per high-churn resource and monitor 410-Gone rates. The apiserver exposes watch-cache Prometheus metrics including apiserver_watch_cache_capacity (the current per-resource buffer capacity) and apiserver_terminated_watchers_total; the latter climbing implies watches are being terminated faster than expected, often because the buffer is too small for current churn.

  • WatchList (KEP-3157) is the active optimization frontier and, as of Kubernetes 1.32, is beta-and-default. It eliminates the heavyweight initial LIST that every controller emits on startup — on a 10k-Pod cluster, controllers’ coordinated startup currently produces a multi-gigabyte LIST burst, and the streaming form avoids materializing each collection in apiserver memory all at once.

  • ConsistentListFromCache (KEP-2340) turns the formerly-expensive “consistent LIST” into a watch-cache read gated on an etcd progress notification, dramatically reducing the etcd LIST QPS that previously was the canonical “what is bottlenecking my cluster” answer. It was alpha in 1.28 and reached beta in 1.31 (requires etcd v3.4.31+/v3.5.13+).

  • Bookmark frequency is server-configurable but typically defaults to roughly per-minute. For very long-quiet watches (e.g., a per-tenant resource a CRD operator rarely touches), this is fine; for sub-minute reconnect needs, the bookmark cadence may be inadequate and operators add explicit periodic LIST refreshes as defense in depth.

  • API Priority and Fairness is the modern flow-control mechanism that prevents one chatty watcher from starving another. On busy clusters, watch traffic is bucketed by priority level; misclassified controllers can find their watches starved during high LIST traffic spikes. (It graduated to GA and is on by default in current releases — the older --enable-priority-and-fairness flag corresponds to a feature that is now always on.) See API Priority and Fairness.

See Also

Graph View

Backlinks