vault

ReplicaSet

18 min read

ReplicaSet

A ReplicaSet maintains a stable set of N identical Pod replicas at any given time. It is the workload controller responsible for the “make sure there are exactly replicas Pods matching this selector” invariant. The Kubernetes documentation defines it: “A ReplicaSet’s purpose is to maintain a stable set of replica Pods running at any given time. As such, it is often used to guarantee the availability of a specified number of identical Pods” (kubernetes.io — ReplicaSet). A ReplicaSet has three load-bearing fields: a Pod template (spec.template) describing the Pods it will create, a selector (spec.selector) describing which Pods it claims as its own, and a replica count (spec.replicas). The replicaset controller, which lives inside kube-controller-manager, runs a level-triggered control loop over these fields: count Pods matching the selector, compare to spec.replicas, create or delete the difference, repeat. ReplicaSet is the direct successor to the deprecated ReplicationController, distinguished by a richer selector model that supports matchExpressions (set-based selectors) in addition to matchLabels (equality-based selectors). In practice, you almost never create a ReplicaSet directly. The recommended pattern is to create a Deployment, which itself creates and manages a ReplicaSet for each version of your app. The ReplicaSet exists as its own resource — rather than being absorbed into Deployment — because of a clean separation of concerns: ReplicaSet owns “maintain count,” Deployment owns “manage rollout.” The Deployment manages the rollout by creating a new ReplicaSet for the new version, scaling its replicas up while scaling the old ReplicaSet’s replicas down. Each ReplicaSet remains simple; the rollout choreography lives in the layer above.

Mental Model

flowchart TB
    subgraph "kube-controller-manager"
        RSC[ReplicaSet Controller<br/>watches: ReplicaSet, Pod]
    end
    RS[ReplicaSet rs-abc<br/>spec.replicas: 3<br/>spec.selector: app=web<br/>spec.template: ...]
    POD1[Pod rs-abc-x9k2j<br/>labels: app=web<br/>ownerRef: rs-abc]
    POD2[Pod rs-abc-q7m4p<br/>labels: app=web<br/>ownerRef: rs-abc]
    POD3[Pod rs-abc-7n3vw<br/>labels: app=web<br/>ownerRef: rs-abc]
    API[(kube-apiserver)]
    DEPLOY[Deployment web<br/>owns rs-abc]
    DEPLOY -- "creates / scales" --> RS
    RS -- "spec read via watch" --> RSC
    RSC -- "list Pods matching<br/>label app=web<br/>(observed count)" --> API
    RSC -- "create or delete<br/>to reconcile count" --> API
    API --> POD1
    API --> POD2
    API --> POD3
    POD1 -.->|ownerReferences| RS
    POD2 -.->|ownerReferences| RS
    POD3 -.->|ownerReferences| RS

What this diagram shows. A ReplicaSet is a small declarative spec — three fields — over which the ReplicaSet controller in kube-controller-manager runs a reconcile loop. The controller watches both ReplicaSet and Pod resources via the API server’s watch interface. On every reconcile (triggered by watch events on either resource), it lists all Pods matching the ReplicaSet’s spec.selector, counts them (with some filtering — see “deletion preference” below), and compares to spec.replicas. If count < replicas, it creates Pods from spec.template (with the ReplicaSet set as the ownerReferences on each new Pod). If count > replicas, it deletes Pods. The owner-reference machinery (see Owner References and Garbage Collection) ties each Pod back to its ReplicaSet, so that deleting the ReplicaSet cascade-deletes the Pods. Above the ReplicaSet, a Deployment (when present) owns the ReplicaSet itself — that’s how rollouts work: the Deployment creates a new ReplicaSet for each version, gradually shifts replicas between old and new. The insight to extract is that the ReplicaSet is deliberately stupid: it does one thing, “maintain N Pods matching S,” and does nothing about updates. Decoupling “maintain count” from “manage version transitions” is what lets the Deployment’s rollout logic be cleanly layered on top — and why the ReplicaSet’s Pod template is immutable (any spec change requires a new ReplicaSet).

Mechanical Walk-through

The reconcile loop

The ReplicaSet controller’s reconcile function (in pkg/controller/replicaset in the K8s source) performs, on every wake-up:

  1. Read the ReplicaSet object from the informer cache by the {namespace, name} from the work queue. If not found, exit (the RS has been deleted; owner-reference garbage collection handles Pod cleanup).
  2. List all Pods in the same namespace matching spec.selector from the local Pod cache.
  3. Adopt orphan Pods. If any Pod matches the selector but has no ownerReferences, the controller adds itself as the owner (this is how acquisitions work — see “Pod adoption” below).
  4. Release Pods that no longer match. If a Pod previously owned by this RS no longer matches the selector (because someone changed its labels), the controller releases ownership.
  5. Compute diff = len(matchingPods) - spec.replicas.
  6. If diff < 0 (under-replicated): create -diff Pods from spec.template, with ownerReferences pointing back to this RS. Pod names are generated via <rs-name>-<5-char-hash> (e.g., frontend-7xqwp).
  7. If diff > 0 (over-replicated): select diff Pods to delete, ranked by deletion preference (below), and issue Delete calls.
  8. Update .status.replicas, .status.readyReplicas, .status.availableReplicas, .status.observedGeneration on the ReplicaSet’s /status subresource.

The reconcile is idempotent and level-triggered (see Kubernetes Control Loop Pattern): a reconcile that’s interrupted, restarted, or re-fired produces the same effect as a single reconcile. The controller never relies on “what changed” — it always reads current state and acts to converge.

Pod adoption

ReplicaSets can adopt orphan Pods that match their selector. The K8s docs state: “A ReplicaSet can acquire Pods that match its label selector and have no OwnerReference or where the OwnerReference is not a Controller.” This means if you create a bare Pod with labels: {app: web} in a namespace where a ReplicaSet selects app=web, the controller will set itself as the bare Pod’s owner. If that brings the count over spec.replicas, the controller will then delete the extras (preferring the just-adopted one, if it isn’t ready).

This adoption mechanism is mostly invisible in practice — operators rarely create bare Pods that happen to match a ReplicaSet’s selector — but it’s the conceptual underpinning of how Deployments hand Pods between successive ReplicaSets during rollback.

Deletion preference order

When scaling down, the controller doesn’t pick Pods arbitrarily. The ranking is implemented by the ActivePodsWithRanks comparator (Less method) in pkg/controller/controller_utils.go in the K8s source. The comparator applies the following criteria in strict order — the first criterion that distinguishes two Pods decides which one is deleted first:

  1. Unassigned-node Pods first. A Pod that has not yet been scheduled to a node (empty spec.nodeName) is deleted before any scheduled Pod.
  2. Pod phase: Pending < Unknown < Running. A Pending Pod is deleted before an Unknown-phase Pod, which is deleted before a Running Pod.
  3. Not-ready before ready. Among same-phase Pods, those whose Ready condition is false are deleted before ready ones.
  4. Lowest controller.kubernetes.io/pod-deletion-cost first. This annotation is an explicit integer hint (default 0); a lower value means delete sooner. It is the formal operator escape hatch.
  5. Higher node-colocation rank first. Pods running on a node that hosts more of this controller’s other matching Pods are deleted first — this spreads the surviving Pods across nodes, preserving failure-domain diversity.
  6. More-recently-ready first. Among ready Pods, the one that became ready more recently is deleted first (compared on a logarithmic time scale), protecting Pods that have been serving — and warming caches — longer.
  7. Higher container-restart-count first. A Pod whose containers have restarted more often is deleted before a more-stable one.
  8. More-recently-created first. As the final tiebreaker, the newer Pod (by creationTimestamp, on a logarithmic scale) is deleted first.

The intent is to delete less useful Pods first: unscheduled and pending Pods that aren’t serving traffic anyway, unready Pods, Pods crowding a node, recently-started Pods that haven’t paid off their warm-up cost, and flapping Pods with high restart counts. The pod-deletion-cost annotation (criterion 4) is the only knob an operator controls directly — set it high on Pods you want to keep (e.g., a Pod that has accumulated valuable in-memory state). The annotation’s semantics are documented at pod-deletion-cost; the full ordering, however, lives only in the controller source cited above, not in the prose docs.

Uncertain

Verify: the exact wording and ordering of criteria 5–8 (node-colocation rank, ready-time, restart count, creation time). Reason: this ordering is read from the ActivePodsWithRanks comparator in the K8s master branch source (pkg/controller/controller_utils.go), not from a stable, versioned doc page — the comparator has been refined across releases and is not part of any API contract. To resolve: pin the comparator source to the exact K8s release tag you operate against (e.g. release-1.36) and re-read the Less method.

Immutability constraints

Two critical immutability rules:

  • spec.selector is immutable. Once set, you cannot change it. (Attempting to do so via API returns a validation error.) Why: changing the selector could orphan or wrongly-adopt Pods, breaking the count invariant. The only way to change the selector is to delete and recreate the ReplicaSet.
  • spec.template is effectively immutable, but only for the template’s metadata-labels-that-determine-selector-matching. Technically spec.template can be edited; however, doing so does not update existing Pods (those keep the old template), and the new template only applies to future Pods. This is the property Deployment exploits: the Deployment never edits an existing ReplicaSet’s template; it creates a new ReplicaSet with the new template, scales it up, and scales the old one down. The two ReplicaSets coexist during rollout, distinguished by a pod-template-hash label that the Deployment computes and adds.

The K8s docs are explicit on the replica default: “You can specify how many Pods should run concurrently by setting .spec.replicas. The ReplicaSet will create/delete its Pods to match this number. If you do not specify .spec.replicas, then it defaults to 1.” The “template is immutable” shorthand is almost right and the correct mental model is now confirmed against the docs: the K8s API server does allow editing spec.template of a ReplicaSet (there is no validation rejection — unlike the selector), but the change does not retroactively update existing Pods. The ReplicaSet docs state it directly: a ReplicaSet “will not make any effort to make existing Pods match a new, different pod template” — a changed template applies only to Pods created after the edit. Functionally this is equivalent to immutability for any operational purpose (you cannot roll out a new image by editing the RS template; existing Pods keep the old spec), which is exactly why higher-level controllers (Deployment) create new ReplicaSets rather than edit existing ones. The genuinely immutable field is spec.selector (covered above) — editing it is rejected by API validation.

Status reporting

The ReplicaSet’s .status exposes:

  • .status.replicas — currently observed Pod count.
  • .status.readyReplicas — Pods whose Ready condition is true.
  • .status.availableReplicas — Pods that have been ready for .spec.minReadySeconds (avoids counting freshly-promoted-to-ready Pods that might still be flapping).
  • .status.fullyLabeledReplicas — Pods that match the template’s labels exactly (not just the selector). Useful diagnostic when labels drift.
  • .status.observedGeneration — the metadata.generation value the controller has reconciled. If observedGeneration < generation, the controller hasn’t caught up yet.

Configuration / API Surface

A bare ReplicaSet (almost never written by humans):

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: frontend
  labels:
    app: guestbook
    tier: frontend
spec:
  # ---------- (1) Desired replica count ----------
  replicas: 3                          # Default 1 if omitted.
 
  # ---------- (2) Selector (immutable!) ----------
  # The set of labels this RS claims. MUST be matched by spec.template.metadata.labels.
  selector:
    matchLabels:
      tier: frontend                   # equality-based selector
    # OR (more expressive) — set-based selector:
    # matchExpressions:
    #   - { key: tier, operator: In, values: [frontend, web] }
 
  # ---------- (3) Minimum ready time ----------
  # A Pod must be Ready for at least 10s to count as "available." Smooths
  # over flappy probes during scale-up.
  minReadySeconds: 10
 
  # ---------- (4) Pod template ----------
  template:
    metadata:
      labels:
        tier: frontend                 # MUST match spec.selector.matchLabels
    spec:
      containers:
        - name: php-redis
          image: us-docker.pkg.dev/google-samples/containers/gke/gb-frontend:v5
          ports:
            - containerPort: 80

What this creates. The API server stores the ReplicaSet. The replicaset-controller in kube-controller-manager observes it via watch, counts matching Pods (zero, since this is fresh), and creates 3 Pods named frontend-<5-char-hash> (e.g., frontend-7xqwp, frontend-9m3vk, frontend-r8nq2). Each new Pod has ownerReferences: [{kind: ReplicaSet, name: frontend, controller: true}]. The ReplicaSet’s .status.replicas becomes 3.

Operational commands:

# Manually scale to 5.
kubectl scale rs frontend --replicas=5
 
# Delete the ReplicaSet, cascading to its Pods (default).
kubectl delete rs frontend
 
# Delete the ReplicaSet but leave Pods orphaned.
kubectl delete rs frontend --cascade=orphan
 
# Show the deletion-cost annotation in action.
kubectl annotate pod frontend-7xqwp controller.kubernetes.io/pod-deletion-cost=1000
# (now this Pod is the last to be deleted on scale-down)

The annotation controller.kubernetes.io/pod-deletion-cost is the formal escape hatch documented at pod-deletion-cost.

The realistic use case: ReplicaSets as managed by Deployment. You don’t write the RS; you write the Deployment, and the RS appears:

apiVersion: apps/v1
kind: Deployment
metadata: { name: frontend }
spec:
  replicas: 3
  selector: { matchLabels: { app: frontend } }
  template:
    metadata: { labels: { app: frontend } }
    spec:
      containers:
        - { name: app, image: myorg/frontend:v1.0 }
---
# After applying, observe:
# $ kubectl get rs
# NAME                  DESIRED   CURRENT   READY   AGE
# frontend-58dc7bc848   3         3         3       2m
#
# The hash 58dc7bc848 is the pod-template-hash, computed from the
# Deployment's spec.template. Updating the Deployment's image creates
# a NEW RS frontend-<new-hash> alongside the old one.

Failure Modes

  1. Selector matches Pods you didn’t create. A new ReplicaSet with selector: app=web in a namespace that already has bare Pods labelled app=web will adopt them. If the count plus the adopted Pods exceeds replicas, the controller deletes the extras. The “extras” might be the just-adopted long-running Pods, not the freshly-created ones. Symptom: Pods you thought were independent vanish when an RS is created. Mitigation: use unique selectors (include a deployment-specific label like deployment: foo-v2); never reuse generic labels.

  2. Selector unique to RS template, no overlap with existing Pods. A correctly-isolated ReplicaSet whose template-labels match nothing pre-existing. Best case. The Deployment-managed pod-template-hash label achieves this automatically.

  3. spec.replicas set to 0 by mistake. ReplicaSet deletes all Pods; service goes dark. This is sometimes intentional (drain to zero before deletion) but more often a bad kubectl patch. Recovery: scale back up. No data is lost from the ReplicaSet itself; ephemeral Pod data is gone.

  4. Editing spec.selector (rejected). API server returns selector is immutable error. The only fix is to delete and recreate the ReplicaSet.

  5. Editing spec.template.spec.containers[0].image directly. The change is accepted by the API server but does not propagate to existing Pods. New Pods created on scale-up get the new image; existing Pods keep the old. Symptom: a heterogeneous fleet running two versions, no clear rollout signal. The Deployment controller avoids this entirely by never editing existing RS templates.

  6. Orphan Pods after --cascade=orphan delete. The RS is gone but its Pods remain, no longer owned by anything. They keep running but won’t be replaced on node failure (no controller is watching them). Useful for migrating Pods between owners; dangerous as a long-term state. Recovery: create a new RS with matching selector, which will adopt the orphans.

  7. pod-template-hash collision (extraordinarily rare). Two distinct templates that hash to the same 5-character suffix. Symptom: a Deployment update creates an “old” RS that matches the hash of a Deployment from months ago, instead of creating a new one. K8s mitigates by using a substantial hash space and adding a salt; in practice this is a 1-in-millions probability not worth designing around.

  8. minReadySeconds masking real failures. A Pod that flaps Ready/NotReady within minReadySeconds is never marked Available, so the rollout never advances. Symptom: a Deployment stuck 0/3 available despite 3/3 ready. Diagnostic: look at probe configurations, container restarts.

  9. Manually-deleted Pods just respawn. A frequent surprise for newcomers: kubectl delete pod frontend-xyz deletes the Pod, but the RS immediately creates a replacement. To actually stop a Pod, delete (or scale down) the RS, not the Pod. The Pod is mortal; the RS is the persistent intent.

  10. Selector-only label drift. If you kubectl label pod frontend-xyz tier=removed --overwrite, the Pod no longer matches the RS’s selector. The RS releases ownership, then notices count is too low, creates a new Pod. The released Pod becomes orphan but stays running. Useful for “extract one Pod for debugging” workflows; surprising the first time it happens.

Alternatives and When to Choose Them

  • Deployment (always prefer over raw ReplicaSet). Deployment manages a ReplicaSet for you and adds rolling-update/rollback semantics. The K8s docs are explicit: “we recommend using Deployments instead of directly using ReplicaSets, unless you require custom update orchestration or don’t require updates at all.” Default to Deployment for stateless workloads.

  • StatefulSet. Like a ReplicaSet but with stable identities, ordered creation, and stable persistent storage. Choose when each Pod replica is distinguishable (Postgres primary vs. replica, Cassandra nodes with shard ownership). The ReplicaSet’s “every Pod is identical and fungible” model doesn’t fit stateful workloads.

  • DaemonSet. “One Pod per (matching) node,” not “N Pods total.” Choose for node-level agents (log shipper, CNI plugin, monitoring agent). The set is gated by node labels, not a numeric replicas count.

  • Job / CronJob. Run-to-completion batch workloads, not long-running services. Job’s parallelism and completions fields are similar in spirit to ReplicaSet’s replicas, but the controller knows about completion (success exit code) — ReplicaSet doesn’t.

  • ReplicationController (deprecated). The pre-apps/v1 ancestor of ReplicaSet. Still in the API for backward compatibility; only supports equality-based selectors (no matchExpressions). Choose only if you’re maintaining a very old cluster that pre-dates apps/v1. New work should never use this.

  • Custom controller / Operator. When your replication invariant is more complex than “N identical Pods” — for example, “1 leader and N followers with leader election,” or “3 Pods, one per zone” — write your own controller via Kubebuilder + controller-runtime. The ReplicaSet’s simple count-invariant doesn’t fit.

  • Bare Pod (no controller). A Pod created directly without any controller above it. On node failure, it’s gone forever — no replacement, no replication. Use only for one-shot debugging or Static Pods for control-plane bootstrapping.

Production Notes

  • The pod-template-hash mechanism is what makes Deployment rollouts work cleanly. Every Deployment’s spec.template is hashed (via a SHA-256-based scheme), and the hash is added as a label both to the ReplicaSet (in spec.selector and spec.template.metadata.labels) and to all Pods it creates. Two ReplicaSets with different templates have different hashes, hence non-overlapping Pod sets, hence can coexist during rollout. The Deployment scales the new RS up and the old RS down; the ReplicaSets themselves remain ignorant of the rollout — they just maintain their own count.

  • Why a ReplicaSet wasn’t merged into Deployment. The 2015 design discussion (preserved in the community design proposals) is explicit about the separation of concerns: a ReplicaSet “just maintains count” and is a building block. The Deployment is one consumer; rolling-update tooling, blue-green deployments via Argo Rollouts, and custom controllers are others. Merging them would couple count-maintenance to update-strategy, foreclosing alternative update strategies.

  • Argo Rollouts (argoproj.io/argo-rollouts) is the canonical example of a non-Deployment ReplicaSet consumer. The Rollout CRD creates and manages ReplicaSets directly, implementing canary and blue-green deployment strategies that go beyond Deployment’s rolling-update model. Each Rollout maintains a stable RS and a canary RS; the controller shifts replicas between them and between Services according to user-defined analysis rules.

  • Operational visibility. kubectl get rs shows the ReplicaSets under each Deployment; after a rolling update, you’ll see two: one old (replicas: 0, current: 0) and one new (replicas: N, current: N). The old RS isn’t deleted (so rollback can re-scale it); it accumulates as a no-op artifact. The Deployment’s revisionHistoryLimit (default 10) caps how many old RSes are retained.

  • k8s.af stories. Several outage write-ups blame “the ReplicaSet” for a problem that’s really at the Deployment or admission layer: a selector overlap caused two Deployments to fight over the same Pods, a misconfigured pod-deletion-cost annotation kept a broken Pod around forever, a minReadySeconds: 600 ground a rollout to a halt because Pods couldn’t accumulate 10 minutes of readiness. The ReplicaSet is rarely the root cause; it’s the messenger.

  • Scale limits. A single ReplicaSet with 5,000+ replicas stresses the controller’s list operations and the API server’s watch fan-out. The K8s scale guidance (SIG-Scalability) is to keep individual workloads under ~5,000 Pods and shard larger workloads across multiple Deployments. At that scale you should also be running EndpointSlice (default since 1.21) rather than legacy Endpoints, which has its own scale limits.

See Also

Graph View

Backlinks