Pod Affinity and Anti-Affinity

Pod (inter-pod) affinity and anti-affinity (spec.affinity.podAffinity / podAntiAffinity) constrain a Pod’s placement relative to other Pods already running — schedule this Pod near Pods matching a label selector (affinity) or away from them (anti-affinity), where “near”/“away” is scoped by a topologyKey (Assigning Pods to Nodes). Unlike Node Affinity, which matches against node labels, this matches against Pod labels and a topology domain. It is the canonical hammer for two opposite goals: anti-affinity to spread replicas across nodes or zones for high availability, and affinity to co-locate coupled workloads (a cache beside its consumer). Its great cost is computational: evaluating “is there a matching Pod in this topology domain” scans Pod state across the cluster on every candidate node, every scheduling decision — informally O(Pods × Nodes) per pass, behaving like an O(N²) blow-up as a workload scales — which is why Topology Spread Constraints exists as a cheaper, modern replacement for the spread use case.

Mental Model

The topologyKey is the load-bearing concept. It names a node label whose distinct values partition the cluster into topology domains. kubernetes.io/hostname makes each node a domain; topology.kubernetes.io/zone makes each zone a domain. The rule then reads: “for the domain this candidate node belongs to, is there (affinity) / is there not (anti-affinity) a Pod matching the selector?”

flowchart TD
    subgraph ZA["Zone us-east-1a"]
        NA1["Node A1<br/>web replica"]
        NA2["Node A2 (empty)"]
    end
    subgraph ZB["Zone us-east-1b"]
        NB1["Node B1<br/>web replica"]
        NB2["Node B2 (empty)"]
    end
    NEW["New 'web' Pod<br/>podAntiAffinity<br/>selector app=web<br/>topologyKey=hostname"] -->|A1 domain has app=web| RA1[A1 rejected]
    NEW -->|A2 domain empty| OK1[A2 feasible]
    NEW -->|B1 domain has app=web| RB1[B1 rejected]
    NEW -->|B2 domain empty| OK2[B2 feasible]

What this diagram shows and the insight to extract. A new web Pod with hard anti-affinity on app=web and topologyKey: kubernetes.io/hostname. Each node is its own domain; nodes A1 and B1 already host a web Pod so they are infeasible; A2 and B2 are empty so they qualify. The insight: with topologyKey: hostname the rule means “one web Pod per node”; switch the key to topology.kubernetes.io/zone and the same rule means “one web Pod per zone” — radically different placement from a single field change. Choosing the topology key is choosing the granularity of the constraint, and getting it wrong is the most common configuration error in this area.

Mechanical Walk-through

Each affinity term carries:

• a labelSelector — which Pods this term is about (e.g. app=web), using matchLabels and/or matchExpressions;
• a topologyKey — the node label defining the domain (required; an empty topologyKey is rejected);
• optionally namespaces / namespaceSelector — which namespaces the matched Pods may live in. The default — and a frequent surprise — is the incoming Pod’s own namespace only. An empty namespaceSelector: {} matches all namespaces;
• optionally matchLabelKeys / mismatchLabelKeys — keys whose values are read from the incoming Pod’s labels and merged into the selector. matchLabelKeys adds key In (incoming-value); mismatchLabelKeys adds key NotIn (incoming-value).

Like Node Affinity, both podAffinity and podAntiAffinity come in hard and soft forms:

• requiredDuringSchedulingIgnoredDuringExecution — a hard list of terms (ANDed; every term must hold). For affinity, the candidate node’s domain must contain a matching Pod; for anti-affinity, it must not. Evaluated by the InterPodAffinity Filter plugin (see Scheduling Framework).
• preferredDuringSchedulingIgnoredDuringExecution — a soft list of {weight, podAffinityTerm} pairs, folded into the Score phase by the InterPodAffinity Score plugin (default scheduler weight ×2).

Both share the IgnoredDuringExecution suffix and node affinity’s rationale: a running Pod is never evicted if the inter-pod condition later changes (a co-located Pod is deleted, a label flips). There is no RequiredDuringExecution form.

matchLabelKeys — the rolling-update fix. During a Deployment rolling update two ReplicaSets coexist; their Pods share app=web but differ in pod-template-hash. Plain podAntiAffinity on app=web then counts the outgoing revision’s Pods against the incoming revision — old Pods occupy the topology domains, and new Pods cannot find a free domain, stalling the rollout (the inverse problem hits podAffinity: new Pods co-locate with soon-to-die old Pods). matchLabelKeys: [pod-template-hash] fixes this: the scheduler reads the incoming Pod’s pod-template-hash value and appends pod-template-hash In (that-value) to the selector, so the term considers only Pods of the same revision (Kubernetes 1.31 blog). Version history: introduced alpha in 1.29, beta (gate MatchLabelKeysInPodAffinity on by default) in 1.31, and stable (GA) in 1.33. mismatchLabelKeys — the inverse, appending key NotIn (value), useful for tenant isolation (“never co-locate with another tenant’s Pods”) — trails as a beta-level field.

Why it is expensive. To decide if a candidate node satisfies an inter-pod term, the scheduler must know, for that node’s topology domain, whether a matching Pod exists. Building that knowledge means examining the labels of every other Pod in the cluster (or every Pod in the relevant domains), and this happens per candidate node, per scheduling decision. The PreFilter phase amortizes some of this by building a topology-pair term index once per cycle, but the fundamental cost scales with the product of Pods and nodes. The Kubernetes docs explicitly warn that inter-pod affinity/anti-affinity “require substantial amounts of processing which can slow down scheduling in large clusters significantly” and are “not recommended in clusters larger than several hundred nodes.” SIG-Scheduling has an open optimization track to special-case host-scoped rules (k8s issue #138314) — isolating topologyKey: kubernetes.io/hostname rules so they avoid the cluster-wide topology scan.

A worked perf-cliff example. Take a 1000-node cluster running 30,000 Pods, and schedule a new Deployment whose template has hard podAntiAffinity on topologyKey: zone. For each of (up to 1000) candidate nodes the scheduler must determine which zone the node is in and whether any of the ~30,000 Pods matching the selector already occupy that zone — and PreFilter must first build the topology index over those 30,000 Pods. A single scheduling decision can add hundreds of milliseconds; multiply by every Pod in a large rollout and scheduler_e2e_scheduling_duration_seconds p99 crosses a second, throughput collapses well below the ~100 Pods/s baseline, and the whole rollout drags. The same workload expressed as a Topology Spread Constraints maxSkew rule costs far less because PodTopologySpread maintains per-domain counts rather than re-deriving “does a matching Pod exist” per node.

Hard anti-affinity couples replica count to domain count. Hard anti-affinity does not just consider one term — it constrains every Pod of the same group. A Deployment of 5 replicas with hard hostname anti-affinity needs at least 5 nodes; replica 6 on a 5-node cluster stays permanently Pending with 0/N nodes are available: N node(s) didn't match pod anti-affinity rules (k8s issue #131467). Hard anti-affinity silently makes replica count ≤ domain count an invariant.

Configuration / API Surface

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 6
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      affinity:
        podAntiAffinity:                                    # SPREAD the replicas
          preferredDuringSchedulingIgnoredDuringExecution:  # SOFT — won't stall at 6 nodes
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values: [web]
                topologyKey: kubernetes.io/hostname         # one per node, preferentially
                matchLabelKeys: [pod-template-hash]         # only count my own revision
        podAffinity:                                        # CO-LOCATE with the cache
          requiredDuringSchedulingIgnoredDuringExecution:   # HARD — must be in the cache's zone
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values: [redis-cache]
              topologyKey: topology.kubernetes.io/zone
      containers:
        - name: web
          image: web:latest

Line-by-line. The podAntiAffinity block uses the preferred (soft) form deliberately: with topologyKey: kubernetes.io/hostname and weight: 100 the scheduler strongly prefers a different node per web replica, but if only 4 nodes exist all 6 replicas still schedule (doubling up on the least-loaded nodes) rather than leaving 2 Pods Pending — the safe choice for a Deployment whose replica count may exceed node count. Had this been required, replicas 5 and 6 would be stuck without 6 nodes. matchLabelKeys: [pod-template-hash] scopes the anti-affinity to the incoming Pod’s own ReplicaSet so a rolling update is not blocked by the outgoing revision. The podAffinity block uses the required (hard) form with topologyKey: topology.kubernetes.io/zone: every web Pod must land in a zone that already runs a redis-cache Pod — co-locating the web tier with its cache to cut cross-zone latency and inter-AZ data-transfer billing. Note the asymmetry of good practice: anti-affinity is usually soft (HA is a preference; you rarely want it to make Pods unschedulable), affinity is often hard (co-location is frequently a correctness or cost requirement).

The InterPodAffinity plugin also takes a scheduler-level argument, hardPodAffinityWeight (in pluginConfig, default 1), which scales how much a satisfied hard affinity term contributes to scoring — orthogonal to the per-rule weight on soft terms.

Failure Modes

• Hard anti-affinity deadlock. A Deployment with required hostname anti-affinity and more replicas than nodes leaves the surplus Pods permanently Pending: 0/N nodes are available: N node(s) didn't match pod anti-affinity rules. The fix is almost always switching to preferred, or using Topology Spread Constraints with a maxSkew.
• Scheduler latency collapse at scale. Inter-pod affinity on a large cluster pushes scheduler_e2e_scheduling_duration_seconds p99 past a second; the cost is the per-decision Pod scan described above. Mitigation: avoid hard inter-pod rules past a few hundred nodes; prefer topology spread; scope labelSelector and namespaces tightly.
• topologyKey typo or missing node label. If nodes lack the named topology label, the term matches nothing and behavior is silently wrong — anti-affinity on a key no node carries imposes no constraint at all; affinity on it makes every node infeasible. Always confirm the key exists with kubectl get nodes -L <key>.
• Namespace scoping surprise. By default an inter-pod term matches Pods only in the incoming Pod’s own namespace. A team expecting cluster-wide co-location forgets namespaces/namespaceSelector and the rule silently considers far fewer Pods than intended — affinity then “finds nothing” and the Pod is infeasible, or anti-affinity “sees nothing” and over-packs.
• Rolling-update stall without matchLabelKeys. Hard anti-affinity on app=X counts the outgoing revision; new Pods cannot find a free domain and the rollout wedges. Add matchLabelKeys: [pod-template-hash].
• Self-matching with too-coarse a key. An anti-affinity term whose selector matches the incoming Pod’s own labels is normal and intended for spread — but hard self-anti-affinity on a coarse topologyKey (e.g. region) makes the second replica unschedulable if there is only one region. Match key granularity to replica count.

Alternatives and When to Choose Them

• Topology Spread Constraints — the modern, much cheaper answer for the spread use case. Where anti-affinity is binary (“a matching Pod is present in this domain or not”) and scans per node, topology spread expresses “keep the count per domain within maxSkew of every other domain” and the PodTopologySpread plugin maintains domain counts far more efficiently. It also gains minDomains, nodeAffinityPolicy/nodeTaintsPolicy, and matchLabelKeys of its own. For “spread my replicas across zones/nodes for HA”, reach for topology spread first; use anti-affinity only for the strict “never two on the same node” hard requirement.
• Node Affinity — when placement depends on node properties, not other Pods. Orthogonal; frequently combined.
• Taints and Tolerations — when you want a node to repel Pods by default rather than Pods to attract/repel each other.
• Plain inter-pod affinity/anti-affinity (this note) — keep it for: hard “one-per-node” requirements (a singleton that is not a DaemonSet), and co-location (podAffinity), which topology spread does not do — spread only spreads, it cannot pull Pods together.

The decision rule: co-locate → podAffinity. Spread for HA → Topology Spread Constraints. Strict one-per-domain → hard podAntiAffinity.

Production Notes

• HA replica spread is the textbook anti-affinity use: stateless Deployments historically used preferred podAntiAffinity on hostname so a single node failure never took out every replica. Most teams have since migrated this pattern to Topology Spread Constraints for the lower scheduler cost and the explicit maxSkew knob — but the soft anti-affinity form remains a perfectly valid choice for small clusters.
• StatefulSet quorum spread. Databases and quorum systems (etcd, Kafka, ZooKeeper) use hard podAntiAffinity on topology.kubernetes.io/zone so a zone outage cannot break quorum. Here the hardness is justified — losing quorum is strictly worse than a Pending Pod — and the replica count (3, 5) is fixed and small, so the “replicas ≤ domains” coupling is acceptable.
• Cache co-location. podAffinity placing application Pods in the same zone as their Redis/Memcached tier cuts cross-AZ latency and inter-zone data-transfer billing — a genuine affinity use that topology spread cannot replace, since spread only separates and never attracts.
• Tenant isolation. mismatchLabelKeys (e.g. on a tenant label) keeps one tenant’s Pods off domains occupied by another tenant’s Pods without hardcoding tenant values into every manifest.
• Large-cluster guidance is a hard constraint, not a suggestion. The Kubernetes docs caution that inter-pod affinity is “not recommended” beyond several hundred nodes; Cast AI and other operators report it adding hundreds of milliseconds per scheduling decision in 500+ node, 10,000+ Pod clusters. Treat the warning as a design boundary: at scale, spread workloads belong on Topology Spread Constraints, and SIG-Scheduling’s host-scoped-affinity optimization work exists precisely because this cost has bitten production users.

See Also

• Topology Spread Constraints — the cheaper modern replacement for the spread use case
• Node Affinity — placement against node labels rather than Pod labels
• Node Selector — the simplest node-label constraint
• Taints and Tolerations — node-side repulsion, the inverse mechanism
• Kubernetes Scheduler Architecture — where the per-decision Pod scan cost is felt
• Scheduling Framework — the InterPodAffinity plugin and its extension points
• kube-scheduler — the scheduler component
• Pod Disruption Budget — protects spread Pods during voluntary disruption
• StatefulSet — common consumer of hard zone anti-affinity
• Kubernetes MOC — parent index