vault

Multi-Cluster Kubernetes

15 min read

Multi-Cluster Kubernetes

Multi-cluster Kubernetes is the operational discipline of running workloads across more than one Kubernetes cluster — and treating that fleet of clusters as a coherent platform rather than a pile of unrelated installations. A single Kubernetes cluster is an extraordinarily capable abstraction, but it has hard ceilings: a documented scale limit of 5,000 nodes / 150,000 pods / 300,000 containers per cluster (Kubernetes — Considerations for large clusters), a single etcd keyspace whose loss is total, a single kube-apiserver failure domain, and a single upgrade/policy boundary. The moment any of those ceilings — scale, blast radius, regulatory geography, environment isolation, edge latency — becomes load-bearing, the answer is another cluster, and from that point the problem is no longer “how do I run a cluster” but “how do I run a fleet.” This note is the umbrella for Kubernetes MOC §17: it covers why one cluster eventually isn’t enough, the new hard problems multi-cluster introduces (cross-cluster service discovery, identity federation, config propagation, observability aggregation, a single pane of glass), and the solution landscape — Cluster Federation, Karmada, the Multi-Cluster Services API, GitOps-to-many-clusters, and Cluster API for the lifecycle layer underneath.

Mental Model

The mental model is the cluster as a unit of failure, jurisdiction, and capacity. A cluster is not just “where pods run” — it is simultaneously (1) a blast-radius boundary (a bad rollout, a misbehaving operator, an etcd corruption, an API server meltdown affects exactly one cluster), (2) a jurisdictional boundary (one region, one trust domain, one set of compliance controls), and (3) a capacity boundary (one scheduler, one etcd, one set of scale limits). Multi-cluster is the deliberate decision to partition along one or more of those axes — and then pay the coordination cost of stitching the partitions back together where workloads need to span them.

flowchart TB
    subgraph WHY["Why partition into multiple clusters"]
        BR["Blast radius<br/>bad rollout / etcd loss<br/>hits one cluster only"]
        SCALE["Scale ceiling<br/>~5000 nodes / 150k pods<br/>per cluster"]
        REG["Regulatory / data residency<br/>EU data in EU cluster"]
        ENV["Environment separation<br/>prod / staging / dev"]
        EDGE["Edge / regional latency<br/>cluster near the users"]
        UPG["Upgrade isolation<br/>canary the K8s version itself"]
    end
    subgraph FLEET["The fleet you now have to manage"]
        C1["Cluster A<br/>us-east-1"]
        C2["Cluster B<br/>eu-west-1"]
        C3["Cluster C<br/>edge / staging"]
    end
    WHY --> FLEET
    subgraph HARD["...and the new hard problems"]
        DISC["Cross-cluster service discovery"]
        IDENT["Identity / trust federation"]
        CFG["Config propagation to N clusters"]
        OBS["Observability aggregation"]
        PANE["Single pane of glass"]
    end
    FLEET --> HARD
    subgraph SOLNS["Solution layers"]
        LIFECYCLE["Lifecycle: Cluster API, kOps, managed services"]
        ORCH["Orchestration: Karmada, KubeFed (retired), OCM"]
        DELIVERY["Delivery: ArgoCD ApplicationSet, Flux"]
        MESH["Connectivity: multi-cluster mesh, MCS API, ClusterMesh"]
    end
    HARD --> SOLNS

What this shows. The top row is the forcing functions — the six reasons a single cluster stops being enough. The middle is the fleet you end up with. The third row is the price: every problem a single cluster solved for free (a service name resolves; a workload’s identity is trusted; config lands everywhere; metrics aggregate; one dashboard shows everything) becomes a distributed-systems problem again the moment there are two clusters. The bottom row is the solution stack, and the key insight is that it is four distinct layers, each with its own tools: cluster lifecycle (creating/upgrading/deleting clusters — Cluster API), workload orchestration (deciding which clusters a workload runs in — Karmada), application delivery (GitOps reconciliation to many clusters — ArgoCD ApplicationSet, Flux), and connectivity (cross-cluster networking and discovery — Multi-cluster Service Mesh, the MCS API, Cilium ClusterMesh). Confusing the layers — “we’ll use Karmada to upgrade clusters” or “we’ll use Cluster API to route traffic” — is the most common multi-cluster design error.

Mechanical Walk-through

Why one cluster isn’t enough — the six forcing functions, in detail.

Blast-radius isolation. A Kubernetes cluster’s control plane is a shared fate. A bad admission webhook that rejects all writes, an etcd disk filling up, a kube-apiserver OOM, a Custom Resource Definition whose conversion webhook wedges — any one of these can render an entire cluster non-functional, and every workload in it loses the ability to scale, roll out, or recover. Splitting workloads across multiple clusters bounds the damage: a control-plane failure in cluster A leaves cluster B untouched. This is the multi-cluster argument that even small organizations eventually feel, and it is the direct antidote to the Cluster as God Object Anti-Pattern — one shared cluster for everything is one shared failure for everything. It is also the Kubernetes-level expression of Cell-Based Architecture: a “cell” is, very often, a cluster.

Scale ceilings. Kubernetes publishes a tested scale envelope — no more than 5,000 nodes, 150,000 total pods, 300,000 total containers, and 100 pods per node (Kubernetes — Considerations for large clusters). These are not arbitrary: they are where etcd’s database size and watch throughput, the API server’s memory (OpenAI observed up to ~70 GB of heap per API server in a 7,500-node cluster — OpenAI), and the scheduler’s bind throughput stop being comfortable. Some operators push past 5,000 nodes with heavy tuning (a dedicated Events etcd, custom schedulers, label/field-selector discipline on every controller), but the recommended answer beyond the envelope is more clusters, not a bigger one.

Regulatory and data-residency. GDPR-style data-residency rules can require that EU users’ data never leave EU infrastructure. A per-region cluster makes the boundary physical and auditable: the EU cluster’s nodes, etcd, and storage are all in the EU, and there is no codepath by which a US-region scheduler could place an EU workload on US hardware. One global cluster cannot make that guarantee structurally.

Environment separation. Production, staging, and development are different trust and stability domains. A shared cluster with namespace-based separation works until a staging workload exhausts a shared node pool, a dev engineer’s overly broad Kubernetes RBAC ClusterRole reaches production objects, or a staging admission webhook is mistakenly cluster-scoped. Separate clusters make the separation a hard wall.

Edge and regional latency. User-facing latency is dominated by geography. A cluster per region (or per edge site — see KubeEdge) puts compute physically close to users. There is no way to make a single us-east-1 cluster serve Sydney users with low latency.

Upgrade isolation. Kubernetes ships a minor version per quarter. Upgrading the control plane is the riskiest routine operation an operator performs (Kubernetes Cluster Upgrade). With multiple clusters, the Kubernetes version itself can be canaried: upgrade one low-stakes cluster, bake, then roll the fleet. With one cluster, every upgrade is all-or-nothing.

The new hard problems. Having split into a fleet, five capabilities that a single cluster provided for free must now be rebuilt:

  1. Cross-cluster service discovery. Within a cluster, payments.default.svc.cluster.local resolves via CoreDNS. Across clusters it does not — cluster A has no idea cluster B has a payments Service. The Kubernetes-native answer is the Multi-Cluster Services (MCS) API (KEP-1645): a ServiceExport object marks a Service as fleet-visible, a controller creates a corresponding ServiceImport in every other cluster of the ClusterSet, and the exported service becomes resolvable at <service>.<namespace>.svc.clusterset.local. Mesh-based (Multi-cluster Service Mesh) and CNI-based (Cilium ClusterMesh) approaches solve the same problem at different layers.
  2. Identity / trust federation. A workload’s identity (a ServiceAccount token, a mesh SPIFFE identity) is cluster-scoped. For cross-cluster mTLS to work, the clusters must share a trust root — a common root CA issuing per-cluster intermediates. See Multi-cluster Service Mesh.
  3. Config propagation. A ConfigMap, a NetworkPolicy, an RBAC policy must now land in N clusters consistently. This is the job of Karmada (propagation policies), GitOps fan-out (ArgoCD ApplicationSet, Flux), or the retired KubeFed.
  4. Observability aggregation. kubectl get pods shows one cluster. Fleet-wide observability needs metrics, logs, and traces aggregated across clusters — Thanos/Mimir for federated Prometheus, a central logging backend, distributed tracing that follows requests across cluster boundaries.
  5. A single pane of glass. Operators need one place to see and act on the whole fleet — which clusters exist, what versions they run, what is deployed where. This is what “fleet management” platforms (Rancher, Karmada’s dashboard, Open Cluster Management, GKE Fleets, EKS fleet tooling) provide.

The solution landscape — how it evolved. The first attempt was federation: a control plane above the clusters with its own API. Federation v1 (~2016) and then KubeFed / Federation v2 (Cluster Federation) tried this; both were abandoned, and KubeFed was archived by SIG-Multicluster on 3 January 2023 (SIG Multicluster archival announcement) — the SIG chairs framed it as clarifying that “federation” as a single API-above-clusters was not the direction the SIG was pursuing (archival is not deletion: the code remains on GitHub for reference/forking). The ecosystem then split the problem along four layers: GitOps to many clusters (ArgoCD ApplicationSet with a cluster generator, Flux) handles the config-propagation slice declaratively; Karmada (CNCF Incubating since 12 December 2023, per CNCF) and Open Cluster Management (CNCF Sandbox since 9 November 2021, per CNCF) handle workload orchestration (scheduling/spreading workloads across clusters with policy); the MCS API standardizes discovery; service meshes and ClusterMesh handle connectivity; and Cluster API (a SIG Cluster Lifecycle subproject, at v1.12 as of January 2026, serving v1beta1 with v1beta2 in development — Kubernetes blog) handles cluster lifecycle underneath all of it. There is no single tool that “does multi-cluster” — there is a layered stack, and a real platform picks one tool per layer.

Configuration / API Surface

There is no single “multi-cluster” object in core Kubernetes — multi-cluster is a pattern, expressed through several APIs. The most standardized is the Multi-Cluster Services API (SIG Multicluster). To expose a Service to the fleet, you create a ServiceExport in the cluster that owns the Service:

apiVersion: multicluster.x-k8s.io/v1alpha1   # MCS API group; the served CRD is still v1alpha1 (as of 2026)
kind: ServiceExport                          # "publish this Service to the ClusterSet"
metadata:
  name: payments                             # MUST match the name of an existing Service
  namespace: commerce                        # and its namespace — MCS uses namespace sameness

Line-by-line: ServiceExport carries no spec — its mere existence, with a name/namespace matching a real Service (Kubernetes), is the signal. An MCS controller (Cilium ClusterMesh, GKE MCS, AWS Cloud Map MCS controller, Submariner) watches ServiceExport objects and, for every other cluster in the ClusterSet (the set of clusters explicitly joined together), creates a ServiceImport:

apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ServiceImport                          # auto-created by the MCS controller — do not hand-write
metadata:
  name: payments
  namespace: commerce
spec:
  type: ClusterSetIP                         # a virtual IP fronting endpoints from ALL clusters
  ports:
    - port: 8080
      protocol: TCP

The ServiceImport is the in-cluster representative of the multi-cluster service: it behaves like a regular Service, gets a ClusterSetIP, and is resolvable at payments.commerce.svc.clusterset.local. A consumer pod in any ClusterSet cluster can call that name and reach endpoints in any exporting cluster. MCS relies on namespace sameness — namespace commerce means the same thing in every cluster — which is itself a fleet-wide governance assumption worth being explicit about.

The ClusterSet itself — the explicit grouping of clusters that agree to share services — and a cluster’s identity within it are defined not by the MCS API but by the companion about-api project (kubernetes-sigs/about-api). A cluster declares its identity by creating a ClusterProperty object with the well-known name cluster.clusterset.k8s.io (a unique per-cluster ID) and clusterset.k8s.io (the name of the ClusterSet it belongs to). Implementations such as Karmada read this ClusterProperty to learn a registering cluster’s ID; without a consistent ClusterSet identity scheme, the MCS controllers cannot reason about which clusters are peers. This is the membership substrate beneath the discovery layer.

For config propagation the canonical declarative surface is a GitOps fan-out. An ArgoCD ApplicationSet with a cluster generator templates one Application per registered cluster:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: platform-baseline
spec:
  generators:
    - clusters:                              # one entry per cluster Secret registered with Argo CD
        selector:
          matchLabels:
            fleet: production                # only clusters labelled fleet=production
  template:
    metadata:
      name: 'baseline-{{name}}'              # {{name}} expands per cluster
    spec:
      destination:
        server: '{{server}}'                 # the cluster's API endpoint, from its Secret
        namespace: platform
      source:
        repoURL: https://git.example.com/platform
        path: baseline

Adding a cluster is then a one-line change: register its Secret with the fleet=production label and the ApplicationSet automatically generates and syncs an Application to it (Argo CD — Cluster Generator).

Failure Modes

Treating a fleet as a single cluster (or vice versa). The two opposite errors. Treating a fleet as one cluster — assuming a service name resolves everywhere, that an RBAC policy applied once is fleet-wide — produces silent gaps. Treating one big cluster as if it were a safe fleet — putting prod and dev in one cluster with “just namespaces” — produces the Cluster as God Object Anti-Pattern. The discipline is naming the partition axis explicitly per cluster.

Configuration drift across clusters. Without a propagation mechanism, clusters diverge: cluster B is two K8s minors behind, cluster C has a stale NetworkPolicy, cluster A’s quota was hand-edited. Drift is the default state of any fleet not under GitOps or an orchestrator. The fix is every cluster reconciled from a single source of truth (GitOps, Karmada).

Cross-cluster discovery without trust. Wiring cross-cluster connectivity (a mesh East-West gateway, ClusterMesh tunnels, MCS) without first federating identity means traffic either fails mTLS or, worse, is sent unauthenticated. Trust federation must precede connectivity.

Observability blind spots. A fleet whose metrics are not aggregated produces incidents where the operator literally cannot see the failing cluster. Each new cluster must be onboarded into the central observability backend as part of provisioning, not after.

Quadratic operational cost. Each cluster is a control plane to patch, an etcd to back up, an upgrade to schedule, a set of add-ons to keep current. Ten clusters is ten times the day-2 surface unless that surface is automated (Cluster API, managed services). Multi-cluster without automation trades one big problem for ten medium ones.

ClusterSet namespace-sameness violations. The MCS API assumes namespace X means the same tenant/app in every cluster. If clusters disagree — namespace payments is team A in one cluster and team B in another — ServiceExport/ServiceImport will cross-wire services. Namespace governance is a prerequisite, not a detail.

Alternatives and When to Choose Them

One big cluster with hard multi-tenancy. Before going multi-cluster, exhaust single-cluster isolation: Namespace + Kubernetes RBAC + ResourceQuota + NetworkPolicy, or virtual clusters (vCluster). The MOC’s decision framework is explicit — RBAC over namespaces, namespaces over clusters, clusters over single-tenant tooling; climb only when you’ve felt the layer below’s limits. One cluster is cheaper to operate; choose multi-cluster only when a forcing function above is genuinely load-bearing.

Multiple fully independent clusters (no fleet tooling). Many organizations run several clusters with zero cross-cluster coordination — separate teams, separate GitOps repos, no cross-cluster service calls. This is legitimate and simple. “Multi-cluster Kubernetes” as a platform discipline is only needed when workloads or governance must span clusters.

Cell-based architecture. Cell-Based Architecture is the architectural-pattern parent: each “cell” is an isolated, independently-failing replica of the stack, and a Kubernetes cluster is the most common cell substrate. If you arrived at multi-cluster via blast-radius reasoning, you are doing cell-based architecture; read that note for the routing-and-sharding half of the pattern.

Managed fleet services. GKE Fleets / Anthos, EKS fleet tooling, Azure Arc, and Rancher each provide an opinionated single-pane-of-glass over multiple clusters. Choosing one trades flexibility for a coherent out-of-the-box experience — reasonable for organizations that don’t want to assemble the four-layer stack themselves.

Production Notes

The CNCF’s own framing (CNCF — Karmada and Open Cluster Management) is that multi-cluster fleet management has no single winning solution because the scenarios — DR, geo-distribution, scale, isolation — are too diverse for one tool; the practical reality is a per-layer stack.

OpenAI’s scaling-to-7,500-nodes write-up is the canonical “push one cluster hard” counter-data-point: with enormous tuning effort (dedicated Events etcd, custom scheduling, strict client discipline) a single cluster reached 7,500 nodes — but the post reads as a catalogue of why most organizations should not try, and motivates the multi-cluster default for anyone without a dedicated platform team.

GKE and EKS both document fleet/multi-cluster patterns as first-class (GKE — planning large clusters, EKS — Kubernetes Scaling Theory) and both recommend more, smaller clusters over fewer giant ones once the fleet exceeds a handful of clusters — the operational simplicity of a sub-5,000-node cluster outweighs the coordination cost when fleet tooling is in place.

On the MCS API’s maturity — a worked example of “stage” versus “served version” diverging. KEP-1645 carries stage: beta in its kep.yaml, which is sometimes misread as “the API is beta.” It is not: the reference implementation in kubernetes-sigs/mcs-api still serves the CRDs at multicluster.x-k8s.io/v1alpha1 — there is no v1alpha2 or v1beta1 package in the repository as of 2026 — and every downstream implementation (Cilium ClusterMesh, GKE MCS, Submariner, Karmada’s multi-cluster service discovery) consumes v1alpha1 objects. The KEP’s “beta” label refers to the enhancement-process stage (the design is considered stable enough to build on), not to a beta-versioned, conversion-guaranteed Kubernetes API. The practical consequence: the apiVersion: multicluster.x-k8s.io/v1alpha1 in the YAML above is correct and current, but because it is still an alpha-versioned CRD it carries no Kubernetes API deprecation/conversion guarantees — a future bump to v1beta1 could change field names. Pin to a specific implementation’s revision in production.

See Also

Graph View

Backlinks