Multi-cluster Service Mesh

A multi-cluster service mesh extends a single mesh’s guarantees — mutual-TLS identity, retries, traffic policy, observability — across more than one Kubernetes cluster, so that workloads in different clusters communicate as if they were one mesh. A single-cluster mesh (Istio, Linkerd, Cilium Service Mesh) gives every workload a cryptographic identity and a uniform L7 control plane within one cluster’s network and one CoreDNS namespace. The moment a second cluster exists, three things break: workloads in cluster B are invisible to cluster A’s service discovery; cluster B’s mesh identities are issued by a different certificate authority that cluster A does not trust; and there is no network path. A multi-cluster mesh solves all three. It rests on three pillars: a shared trust root — a common root CA so that mTLS identities issued in any cluster are mutually verifiable; a control-plane topology — multi-primary (every cluster runs its own control plane) versus primary-remote (one control plane serves remote clusters); and East-West gateways — the dedicated ingress/egress points that carry cross-cluster mesh traffic. On top of those sits cross-cluster service discovery (endpoints from cluster B appear in cluster A) and locality-aware load balancing and failover (prefer same-zone, fail over to another cluster). This is the mesh-layer answer to Multi-Cluster Kubernetes’s connectivity problem.

Mental Model

The mental model is one logical mesh stretched over multiple cluster-shaped network islands, glued by a shared identity root and gateways. Each cluster keeps its own data plane (sidecars or eBPF) and, depending on topology, its own or a shared control plane. What unifies them into one mesh is not a shared network — clusters often sit on entirely separate networks — but a shared trust domain: every workload’s identity is a SPIFFE-style ID like spiffe://mesh.example.com/ns/payments/sa/checkout, and because every cluster’s mesh CA chains to the same root, a sidecar in cluster A can validate a certificate presented by a workload in cluster B. Discovery is then a matter of each cluster’s control plane learning about the other clusters’ endpoints; connectivity is a matter of routing cross-cluster traffic through East-West gateways.

flowchart TB
    ROOT["Shared Root CA<br/>(common trust domain: mesh.example.com)"]
    ROOT -->|"issues intermediate CA"| CAA
    ROOT -->|"issues intermediate CA"| CAB
    subgraph CLA["Cluster A — network 1"]
        CAA["mesh CA (intermediate)"]
        CPA["mesh control plane<br/>(istiod / linkerd / cilium)"]
        APP_A["checkout pod + proxy"]
        EWA["East-West Gateway A"]
    end
    subgraph CLB["Cluster B — network 2"]
        CAB["mesh CA (intermediate)"]
        CPB["mesh control plane"]
        APP_B["payments pod + proxy"]
        EWB["East-West Gateway B"]
    end
    CPA -.->|"discovers B's endpoints<br/>(reads B's API / mirrors services)"| CPB
    CPB -.->|"discovers A's endpoints"| CPA
    APP_A -->|"call payments.*"| EWA
    EWA -->|"mTLS, identity-checked"| EWB
    EWB --> APP_B

What this shows. A single root CA sits above both clusters and issues each cluster’s mesh an intermediate CA. Because both intermediates chain to the same root, the proxy in front of checkout (cluster A) can cryptographically verify the identity certificate presented by payments (cluster B) — and vice versa — even though the two clusters share no network. Each control plane discovers the other cluster’s services (Istio’s istiods read each other’s clusters; Linkerd mirrors services; Cilium exchanges state via clustermesh-apiserver). When checkout calls payments, traffic leaves cluster A through East-West Gateway A, crosses the network to East-West Gateway B, and is delivered to payments — the whole hop protected by mesh mTLS, so even though the gateway may be exposed on the public internet, only a peer with a trusted workload certificate can actually use it. The insight to extract: the unifying substance of a multi-cluster mesh is shared identity, not shared networking. Get the trust root right and the rest is plumbing; get it wrong and nothing else matters.

Mechanical Walk-through

Pillar 1 — shared trust / common root CA. A standalone mesh generates its own self-signed root CA and issues workload certs from it. Two such meshes have unrelated roots: cluster A’s proxies will reject cluster B’s certificates outright. The fix is a shared trust domain backed by a common root CA (Istio — multicluster install). In Istio you generate one root CA, derive a per-cluster intermediate CA, and install each intermediate as the cacerts secret in that cluster’s istiod; istiod then issues workload certs from the intermediate, and because both intermediates chain to the shared root, cross-cluster mTLS validates. Linkerd has the analogous model — a shared trust anchor with per-cluster issuer certificates. The trust domain itself (e.g., mesh.example.com) is part of every SPIFFE identity (SPIFFE concepts); all clusters in one mesh share one trust domain. Getting this wrong is the #1 multi-cluster-mesh failure: clusters simply cannot talk.

Pillar 2 — control-plane topology. Two canonical shapes (Istio — deployment models):

Multi-primary. Every cluster runs its own full mesh control plane (its own istiod). Each control plane has API access to all clusters and so discovers every cluster’s services and configures its local proxies for the whole mesh. The advantage is no cross-cluster control-plane dependency — each cluster’s control plane is self-sufficient, and losing one cluster does not deprive the others of a control plane. This is the high-availability choice and the recommended default for production.
Primary-remote. One cluster runs the control plane (the primary); other clusters (remotes) run only a data plane and connect to the primary’s control plane across the network. Fewer control planes to operate, but the remotes have a hard dependency on the primary: if the primary cluster’s control plane is unreachable, the remotes cannot get new configuration or fresh certificates. Suitable when remote clusters are small or operationally constrained and the primary is highly available.

These combine with a network topology: single-network (all clusters’ pods are on one flat, mutually-routable network — direct pod-to-pod is possible) or multi-network (clusters are on separate networks — all cross-cluster traffic must traverse East-West gateways). Production multi-cluster is usually multi-network.

Pillar 3 — East-West gateways. Where North-South traffic is client↔cluster, East-West traffic is cluster↔cluster service traffic. On separate networks, a cluster cannot reach another cluster’s pod IPs directly, so each cluster runs an East-West gateway: a mesh-managed proxy that exposes that cluster’s services to other clusters. Cross-cluster calls egress through the caller’s side and ingress through the callee cluster’s East-West gateway. Crucially, although the gateway may be reachable on the public internet, it does not terminate trust — it passes through mTLS, so the receiving workload still authenticates the caller’s workload identity (Istio — multi-primary multi-network). A request from cluster B is accepted only if it carries a certificate from the shared trust domain — exactly as if the caller were a local pod.

Cross-cluster service discovery. For checkout in A to call payments in B, A’s mesh must know B’s payments endpoints. Three mechanisms, one per mesh:

Istio — in multi-primary, each istiod is given credentials (a remote secret) to read every other cluster’s Kubernetes API; it watches all clusters’ Services/EndpointSlices and programs every proxy with the union. A Service of the same name in multiple clusters becomes one logical service with endpoints in all of them.
Linkerd — uses an explicit service mirror. The linkerd multicluster link command creates a Link custom resource holding the target cluster’s gateway address, gateway identity, and a label selector. The service-mirror controller then watches the target cluster and, for each matching Service, creates a mirror Service locally (named <svc>-<target-cluster>) whose endpoint is the target cluster’s gateway (Linkerd — multicluster). Discovery is thus explicit and opt-in per service — a deliberate Linkerd design choice favoring clarity over automatic global visibility. Current Linkerd (2.19, as of 2026) offers two flavors of this: the original gateway mode just described, where cross-cluster traffic hops through a per-cluster gateway, and a newer Pod-to-Pod mode (selected via the Link’s remoteDiscoverySelector) that mirrors only the target Service’s endpoints and routes directly to remote pods when the clusters share flat network connectivity — eliminating the gateway hop, analogous to Cilium’s direct-routing property. The Link API is still multicluster.linkerd.io/v1alpha1 in either mode.
Cilium ClusterMesh — each cluster runs a clustermesh-apiserver that publishes that cluster’s endpoints, services, and security identities; agents in peer clusters consume it, giving direct pod-to-pod connectivity across clusters without per-hop proxying, plus global services (annotate a Service service.cilium.io/global: "true" to make its backends span clusters). Cilium also implements the Kubernetes Multi-Cluster Services API (ServiceExport/ServiceImport) (Cilium — MCS API).

Locality-aware load balancing and failover. Once a service has endpoints in several clusters/zones, the mesh should prefer the closest — same zone, then same region, then another cluster — to minimize latency and cross-zone/cross-region data-transfer cost. This is locality-aware load balancing. Coupled with it is locality failover: if all local endpoints are unhealthy, traffic fails over to endpoints in another zone or cluster. Istio expresses this with DestinationRule localityLbSetting (distribute and failover rules over region/zone/subzone); the effect is a mesh that keeps traffic local for cost and latency in the steady state but transparently spills cross-cluster during a localized outage — automatic multi-cluster resilience without application changes.

Configuration / API Surface

The defining cross-cluster object differs per mesh. For Linkerd, the Link resource (created by linkerd multicluster link) is the unit of cross-cluster wiring:

apiVersion: multicluster.linkerd.io/v1alpha1
kind: Link
metadata:
  name: east                                  # the name of the TARGET cluster being linked
  namespace: linkerd-multicluster
spec:
  targetClusterName: east                     # cluster whose services will be mirrored here
  targetClusterDomain: cluster.local          # the target's cluster DNS domain
  gatewayAddress: 203.0.113.40                 # the target cluster's multicluster gateway IP
  gatewayPort: 4143                            # mesh gateway port
  gatewayIdentity: linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local
  selector:                                   # WHICH target services to mirror
    matchLabels:
      mirror.linkerd.io/exported: "true"

Line-by-line: targetClusterName / targetClusterDomain identify the cluster being linked. gatewayAddress + gatewayPort are the target’s East-West gateway — where mirrored traffic is sent. gatewayIdentity is the gateway’s mesh identity, which the service-mirror’d traffic will mTLS-authenticate against (this is the trust check in action). The selector is the opt-in mechanism: only Services in the target cluster labeled mirror.linkerd.io/exported: "true" are mirrored — Linkerd does not expose services across clusters by default; a service owner explicitly labels a Service to publish it. The service-mirror controller then creates a local payments-east Service whose endpoints point at the target gateway, and in-cluster callers address payments-east like any other Service.

For the Kubernetes-native path (used by Cilium and others), the surface is the MCS API ServiceExport — a controller-watched marker with no spec — which causes a ServiceImport resolvable at <service>.<namespace>.svc.clusterset.local to appear in every peer cluster (KEP-1645); see Multi-Cluster Kubernetes for that YAML. For Cilium ClusterMesh, a simpler per-Service annotation makes a service global:

apiVersion: v1
kind: Service
metadata:
  name: payments
  namespace: commerce
  annotations:
    service.cilium.io/global: "true"          # backends in ALL meshed clusters serve this Service
    service.cilium.io/affinity: "local"       # prefer local backends; spill cross-cluster on failure
spec:
  selector: {app: payments}
  ports:
    - port: 8080

service.cilium.io/global: "true" tells ClusterMesh to treat same-named payments Services across all connected clusters as one logical service with a combined endpoint set; service.cilium.io/affinity: "local" is the locality-aware-routing knob — keep traffic in-cluster while local backends are healthy, fail over to remote backends otherwise (the other accepted values are remote, useful for draining a cluster for maintenance, and none, the default, which load-balances across all clusters without preference) (Cilium — Service Affinity). A third related annotation, service.cilium.io/shared (default "true"), gates whether this cluster’s backends are advertised to peers at all — set it "false" to consume a global service without contributing endpoints to it. Affinity governs routing preference; shared governs export — they are orthogonal.

For Istio, cross-cluster discovery is not a single object but the act of installing each cluster’s mesh CA from a shared root and registering each remote cluster’s API access (a remote secret) with every istiod; locality LB is then a DestinationRule.localityLbSetting.

Failure Modes

Mismatched trust roots. The cardinal failure. If clusters’ mesh CAs do not chain to a common root, every cross-cluster mTLS handshake fails — services are simply unreachable across clusters with opaque TLS errors. Verify the shared-root setup first; nothing else works until it does.

Trust domain mismatch. Even with a shared CA, if clusters are configured with different trust domains the SPIFFE identities do not match the expected pattern and authorization policies reject the traffic. All clusters in one mesh must share one trust domain.

East-West gateway exposure and outage. The East-West gateway is often internet-reachable. It is protected by mTLS — but a misconfiguration that disables mTLS, or an authorization-policy gap, turns it into an open door into the mesh. Conversely, if a cluster’s East-West gateway is down, all inbound cross-cluster traffic to that cluster fails while in-cluster traffic is fine — a partial, confusing outage. The gateway must be HA and monitored as critical infrastructure.

Primary-remote control-plane dependency. In primary-remote topology, an outage of the primary cluster’s control plane leaves remote clusters unable to receive new config or renew certificates. Once existing workload certs expire, remote-cluster mTLS breaks even though the remote cluster itself is healthy. Multi-primary avoids this; primary-remote demands a highly-available primary.

Cross-cluster discovery staleness or storms. Istio multi-primary has every istiod watching every cluster’s API — at fleet scale this is a lot of watches and a lot of config recomputation; a flapping cluster can cause config churn mesh-wide. Linkerd’s per-service mirroring bounds this but means a forgotten label leaves a service unreachable cross-cluster.

Locality failover thrash and surprise cost. Misconfigured locality settings can send all traffic cross-region (latency spike, egress-cost spike) or fail to fail over (a local outage takes down the service despite healthy remote endpoints). Locality LB must be tested with deliberate fault injection.

Network MTU / overlay interaction. Cross-cluster traffic through gateways or ClusterMesh tunnels adds encapsulation; MTU mismatches cause silent fragmentation or black-holed large packets — a classic, hard-to-diagnose multi-network mesh problem.

Alternatives and When to Choose Them

Single large cluster instead of a multi-cluster mesh. If the only reason for multiple clusters is “we have many services,” a single cluster with a single-cluster mesh is far simpler. Multi-cluster mesh is justified only when the clusters are needed for the reasons in Multi-Cluster Kubernetes (blast radius, regions, regulation, scale). Do not adopt multi-cluster mesh complexity to solve a problem one cluster already solves.

Mesh-free cross-cluster connectivity — MCS API / ClusterMesh / Submariner. If you need cross-cluster service discovery and connectivity but not L7 mesh features (retries, traffic shifting, fine-grained authz), the Kubernetes MCS API, Cilium ClusterMesh’s plain global services, or Submariner provide it more cheaply — no sidecars, no mesh control plane. Choose these when the requirement is connectivity, not full mesh policy.

API gateway / explicit cross-cluster endpoints. The simplest model: clusters expose services via ordinary ingress / API gateways and call each other over public or private endpoints, with TLS terminated conventionally. No mesh federation, no shared trust root — but also no transparent mTLS identity, no locality LB, no uniform policy. Adequate for a few coarse cross-cluster integrations; it does not scale to many fine-grained service-to-service calls.

Choosing the mesh. Istio has the richest multi-cluster story (multi-primary/primary-remote × single/multi-network in mature sidecar mode, plus ambient multi-cluster now at Beta — see Production Notes) and the most knobs — choose it when you need that flexibility and can run the operational weight. Linkerd’s explicit Link + service-mirror model is simpler and more predictable — choose it when you want clarity and minimal surface. Cilium Service Mesh / ClusterMesh integrates meshing with the CNI and gives direct pod-to-pod cross-cluster connectivity without sidecars — choose it when Cilium is already the CNI and sidecarless is a priority.

Production Notes

The Istio documentation treats multi-cluster as a first-class, heavily-documented topic, enumerating the full topology matrix — multi-primary vs primary-remote × single-network vs multi-network (Istio — deployment models). The consistent production guidance is multi-primary, multi-network for serious deployments: it removes the cross-cluster control-plane dependency, and multi-network is the realistic assumption since clusters in different regions or VPCs rarely share a flat network.

Multi-cluster has also reached Istio’s sidecarless ambient mode (see Ambient Mesh), and dating its maturity matters because it moves fast: ambient multicluster was introduced alpha in Istio 1.27 (Istio blog — ambient multicluster, 2025), and ambient multi-network multicluster was promoted to Beta in Istio 1.29 in 2026 (Istio blog — ambient multi-network multicluster Beta). The ambient cross-cluster data path differs from the sidecar one: instead of an Envoy East-West gateway pair, the source node’s ztunnel dials the remote cluster’s east-west gateway using nested HBONE (HTTP-Based Overlay Network Environment) tunnels — an outer HBONE leg authenticating ztunnel↔gateway and an inner leg providing end-to-end ztunnel↔ztunnel mTLS, so trust is still terminated at the workload, not the gateway. As of the Beta (mid-2026) it carries real limitations: only the multi-primary topology and one network per cluster are supported, there is no cross-cluster L7 failover, no direct pod addressing or headless services, and service/waypoint configuration must be uniform across clusters — so treat ambient multicluster as production-promising but not yet at sidecar-mode parity. Vendor write-ups (Tetrate — multicluster Istio) emphasize that the East-West gateway plus shared-CA setup is the part teams most often get wrong, and that automating certificate issuance from the shared root (rather than hand-managing cacerts secrets) is essential at fleet scale.

Linkerd’s design philosophy shows through in its multi-cluster model: the explicit, per-service, label-gated Link/service-mirror mechanism (Linkerd — multicluster) trades Istio’s automatic global visibility for predictability — an operator can see exactly which services cross which cluster boundary by reading Link resources and mirror Services. This matches Linkerd’s broader stance of a small, comprehensible feature surface.

Cilium ClusterMesh’s distinguishing production property is no extra proxy hop: cross-cluster traffic goes pod-to-pod over the CNI’s own tunneling/routing, so the latency and resource overhead of gateway-mediated mesh traffic is avoided (Cilium — Cluster Mesh). The trade-off is tighter coupling to the network layer and a requirement that all clusters run Cilium with non-overlapping Pod CIDRs.

The API surface above was verified against current documentation (May 2026): the Linkerd Link resource is still multicluster.linkerd.io/v1alpha1 and the default multicluster gateway port is still 4143 in current Linkerd (2.19 stable / edge) (Linkerd — multicluster). Cilium’s service.cilium.io/global is confirmed in the current ClusterMesh service-discovery docs (Cilium — Load-balancing & Service Discovery), and service.cilium.io/affinity is a real, separately-documented annotation accepting local, remote, or none (default), with local meaning “prefer same-cluster backends and fail over to remote on local failure” (Cilium — Service Affinity). The closely-related service.cilium.io/shared annotation (default true) controls whether a cluster’s local backends are exported to peers at all — distinct from affinity, which controls routing preference among already-shared backends. Cilium’s implementation of the Kubernetes Multi-Cluster Services API remains documented as Beta (Cilium — MCS API).

vault

Explorer

Multi-cluster Service Mesh

Multi-cluster Service Mesh

Mental Model

Mechanical Walk-through

Configuration / API Surface

Failure Modes

Alternatives and When to Choose Them

Production Notes

See Also

Graph View

Table of Contents