vault

Kubernetes Multi-tenancy Models

11 min read

Kubernetes Multi-tenancy Models

Multi-tenancy in Kubernetes is the practice of sharing one cluster (or one fleet) across multiple tenants — teams, customers, environments — without them interfering with, observing, or compromising one another. There is no single “multi-tenancy feature”; instead there is a spectrum of isolation models, trading isolation strength against cost and operational overhead (kubernetes.io — Multi-tenancy). The spectrum runs from namespaces as tenants (cheap, but the control plane, the nodes, and the Linux kernel are all shared) through virtual control planes and node-level isolation to one cluster per tenant (full isolation, full cost). The choice is governed by one dominant question: how much do the tenants trust each other? Internal engineering teams who would never deliberately attack each other can share far more than mutually-hostile SaaS customers running arbitrary code. The load-bearing truth underneath all of it: a Kubernetes namespace is not a security boundary — it is an organizational and policy boundary, and a kernel-level escape crosses it as if it were not there.

Mental Model

flowchart LR
    subgraph WEAK["weaker isolation · lower cost"]
        NS["Namespaces as tenants<br/>('soft multi-tenancy')<br/>shared CP, nodes, kernel"]
    end
    subgraph MID["intermediate isolation"]
        VC["Virtual control planes<br/>(vCluster)<br/>own API server per tenant,<br/>shared nodes & kernel"]
        NODE["Node-level isolation<br/>dedicated node pools /<br/>sandboxed runtimes<br/>(gVisor, Kata)"]
    end
    subgraph STRONG["stronger isolation · higher cost"]
        CLUSTER["Cluster per tenant<br/>('hard multi-tenancy')<br/>own CP, own nodes, own kernel"]
    end
    NS --> VC --> NODE --> CLUSTER
    TRUST["trust ↓ / hostility ↑ / compliance ↑<br/>───────────────────────────▶"]

What this diagram shows. The four models are not alternatives so much as a gradient. Moving rightward buys stronger isolation — first an isolated API surface, then an isolated kernel, finally an isolated control plane — and pays for it in money (more clusters, more nodes, sandbox overhead) and operational toil (more things to upgrade, patch, monitor). The driver along the axis is the trust relationship between tenants plus any compliance regime that mandates a hard boundary. The insight to extract: there is no “correct” model in the abstract; there is only the model that matches a specific tenant trust level and a specific cost tolerance. Most real organizations run more than one model at once — soft multi-tenancy for internal teams, hard for regulated workloads.

Mechanical Walk-through

Model 1 — Namespaces as tenants (“soft multi-tenancy”)

The cheapest model: give each tenant a Namespace (or a set of them) and stack the namespace-scoped controls on top.

  • Namespace — the partition itself; the unit that RBAC, quota, and policy scope to.
  • Kubernetes RBAC — Roles and RoleBindings restrict each tenant’s identities to their own namespace’s objects.
  • ResourceQuota + LimitRange — bound how much CPU/memory/storage/objects a tenant may consume, so one tenant cannot starve the others (the “noisy neighbour” defence).
  • NetworkPolicy — a pod-level firewall; without it, every Pod in the cluster can reach every other Pod across namespace lines (the flat-network default). A default-deny NetworkPolicy per tenant namespace is mandatory for any real isolation.
  • Pod Security Standards / Pod Security Admission — enforce the Restricted profile so tenants cannot run privileged Pods, host-mount, or escalate.

What this model does not isolate: the control plane (kube-apiserver, etcd, the scheduler are shared — a CRD or webhook installed by one tenant is cluster-global), the nodes (Pods from different tenants land on the same machine), and crucially the Linux kernel (every container on a node shares one kernel; a kernel exploit or container-escape from one tenant’s Pod compromises every co-located tenant). This is precisely why the docs insist namespaces are not a security boundary. Soft multi-tenancy is appropriate when tenants are trusted — internal teams who might accidentally misbehave but would not deliberately attack — and the goal is organizational tidiness, fair resource sharing, and accident containment, not defence against a hostile actor.

Model 2 — Virtual control planes

The next step up gives each tenant a syntactically real, private API server while still scheduling their workloads onto shared host-cluster nodes. The dominant implementation is vCluster: a tenant’s “cluster” is a Pod (or set of Pods) running an actual kube-apiserver plus a lightweight datastore (SQLite/etcd) and a syncer inside one namespace of a host cluster. The tenant gets full API freedom — they can install their own CRDs, run cluster-scoped controllers, pick their own Kubernetes minor version, hold cluster-admin on their virtual cluster — none of which leaks into the host or into sibling tenants, because each virtual cluster’s API server and etcd are entirely separate objects. The syncer copies the tenant’s Pods down to the host cluster’s real nodes for execution.

What this isolates that namespaces do not: the control plane (each tenant’s API server, CRDs, RBAC, and API version are fully their own). What it still shares: the host cluster’s nodes and therefore the kernel — a virtual cluster’s Pod runs as an ordinary container on a host node, so a container escape still crosses tenants. Virtual control planes are the sweet spot for “I need to give tenants cluster-admin-grade freedom and CRD installation rights, but I don’t trust them with the kernel and I can’t afford a cluster each.”

Model 3 — Node-level isolation

Two distinct techniques live here, often combined.

  • Dedicated node pools per tenant — use Taints and Tolerations (taint a node pool with the tenant key, give only that tenant’s Pods the matching toleration) plus Node Affinity to guarantee a tenant’s Pods land only on that tenant’s nodes. This isolates the blast radius of node sharing — a noisy or compromised Pod can only affect its own tenant’s nodes — without isolating the kernel of a single node from the Pods on it.
  • Sandboxed runtimes — swap the default runc container runtime for a kernel-isolating one, selected per-Pod via RuntimeClass:
    • gVisor (runsc) — a user-space kernel written in Go that intercepts the container’s syscalls, so the container never talks to the host kernel directly. Strong isolation, some syscall-compatibility and performance cost.
    • Kata Containers — runs each Pod inside a lightweight hardware-virtualized VM with its own guest kernel. Even stronger isolation (a real VM boundary), higher per-Pod overhead (accounted via Pod Overhead).

Sandboxed runtimes are the answer to the kernel-sharing weakness that both soft multi-tenancy and virtual control planes leave open. They are what makes it defensible to run arbitrary, untrusted tenant code (a CI runner executing customer code, a serverless function platform) on shared nodes.

Model 4 — Cluster per tenant (“hard multi-tenancy”)

The maximal model: each tenant gets a wholly separate cluster — its own control plane, its own nodes, its own kernel, its own etcd. This is the only model that fully isolates the control plane and the kernel with no shared substrate. It is also the most expensive (a control plane per tenant, idle-capacity multiplied across clusters) and the most operationally heavy (every cluster is one more thing to upgrade, patch, back up, monitor). Tooling — Cluster API for declarative cluster lifecycle, managed offerings like Amazon EKS/Google GKE/Azure AKS, or “control plane as a service” patterns like Kamaji — exists specifically to make running many clusters tractable. Hard multi-tenancy is mandated when tenants are mutually hostile (different paying customers running arbitrary workloads) or when a compliance regime (PCI-DSS, HIPAA, FedRAMP, data-residency law) requires a demonstrable hard boundary that a shared control plane cannot provide.

Configuration / API Surface

Soft multi-tenancy is not a single object — it is a stack of namespace-scoped objects applied per tenant. The minimum viable tenant baseline:

# 1. The partition itself, labelled for Pod Security Admission enforcement.
apiVersion: v1
kind: Namespace
metadata:
  name: tenant-acme
  labels:
    pod-security.kubernetes.io/enforce: restricted      # block privileged Pods, host mounts
    pod-security.kubernetes.io/enforce-version: latest
---
# 2. A compute + object cap so this tenant cannot starve the cluster.
apiVersion: v1
kind: ResourceQuota
metadata: { name: tenant-acme-quota, namespace: tenant-acme }
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    pods: "100"
---
# 3. Per-Pod defaults so the quota above is usable (see [[LimitRange]]).
apiVersion: v1
kind: LimitRange
metadata: { name: tenant-acme-limits, namespace: tenant-acme }
spec:
  limits:
  - type: Container
    defaultRequest: { cpu: 100m, memory: 128Mi }
    default:        { cpu: 500m, memory: 256Mi }
---
# 4. Default-deny: without this, the flat cluster network lets any Pod reach this namespace.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny-ingress, namespace: tenant-acme }
spec:
  podSelector: {}                                       # every Pod in the namespace
  policyTypes: [Ingress]                                # ingress not listed below ⇒ denied
---
# 5. RBAC: bind the tenant's group to a Role scoped to this namespace only.
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: tenant-acme-admin, namespace: tenant-acme }
subjects: [{ kind: Group, name: "tenant:acme", apiGroup: rbac.authorization.k8s.io }]
roleRef: { kind: ClusterRole, name: admin, apiGroup: rbac.authorization.k8s.io }

For node-level isolation, a tenant’s Pods additionally carry a tolerations block matching a tenant-taint and, for kernel isolation, spec.runtimeClassName: gvisor. For a virtual control plane, the tenant interacts with a separate kubeconfig pointing at their vCluster’s API server entirely.

Failure Modes

Treating a namespace as a security boundary. The recurring catastrophic mistake: running hostile, arbitrary tenant code in plain namespaces. A container escape (a kernel CVE, a misconfigured hostPath, a privileged Pod that PSA failed to block) crosses every namespace on the node instantly. Symptom: a single compromised Pod yields the whole node and every tenant on it. Fix: do not run untrusted code without a sandboxed runtime or a separate cluster.

Forgetting the default-deny NetworkPolicy. Namespaces partition names, not network reachability. Without a default-deny policy, a Pod in tenant A can open a TCP connection straight to a Pod in tenant B. Symptom: tenants can scrape each other’s services. Fix: a default-deny NetworkPolicy per tenant namespace, plus a CNI that actually enforces policy (Calico, Cilium).

Cluster-scoped escape hatches. Even with perfect namespace RBAC, CRDs, ValidatingWebhookConfigurations, PriorityClasses, and ClusterRole objects are cluster-global. A tenant granted the ability to create any of these affects every other tenant. Symptom: one tenant installs a mutating webhook that intercepts all Pod creates cluster-wide. Fix: never grant tenants cluster-scoped write verbs in soft multi-tenancy; if they need CRDs, give them a virtual control plane.

Quota without LimitRange. A tenant’s compute ResourceQuota with no LimitRange rejects every Pod that omits requests/limits — the tenant experiences their namespace as broken. Fix: always pair the two.

Control-plane noisy neighbour. Soft and virtual-control-plane models share the host etcd / API server. A tenant generating millions of objects or hammering LIST/WATCH degrades the API for everyone; API Priority and Fairness mitigates but does not eliminate this. Hard multi-tenancy is the only complete fix.

Alternatives and When to Choose Them

The four models are the alternatives; the decision is which to pick.

ModelIsolates CP?Isolates kernel?CostChoose when
Namespaces (soft)NoNoLowestTrusted internal teams; accident containment + fair sharing
Virtual control plane (vCluster)YesNoLow–mediumTenants need cluster-admin/CRDs; kernel trust still acceptable
Node pools + sandboxed runtimePartialYes (sandbox)MediumUntrusted code must run on shared infrastructure
Cluster per tenant (hard)YesYesHighestMutually-hostile tenants; compliance mandates a hard boundary

Decision factors, in priority order: (1) trust level — internal teams vs hostile SaaS customers running arbitrary code; (2) compliance — a regime that mandates a hard boundary removes the choice; (3) cost — a control plane and idle capacity per tenant is real money; (4) operational overhead — every cluster multiplies the upgrade/patch/backup surface. Note these models compose: a single organization commonly runs soft multi-tenancy for internal teams and a separate hard cluster for regulated workloads.

Production Notes

  • “Namespaces are not a security boundary” is the most-repeated multi-tenancy lesson. It appears verbatim in the Kubernetes docs and in essentially every production write-up. A namespace contains accidents; it does not contain attackers.
  • Most real fleets are multi-model. The CNCF multi-tenancy guidance and vendor write-ups converge on the same picture: internal teams on soft multi-tenancy, untrusted or regulated workloads on hard. Picking one model for the whole organization is usually wrong.
  • vCluster has become the default “middle” answer. Giving tenants a virtual control plane sidesteps the two biggest soft-multi-tenancy pains — cluster-scoped escape hatches and CRD conflicts — at a fraction of cluster-per-tenant cost. Its remaining gap (shared kernel) is closed by layering a sandboxed runtime underneath.
  • Sandboxed runtimes are the price of running arbitrary code. CI systems, serverless platforms, and notebook services that execute customer code on shared nodes almost universally reach for gVisor or Kata via RuntimeClass; plain runc on a shared node is indefensible for hostile workloads.
  • Climb the ladder only when you’ve felt the limits below. Each model up is more isolation and more cost and toil. The Kubernetes MOC decision framework puts it bluntly: RBAC over namespaces, namespaces over clusters — climb only when the layer below has demonstrably failed you.

See Also

Graph View

Backlinks