vault

Kamaji

10 min read

Kamaji

Kamaji is a project (by Clastix) that implements the hosted control planes pattern: it turns an ordinary Kubernetes cluster into a management cluster that runs the control planes of many tenant clusters — each tenant’s kube-apiserver, scheduler, and controller-manager running as ordinary Deployment Pods in the management cluster (Kamaji — Concepts). The tenant’s worker nodes are then real, separate machines (VM or bare metal) that join their hosted control plane over the network — so each tenant ends up with a genuinely separate cluster: its own nodes, its own kernel, its own CNI. What Kamaji collapses is not the cluster boundary but the cost of operating control planes: instead of three dedicated control-plane machines per tenant cluster, every tenant control plane is a few Pods on shared management-cluster infrastructure, and tenant state lives in a shared, multi-tenant datastore (Kamaji — Datastore). This is the same architectural pattern the hyperscalers use internally for EKS/GKE and that Red Hat ships as HyperShift; Kamaji makes it self-hostable, and doubles as a Cluster API control-plane provider.

Mental Model

flowchart TB
    subgraph MGMT["Management Cluster (one real cluster, you operate it)"]
        KAM["Kamaji operator"]
        subgraph TCP1["TenantControlPlane: acme (Deployment)"]
            A1["kube-apiserver"]
            S1["scheduler"]
            C1["controller-manager"]
        end
        subgraph TCP2["TenantControlPlane: globex (Deployment)"]
            A2["kube-apiserver"]
            S2["scheduler"]
            C2["controller-manager"]
        end
        DS[("Shared datastore<br/>etcd  /  PostgreSQL·MySQL via kine")]
    end
    KAM -- "reconciles" --> TCP1 & TCP2
    A1 --> DS
    A2 --> DS
    subgraph WACME["acme worker nodes (real machines)"]
        WN1["kubelet"] & WN2["kubelet"]
    end
    subgraph WGLOBEX["globex worker nodes (real machines)"]
        WN3["kubelet"] & WN4["kubelet"]
    end
    WN1 & WN2 -- "join, watch" --> A1
    WN3 & WN4 -- "join, watch" --> A2

What this shows. Two distinct planes. In the management cluster, each tenant’s control plane is a Deployment (replicated for HA like any Deployment) reconciled by the Kamaji operator from a TenantControlPlane custom resource. All tenant API servers read and write a shared datastore — one etcd or one SQL database serving many tenants, each tenant’s keys logically partitioned. Out in the world, each tenant’s worker nodes are real machines running real kubelets that join their hosted API server. The insight to extract: Kamaji virtualizes the control plane but not the data plane — unlike vCluster, the tenant’s Pods do not land on shared nodes; they land on the tenant’s own nodes. A Kamaji tenant cluster is a real, kernel-isolated cluster whose only “virtual” aspect is where its brain runs.

Mechanical Walk-through

The TenantControlPlane resource. Kamaji is a Kubernetes operator. You install it in a management cluster, and it registers a CRD: TenantControlPlane (kamaji.clastix.io). Creating one such object triggers the Kamaji operator to materialize, in the management cluster, a complete tenant control plane: a Deployment running the tenant’s kube-apiserver (plus scheduler and controller-manager), a Service exposing that API server (ClusterIP, NodePort, LoadBalancer, or via Ingress) so worker nodes can reach it, the PKI (CA and certificates) for the tenant cluster, and an admin kubeconfig published as a Secret. The tenant control plane is just Kubernetes workloads — it benefits from the management cluster’s own scheduling, autoscaling, monitoring, and HA: scale the tenant control plane by bumping the Deployment’s replica count; the management cluster’s scheduler spreads the replicas; the management cluster’s node failures are handled by ordinary Pod rescheduling.

The shared datastore. This is Kamaji’s defining decision and the reason it scales. A normal Kubernetes cluster bundles its own etcd. Kamaji instead decouples the control plane from its datastore (Kamaji — Datastore) so many tenant control planes share one datastore. Three backends are supported:

  • etcd — the default. A DataStore resource points at an etcd cluster running in (or alongside) the management cluster; many TenantControlPlanes are assigned to it, each tenant’s data isolated by key prefix.
  • SQL via kine — Kamaji integrates the kine shim (the same shim k3s uses), which makes a PostgreSQL- or MySQL-compatible database speak the etcd API. The tenant API server thinks it talks to etcd; kine translates to SQL. This lets a managed cloud database (RDS, Cloud SQL) back hundreds of tenant clusters — operationally far simpler than running hundreds of etcd clusters.
  • NATS — experimental; multi-tenancy on NATS is noted as not yet fully supported.

Kamaji supports pools of datastores — tenants are assigned to a datastore based on resource needs or policy — and live migration of a tenant’s data between datastores of the same backend type.

How worker nodes join. Once the tenant control plane is up, the tenant provisions worker machines and runs kubeadm join (or equivalent) pointed at the tenant’s API server endpoint with the join token. From that point the worker’s kubelet watches the tenant API server, the tenant scheduler places the tenant’s Pods on those workers, and the cluster behaves exactly like a standalone cluster. Communication is strictly one-way (Kamaji — Concepts): tenant clusters have no access to or awareness of the management cluster, and tenant clusters cannot see each other.

As a Cluster API provider. Cluster API (CAPI) splits cluster lifecycle into a bootstrap provider, an infrastructure provider, and a control-plane provider. Kamaji plugs in as the control-plane provider (Kamaji CAPI): instead of CAPI’s default KubeadmControlPlane (which provisions dedicated control-plane machines), the KamajiControlPlane materializes the control plane as Pods in the management cluster. The CAPI infrastructure provider still provisions the tenant’s worker machines. The result is fully declarative, GitOps-able cluster lifecycle where control planes cost Pods, not machines.

Configuration / API Surface

A minimal TenantControlPlane:

apiVersion: kamaji.clastix.io/v1alpha1
kind: TenantControlPlane
metadata:
  name: acme
  namespace: tenants
spec:
  controlPlane:
    deployment:
      replicas: 2                       # tenant API server runs as a 2-replica Deployment (HA)
    service:
      serviceType: LoadBalancer         # how worker nodes reach this tenant's API server
  kubernetes:
    version: v1.30.2                    # this tenant's Kubernetes version — independent of others
    kubelet:
      cgroupfs: systemd
  dataStore: default-etcd               # which shared DataStore this tenant's state lives in
  networkProfile:
    port: 6443
    certSANs:                           # extra SANs on the API server cert
      - acme.k8s.example.com
  addons:
    coreDNS: {}                         # Kamaji can deploy in-tenant addons
    konnectivity: {}                    # control-plane→node tunnelling

The referenced DataStore:

apiVersion: kamaji.clastix.io/v1alpha1
kind: DataStore
metadata:
  name: default-etcd
spec:
  driver: etcd                          # or 'PostgreSQL' / 'MySQL' (kine-backed)
  endpoints:
    - etcd-0.etcd.kamaji-system.svc:2379
  tlsConfig: { ... }                    # mTLS to the datastore

Line-by-line: controlPlane.deployment.replicas is literally a Deployment replica count — HA is “more Pods,” not “more machines.” service.serviceType decides reachability of the tenant API server from its worker nodes. kubernetes.version is per tenant — two tenants on one management cluster can run different Kubernetes minor versions. dataStore names the shared backend; switching it (same driver) triggers a live migration. konnectivity sets up the tunnel that lets the hosted control plane reach back into the tenant’s node network for kubectl exec/logs/webhooks.

Failure Modes

Shared datastore as the blast radius. The flip side of one datastore for many tenants: datastore failure, corruption, or saturation affects every tenant assigned to it. A noisy tenant generating millions of objects degrades sibling tenants. Mitigation: datastore pools (isolate noisy or critical tenants on their own datastore), monitoring per-tenant object counts, and API Priority and Fairness inside each tenant API server.

Management-cluster failure cascades. If the management cluster goes down, every tenant control plane goes down with it — worker nodes keep running existing Pods (the kubelet is autonomous) but no scheduling, scaling, or kubectl works for any tenant until the management cluster recovers. The management cluster must therefore be the most robust, best-monitored cluster in the fleet; its HA is the fleet’s HA.

kine / SQL semantic mismatch. kine emulates etcd’s watch and revision semantics on top of SQL. Under very high write rates the emulation can lag etcd’s performance, and some etcd features (precise compaction behaviour) differ. For very large or write-heavy tenant clusters, real etcd is the safer backend.

Network reachability between planes. Worker nodes must reach the tenant API server Service, and the hosted control plane must reach back into worker nodes (for exec, logs, metrics) — typically via konnectivity. Misconfigured networking between the management cluster’s LoadBalancer/NodePort and the tenant’s node network is the most common bring-up failure: nodes show NotReady or kubectl logs hangs.

Certificate and token lifecycle. Each tenant cluster has its own PKI managed by Kamaji. Expired CAs, un-rotated certificates, or stale join tokens break worker enrolment. Kamaji automates rotation, but operators must monitor it as they would any control plane’s PKI.

Alternatives and When to Choose Them

vCluster. The most-confused comparison. Both run tenant control planes as Pods. The decisive difference: vCluster shares the host cluster’s worker nodes (tenant Pods are synced down onto shared nodes, shared kernel), whereas Kamaji’s tenants have their own worker nodes (a real, kernel-isolated, separate cluster). Choose vCluster for the cheapest, fastest API-level isolation when sharing nodes is fine; choose Kamaji when each tenant must have genuinely separate nodes and kernel but you still want to avoid running dedicated control-plane machines.

Dedicated control-plane machines (Self-Managed Kubernetes / kubeadm). The classic model: 3 or 5 control-plane VMs per cluster, each running etcd + API server + scheduler + controller-manager. Strong isolation, but the per-cluster machine and operational cost is exactly what Kamaji exists to eliminate. Choose dedicated machines for a small number of clusters where the management-cluster dependency is unwanted; choose Kamaji at fleet scale.

Managed Kubernetes (EKS / GKE / AKS). The cloud providers run hosted control planes for you — EKS/GKE are hosted-control-plane systems internally. Choose managed when you are on one cloud and want zero control-plane operations. Choose Kamaji when you need hosted control planes on-prem, across heterogeneous infrastructure, or under your own operational control — Kamaji is the self-hostable version of what the clouds do internally.

Red Hat HyperShift. OpenShift’s hosted-control-planes implementation — the same pattern, scoped to OpenShift. Kamaji is the vanilla-Kubernetes, vendor-neutral equivalent.

Production Notes

  • The pattern, not the project, is the point. “Hosted control planes” — control planes as workloads on a management cluster — is now an industry-standard pattern: it underlies EKS, GKE, OpenShift HyperShift, and Kamaji. Internalize the pattern and all four become legible.
  • Density economics. A management cluster can host hundreds of tenant control planes. Compared to dedicated control-plane machines, the saving is the eliminated idle capacity of 3 × N control-plane VMs across N clusters.
  • Edge and telco. Kamaji is popular for edge fleets — many small clusters at the edge, each with a handful of worker nodes, whose control planes are uneconomical to run on-site. Hosting them centrally on one management cluster is the natural fit.
  • CAPI integration is the production-grade path. Using Kamaji as a CAPI control-plane provider makes the whole fleet declarative: a tenant cluster (control plane + worker machines) is described in Git and reconciled — the same GitOps discipline applied to clusters themselves.
  • The management cluster is sacred. Every honest write-up converges on this: the management cluster is a single point of failure for the entire fleet’s control planes. Treat it as tier-0 infrastructure — multi-AZ, heavily monitored, conservatively upgraded.

See Also

Graph View

Backlinks