Kubernetes Cluster Architecture

A Kubernetes cluster is a set of machines — physical or virtual — that collectively run containerized workloads under a single, consistent declarative API. Topologically the cluster cleaves into two roles: a control plane (one or more machines running the kube-apiserver, etcd, kube-scheduler, kube-controller-manager, and cloud-controller-manager) that owns the system’s desired state and decides what should run where; and a fleet of worker nodes (each running a kubelet, a kube-proxy or equivalent, and a CRI-compliant container runtime such as containerd or CRI-O) that actually execute Pods (Kubernetes — Cluster Architecture). Historically the control-plane machines were called masters in K8s documentation and tooling; following a 2020 SIG-Architecture recommendation and KEP-2067, the terminology was changed to control-plane node for inclusivity reasons. The rollout was staged over several minor releases: Kubernetes 1.20 introduced the node-role.kubernetes.io/control-plane label alongside the legacy master label and deprecated the latter; 1.24 stopped adding the master label to new control-plane nodes (and kubeadm upgrade apply removed it from existing ones) while applying both taints; 1.25 stopped applying the master taint, leaving only control-plane; and 1.26 removed the lingering toleration for the old taint from core components such as CoreDNS (KEP-2067; kubeadm 1.24 migration PR #107533). High-availability sizing is dictated by Raft, the consensus algorithm under etcd: clusters use 3 or 5 etcd members (always odd, for stable Raft quorum), with 3 tolerating one failure and 5 tolerating two. Kubeadm supports two HA topologies — stacked etcd (etcd colocated on control-plane nodes; simpler, default) and external etcd (etcd on dedicated machines; more isolation, more hardware). The cluster’s boundary is sharp: Pods, Services, controllers, kubelets, kube-proxy run inside; image registries, external databases, cloud load balancers, identity providers (OIDC), DNS servers outside the cluster’s own CoreDNS live outside. Understanding this topology is foundational because every other note in Kubernetes MOC either runs on this topology or extends it.

Mental Model

flowchart TB
    subgraph EXT["Outside the Cluster"]
        DEV["Engineer (kubectl)"]
        CI["CI/CD & GitOps controllers"]
        REG["Container Image Registry<br/>(ECR, GCR, GHCR, Harbor)"]
        DB["External managed DB<br/>(RDS, Cloud SQL)"]
        IDP["Identity Provider<br/>(OIDC, Cloud IAM)"]
        LB["Cloud Load Balancer<br/>(ALB/NLB/GLB)"]
        USERS[Internet users]
    end
    USERS --> LB
    LB --> NODE1
    LB --> NODE2
    LB --> NODE3
    DEV --> APISERVER
    CI --> APISERVER

    subgraph CP["Control Plane (1 or 3+ machines, HA)"]
        APISERVER["kube-apiserver<br/>(REST front door)"]
        ETCD[("etcd<br/>(Raft 3/5 nodes)")]
        SCHED["kube-scheduler"]
        KCM["kube-controller-manager"]
        CCM["cloud-controller-manager"]
    end
    APISERVER <--> ETCD
    SCHED --> APISERVER
    KCM --> APISERVER
    CCM --> APISERVER
    CCM -- "cloud SDK" --> CLOUD["Cloud Provider API"]

    subgraph NODES["Worker Node Fleet (1..N machines, may be node pools)"]
        subgraph NODE1["Worker Node 1"]
            K1[kubelet]
            P1[kube-proxy]
            CR1["Container Runtime<br/>(containerd)"]
            POD1[Pods]
        end
        subgraph NODE2["Worker Node 2"]
            K2[kubelet]
            P2[kube-proxy]
            CR2[containerd]
            POD2[Pods]
        end
        subgraph NODE3["Worker Node N"]
            K3[kubelet]
            P3[kube-proxy]
            CR3[containerd]
            POD3[Pods]
        end
    end
    K1 -- "watch + heartbeat" --> APISERVER
    K2 -- "watch + heartbeat" --> APISERVER
    K3 -- "watch + heartbeat" --> APISERVER
    CR1 -- "pull image" --> REG
    CR2 -- "pull image" --> REG
    CR3 -- "pull image" --> REG
    POD1 -. "SQL" .-> DB
    POD2 -. "SQL" .-> DB
    APISERVER -. "OIDC validate" .-> IDP

What this shows. Inside the cluster boundary, the control plane is the brain (one or more machines, HA in production) and the worker nodes are the muscle (1..N, often partitioned into node pools by instance type or workload class). Every actor talks to the API server; only the API server talks to etcd; etcd is the single source of truth. Outside the cluster boundary live the things K8s uses but doesn’t own: the image registry that serves container images, the external databases and message brokers that hold business state too important to risk inside the cluster, the identity provider that authenticates engineers, and the cloud load balancers that bridge internet traffic to the node fleet. The insight to extract: the cluster’s external dependencies are deliberate and named, and a failure-mode analysis must consider both the in-cluster components and the cross-boundary ones (registry availability, IdP availability, database health).

Mechanical Walk-through

A K8s cluster’s life begins with bootstrap. The canonical tool is kubeadm (Kubernetes — kubeadm), which provisions a first control-plane machine by generating a Certificate Authority (CA), creating TLS certificates for each control-plane component, writing static-Pod manifests for the API server, scheduler, and controller-manager into /etc/kubernetes/manifests/, and starting a local kubelet that picks up those manifests and runs them as Static Pods (Pods managed directly by kubelet, not by the API server). At that point the first API server is up; kubeadm proceeds to initialize the etcd member (also as a static Pod on the same machine in the stacked topology), apply core add-ons (CoreDNS, kube-proxy as a DaemonSet), and emit a kubeadm join command. Additional control-plane nodes and worker nodes use that join command — which contains a bootstrap token plus the CA’s discovery hash — to register themselves with the cluster.

Once running, the cluster’s behavioral substrate is the API server’s authority and etcd’s persistence. Every state-changing action — a user kubectl apply, a controller writing back status, a kubelet reporting node conditions — is an HTTP call against the API server, authenticated and authorized, validated, admission-controlled, and finally written to etcd. Reads are also through the API server (controllers rarely read etcd directly; the API server is the indirection that makes RBAC, audit logging, and watch fan-out possible). Worker nodes never touch etcd; they only talk to the API server.

In a single-master test cluster (kind, minikube, k3s by default), the control plane runs on one machine. Losing that machine loses the cluster — workloads keep running on worker nodes for a while (kubelets continue to drive containers from their last known assignments), but no scheduling, no scaling, no rollouts can happen, and after the kubelet heartbeat misses the cutoff the nodes are eventually marked NotReady from the still-not-running controller-manager perspective. This is fine for development but unacceptable for production.

For production HA, the control plane is replicated. The minimum is three control-plane nodes spread across failure domains (availability zones in a cloud, racks on-prem). The API server is stateless — clients (kubelets, kubectl, controllers) hit any of the three through a load balancer, and any API server can serve any request. etcd is replicated via Raft consensus (Raft paper): each etcd member holds the full state, one is the elected leader, writes go to the leader and are propagated to followers, and a write is durable once a quorum (majority) of members has it. The quorum math is unforgiving: with 3 members, quorum is 2, so 1 failure is tolerable; with 5, quorum is 3, so 2 failures are tolerable; with 4 (an even number), quorum is still 3, so only 1 failure is tolerable and you’ve paid for a node that adds no fault tolerance. This is why etcd cluster sizes are always odd. Beyond 5–7 the write latency overhead of Raft propagation across more members outweighs the marginal availability — production etcd clusters above 7 nodes are vanishingly rare. The control plane’s scaling ceiling is ultimately set here: the upstream-supported envelope is 5,000 nodes / 150,000 Pods per cluster (see the scalability numbers in Kubernetes), and the binding constraint at that scale is etcd’s write throughput and storage quota, not the number of API-server replicas (which are stateless and scale horizontally behind the load balancer).

Within the HA control plane there are two ways to lay out etcd, formalized in the kubeadm documentation as topologies (Kubernetes — HA topology):

Stacked etcd topology colocates etcd on the same machines as the API server, scheduler, and controller-manager. Three control-plane nodes means three etcd members; the cluster needs only three machines for the entire control plane. The trade-off is failure-domain coupling: a single machine failure removes both an API server and an etcd member. Two simultaneous machine failures cost a control-plane redundancy and break etcd quorum (only one of three etcd members remains, below the quorum of 2). For small to medium production clusters, the simplicity wins — this is what kubeadm init does by default.

External etcd topology runs etcd on a dedicated set of three (or more) machines, separate from the API servers. The control plane needs at least 3 etcd + 3 API server = 6 machines. The reward is failure-domain isolation: an API server crash doesn’t take down an etcd member, and an etcd member crash doesn’t take down an API server. Operators who care intensely about cluster availability or who run very large clusters (etcd’s CPU/memory profile diverges from the API server’s at scale, making dedicated nodes economically rational) choose this topology. Managed K8s services almost universally use external etcd patterns under the hood, even though the topology is invisible to the customer.

Within the worker fleet, machines are usually grouped into node pools (the managed-K8s term; in self-managed K8s it’s just labeled node groups). A node pool is a homogeneous set of nodes — same instance type, same OS image, same labels and taints — that the Cluster Autoscaler or Karpenter can scale up and down as a unit. Common node-pool patterns: a “general” pool of medium x86 instances; a “memory” pool of large memory-optimized instances for caching workloads; a “GPU” pool for ML workloads tainted so only GPU-tolerating Pods land there; a “spot” pool of preemptible/spot instances for fault-tolerant batch jobs. Labels (workload-type=batch) and taints (workload-type=batch:NoSchedule) wire Pods to pools; see Taints and Tolerations and Node Affinity.

The cluster’s boundary is sharp and worth memorizing. Inside the cluster live Pods (and the controllers that manage them), Services, ConfigMaps, Secrets, the kubelet and kube-proxy on each node, the CNI plugin’s daemon, and the CSI plugin’s daemons. Cluster-internal DNS for Service names is served by CoreDNS, typically running as a Deployment in the kube-system namespace. Outside the cluster: the container image registry (ECR / GCR / Harbor / DockerHub) that the runtime pulls from; any external databases and message brokers (RDS / Cloud SQL / MSK / managed Kafka) — see the In-Cluster Database Anti-Pattern note for why; the cloud load balancers that bridge internet traffic into Services of type LoadBalancer (provisioned by the cloud-controller-manager but living in the cloud’s network plane, not the cluster’s); the identity provider used for human OIDC authentication and (for workload identity) the cloud IAM that issues short-lived tokens; and observability backends (Prometheus may run inside, but long-term storage like Thanos or Grafana Cloud often lives outside). The “what is inside vs outside” question is the first sanity check on any cluster-design diagram.

Configuration / API Surface

The cluster’s topology is reflected in the cluster’s own API objects. A Node object exists for every machine in the cluster; control-plane nodes carry the node-role.kubernetes.io/control-plane label (and a matching NoSchedule taint that keeps user workloads off). A trimmed kubectl get nodes -o yaml for a kubeadm-bootstrapped cluster:

apiVersion: v1
kind: Node
metadata:
  name: cp-1.example.local
  labels:
    node-role.kubernetes.io/control-plane: ""                # control-plane marker (1.20+ name)
    kubernetes.io/hostname: cp-1.example.local
    kubernetes.io/os: linux
    kubernetes.io/arch: amd64
    topology.kubernetes.io/region: us-east-1
    topology.kubernetes.io/zone: us-east-1a
spec:
  taints:
    - key: node-role.kubernetes.io/control-plane
      effect: NoSchedule                                     # keep user workloads off CP
  podCIDR: 10.244.0.0/24
status:
  capacity: {cpu: "8", memory: "32Gi", pods: "110"}
  allocatable: {cpu: "7800m", memory: "30Gi", pods: "110"}
  conditions:
    - type: Ready
      status: "True"
  nodeInfo:
    kubeletVersion: v1.36.0
    containerRuntimeVersion: containerd://2.1.0
    osImage: Ubuntu 24.04.2 LTS
    kernelVersion: 6.8.0-generic

Line-by-line: the node-role.kubernetes.io/control-plane label is the modern (1.20+) name for what was historically node-role.kubernetes.io/master. KEP-2067 prescribed introducing the new label in parallel in 1.20, deprecating the legacy master label, and eventually removing it (KEP-2067). The topology.kubernetes.io/zone label tells the scheduler which AZ a node is in, and is used by Topology Spread Constraints and zone-aware Service routing. The NoSchedule taint with the control-plane key is what keeps workload Pods off control-plane nodes — without it, kubelet would happily schedule arbitrary Pods on the API server’s machine, with predictable disaster.

For a kubeadm HA cluster, a typical sizing decision matrix is:

Use case	CP nodes	etcd location	Worker nodes	Notes
Local dev (kind, minikube)	1	Stacked	0–3	Single binary acceptable
Small prod, single-region	3	Stacked	5–50	Simplest HA
Medium prod, multi-AZ	3	Stacked	50–500	One CP per AZ
Large prod, multi-region active-passive	3 + 3	Stacked per region	500+ each	Two independent clusters
Largest prod, very write-heavy	3	External (5 etcd)	1000+	etcd isolation; bigger etcd
Edge / IoT (k3s)	1 (HA via Postgres/etcd)	k3s embedded	1–100	SQLite or embedded etcd

Managed services (EKS / GKE / AKS) hide all of this. The customer provisions a cluster, and AWS/Google/Microsoft run the API servers, etcd, scheduler, and controller-manager on their own infrastructure. The customer sees only worker nodes (managed node groups, or in EKS Fargate, no nodes at all). The control plane is billed per cluster-hour (Amazon EKS charges $0.10/hour\ast \ast perclusterwhiletheKubernetesversionisinstandardsupport,risingto\ast \ast$0.60/hour once a version enters extended support after ~14 months — as of May 2026, EKS pricing). The trade-off is operational simplicity for less control: managed services pick the etcd topology, the control-plane sizing, the upgrade cadence, and the etcd backup policy on the customer’s behalf.

Failure Modes

Loss of etcd quorum. The single most catastrophic cluster-wide failure. With 3 etcd members, losing 2 simultaneously means no writes succeed (no scheduling, no autoscaling, no status updates), though existing Pods continue running because the kubelets already have their assigned Pods cached. Recovery is non-trivial: either restore the lost members from their own state (if the disks survived) or restore the entire cluster from an etcdctl snapshot save backup, which is a control-plane reboot. Production clusters must (a) run etcd on dedicated fast disks (SSD, not network-attached unless very high quality), (b) snapshot etcd at least daily and store snapshots off-cluster, and (c) periodically drill restore procedures.

Split brain in stacked etcd across AZ outage. Stacked etcd with one member per AZ across three AZs is robust to one AZ outage. Across two AZs (a common cost-cutting mistake), losing one AZ takes 2 of 3 etcd members and breaks the cluster. This is why production guidance is three AZs for stacked etcd, not two.

Control-plane LB outage. If kubelets reach the API server through a single load balancer and that LB fails, every node simultaneously loses control-plane connectivity. Pods continue running on existing assignments (kubelet falls back to cached state) but nothing new happens. The remediation is making the API server LB itself redundant — cloud-managed LBs are typically AZ-redundant; in self-managed clusters the LB (HAProxy, kube-vip) needs its own HA story.

Image registry outage. A node-rotation event (autoscaler adds capacity, a Deployment rolls out, a node reboots) triggers a cascade of image pulls. If the registry is unreachable, no new Pods start. This is one of the most common production K8s outages. Mitigations: cache images on nodes (containerd pull caching), pre-pull via a DaemonSet, use a registry mirror (pull-through cache) close to the cluster, and consider replicating registries across regions.

Network plugin daemon failure. The CNI plugin’s per-node daemon (Calico node, Cilium agent, AWS VPC CNI) is responsible for wiring up Pod networking. If it crashes on a node, new Pods cannot get IPs and existing Pods can lose connectivity depending on the failure mode. The mitigation is treating the CNI daemon as a critical system component (DaemonSet with restart and resource guarantees, alert on per-node CNI failures).

Static-Pod kubelet wedge. kubeadm clusters run the control-plane components as static Pods managed by the kubelet on each control-plane node. If the kubelet on a control-plane node hangs, its API server stops; the load balancer should mark it unhealthy and route to the other two. But if the kubelet is what wedged (rare but real — kernel issue, disk full), recovery may require manual node reboot. This is one of the operational reasons external etcd topology can be preferred at very large scale: it removes a single coupling.

Cloud-controller-manager misconfiguration. The CCM provisions cloud LBs for Service type=LoadBalancer and attaches cloud volumes for PVCs. Misconfigured IAM (the CCM cannot create LBs) leads to Services stuck in Pending indefinitely. The error surfaces only as a Service event, easy to miss.

Alternatives and When to Choose Them

Single-binary K8s distributions (k3s, k0s, microk8s). k3s collapses the control plane into a single Go binary (~50 MB), replaces etcd with SQLite by default (with optional embedded etcd or external Postgres/MySQL for HA), and bundles common add-ons. It runs in 512 MB of RAM and is the de facto edge K8s distribution. Choose k3s when running on resource-constrained hardware (Raspberry Pi, branch office servers) or when the simplicity payoff justifies trading some upstream features.

Managed Kubernetes (EKS, GKE, AKS, OKE). The control plane is the cloud provider’s responsibility; the customer only operates worker nodes (or in serverless variants like EKS Fargate / GKE Autopilot, not even those). Choose managed K8s as the default. The per-cluster fee is dwarfed by the engineering time saved on control-plane operations, etcd backups, version upgrades, and HA debugging. Reasons to not use managed services: regulatory requirements (some industries require on-premise infrastructure), very large scale where the per-cluster fee becomes meaningful (rare; most fleets are under 50 clusters), or strong multi-cloud abstraction needs that argue for a uniform self-managed distribution.

Hyperscale-only platforms (Borg, hyperscaler internal schedulers). Google still runs Borg internally for some workloads; Facebook/Meta uses Tupperware/Twine; Twitter historically used Mesos+Aurora. These are not Kubernetes and not externally consumable, but the architectural lineage of K8s explicitly draws from Borg (Burns et al. 2016). The distinction matters in interviews: knowing that K8s is “Borg lessons applied to a community-friendly open-source design” is the framing that makes K8s’ architectural choices feel inevitable rather than arbitrary.

Non-Kubernetes orchestrators (Nomad, ECS, Titus). HashiCorp Nomad is a simpler scheduler that handles containers, VMs, and bare binaries; ecosystem is smaller but operational footprint is lighter. AWS ECS is AWS-specific and integrates tightly with the AWS console. Netflix’s Titus is internal-only. Choose these when Kubernetes’ ecosystem isn’t a tiebreaker and you prefer simplicity (Nomad) or AWS-native integration (ECS).

Production Notes

Google’s GKE — the most Kubernetes-native managed offering, owing to its Borg-and-Kubernetes lineage — documents its cluster architecture publicly (cloud.google.com/kubernetes-engine/docs/concepts/cluster-architecture). GKE runs the control plane on Google-managed VMs, with etcd replicated across three zones; customers cannot see the control-plane VMs. GKE Autopilot extends this further: customers don’t manage worker nodes either, paying per-Pod resources directly. This is the architecture-of-K8s taken to its logical conclusion — the cluster becomes a hosted API surface, the topology vanishes from the customer’s mental model.

Amazon EKS (docs.aws.amazon.com/eks) runs the control plane on multi-AZ-replicated AWS infrastructure. EKS publishes a Cluster Endpoint (a Network LB in front of the API servers) which customers’ kubectl and kubelets target. EKS-Anywhere is the on-prem variant; EKS on Outposts runs control planes on AWS hardware in customer data centers. EKS’s HA story is invisible to the operator — the customer just gets a cluster with stated SLA — but is fundamentally the same external-etcd-and-multi-API-server topology described above.

Azure AKS (learn.microsoft.com/en-us/azure/aks/core-aks-concepts) historically distinguished the free-tier (single-replica control plane, no SLA) from the paid Uptime SLA tier (multi-replica HA control plane, 99.95 % SLA). The architectural lesson: HA control plane is not free, and operators choosing “the cheapest managed K8s” may inadvertently choose a non-HA topology that fails harder than expected.

Kelsey Hightower’s “Kubernetes The Hard Way” (github.com/kelseyhightower/kubernetes-the-hard-way) is the canonical pedagogical exercise for understanding cluster architecture — building a 3-node cluster by hand, provisioning CA and certificates, configuring each component flag-by-flag, debugging the kubelet TLS bootstrap. Most production engineers do not need to do this once; once is enough to demystify “what is in this cluster.”

Shopify’s engineering blog has discussed running large multi-tenant K8s clusters serving the Black Friday / Cyber Monday traffic burst, with thousands of nodes across multiple clusters and active capacity planning to keep the scheduler latency manageable. At that scale they have observed scheduler-throughput ceilings (~100-200 Pods/sec) that motivated some workload-specific custom schedulers — a reminder that the default scheduler, while excellent, is not infinite.

The node-role.kubernetes.io/master deprecation/removal timeline is now fully resolved against KEP-2067 and the implementing kubeadm changes: introduced-in-parallel and deprecated in 1.20, the master label dropped from new/upgraded control-plane nodes in 1.24 (PR #107533), the master taint no longer applied in 1.25, and the residual toleration removed in 1.26. Any modern (>= 1.26) cluster therefore carries only node-role.kubernetes.io/control-plane; the legacy key should not be relied upon.

See Also

• Kubernetes MOC — parent map
• Kubernetes Control Plane — the brain side of the topology
• Kubernetes Node — the muscle side
• Kubernetes Object Model — the shape of the state etcd persists
• kube-apiserver, etcd, kube-scheduler, kube-controller-manager, cloud-controller-manager — individual control-plane components
• kubelet, kube-proxy, Container Runtime Interface — individual node components
• Static Pods — how the control plane bootstraps itself on kubeadm clusters
• Raft — the consensus algorithm under etcd
• etcd Backup and Restore — DR for the state of record
• Kubernetes Cluster Upgrade — operating the topology across versions
• Managed Kubernetes Services — the managed variants of this topology
• Multi-Cluster Kubernetes — when one cluster isn’t enough
• Container Orchestration Architecture — the architectural pattern this topology implements