vault

Kubernetes Control Plane

18 min read

Kubernetes Control Plane

The Kubernetes control plane is the collection of processes that own the cluster’s desired state and orchestrate convergence of the observed state toward it. It consists of five canonical components: kube-apiserver (the REST front door — the only process that talks to etcd, the only process every other component talks to), etcd (the distributed Raft-backed key-value store that persists every API object), kube-scheduler (assigns Pods to Nodes via a filter-and-score pipeline), kube-controller-manager (a single binary that bundles the built-in controllers — Deployment, ReplicaSet, Node, Service, EndpointSlice, ServiceAccount, Job, CronJob, and dozens of others), and cloud-controller-manager (cloud-provider-specific controllers — Service LoadBalancer provisioning, cloud Node lifecycle, cloud Route table programming — split out so the core is cloud-agnostic) (Kubernetes — Components). In production these processes run on dedicated control-plane nodes (3 or 5 of them for HA, see Kubernetes Cluster Architecture); in managed Kubernetes (EKS, GKE, AKS) they run on cloud-provider infrastructure entirely invisible to the customer. Two architectural invariants give the control plane its essential character. First, every component talks only to the API server — controllers, schedulers, kubelets, and cloud-controller-manager all read state and write state through the API server’s REST interface, never directly to etcd or to each other. This is the single most consequential architectural choice in Kubernetes: it concentrates authentication, authorization, validation, admission, audit, and watch fan-out in one place and means every other component can be stateless. Second, controllers and the scheduler use Raft-backed lease-based leader election in HA topologies — multiple replicas of the scheduler and controller-manager run for redundancy, but only one is the active leader at any moment, with the leadership recorded as a Lease object in etcd (via the kube-apiserver) and renewed periodically (Kubernetes — Leases). Loss of the leader (process crash, network partition, host failure) triggers a re-election within seconds.

Mental Model

flowchart TB
    subgraph CP["Control Plane (one logical unit, 3+ replicas in HA)"]
        APIS["kube-apiserver<br/>(stateless, horizontally scalable)<br/>port 6443"]
        ETCD[("etcd<br/>(Raft, 3 or 5 members)")]
        SCHED["kube-scheduler<br/>(leader-elected)"]
        KCM["kube-controller-manager<br/>(leader-elected; bundles N controllers)"]
        CCM["cloud-controller-manager<br/>(leader-elected; cloud-specific controllers)"]
    end
    APIS <--> ETCD
    SCHED -- "watch unscheduled Pods,<br/>write Pod/binding" --> APIS
    KCM -- "watch all builtins,<br/>write status &amp; children" --> APIS
    CCM -- "watch Services/Nodes,<br/>write status" --> APIS
    CCM -- "create LBs, attach volumes,<br/>tag Nodes" --> CLOUD["Cloud Provider SDK"]

    subgraph CLIENTS["Clients (outside control plane)"]
        KCT[kubectl]
        KL[kubelets on Nodes]
        CTRL[3rd-party controllers]
        WHK[admission webhooks]
    end
    KCT --> APIS
    KL --> APIS
    CTRL --> APIS
    APIS <-- "admission callbacks" --> WHK

    LE[("Lease objects<br/>kube-system namespace")]
    SCHED -. renews lease .-> LE
    KCM -. renews lease .-> LE
    CCM -. renews lease .-> LE
    LE <-- stored in --> ETCD

What this shows. The five components form a tight tripod: API server in the middle, etcd behind it (talked to only by the API server), and the three controller-class processes (scheduler, controller-manager, cloud-controller-manager) talking to it through the same REST interface that external clients use. The leader-election Leases — one per leader-elected component — sit in kube-system namespace and are persisted in etcd through the API server like any other K8s object. The insight to extract: the control plane is not really “one program with five modules” but five independent processes that converse exclusively through a single, well-defined REST API. This is the architectural property that makes managed Kubernetes possible — the cloud provider can run these five processes on whatever infrastructure they like, in whatever topology, with whatever monitoring, and customers cannot tell the difference because the only contract is “the API server is reachable at this URL with this certificate.”

Mechanical Walk-through

The control plane is what gives Kubernetes its declarative, eventually-consistent behavior. Conceptually it implements the formula: desired state in etcd → controllers observe → controllers act → observed state converges → controllers observe convergence → loop quiesces. Each of the five components plays a specific role.

kube-apiserver is the API server. It is a horizontally scalable, stateless HTTP/2 server (port 6443 by default) that exposes the Kubernetes REST API. Every request — kubectl apply, a controller’s watch, a kubelet’s status update — goes through an admission pipeline (authenticate → authorize via RBAC → run mutating admission webhooks → validate against OpenAPI schema → run validating admission webhooks → persist to etcd → emit watch event) and the API server is where all of that happens. The API server is also the only process in the cluster that talks to etcd directly; controllers and kubelets never touch etcd. The API server can be replicated arbitrarily — putting three or five behind a load balancer is standard for HA — and adding API servers is the standard scaling lever for very large clusters. Stateless does not mean memoryless: each API server caches recently-watched objects in an in-process watch cache that serves most reads without hitting etcd, and the cache invalidation is via etcd’s own watch stream.

etcd is the distributed key-value store under the API server (etcd.io). It uses the Raft consensus algorithm (Ongaro & Ousterhout 2014) for replicated, linearizable writes. Every API server write is durable once a Raft quorum of etcd members commits it. etcd holds the entirety of cluster state: every Pod, every ConfigMap, every Secret, every Node status, every Lease, every CRD instance — all of it. The key layout is hierarchical (/registry/<resource>/<namespace>/<name> or /registry/<resource>/<name> for cluster-scoped). etcd’s HA story dictates the control plane’s HA story: 3 etcd members tolerate 1 failure, 5 tolerate 2; the size is always odd. See Kubernetes Cluster Architecture for the stacked-vs-external etcd topology choice. Losing etcd quorum is losing the cluster’s brain — recovery is from etcdctl snapshot save backups.

kube-scheduler (kube-scheduler reference) watches the API server for Pods whose spec.nodeName is empty (unscheduled Pods) and, for each, runs a filter-and-score pipeline to pick a node. The Filter phase eliminates nodes that don’t satisfy hard constraints (insufficient CPU/memory, taints the Pod doesn’t tolerate, node selectors that don’t match, volume zone restrictions, Pod Affinity and Anti-Affinity requirements). The Score phase ranks surviving nodes by soft criteria (resource balance, Topology Spread Constraints, Node Affinity preferences, image-locality bonus). The highest-scoring node wins; ties break randomly. The scheduler then writes a Binding subresource that updates the Pod’s spec.nodeName. The scheduler is single-leader via leader election — even when three replicas are deployed (typical HA), only the leader does work. This is a deliberate consistency choice: parallel schedulers could write conflicting bindings for the same Pod. See Scheduling Framework for the full plugin extensibility model, and Kubernetes Scheduler Architecture for the internal three-queue/two-cycle machinery.

kube-controller-manager (kube-controller-manager reference) is a single binary that bundles many built-in controllers into one process. Bundled controllers include: the Node controller (watches Node objects, marks them NotReady when kubelet heartbeats lapse, evicts Pods after a grace period); the Deployment controller (turns Deployment spec changes into ReplicaSet rollouts); the ReplicaSet controller (ensures N Pods exist for each ReplicaSet); the EndpointSlice controller (turns Service+selector into the list of backend Pod IPs); the ServiceAccount + Token controllers (mint default ServiceAccounts and projected tokens); the StatefulSet, DaemonSet, Job, and CronJob controllers; the PersistentVolume binder and the PersistentVolumeClaim protection controller; the Namespace controller (handles namespace deletion cascade); the garbage collector (deletes objects whose owners are gone); the HorizontalPodAutoscaler controller; and several dozen others. Each runs as an independent goroutine inside the binary, each with its own informer cache and workqueue. Like the scheduler, kube-controller-manager is single-leader via leader election — the bundling-into-one-process is an operational simplification that doesn’t change the leader-election semantics.

cloud-controller-manager (Kubernetes — Cloud Controller Manager) is the cloud-specific subset of controllers, split out from kube-controller-manager so the core Kubernetes codebase can be cloud-agnostic. The CCM runs three canonical controllers: the Node controller (watches the cloud provider’s API to detect when an underlying instance has been deleted, and removes the corresponding K8s Node), the Service controller (watches Service objects of type LoadBalancer and provisions/reconfigures the corresponding cloud LB — AWS NLB/ALB, GCP Network/Global LB, Azure Load Balancer), and the Route controller (programs cloud route tables for Pod-to-Pod routing on networks that need it). Different clouds ship different CCMs (cloud-provider-aws, cloud-provider-gcp, cloud-provider-azure, plus community providers for OpenStack, vSphere, IBM Cloud, etc.). On managed K8s the CCM is invisible to customers; on self-managed clusters it must be installed and configured with cloud credentials. CCM is leader-elected; only one is active at a time.

The architectural cleanliness of “every component talks only to the API server” is worth dwelling on. Consider the alternative: if the scheduler talked directly to kubelets, you would need authn/authz between every (scheduler, kubelet) pair, schedulers would need to know kubelet endpoints (which change as nodes come and go), and replacing the scheduler would mean reconfiguring every kubelet. The API-server-as-hub design lets the scheduler watch unscheduled Pods (one watch against one URL), write a binding (one PATCH against one URL), and forget about kubelets entirely — the kubelet that owns the chosen node has its own watch and picks up the assignment independently. The same pattern recurs everywhere: third-party operators (Prometheus Operator, cert-manager, Crossplane) plug in by talking to the API server, never to each other; new admission webhooks insert themselves into the request pipeline without modifying any other component; observability tools watch resource events through the API server’s stream. This is why the cluster’s API surface is also its extensibility surface.

Leader election for the scheduler, controller-manager, and cloud-controller-manager is implemented via Lease objects in the coordination.k8s.io/v1 API group (Kubernetes — Leases). Each component, on startup, attempts to acquire a Lease in kube-system (the scheduler’s is named kube-scheduler, the controller-manager’s is kube-controller-manager, the CCM’s is cloud-controller-manager). Whichever replica succeeds in writing its identity to the Lease’s holderIdentity field becomes leader and starts doing real work. The leader periodically renews the lease (updating spec.renewTime); the renewal cadence is roughly leaseDurationSeconds ÷ 2 so the lease never lapses while the leader is healthy (Kubernetes — Leases). Non-leaders periodically check whether the lease has expired (its renewTime is older than leaseDurationSeconds ago) and, if so, try to acquire it themselves. The classic library defaults are a 15-second lease duration, a 10-second renew deadline, and a 2-second retry period — so a leader’s death is detected within ~15 s and a new leader takes over within another few seconds. The kube-controller-manager / kube-scheduler flags --leader-elect-lease-duration, --leader-elect-renew-deadline, and --leader-elect-retry-period (or the equivalent LeaderElectionConfiguration fields) tune these. K8s 1.36 added an alpha feature (ControllerManagerReleaseLeaderElectionLockOnExit, disabled by default) that has the kube-controller-manager actively release its lease during a graceful shutdown rather than waiting for the lease TTL to expire, cutting transition latency (Kubernetes — Leases). The same Lease primitive is used by third-party operators that need HA: controller-runtime ships a leader-election helper that defaults to a Lease in the operator’s namespace.

A newer, distinct mechanism sits alongside the classic lease-grab: Coordinated Leader Election (feature gate CoordinatedLeaderElection), beta in Kubernetes 1.36 (introduced as alpha in 1.33) and disabled by default, with kube-controller-manager and kube-scheduler as the two components wired to use it (Kubernetes — Coordinated Leader Election). The classic scheme is first-come-first-served: whichever replica happens to grab the lease wins, regardless of its version. That is a problem during a control-plane upgrade — you generally want the older component version to lead while a mixed-version control plane is in flight, so that newer replicas (which may write data the old ones cannot read) do not take over mid-rollout. Coordinated leader election fixes this by having candidates publish a LeaseCandidate object (in coordination.k8s.io/v1beta1) carrying their binary version and emulation version; a controller in the API server then picks the leader by a strategy — the only built-in strategy, OldestEmulationVersion, prefers the candidate with the lowest emulation version, then lowest binary version, then oldest creation timestamp. Enabling it requires both --feature-gates=CoordinatedLeaderElection=true and --runtime-config=coordination.k8s.io/v1beta1=true on the API server. The takeaway: classic leader election answers “is the leader alive?”, whereas coordinated leader election additionally answers “which candidate should lead given a version skew?”

Configuration / API Surface

A kubeadm-bootstrapped control-plane node runs the four control-plane processes as static Pods (Pods managed directly by the local kubelet, not via the API server) defined in /etc/kubernetes/manifests/. A trimmed kube-apiserver static Pod manifest:

apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  hostNetwork: true                                           # uses node's network namespace
  priority: 2000001000                                        # system-cluster-critical
  containers:
    - name: kube-apiserver
      image: registry.k8s.io/kube-apiserver:v1.36.1            # current stable line as of 2026-05
      command:
        - kube-apiserver
        - --advertise-address=10.0.0.10                       # cluster-routable IP of this CP node
        - --etcd-servers=https://127.0.0.1:2379               # local etcd in stacked topology
        - --service-cluster-ip-range=10.96.0.0/12             # ClusterIP CIDR
        - --authorization-mode=Node,RBAC
        - --enable-admission-plugins=NodeRestriction,PodSecurity,...
        - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
        - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
        - --client-ca-file=/etc/kubernetes/pki/ca.crt         # who signs valid client certs
        - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
        - --enable-bootstrap-token-auth=true
        - --service-account-key-file=/etc/kubernetes/pki/sa.pub
        - --service-account-issuer=https://kubernetes.default.svc.cluster.local
        - --encryption-provider-config=/etc/kubernetes/pki/encryption-config.yaml
      livenessProbe:
        httpGet: {path: /livez, port: 6443, scheme: HTTPS}
      ports:
        - containerPort: 6443

Line-by-line: --etcd-servers points the API server at its etcd backend; in stacked etcd topology this is localhost, in external etcd it is the etcd cluster’s LB endpoint. --authorization-mode=Node,RBAC enables the Node authorizer (which restricts kubelets to objects related to their own node) followed by RBAC. --enable-admission-plugins lists the built-in admission controllers that run on every request — NodeRestriction (prevents a kubelet from editing other nodes’ objects), PodSecurity (the modern Pod Security Admission controller replacing the deprecated PodSecurityPolicy), and many others. --encryption-provider-config enables at-rest encryption for Secrets and other sensitive resources before they hit etcd; without it, Secrets are merely base64-encoded.

The kube-controller-manager and kube-scheduler manifests look similar but with their own command-line flags. The kube-scheduler additionally accepts a KubeSchedulerConfiguration resource describing its plugin pipeline (which Filter, Score, Bind plugins are enabled and in what order); see Scheduling Framework for details.

Leader-election Lease objects are visible via the API:

$ kubectl -n kube-system get lease
NAME                      HOLDER                                    AGE
kube-controller-manager   cp-1.example.local_a1b2c3d4...            72d
kube-scheduler            cp-2.example.local_e5f6a7b8...            72d
cloud-controller-manager  cp-1.example.local_c9d0e1f2...            72d
 
$ kubectl -n kube-system get lease kube-scheduler -o yaml
spec:
  holderIdentity: cp-2.example.local_e5f6a7b8-...
  leaseDurationSeconds: 15
  renewTime: "2026-05-15T12:34:56.789012Z"
  acquireTime: "2026-03-04T09:11:23.456789Z"

The holderIdentity shows which replica is currently active; renewTime is updated on each renewal. When this manifest is published every ~5 s (the default refresh interval), other replicas know the leader is alive.

Failure Modes

API server overload. Heavy watch fan-out (e.g., a large cluster with thousands of controllers each watching all Pods) can drive the API server’s CPU and memory hard; symptoms are 429 Too Many Requests responses, watch-stream disconnects, and rising tail latencies on kubectl get. Mitigations include client-side rate limiting (controller-runtime defaults are tuned for this), Priority and Fairness (flowcontrol.apiserver.k8s.io API; lets operators give critical controllers preferred queues), and horizontal scaling (add more API server replicas).

etcd write latency degradation. Slow disks under etcd (network-attached storage, contention from other workloads) cause Raft commit latencies to spike. Symptoms: kubectl apply takes seconds rather than tens of ms, controllers fall behind their watch streams, status updates lag. etcd must run on dedicated, fast local SSDs; cloud-managed etcd on burst-balance EBS has bitten many production teams.

Stuck leader-election Lease. If the active controller-manager hangs (doesn’t crash — just stops doing work but keeps renewing its Lease), no other replica can take over. The Lease renewal continues, masking the failure. The fix is monitoring at the behavior level (e.g., are Deployments converging?) rather than at the liveness level (is the process alive?). Some operators add a synthetic “canary Deployment” that gets continuously reconciled to detect this class of failure.

Admission webhook flakiness. If an admission webhook’s endpoint is unreachable or slow, every API server write that triggers it is delayed. A misconfigured webhook with failurePolicy: Fail can hard-block all API writes; with failurePolicy: Ignore it produces silent gaps in policy enforcement. The defense: tight timeouts, narrow objectSelector/namespaceSelector scoping, and well-monitored webhook endpoints.

etcd defragmentation pause. etcd periodically requires defragmentation to reclaim freed key space. The defragmentation call blocks the etcd member; in a 3-member cluster, defragmenting members one at a time briefly removes that member from quorum participation. Misconfigured automation that defragments two members simultaneously can break quorum. Production runbooks must serialize defragmentation.

Single-zone control plane. A control plane whose three nodes all live in one AZ loses everything during an AZ outage. The fix is spreading control-plane nodes across at least three AZs. On managed K8s the cloud handles this; on self-managed it’s an operator responsibility easily overlooked.

Token signing key rotation incomplete. The API server uses signing keys for ServiceAccount tokens (--service-account-key-file). Rotating these keys without configuring the API server to accept both old and new keys during the rotation window invalidates every in-flight token — Pods can’t authenticate to the API server, status updates stall, and the cluster fails in a hard-to-diagnose pattern.

Alternatives and When to Choose Them

Managed control plane (EKS / GKE / AKS). The cloud runs the control plane on its own infrastructure. The customer sees only an API server endpoint and a kubeconfig. EKS charges ~$0.10/cluster-hour, GKE charges similarly (one zonal cluster free under some tiers), AKS varies by SLA tier. The savings versus self-managed are enormous: no etcd backup, no version upgrades of control-plane components, no leader-election debugging, no admission-webhook ops. Choose managed unless cost-at-very-large-scale, regulatory isolation, or on-prem deployment forces self-managed.

Self-managed via kubeadm. kubeadm bootstraps a control plane on machines the operator provisions (VMs or bare metal). The operator runs etcd, manages CA and certificate rotation, upgrades control-plane components quarterly. Tooling like Cluster API (Cluster API) automates kubeadm-based cluster lifecycle as Kubernetes resources themselves. Choose self-managed when managed services aren’t viable or when the operator needs deep control (custom admission plugins, custom scheduler plugin builds, very large clusters).

Lightweight self-managed (k3s, k0s). k3s collapses the control-plane processes into a single binary, uses SQLite or embedded etcd, and runs in 512 MB of RAM. Choose for edge, IoT, branch-office, or developer environments. Not recommended for very large clusters; SQLite throughput is the bottleneck.

Kamaji / Hosted Control Planes pattern. Kamaji (and similar projects) run K8s control planes as Pods inside a management cluster — the customer’s cluster is just worker nodes, and the control plane is a tenant Deployment on the platform team’s cluster. Choose for very large fleets where operating per-cluster control planes on dedicated machines is uneconomical. Adds a layer of indirection but pays off at scale.

Production Notes

GKE’s hosted control plane runs in Google’s infrastructure and is replicated across three zones automatically (Google GKE — Cluster Architecture). The control plane is reachable via a public endpoint (or private endpoint for private clusters), with the customer’s connection authenticated either by IAM (gcloud) or by short-lived tokens. GKE’s “regional clusters” replicate the control plane across the region’s zones automatically; “zonal clusters” do not and have lower SLA. The single most common GKE production lesson is “use regional clusters, not zonal” — the cost difference is small and the resilience improvement is large.

EKS (Amazon EKS) runs the control plane across multiple AZs by default, with the API server reachable via a customer-facing endpoint. EKS’s Node IAM Role and aws-auth ConfigMap historically wired AWS IAM identities to Kubernetes users; the modern replacement is EKS Access Entries (a first-class K8s/IAM mapping) and EKS Pod Identity (for workload identity to AWS services). EKS publishes a 99.95 % SLA for the control plane.

AKS distinguishes the free tier (no SLA, single-replica control plane) from the Uptime SLA tier (multi-AZ replica HA, 99.95 %). Production AKS clusters should always be on the paid Uptime SLA tier; the “free” cluster is for development only.

Spotify’s engineering blog (search for K8s posts on engineering.atspotify.com) discussed lessons from running self-managed K8s before migrating to GKE — many lessons translated as control-plane operational toil they no longer had to do. Shopify, similarly, transitioned away from self-managed control planes to managed ones for the BFCM-scale clusters, citing the operational drag of upgrades and etcd ops.

A subtle production note: the API server’s --max-requests-inflight / --max-mutating-requests-inflight flags (default 400 / 200) cap concurrent API server work. Very large clusters or noisy controller fleets need these tuned up plus Priority and Fairness rules to keep critical workloads (kubelets, leader-election renewals) from being starved by chatty third-party operators.

Kelsey Hightower’s Kubernetes The Hard Way walks through building a control plane manually, flag by flag, certificate by certificate. The tutorial’s pedagogical value is that after completing it the operator understands why every flag in the static-Pod manifest matters — and why managed K8s exists.

Uncertain

Verify: the precise numeric defaults for leader election — 15 s lease duration, 10 s renew deadline, 2 s retry period. Reason: the official Leases and Coordinated-Leader-Election docs describe these fields but do not publish the numeric defaults; they live in the leader-election library’s RecommendedDefaultLeaderElectionConfiguration in source, and the kube-controller-manager / kube-scheduler CLI reference pages were truncated on fetch. These values have been the de-facto library defaults for many releases, but a given distribution (or kubeadm) may override them. To resolve: run kube-scheduler --help / kube-controller-manager --help on the installed version, or read defaultLeaderElect* in k8s.io/apiserver/pkg/apis/config/component-base. The ControllerManagerReleaseLeaderElectionLockOnExit (alpha, 1.36) and CoordinatedLeaderElection (beta, 1.36) status claims above are confirmed against primary docs and are not in doubt.

See Also

Graph View

Backlinks