vault

kube-controller-manager

16 min read

kube-controller-manager

The kube-controller-manager is the control-plane process that bundles ~30 built-in controllers into a single binary, running each as its own goroutine inside one OS process so they can share an informer cache, a workqueue infrastructure, and a leader-election lease (Kubernetes — Components, Controllers concept, kube-controller-manager flag reference). Every built-in higher-level Kubernetes behavior — ReplicaSets maintaining Pod counts, Deployments rolling forward, StatefulSets ordering Pod creation, Jobs running to completion, EndpointSlices wiring Services to Pods, ServiceAccounts being auto-created in new namespaces, certificate signing requests being approved, garbage collection of orphaned dependents, horizontal pod autoscaling — runs inside this single process. The bundling is a deliberate optimization: thirty independent processes would each open their own watches against the kube-apiserver, each warm their own informer caches, and each negotiate their own leader-election lease, multiplying apiserver load and memory by 30. By contrast, one binary with one set of informers and one lease — held in kube-system/kube-controller-manager via a Lease object so only one replica is active at a time across HA control planes — gets that 30× efficiency essentially for free.

Mental Model

The kube-controller-manager is a process whose primary job is to host other goroutines. Each controller goroutine is an instance of the universal Kubernetes Control Loop Pattern: watch a set of resources, diff observed against desired, take corrective action via the apiserver, requeue on failure. The controllers are completely independent of each other except that they share informer cache (a per-process in-memory mirror of apiserver objects) and an apiserver client. Their collective output is what gives Kubernetes its self-healing, declarative semantics.

flowchart TB
    subgraph "kube-controller-manager process"
      LEADER[Leader Election<br/>Lease: kube-system/kube-controller-manager]
      LEADER -->|won lease| RUN[Run all controllers]
      RUN --> SHARED[Shared Informer Cache<br/>+ Workqueue Infrastructure]
      SHARED --> RS[ReplicaSet<br/>Controller]
      SHARED --> DEP[Deployment<br/>Controller]
      SHARED --> SS[StatefulSet<br/>Controller]
      SHARED --> DS[DaemonSet<br/>Controller]
      SHARED --> JOB[Job<br/>Controller]
      SHARED --> CRON[CronJob<br/>Controller]
      SHARED --> NODE[Node<br/>Controller]
      SHARED --> ENDP[EndpointSlice<br/>Controller]
      SHARED --> NS[Namespace<br/>Controller]
      SHARED --> SA[ServiceAccount<br/>+ Token Controllers]
      SHARED --> GC[GarbageCollection<br/>Controller]
      SHARED --> RQ[ResourceQuota<br/>Controller]
      SHARED --> HPA[HorizontalPodAutoscaler<br/>Controller]
      SHARED --> CSR[CSR Signing<br/>Controller]
      SHARED --> PV[PersistentVolume<br/>Binder]
      SHARED --> TTL[TTL / TTL-After-Finished<br/>Controllers]
      SHARED --> ROOT[Root CA Pub<br/>Controller]
      SHARED --> ETC[... ~30 total]
    end
    APISERVER[kube-apiserver] <-->|watch + write| SHARED

What this diagram shows. The single process runs one leader-election loop and, upon winning the lease, starts every embedded controller. Each controller subscribes to a slice of the apiserver’s resources via the shared informer cache (one watch per resource kind, fanned out in-process to every interested controller), processes events through its own workqueue, and writes back through the shared apiserver client. The insight to extract is the shared cache as the engineering reason for bundling: without it, each controller’s individual informer would keep its own copy of every Pod, every Node, every Deployment in memory, and would open its own apiserver watch. With it, every controller goroutine reads from the same in-process Go data structures, paying memory once for the cache and apiserver-side once for the watch.

Mechanical Walk-through

On startup the kube-controller-manager binary parses its hundreds of flags (a representative subset is below) and connects to the apiserver. The first action is leader election (Leases): a Lease object named kube-controller-manager in the kube-system namespace is the contended resource. The process attempts to acquire the lease via an optimistic Update (succeeds only if the current holderIdentity is empty or expired); if successful, this replica becomes the active leader and starts running controllers. If another replica already holds it, this replica enters standby, periodically polling to detect a missed renewal (default lease duration 15 s, renew interval 10 s, retry interval 2 s).

Once leader, the process starts the shared informer factory. This is a singleton in-process object that maintains one watch per resource kind against the apiserver and delivers events to every controller goroutine that registered an interest. The factory exposes a typed indexer per resource (PodLister, NodeLister, etc.) backed by a thread-safe map keyed by namespace/name; controllers read this map directly instead of querying the apiserver, which is the single biggest reason controllers are cheap at scale.

Each controller registers an AddEventHandler callback (typically the trio AddFunc, UpdateFunc, DeleteFunc) that enqueues affected object keys into the controller’s own workqueue. The workqueue is a rate-limited, deduplicated FIFO with exponential backoff: if a controller’s sync function returns an error, the key is re-queued with a doubled delay (up to a cap). Each controller runs a configurable number of worker goroutines (typical default 5, see --concurrent-deployment-syncs) that pull keys from the queue and call the controller’s sync function.

The sync function is where the actual reconciliation logic lives. It is a pure function of the form sync(key) -> error: read the desired state for key from the indexer, read the actual state for key from the indexer, compute the diff, issue API calls via the apiserver client to converge actual to desired, return nil on success or an error to trigger requeue. Crucially, the sync is idempotent and re-entrant: calling it twice with the same input produces the same result, so a missed event or out-of-order delivery is harmless.

The complete list of built-in controllers, derived from the --concurrent-*-syncs flag set, is approximately:

  • Workload controllers: ReplicationController (legacy, pre-apps/v1), ReplicaSet, Deployment, StatefulSet, DaemonSet, Job, CronJob.
  • Service/networking controllers: Endpoints (legacy), EndpointSlice, EndpointSlice Mirroring (back-fills v1 Endpoints from EndpointSlices), Service Account Token Cleanup.
  • Namespace lifecycle: Namespace (drives deletion-cascade), ServiceAccount (auto-creates default SA per namespace), ServiceAccount Token (issues legacy SA tokens).
  • Node lifecycle (when not using cloud-controller-manager for them): Node Lifecycle / Node Controller (monitors node health and applies NoExecute/NoSchedule taints when a node goes unhealthy), Taint Eviction Controller (a now-separate controller that evicts Pods from NoExecute-tainted nodes after the grace period — decoupled from the node-lifecycle controller via the SeparateTaintEvictionController gate, which graduated to GA in Kubernetes 1.34 and can be turned off with --controllers=-taint-eviction-controller, per Decoupled Taint Manager Is Now Stable), Node IPAM / NodeCIDR allocator (assigns Pod CIDRs to nodes when --allocate-node-cidrs is set).
  • Volume controllers: PersistentVolume Binder (binds PVCs to PVs and handles reclaim policies), PersistentVolume Protection (finalizer), Attach/Detach Controller (orchestrates volume attach/detach via Container Storage Interface), Ephemeral Volume Controller.
  • Autoscaling: HorizontalPodAutoscaler (reads metrics, adjusts replica counts), Pod Disruption Budget.
  • Policy and resource management: ResourceQuota, LimitRange Defaults (mutating admission has its part; the controller enforces post-hoc).
  • Garbage collection: GarbageCollector (the generic finalizer-aware engine that cascades deletes via ownerReferences), TTL Controller (sets node-creation TTL labels), TTL-After-Finished (deletes completed Jobs/Pods after the configured TTL).
  • Certificate / identity: CSR Signer (signs Certificate Signing Requests using cluster CA), CSR Approver, Root CA Publisher (publishes cluster CA into kube-root-ca.crt ConfigMap in every namespace), Bootstrap Signer (signs bootstrap tokens for kubeadm-style join), Cluster Role Aggregation (composes aggregated cluster roles via label selectors).
  • Resource Claim controller (for Dynamic Resource Allocation).
  • Device Taint Eviction: evicts Pods that use Dynamic-Resource-Allocation devices which have become tainted (e.g., a DRA driver marks a GPU unhealthy, or an admin creates a DeviceTaintRule). This is the device-level analogue of node taint eviction; per the Kubernetes v1.36 sneak peek the underlying device-taints feature graduates to beta (on by default) in 1.36 after being alpha from 1.33.

The --cloud-provider flag historically caused the controller manager to additionally start cloud-specific controllers (Route, Service for cloud load balancers, parts of Node). These have been migrated out to the cloud-controller-manager via KEP-2392; the modern best practice is to run --cloud-provider=external and let the cloud-controller-manager handle cloud integration.

A worked example: the Deployment → ReplicaSet → Pod chain. A user kubectl applys a Deployment with replicas: 3. The apiserver writes the Deployment to etcd. The Deployment controller’s watch fires; its sync function reads the Deployment, observes no matching ReplicaSet exists at the current pod-template-hash, and creates one. The apiserver writes the ReplicaSet. The ReplicaSet controller’s watch fires; its sync function counts existing Pods matching the ReplicaSet’s selector (0), computes the diff (need 3), and creates 3 Pod objects via the apiserver. The Pods exist in etcd with no spec.nodeName; the kube-scheduler (a separate process) picks them up. Each of these steps is a separate controller’s sync function running independently inside the same kube-controller-manager process.

Configuration / API Surface

A representative startup invocation for a self-managed HA control plane:

kube-controller-manager \
  --leader-elect=true \
  --leader-elect-lease-duration=15s \
  --leader-elect-renew-deadline=10s \
  --leader-elect-retry-period=2s \
  --leader-elect-resource-lock=leases \
  --leader-elect-resource-name=kube-controller-manager \
  --leader-elect-resource-namespace=kube-system \
  --kubeconfig=/etc/kubernetes/controller-manager.conf \
  --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf \
  --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf \
  --bind-address=127.0.0.1 \
  --client-ca-file=/etc/kubernetes/pki/ca.crt \
  --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt \
  --service-account-private-key-file=/etc/kubernetes/pki/sa.key \
  --root-ca-file=/etc/kubernetes/pki/ca.crt \
  --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt \
  --cluster-signing-key-file=/etc/kubernetes/pki/ca.key \
  --cluster-signing-duration=8760h0m0s \
  --use-service-account-credentials=true \
  --controllers=*,bootstrapsigner,tokencleaner \
  --cluster-cidr=10.244.0.0/16 \
  --allocate-node-cidrs=true \
  --node-cidr-mask-size=24 \
  --cloud-provider=external \
  --concurrent-deployment-syncs=10 \
  --concurrent-replicaset-syncs=10 \
  --concurrent-service-endpoint-syncs=10 \
  --concurrent-gc-syncs=30 \
  --terminated-pod-gc-threshold=12500

Line-by-line. The five --leader-elect-* flags configure the lease-based leader election described above; the defaults (15s / 10s / 2s) are tuned so that a leader transition takes ~5 seconds in the worst case. --leader-elect-resource-lock=leases uses coordination.k8s.io/v1 Leases (deprecated alternatives are configmaps and endpoints).

--use-service-account-credentials=true is the single most important security flag: it makes each individual controller authenticate to the apiserver as its own ServiceAccount (e.g., the deployment controller as system:serviceaccount:kube-system:deployment-controller), so RBAC permissions are scoped per-controller instead of every controller running as cluster-admin. Without this flag, a compromise in any one controller is a cluster-wide compromise.

--controllers=*,bootstrapsigner,tokencleaner selects which controllers to enable: * means “all default-enabled” and the additional names enable opt-in controllers. Use a -name prefix to disable: --controllers=*,-deployment would disable only the Deployment controller (useful in testing or when an external operator replaces a built-in).

The three --cluster-signing-* flags configure the CSR Signer: the cluster CA cert/key that signs Certificate Signing Requests submitted by kubelets at bootstrap (TLS bootstrap) and by other controllers issuing certs to workloads. --cluster-signing-duration=8760h = 1 year is the default lifetime for signed certs.

--cluster-cidr plus --allocate-node-cidrs enables the Node IPAM controller which carves the cluster Pod CIDR (10.244.0.0/16 = 65536 Pod IPs) into per-node /24 subnets and assigns one to each Node as Pods come and go. This is used in clusters where the CNI doesn’t manage Pod CIDRs itself (e.g., kubenet, some Flannel modes); for CNIs like Calico or Cilium that allocate from their own IPAM, set --allocate-node-cidrs=false.

--cloud-provider=external is the modern convention indicating “I am not running in-tree cloud provider integration; the cloud-controller-manager handles it.” The deprecated alternative is naming the provider (aws, gce, azure) directly, which activates the legacy in-tree cloud code.

The --concurrent-*-syncs flags scale the per-controller worker-goroutine count. The default of 5 is fine for small clusters; large clusters with high Pod churn typically bump Deployment, ReplicaSet, Endpoint, and GC syncs to 10-20.

--terminated-pod-gc-threshold=12500 controls the Pod GC controller: when the number of terminated (Succeeded/Failed) Pods exceeds this count, the controller deletes the oldest. The default 12 500 is generous for typical workloads but a busy Job-heavy cluster should pair this with the TTL-After-Finished mechanism for time-based cleanup.

Failure Modes

Leader election thrash. Two replicas alternate holding the lease, each running a fresh informer warm-up on takeover (re-listing every controller’s resources from the apiserver). Cause is usually an overloaded apiserver causing renew requests to time out. Diagnose via leader_election_master_status and leader_election_slowpath_total metrics. The thrash compounds the apiserver overload because each takeover causes a list-storm. Mitigation: lengthen --leader-elect-renew-deadline (e.g., 30s) at the cost of slower failover.

One controller’s bug stalls others. Because all controllers share one process, a panic in one controller crashes the process — although Go’s recover and the controller-runtime’s panic-recovery typically isolate to a single sync. The bigger risk is that a controller takes a lock on a global structure or saturates the workqueue’s rate limiter. Symptom: the affected controller’s _workqueue_depth metric climbs while others remain healthy.

Garbage collector incorrect deletion. The GarbageCollector controller walks the ownerReference graph and cascade-deletes orphans. Bugs in finalizer handling (a controller forgot to remove its finalizer) make resources undeletable; bugs in ownerReference (a controller set the wrong UID) cause GC to delete the wrong children. Symptoms: kubectl delete hangs forever, or unexpected mass deletions. Diagnose with kubectl get <obj> -o jsonpath='{.metadata.finalizers}' and kubectl get events.

HPA scaling oscillation. The HorizontalPodAutoscaler controller reads metrics from metrics.k8s.io every 15 seconds; if a Pod’s metrics spike on startup, the HPA can scale further up before the new Pods stabilize, creating a feedback loop. Mitigations: HPA behavior.scaleUp.stabilizationWindowSeconds, conservative --horizontal-pod-autoscaler-cpu-initialization-period.

Stale informer cache. A controller’s local cache is updated asynchronously; immediately after writing, the controller’s own next sync may see the old state. The fix is to either re-list before deciding, or to encode the just-written resourceVersion into the cache lookup. Most built-in controllers handle this correctly but operators built on controller-runtime often hit this race.

Cloud-provider misconfiguration. Running with --cloud-provider=aws (the deprecated in-tree path) alongside the cloud-controller-manager double-runs the cloud controllers and produces conflicting writes. The modern convention is strict: kube-controller-manager runs --cloud-provider=external, cloud-controller-manager handles all cloud-specific controllers.

RBAC missing for individual controllers. With --use-service-account-credentials=true, each controller authenticates as its own SA; missing RBAC bindings (a new minor version added a new controller that needs new permissions) cause silent skipped reconciliations. Diagnose via apiserver audit log filtering on system:serviceaccount:kube-system:*-controller for Forbidden responses.

Alternatives and When to Choose Them

The kube-controller-manager binary is essentially mandatory: it embeds the built-in controllers Kubernetes depends on. The interesting variants are:

  • Splitting controllers out into separate processes. Some operators (notably OpenShift’s enhanced controllers) re-implement specific controllers and disable the built-in via --controllers=*,-name. This is useful when a controller needs a different release cadence than the rest of the kube-controller-manager.
  • Per-controller binaries (rare). A few large operators (Kueue, Karpenter) run as standalone Deployments with their own leader election; they coexist with kube-controller-manager rather than replacing it.
  • External controllers via controller-runtime. The standard pattern for operators (Operator Pattern) is to run a separate Deployment per operator using sigs.k8s.io/controller-runtime, which provides the same workqueue/informer/leader-election scaffolding. These are functionally identical to the kube-controller-manager’s controllers but run out-of-process for development isolation.
  • Managed services (EKS / GKE / AKS): the kube-controller-manager runs on the cloud-managed control plane and is invisible to the customer. Its operational properties (leader election, scaling) are AWS / Google / Microsoft’s responsibility.

Production Notes

The kube-controller-manager is rarely the bottleneck in well-tuned clusters — the kube-apiserver and etcd surface limits first — but a few production-grade tuning lessons:

  • Tune --concurrent-deployment-syncs and --concurrent-replicaset-syncs upward to ~20-30 on clusters with frequent rollouts. The default of 5 means at most 5 Deployments can sync concurrently; teams with 100+ Deployments rolling out simultaneously (e.g., a coordinated platform release) get serialized behind that limit.

  • Tune --concurrent-gc-syncs upward (default 20 in newer versions) when running many Jobs that produce orphaned Pods; insufficient GC concurrency lets terminated-Pod garbage build up.

  • Watch workqueue_depth, workqueue_adds_total, workqueue_retries_total Prometheus metrics per controller. A consistently non-empty queue for one controller indicates either a tuning issue or that controller’s API surface is saturating.

  • Pin to dedicated nodes via nodeSelector or taints on self-managed control planes. The controller manager’s memory footprint scales with cluster object count (one entry per Pod, per Node, per ConfigMap, etc.); a busy 5000-node cluster has a controller manager using 4-8 GB RAM.

  • Informer cache rebuild time on leader takeover is the controller manager’s dominant failure-mode at extreme scale — not CPU starvation. On a very large cluster, a freshly-elected leader must re-list and re-watch every resource its controllers care about before any reconciliation can resume, a window of seconds-to-tens-of-seconds during which the control loops are effectively paused. Mitigations: longer lease durations to minimize takeovers (fewer rebuilds), and keeping informers warm on standby replicas so a takeover does not start cold. The ControllerManagerReleaseLeaderElectionLockOnExit gate (above) attacks the handoff latency; warm standbys attack the rebuild latency — they are complementary.

    Uncertain uncertain

    Verify: the specific “~30 s to re-list all Pods on a 100K-Pod cluster” figure (originally attributed in this note to a Spotify 2023 retrospective). Reason: the attribution and the exact number were not re-confirmed against a primary source during this pass. Reason it is plausible: informer warm-up cost scaling with object count is well established, but the precise seconds figure is cluster- and version-specific. To resolve: find the originating talk/post or measure on a representative cluster.

  • Disable controllers you don’t need via --controllers=*,-namedone,-nametwo. This reduces apiserver watch load and saves memory. Common candidates for disabling: bootstrapsigner and tokencleaner if not using kubeadm-style bootstrap.

A note on version drift: the exact set of controllers bundled in kube-controller-manager evolves across minor versions — new controllers are added as features graduate (Resource Claim and Device Taint Eviction for Dynamic Resource Allocation; the now-separate Taint Eviction Controller), and old ones are removed (the legacy in-tree cloud controllers, migrated to cloud-controller-manager). The list above reflects the set as of Kubernetes ~1.36 (May 2026); for an authoritative, version-exact enumeration consult kube-controller-manager --help (the --controllers flag help text lists every controller name) or the flag reference.

Faster leader handoff — releasing the lease on exit

By default a standby replica cannot take over until the previous leader’s Lease expires by TTL (lease duration 15 s) — so an orderly leader shutdown still costs up to a full lease duration of dead time. The ControllerManagerReleaseLeaderElectionLockOnExit feature gate, alpha and disabled by default in Kubernetes 1.36, has the kube-controller-manager actively release its leader-election lock during a leader transition (e.g., a clean shutdown or rolling restart) rather than waiting for the TTL to lapse, letting a new leader be elected almost immediately and shrinking transition latency (Leases). Because it is alpha-and-off, it must be explicitly enabled, and its interaction with standby informer-warmth (see Production Notes below) is the property to validate before relying on it operationally.

See Also

Graph View

Backlinks