cloud-controller-manager

The cloud-controller-manager (CCM) is the control-plane component that hosts the cloud-provider-specific controllers — Node controller (cloud-side node initialization and deletion detection), Route controller (configure Pod CIDR routes in the cloud’s VPC), and Service controller (provision and manage cloud load balancers for type: LoadBalancer Services) — split out from the kube-controller-manager so that the Kubernetes core stays cloud-agnostic (Kubernetes — Cloud Controller Manager, Components). This extraction spans two complementary enhancements: KEP-2392 — Cloud Controller Manager defines the new out-of-tree component and its architecture, while KEP-2395 — Removing In-Tree Cloud Providers governs the deletion of the legacy compiled-in code. (The two are routinely conflated; the removal work is KEP-2395, not KEP-2392.) Together they form the multi-year effort that moved AWS, Azure, GCP, OpenStack, vSphere, and other cloud integrations out of the main kubernetes/kubernetes repository into provider-specific repos (kubernetes/cloud-provider-aws, kubernetes/cloud-provider-azure, kubernetes/cloud-provider-gcp, etc.) so cloud providers can iterate on their integrations independently of the Kubernetes release cycle and so the Kubernetes core no longer carries millions of lines of vendor-specific code and SDK dependencies. Kubernetes SIG Cloud Provider describes the completed migration as the largest migration in Kubernetes history — roughly 1.5 million lines of code removed and core-component binary sizes cut by about 40% (Completing the largest migration in Kubernetes history, 2024). The externalization is now finished: the legacy --cloud-provider=<name> integrations were disabled by default in v1.29 (the DisableCloudProviders and DisableKubeletCloudCredentialProviders feature gates went beta-on) and the last in-tree integration code was removed in v1.31 (those gates locked to GA), so from v1.31 onward the only valid --cloud-provider values are the empty string (no integration) or external (Kubernetes 1.29: Cloud Provider Integrations Are Now Separate Components; Kubernetes Removals and Major Changes In v1.31). As of Kubernetes v1.36 (April 2026) running on a cloud therefore requires an external CCM — there is no in-tree fallback left. On managed Kubernetes services (Amazon EKS, Google GKE, Azure AKS) the CCM is part of the hidden control-plane the provider operates; on self-managed cloud clusters (kubeadm on EC2, Cluster API, OpenShift on Azure) the operator deploys the provider-specific CCM as a Deployment alongside the standard kube-controller-manager.

Mental Model

The CCM is a lookalike of the kube-controller-manager: same binary architecture (a single process hosting multiple controller goroutines), same leader-election lease pattern, same workqueue/informer machinery. The difference is what the controllers do: they bridge between Kubernetes API objects and the underlying cloud provider’s API. The CCM is the only control-plane component that holds cloud-IAM credentials, and it is the only component that issues cloud-API calls (CreateLoadBalancer, DescribeInstances, CreateRoute). Everything else in Kubernetes remains cloud-agnostic because the CCM converts cloud-side state into Kubernetes objects (e.g., adding topology.kubernetes.io/zone labels to Nodes after polling the cloud’s instance metadata) and Kubernetes objects into cloud-side actions (e.g., observing a new type: LoadBalancer Service and provisioning an ELB).

flowchart LR
    APISERVER[kube-apiserver] <-->|watch + write| CCM[cloud-controller-manager<br/>provider-specific binary]
    subgraph "CCM process"
      LEADER[Leader Election Lease]
      NODE_CTRL[Node Controller<br/>cloud-side init + deletion detect]
      ROUTE_CTRL[Route Controller<br/>set Pod-CIDR routes in cloud VPC]
      SVC_CTRL[Service Controller<br/>provision cloud LB for<br/>type:LoadBalancer Services]
    end
    CCM --> CLOUD_API[Cloud Provider API<br/>EC2 / Compute Engine / Azure ARM]
    CLOUD_API --> LB[(Cloud Load Balancer<br/>ALB / NLB / GCE LB / Azure LB)]
    CLOUD_API --> ROUTES[(VPC Route Tables)]
    CLOUD_API --> INSTANCES[(EC2 Instances / VMs)]
    KCM[kube-controller-manager<br/>cloud-agnostic] <-->|watch + write| APISERVER
    KCM -.->|no longer runs<br/>cloud controllers| CLOUD_API

What this diagram shows. The CCM sits between the apiserver and the cloud-provider API. It watches Kubernetes objects (Nodes, Services) via the apiserver and translates relevant changes into cloud-API calls; conversely it watches the cloud (via polling and webhooks where available) and reflects state back into Kubernetes via Node labels, Node conditions, and Service status. The insight to extract is that the CCM is the trust boundary for cloud credentials — it is the only component holding cloud-IAM access in a standard install, and its failure mode is “the cluster keeps running but cannot integrate new cloud resources” (existing load balancers keep working; new type: LoadBalancer Services hang in the Pending state). The CCM is also why a managed-services kube-controller-manager never needs --cloud-provider: the cloud-specific work happens elsewhere.

Mechanical Walk-through

A provider-specific CCM binary embeds the CloudProvider interface from k8s.io/cloud-provider:

type CloudProvider interface {
    Initialize(clientBuilder ControllerClientBuilder, stop <-chan struct{})
    LoadBalancer() (LoadBalancer, bool)
    Instances() (Instances, bool)
    InstancesV2() (InstancesV2, bool)
    Zones() (Zones, bool)
    Clusters() (Clusters, bool)
    Routes() (Routes, bool)
    ProviderName() string
    HasClusterID() bool
}

Each provider (cloud-provider-aws, cloud-provider-gcp, cloud-provider-azure, etc.) implements this interface by wrapping its SDK calls. The CCM’s generic controller code calls into these methods, never the cloud SDK directly. This indirection is what makes the CCM portable across clouds at the architectural level — though the operational details (IAM permission models, LB configuration syntax, instance-metadata schemas) differ widely.

On startup the CCM acquires the leader-election lease (kube-system/cloud-controller-manager by default), then starts each enabled controller as a goroutine.

Node Controller. Watches Node objects via the apiserver. For every Node, it calls the cloud provider’s Instances.InstanceID(node.Name) (or the V2 method InstancesV2.InstanceMetadata(node)) to look up the cloud-side instance and populate:

• spec.providerID (e.g., aws:///us-east-1a/i-0abc123) — the canonical cross-reference between Kubernetes Node and cloud instance.
• topology.kubernetes.io/region and topology.kubernetes.io/zone labels, used by zone-aware scheduling and volume binding.
• node.kubernetes.io/instance-type label (e.g., m6i.2xlarge).
• status.addresses (internal IP, external IP, hostname) — used by clients reaching the kubelet.

It also periodically polls the cloud to detect deleted instances: if a cloud instance is terminated (autoscaler scaled down, spot interruption, hardware failure), the Node Controller removes the Node from Kubernetes, which triggers the kube-controller-manager’s Node Lifecycle controller to evict and reschedule the Pods that were on it. This is a critical operational hand-off: without the CCM, deleted cloud instances would persist as ghost Nodes in NotReady state forever.

Route Controller. Watches Nodes and ensures that, for each Node’s allocated Pod CIDR, a route exists in the cloud’s VPC routing table pointing the CIDR at that Node’s instance. This is required for CNIs that depend on cloud routing (e.g., kubenet on AWS, the GCE CNI) rather than on overlay networking. Modern CNIs that allocate from cloud-native VPC IP space (AWS VPC CNI, Azure CNI, GKE Dataplane V2) do not need this controller because Pod traffic flows through native VPC routing without per-node routes.

Service Controller. Watches Services. When a Service of type: LoadBalancer is created, the Service Controller calls LoadBalancer.EnsureLoadBalancer(clusterName, service, nodes) which:

1. Creates (or updates) a cloud load balancer — an AWS NLB or Classic ELB, a GCP Load Balancer, an Azure Load Balancer or Application Gateway depending on annotations.
2. Adds backend members for each Node (or for Nodes matching externalTrafficPolicy: Local semantics).
3. Configures health checks pointing at the node-port the kube-apiserver assigned to the Service.
4. Writes the load balancer’s public IP/hostname back into the Service’s status.loadBalancer.ingress[], which is how clients learn the externally-reachable address.

On Service deletion, the Service Controller calls LoadBalancer.EnsureLoadBalancerDeleted(...) to tear down the cloud LB. Service-Controller bugs are the leading cause of orphaned cloud load balancers — costly resources that survive cluster deletion if the CCM was misconfigured or de-credentialed before the Service was deleted.

Historical Volume Controller (now fully removed). Originally the in-tree cloud code also provided cloud-side volume attach/detach plugins (awsElasticBlockStore, gcePersistentDisk, azureDisk) wired into the kube-controller-manager’s attach-detach controller. These were superseded by Container Storage Interface (CSI) drivers running as separate Deployments per cloud (ebs-csi-driver, pd-csi-driver, azuredisk-csi-driver). CSI Migration — the shim that transparently redirects in-tree volume API calls to the corresponding CSI driver — graduated to GA in v1.25, having been on by default for these three plugins since v1.23 (Kubernetes 1.25 CSI migration status update). The in-tree plugins were then deleted outright: awsElasticBlockStore and azureDisk were removed in v1.27, and gcePersistentDisk was removed in v1.28 (Kubernetes — Volumes). So by current Kubernetes (v1.36, April 2026) there is no in-tree volume handling left at all — every cloud volume goes through its CSI driver, and a PersistentVolume still declaring an in-tree source like awsElasticBlockStore is rejected. This is a separate but parallel removal track to the cloud-provider one above.

The CCM does not start kubelet-side cloud integration. The kubelet also has a --cloud-provider flag that historically caused it to call the cloud SDK directly for instance metadata. Since the in-tree removal (gates locked GA in v1.31), the only values the kubelet accepts are external and the empty string — passing a provider name like aws now aborts startup. The modern flow is --cloud-provider=external, which makes the kubelet taint itself node.cloudprovider.kubernetes.io/uninitialized and wait for the CCM’s Node Controller to populate its providerID and remove the taint before considering itself fully initialized. (Kubelet image-pull credentials, formerly served by the same in-tree code, are now provided out-of-tree by kubelet credential provider plugins under KEP-2133 — see the DisableKubeletCloudCredentialProviders gate, also locked in v1.31.)

Configuration / API Surface

Deploying the AWS CCM on a self-managed kubeadm cluster on EC2 as a Deployment in kube-system:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: aws-cloud-controller-manager
  namespace: kube-system
spec:
  replicas: 2
  selector:
    matchLabels:
      app: aws-cloud-controller-manager
  template:
    metadata:
      labels:
        app: aws-cloud-controller-manager
    spec:
      serviceAccountName: cloud-controller-manager
      hostNetwork: true
      priorityClassName: system-cluster-critical
      nodeSelector:
        node-role.kubernetes.io/control-plane: ""
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          effect: NoSchedule
        - key: node.cloudprovider.kubernetes.io/uninitialized
          value: "true"
          effect: NoSchedule
      containers:
        - name: aws-cloud-controller-manager
          image: registry.k8s.io/provider-aws/cloud-controller-manager:v1.32.0  # pin to a tag matching your cluster minor
          args:
            - --v=2
            - --cloud-provider=aws
            - --leader-elect=true
            - --use-service-account-credentials=true
            - --configure-cloud-routes=false  # AWS VPC CNI handles Pod routing
            - --allocate-node-cidrs=false
            - --controllers=cloud-node,cloud-node-lifecycle,service
          resources:
            requests:
              cpu: 200m
              memory: 256Mi
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cloud-controller-manager:apiserver-authentication-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
  - kind: ServiceAccount
    name: cloud-controller-manager
    namespace: kube-system

Line-by-line. hostNetwork: true because the CCM must reach the cloud’s metadata service (e.g., 169.254.169.254 on EC2) which is only routable from the host network namespace. priorityClassName: system-cluster-critical ensures the CCM is not evicted under node pressure. nodeSelector plus the control-plane toleration pin the CCM to control-plane nodes (separate from worker nodes).

The crucial toleration node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule is the bootstrap chicken-and-egg solution: kubelets started with --cloud-provider=external add this taint to themselves on registration. The CCM tolerates this taint so it can be scheduled on those nodes despite them being “uninitialized.” Once the Node Controller initializes the Node (sets providerID, labels), it removes the taint, allowing other workloads to schedule.

--cloud-provider=aws selects the AWS plugin (alternatives compiled in: gce, azure, openstack, etc.). --use-service-account-credentials=true uses a separate ServiceAccount per controller for finer-grained RBAC. --configure-cloud-routes=false disables the Route Controller because AWS VPC CNI handles Pod-CIDR routing via secondary ENIs; on a cluster using kubenet you’d set true. --controllers=cloud-node,cloud-node-lifecycle,service is an explicit list of which controllers to enable (matches our --configure-cloud-routes=false by omitting the route controller).

The required minimum RBAC (per the docs):

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cloud-controller-manager
rules:
  - apiGroups: [""]
    resources: [events]
    verbs: [create, patch, update]
  - apiGroups: [""]
    resources: [nodes]
    verbs: ["*"]
  - apiGroups: [""]
    resources: [nodes/status]
    verbs: [patch]
  - apiGroups: [""]
    resources: [services]
    verbs: [list, watch]
  - apiGroups: [""]
    resources: [services/status]
    verbs: [patch, update]
  - apiGroups: [""]
    resources: [serviceaccounts]
    verbs: [create]
  - apiGroups: [""]
    resources: [persistentvolumes]
    verbs: [get, list, update, watch]
  - apiGroups: [coordination.k8s.io]
    resources: [leases]
    verbs: [get, create, update]

The cloud-side IAM permissions are separate and provider-specific. For AWS, the CCM’s IAM role typically needs:

• ec2:DescribeInstances, ec2:DescribeRegions, ec2:DescribeAvailabilityZones for the Node Controller.
• elasticloadbalancing:CreateLoadBalancer, DeleteLoadBalancer, RegisterTargets, DescribeLoadBalancers, etc. for the Service Controller.
• (Historical, only with --configure-cloud-routes=true) ec2:CreateRoute, DeleteRoute, DescribeRouteTables.

On EKS/GKE/AKS the user does not see this — the provider runs the CCM with appropriate IAM in their managed control plane.

Failure Modes

Cloud-IAM misconfiguration. The most common failure: a Service of type: LoadBalancer stays Pending because the CCM cannot authenticate to the cloud, or has AccessDenied on elasticloadbalancing:CreateLoadBalancer. Diagnose via kubectl describe service (the Service Controller emits Events) and the CCM’s own logs (kubectl logs -n kube-system deployment/cloud-controller-manager). Mitigation: audit the IAM role attached to the CCM’s ServiceAccount or the underlying instance/pod identity.

Stuck Service deletion → orphaned cloud LB. Service Controller calls EnsureLoadBalancerDeleted on Service deletion; if the call fails (rate-limited, IAM revoked mid-delete, transient cloud-API error), the Service is deleted from Kubernetes but the cloud LB remains, accruing cost. The Service Controller used to leak these silently; modern versions use a service.kubernetes.io/load-balancer-cleanup finalizer to block Service deletion until the cloud teardown succeeds. Always verify finalizer behavior in the specific provider’s CCM version.

Node staying NotReady with uninitialized taint. Kubelet started with --cloud-provider=external but no CCM is running or the CCM cannot reach the cloud API. Nodes stay in this state indefinitely; no workloads schedule on them. Diagnose via kubectl describe node (look for the taint) and CCM logs.

Deleted cloud instances persisting as Nodes. If the CCM is down or has lost its cloud credentials, terminated EC2 instances stay as NotReady Nodes in Kubernetes. After ~5 minutes the kube-controller-manager’s Node Lifecycle controller evicts Pods, but the Node object lingers. This typically manifests as “ghost nodes” in kubectl get nodes that never go away.

Leader-election lease contention with kcm. Historical pre-extraction clusters ran the same cloud controllers in both the kcm and the CCM, fighting over Service status. The Leader Migration procedure handles the rollover by having kcm and CCM share a lease for the cloud controllers during the transition. After full migration, only the CCM runs them.

Cloud-provider rate limits. Each cloud’s API has per-account rate limits (AWS EC2 DescribeInstances: typically 100 req/s; EC2 RunInstances: lower). A cluster with 5000 nodes whose CCM polls every Node every minute can saturate these limits and cause cascade failures. Mitigations: tune --node-monitor-period, increase cloud-account limits via support tickets, separate cluster IAM roles per environment.

Cross-cluster lease/RBAC bleed. The CCM’s leader-election lease is namespaced and identified by holderIdentity; running two clusters in the same AWS account with the same cluster name causes cross-cluster collisions in ClusterID tagging. Always use unique cluster names and tag-isolation.

Alternatives and When to Choose Them

The CCM is essentially mandatory for clusters running on a cloud provider — without it (or its replacement), type: LoadBalancer Services don’t work and cloud-instance lifecycle is invisible. The relevant variants:

• Managed services run their own CCM. EKS uses the AWS CCM (cloud-provider-aws) on the managed control plane. GKE uses cloud-provider-gcp. AKS uses cloud-provider-azure. The CCM binary is invisible to customers but its behavior is observable through Service-Controller side effects (LBs created/destroyed).
• No-cloud clusters skip the CCM entirely. On-prem clusters, kind / minikube / k3s clusters, and bare-metal clusters don’t run a CCM. type: LoadBalancer Services then need a software load balancer like MetalLB or kube-vip to provide the LB virtual IP, and Node initialization happens via the kubelet’s local logic without cloud-side metadata.
• Cluster API providers. Cluster API (Cluster API docs) is the declarative-Kubernetes-cluster-lifecycle project; its “infrastructure providers” (CAPA for AWS, CAPG for GCP, CAPZ for Azure, CAPV for vSphere) bundle a CCM with cluster bootstrapping. Choosing CAPI typically means accepting its bundled CCM choice.
• Custom CCMs for niche clouds. OpenStack, IBM Cloud, Oracle Cloud, Equinix Metal, Hetzner, DigitalOcean, and Linode all maintain their own provider repositories implementing the CloudProvider interface. The architectural pattern is identical; the API surface and IAM model differs per provider.

Production Notes

The CCM is operationally invisible on managed services but critical on self-managed cloud deployments. Lessons from real-world deployments:

• Run 2+ replicas with leader election in HA. A CCM outage stalls new Service-Controller work for the duration; existing LBs continue to serve traffic, but new Services hang Pending.
• Pin CCM IAM to least privilege. A common production mistake is granting the CCM full EC2 / ELB access; instead, scope to the specific actions listed above. The cloud-provider-aws repo publishes minimal-IAM policies; equivalent exists for GCP and Azure.
• Monitor service_controller_loop_duration_seconds and similar metrics. Slow cloud-API responses (regional AWS slowdowns, throttling) surface here first.
• AWS-specific: prefer the AWS Load Balancer Controller (a separate addon, formerly known as ALB Ingress Controller) for type: LoadBalancer Services that need ALB or NLB features beyond what the legacy in-tree code provides. The AWS LB Controller takes over Service Controller work via the service.beta.kubernetes.io/aws-load-balancer-type: external annotation, letting the CCM defer to it (AWS docs). EKS now defaults to this pattern.
• GKE-specific: GKE’s CCM has special handling for Ingress (via GKE Ingress Controller) and Service of type: LoadBalancer that integrates with Google Cloud Load Balancing’s tiered offerings (Standard vs Premium network tier). The choice of LB tier is annotation-driven.
• AKS-specific: AKS’s CCM integrates with both Basic and Standard Azure Load Balancers; Standard SKU is required for AZ-redundant Services.
• Spotify, Airbnb, and Shopify migration write-ups all describe the in-tree-to-CCM migration as one of the more painful Kubernetes upgrades because it required coordinated changes across kubelet config, kcm config, and the new CCM Deployment. Cluster API now hides much of this complexity.

Removal Timeline (resolved)

The two facts that were previously flagged as uncertain are now pinned to primary sources:

• In-tree cloud-provider code is fully removed. The removal proceeded per provider and then via a global feature-gate kill-switch. OpenStack’s in-tree code was removed in v1.26 and AWS’s in v1.27 (each had its own earlier dedicated removal); the remaining providers (GCE, Azure, vSphere) were disabled by default in v1.29 when DisableCloudProviders went beta-on, and the last in-tree integration code across all providers was deleted in v1.31 when that gate (and DisableKubeletCloudCredentialProviders) locked to GA (Kubernetes 1.29 blog; v1.31 removals blog; completing-the-migration blog). From v1.31 onward an external CCM is mandatory on any cloud.
• In-tree volume plugins are fully removed too. CSI Migration went GA in v1.25; the in-tree awsElasticBlockStore and azureDisk plugins were removed in v1.27 and gcePersistentDisk in v1.28 (Kubernetes — Volumes). There is no in-tree volume fallback in current Kubernetes; every cloud volume is served by a CSI driver.

See Also

• Kubernetes MOC — parent index
• Kubernetes Control Plane — broader component family
• kube-apiserver — the CCM’s read-and-write peer
• API Server Request Flow — every CCM read/write to Nodes and Service status traverses this authn → authz → admission → etcd pipeline
• kube-controller-manager — the cloud-agnostic sibling from which CCM was split
• kube-scheduler — sibling control-plane process
• kubelet — the --cloud-provider=external flag is the kubelet-side hand-off
• etcd — where CCM state changes are persisted
• LoadBalancer Service — the Service Controller’s primary output
• Container Storage Interface — replaced the CCM’s historical volume controllers
• Cluster API — bundles a CCM per infrastructure provider
• Amazon EKS — runs cloud-provider-aws CCM on the managed control plane
• Google GKE — runs cloud-provider-gcp CCM
• Azure AKS — runs cloud-provider-azure CCM
• Workload Identity (Kubernetes) — IRSA / Workload Identity for IAM-bound ServiceAccounts (alternative to CCM’s instance-role auth)
• Container Orchestration Architecture — architectural context