vault

Azure CNI

15 min read

Azure CNI

Azure CNI is the family of Azure-native CNI plugins for Azure Kubernetes Service (AKS) that assign Pod IPs from Azure VNet (Virtual Network) address space rather than from a separate overlay CIDR managed by the cluster — making AKS Pods routable from anywhere in the VNet, peered VNets, and on-prem networks reached via ExpressRoute or VPN (learn.microsoft.com/en-us/azure/aks — CNI Networking Overview). The family has evolved through four distinct plugins, two of which are now legacy: kubenet (the original bridge-based overlay, now deprecated), Azure CNI Node Subnet (the original flat-network plugin where Pod IPs come from the node’s subnet; now “legacy”), Azure CNI Pod Subnet (the dynamic IP allocation variant where Pod IPs come from a separate Pod subnet but in the same VNet), and Azure CNI Overlay (Pod IPs come from a logically separate overlay CIDR not in the VNet; the recommended default since GA on April 17, 2023 — azure.microsoft.com/en-us/blog). Layered on top of any of the above, Azure CNI Powered by Cilium swaps the underlying iptables-based dataplane for a Cilium eBPF dataplane while keeping Azure’s IPAM in front — providing kube-proxy-replacement, native NetworkPolicy enforcement, and Hubble-style observability. This note covers all four plugin modes plus the Cilium dataplane variant, contrasts them on the dimensions that matter (IP planning math, scale ceiling, NetworkPolicy implementation, Windows support), and explains when to choose which. See Container Network Interface for the underlying spec, Pod Networking for the per-Pod sandbox plumbing, and Cilium for the eBPF dataplane.

Mental Model

flowchart TB
    subgraph VNET["Azure VNet 10.0.0.0/8"]
        subgraph NSUBNET["Node subnet 10.240.0.0/16"]
            NODE1[AKS Node 1<br/>10.240.0.4]
            NODE2[AKS Node 2<br/>10.240.0.5]
        end
        subgraph PSUBNET["Pod subnet (Azure CNI Pod Subnet)<br/>10.241.0.0/16"]
            POD_DYN1[Pod IP 10.241.0.10]
            POD_DYN2[Pod IP 10.241.0.11]
        end
        OTHER[VM / SQL DB / App Service<br/>sees Pod IP directly]
    end
    subgraph OVERLAY["Overlay Pod CIDR 100.64.0.0/10<br/>(Azure CNI Overlay — outside VNet)"]
        POD_OV1[Pod IP 100.64.0.5]
        POD_OV2[Pod IP 100.64.0.6]
    end
    NODE1 -. "ARP for 10.241.0.10<br/>via VNet routing<br/>(flat)" .- POD_DYN1
    NODE1 -. "encap via Azure SDN<br/>(overlay)" .- POD_OV1
    POD_DYN1 ==> OTHER
    POD_OV1 -- "SNATed to node IP<br/>before leaving VNet" --> OTHER

What this diagram shows. The fundamental split between Azure CNI’s flat modes (top) and overlay mode (bottom). In flat networking — Azure CNI Node Subnet (legacy) or Azure CNI Pod Subnet — Pod IPs live in a VNet subnet (often a dedicated “Pod subnet” distinct from the node subnet), so any other resource in the VNet or a peered VNet can reach the Pod by its IP without SNAT. The cost: the VNet must hold IPs for every Pod that will ever exist, and Pod IP exhaustion ties directly to subnet sizing. In overlay mode (Azure CNI Overlay), Pod IPs come from an overlay CIDR that’s outside the VNet (typically 10.244.0.0/16 or 100.64.0.0/10); Pods talk to each other natively but egress to the VNet is SNATed to the node IP by Azure’s SDN. The trade: overlay decouples Pod IP planning from VNet sizing (you can use a /10 Pod CIDR even on a small VNet) but external resources see node IPs in their access logs, not Pod IPs. The insight to extract: Azure CNI’s four plugin modes are four answers to the question “where does the Pod IP come from?” — and that single design choice cascades through routing, NetworkPolicy availability, and subnet-sizing math.

Mechanical Walk-through

The four plugin modes

Per learn.microsoft.com/en-us/azure/aks — CNI Networking Overview, AKS exposes these CNI choices:

ModeNetwork modelPod IP sourceStatus (2026)Recommended for
kubenetOverlay (bridge + UDR)Pod CIDR managed by AKSLegacy / deprecatedNothing new
Azure CNI Node SubnetFlatNode’s subnetLegacy (still supported)Backward compat
Azure CNI Pod SubnetFlat (dynamic IPAM)Dedicated Pod subnet in VNetCurrent (recommended for flat)Direct VNet-routable Pods
Azure CNI OverlayOverlay (Azure SDN-encapsulated)Overlay CIDR outside VNetDefault since 2023-04Most workloads

Each mode produces a different answer to “how many IPs does my AKS cluster consume from the VNet?” — the load-bearing question for capacity planners.

Mode 1: kubenet (legacy)

The original AKS CNI; uses the upstream kubenet plugin from the Kubernetes project plus User-Defined Routes (UDRs) on the VNet to handle cross-node Pod-to-Pod traffic. Each node creates a Linux bridge (cbr0), allocates Pods a /24 from a 10.244.0.0/16-style cluster Pod CIDR, and the AKS control plane installs UDRs in the node subnet’s route table pointing each node’s Pod CIDR at that node’s VM IP.

Limitations cited by Azure:

  • Max 400 nodes because UDRs are capped at ~400 routes per VNet route table.
  • No Windows node pools.
  • No support for Azure Application Gateway for Containers.
  • No subnet sharing across multiple clusters.
  • Manual UDR management when bringing your own VNet.

Azure documentation explicitly lists kubenet as legacy and recommends Azure CNI Overlay for new clusters. The plugin is no longer merely “on a path toward retirement” — Microsoft has now published a firm end-of-life date: kubenet networking for AKS will be retired on 31 March 2028, after which workloads still running on kubenet will no longer be supported, and Microsoft directs operators to “upgrade to Azure CNI Overlay before that date” (per the AKS legacy CNI concept doc, Important callout). New clusters should therefore not adopt kubenet, and existing kubenet clusters should plan the migration well ahead of the 2028 deadline (see the IPAM/dataplane upgrade guide).

Mode 2: Azure CNI Node Subnet (legacy flat)

The original “real Azure CNI” — every Pod gets a secondary IP from the node’s subnet, allocated up front on cluster create (each node pre-allocates its max-pods IPs in the subnet at startup). Pods are direct VNet citizens; any peered VNet can reach a Pod by IP.

The math: subnet must hold (nodes × max-pods-per-node) + nodes. A 5-node cluster with max-pods=30 consumes 155 IPs from the node subnet — already half of a /24. Adding capacity for autoscaling pushes most clusters to /22 or larger subnets up front.

The mode is now labeled “legacy” — recommended only for backward-compat and for cases where you need AKS to fully manage the VNet (--vnet-subnet-id not set).

Mode 3: Azure CNI Pod Subnet (flat, dynamic IP allocation)

The modern flat-networking plugin (formerly “Azure CNI with Dynamic IP Allocation”). Pod IPs come from a separate Pod subnet in the VNet, distinct from the node subnet. Allocation is dynamic — Pods get IPs on demand from the Pod subnet rather than from a pre-allocated per-node block.

Configuration:

az aks create \
  --network-plugin azure \
  --vnet-subnet-id <node-subnet-id> \
  --pod-subnet-id <pod-subnet-id>

Benefits over the legacy Node Subnet mode:

  • More efficient IP usage (no per-node pre-allocation; only Pods that actually exist consume IPs).
  • Pod subnet can be sized independently of node subnet — typically Pod subnet is much larger.
  • Pods are still VNet-routable (the entire point of flat mode).

This is the recommended choice when you specifically need direct VNet routing of Pod IPs (e.g., a Pod must accept inbound connections from an on-prem system via ExpressRoute).

Mode 4: Azure CNI Overlay (default since 2023-04)

GA on April 17, 2023 (azure.microsoft.com/en-us/blog). Pod IPs come from a separate Pod CIDR not in the VNet (default 10.244.0.0/16, configurable via --pod-cidr). Cross-node Pod-to-Pod traffic is encapsulated by Azure’s underlying SDN (not a Pod-level VXLAN; the host stack handles it) so Pods can use the same IP space across many clusters.

Configuration:

az aks create \
  --network-plugin azure \
  --network-plugin-mode overlay \
  --pod-cidr 192.168.0.0/16

Properties:

  • Up to 250 Pods per node (vs 30–110 typical for flat modes constrained by subnet size).
  • VNet only needs to hold node IPs, not Pod IPs — frees up address planning.
  • Pod egress to VNet is SNATed to node IP — external systems see node IP in their access logs.
  • Pod ingress from VNet requires going through a Service (LoadBalancer, Application Gateway); a peered VNet cannot dial a Pod IP directly.
  • Windows node pools supported.
  • Default mode for new AKS clusters — but only when --network-plugin is omitted entirely. Per the Azure CNI Overlay configuration doc, “If you don’t specify --network-plugin, AKS defaults to Azure CNI Overlay.” There is a sharp gotcha here: if you do pass --network-plugin=azure but omit --network-plugin-mode, AKS “intentionally uses virtual network (node subnet) mode for backward compatibility” — i.e. you silently land in the legacy flat plugin, not Overlay. Getting Overlay therefore means either omitting the plugin flag or explicitly passing --network-plugin-mode overlay. The default Pod CIDR when none is given is 10.244.0.0/16 (same doc).

Azure CNI Powered by Cilium

Layered on top of either Overlay or Pod Subnet modes (learn.microsoft.com/en-us/azure/aks — Azure CNI Powered by Cilium). The Azure IPAM still owns Pod IP assignment; the dataplane (kube-proxy logic, NetworkPolicy enforcement, service load-balancing) is rewritten as eBPF programs loaded by Cilium on each node.

Configuration adds one flag:

az aks create \
  --network-plugin azure \
  --network-plugin-mode overlay \
  --network-dataplane cilium \
  --generate-ssh-keys

Benefits:

  • NetworkPolicy enforcement is native — no separate Calico or Azure Network Policy Manager add-on. Both standard Kubernetes NetworkPolicy and CiliumNetworkPolicy / CiliumClusterwideNetworkPolicy (L3/L4) work.
  • kube-proxy is removed. AKS clusters with Cilium dataplane run no kube-proxy DaemonSet; service VIPs are programmed via eBPF maps.
  • Better scale ceilings. Cilium’s eBPF dataplane handles 10k+ Services per cluster without the iptables blowup that the legacy mode suffers.
  • Linux only (no Windows node pool support as of 2026).
  • L7 policies and FQDN filtering are gated behind Advanced Container Networking Services (ACNS) — an additional Azure feature.

The Cilium version mapping (learn.microsoft.com/en-us/azure/aks — Azure CNI Powered by Cilium):

AKS K8s versionMinimum Cilium version
1.29 (LTS)1.14.20
1.30 (LTS)1.14.20
1.31 (LTS)1.16.16
1.321.17.9
1.331.17.9
1.341.18.6
1.351.18.6

(The doc labels these the minimum Cilium version per AKS Kubernetes release — AKS may run a newer patch within the same minor line; the version a given cluster runs is managed by AKS and tracks this floor.)

The feature’s release timeline is now pinnable: Azure CNI Powered by Cilium entered public preview in late 2022 (the Isovalent announcement is dated 26 October 2022) and reached general availability on 30 May 2023, announced on the Azure updates portal (General availability: Azure CNI powered by Cilium) alongside the Build 2023 wave and the engineering write-up “Azure CNI with Cilium: Most scalable and performant container networking in the Cloud.” GA covered both Overlay and VNet (Pod Subnet) IPAM modes; Node Subnet + Cilium support came later as a separate GA.

Subnet-sizing math

The single most useful planning exercise: how many VNet IPs does each plugin mode consume per N-node cluster with M Pods per node?

Plugin modeNode subnet IPs consumedPod subnet IPs consumedVNet total
kubenetN (just node IPs)0 (Pod CIDR is outside VNet)N
Azure CNI Node SubnetN + N×M (pre-allocated)shared with node subnetN + N×M
Azure CNI Pod SubnetNup to N×M dynamicN + (current Pods)
Azure CNI OverlayN0 (Pod CIDR is outside VNet)N

A 50-node cluster with max-pods=30 consumes:

  • kubenet: 50 IPs in the VNet (plus a per-node bridge IP outside the VNet)
  • CNI Node Subnet (legacy): 50 + 50×30 = 1550 IPs, requiring a /21 minimum
  • CNI Pod Subnet: 50 in node subnet + however many Pods exist in Pod subnet (a /22 for headroom)
  • CNI Overlay: 50 IPs in the VNet; Pod CIDR is whatever overlay size you pick

This is why Azure CNI Overlay became the default: it solves the recurring “I sized the VNet too small and now I can’t scale” failure mode without losing eBPF-grade observability (when Cilium is layered on top).

Configuration / API Surface

A complete CLI invocation matrix:

# Mode 1: kubenet (deprecated; included for reference)
az aks create --name my-cluster \
  --network-plugin kubenet \
  --pod-cidr 10.244.0.0/16
 
# Mode 2: Azure CNI Node Subnet (legacy)
az aks create --name my-cluster \
  --network-plugin azure                           # defaults to node-subnet mode
                                                   # when --network-plugin-mode is omitted
 
# Mode 3: Azure CNI Pod Subnet (flat, dynamic)
az aks create --name my-cluster \
  --network-plugin azure \
  --vnet-subnet-id /subscriptions/.../subnets/nodes \
  --pod-subnet-id /subscriptions/.../subnets/pods  # pod IPs come from this subnet
 
# Mode 4: Azure CNI Overlay (modern default)
az aks create --name my-cluster \
  --network-plugin azure \
  --network-plugin-mode overlay \
  --pod-cidr 192.168.0.0/16                        # pod IPs come from this CIDR
                                                   # (must NOT overlap the VNet)
 
# Overlay + Cilium dataplane
az aks create --name my-cluster \
  --network-plugin azure \
  --network-plugin-mode overlay \
  --network-dataplane cilium \                     # the magic switch
  --pod-cidr 192.168.0.0/16
 
# Pod subnet + Cilium dataplane
az aks create --name my-cluster \
  --network-plugin azure \
  --vnet-subnet-id /subscriptions/.../subnets/nodes \
  --pod-subnet-id /subscriptions/.../subnets/pods \
  --network-dataplane cilium

A representative Pod-side network state when using Overlay + Cilium:

$ kubectl exec -it my-pod -- ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> ...
3: eth0@if78: <BROADCAST,MULTICAST,UP,LOWER_UP>
   inet 192.168.7.42/16 scope global eth0            # from Pod CIDR (overlay)
   link/ether 22:33:44:55:66:77
 
$ kubectl exec -it my-pod -- ip route
default via 192.168.7.1 dev eth0
192.168.0.0/16 dev eth0 scope link
 
# On the host, Cilium has loaded eBPF programs in place of iptables:
$ kubectl -n kube-system exec ds/cilium -- cilium status | grep "kube-proxy"
KubeProxyReplacement:   True   [eth0, eth1]

Failure Modes

  1. Pod CIDR overlaps node subnet or peered VNet CIDR. Most common Azure CNI Overlay mistake. Symptom: cross-node Pod traffic dropped, or Pods can’t reach peered-VNet resources because they look like local traffic. Diagnostic: az network vnet show + kubectl describe configmap kube-system/cilium-config | grep -i cidr. Fix: use 100.64.0.0/10 (RFC 6598 shared address space) or a clearly non-overlapping range.

  2. Subnet IP exhaustion (Azure CNI Node Subnet mode). Cluster cannot scale because all subnet IPs are consumed by per-node Pod pre-allocation. Symptom: new nodes stay NotReady, kubelet logs show CNI IP allocation errors. Fix: migrate to Pod Subnet mode (dynamic) or Overlay (no VNet IPs for Pods).

  3. ipBlock NetworkPolicy doesn’t allow Pod or node IPs (Cilium dataplane). Cilium’s interpretation of ipBlock excludes Pod and node IPs by default (learn.microsoft.com/en-us/azure/aks — Azure CNI Powered by Cilium FAQ); a policy to.ipBlock.cidr: 0.0.0.0/0 does not allow egress to other Pods. Workaround: add namespaceSelector: {} + podSelector: {} to the to clause.

  4. NodePort Services + IPv6 + externalTrafficPolicy=Cluster (Overlay dual-stack). Azure Load Balancer health probes don’t reach IPv6 Pods correctly with cluster traffic policy. Fix: use externalTrafficPolicy: Local for IPv6 services on AKS.

  5. Cilium identity exhaustion under high label churn. Spark workloads (which generate per-pod labels like spark-app-name) push Cilium identity count past 65535 (the v1 identity limit). Mitigation: add label exclusion in cilium-config ConfigMap (only modification AKS permits).

  6. Migration kubenet → Azure CNI Overlay requires cluster re-creation in some cases. Microsoft documents an in-place upgrade path but it’s gated on AKS version and customer subnet configuration; many customers end up doing a blue-green cluster migration.

  7. Windows node pool incompatibility. kubenet and Cilium dataplane don’t support Windows; Windows support requires Azure CNI Overlay (Linux/Windows mixed) without Cilium dataplane, or Azure CNI Node/Pod Subnet without Cilium.

Alternatives and When to Choose Them

  • AWS VPC CNI (analogous flat-VNet style on AWS). Solves the same problem on AWS by leasing ENI secondary IPs to Pods.
  • GKE Dataplane V2 (analogous Cilium-on-cloud-native CNI). GKE’s equivalent of “Azure CNI Powered by Cilium” — runs Cilium eBPF as the dataplane on Google Cloud.
  • Calico on AKS. Supported as a NetworkPolicy provider on AKS clusters using Azure CNI; the policy-only Calico deployment was the de facto choice before Cilium dataplane went broadly available. Now less compelling for new clusters.
  • BYO CNI on AKS. AKS supports “Bring Your Own CNI” — install Calico, Cilium, Flannel, etc. manually. Loses Azure support; useful when a feature isn’t available in the managed dataplane (e.g., custom Cilium config that AKS’s managed ConfigMap doesn’t allow).
  • AKS-managed NGINX ingress + Azure CNI Overlay. The recommended “easy mode” stack for a new AKS deployment: managed CNI + managed ingress + Azure-managed identity.

Production Notes

  • Choose Azure CNI Overlay unless you have a specific reason not to. Microsoft’s own documentation makes this recommendation, and the IP-planning math is dramatically simpler. The cases requiring flat networking — Pod IPs visible to peered VNets or on-prem networks — are increasingly rare as Service Mesh and API Gateway patterns absorb that need.
  • Pair Overlay with Cilium dataplane for production clusters. NetworkPolicy enforcement matters; the eBPF dataplane handles it natively without a separate add-on, and the observability story (Hubble flow logs, ACNS metrics) is operationally valuable.
  • kubenet is on borrowed time. AKS documentation has labeled it “legacy” since 2023; though no specific EOL date is published, new clusters should not use it. Migration to Azure CNI Overlay is the recommended path.
  • ipBlock semantics differ between Azure Network Policy Manager and Cilium. A subtle production gotcha: NetworkPolicies that worked on a Calico- or Azure-NPM-backed cluster may silently break when the cluster migrates to Cilium dataplane because Cilium’s ipBlock excludes Pod/node IPs by default. Audit policies during migration.
  • AKS reserves the right to manage Cilium ConfigMap. Customers cannot edit the cilium-config ConfigMap freely (only label exclusion is permitted). Operators needing fine-grained Cilium control should use BYO CNI instead.

See Also

Graph View

Backlinks