AWS VPC CNI

The Amazon VPC CNI plugin for Kubernetes (amazon-vpc-cni-k8s) is the default and only AWS-supported CNI plugin on Amazon EKS for nodes that run on AWS infrastructure (docs.aws.amazon.com/eks — Pod Networking). Unlike overlay CNIs (Flannel, Cilium in overlay mode), the VPC CNI does not encapsulate Pod traffic — it assigns each Pod a real private IPv4 or IPv6 address from the VPC subnet by allocating secondary IPs (or, with prefix delegation, /28 IPv4 prefixes or /80 IPv6 prefixes) on Elastic Network Interfaces (ENIs) attached to the EC2 worker node. The result is that Pods are first-class VPC citizens: they appear in VPC Flow Logs with their real IPs, security groups can apply to them directly (via Security Groups for Pods SGPP), and routing between Pods on the same VPC needs no overlay — it’s ordinary VPC routing. A cluster must commit at creation time to either IPv4-only or IPv6-only Pods: dual-stack (Dual-Stack Networking) is explicitly rejected by EKS (“Amazon EKS doesn’t support dual-stacked Pods or services, even though Kubernetes does,” per docs.aws.amazon.com/eks — cni-ipv6) and the family cannot be changed after cluster creation. The plugin consists of two pieces (github.com/aws/amazon-vpc-cni-k8s): (1) the CNI binary (aws-cni) invoked per-Pod by containerd to wire up veths inside the Pod namespace, and (2) the L-IPAM daemon (ipamd, running inside the aws-node DaemonSet on every node) — a long-running gRPC service that maintains a warm pool of unused IPs by calling EC2 APIs (AssignPrivateIpAddresses, AttachNetworkInterface) ahead of demand. The tradeoff: Pod density per node is capped by EC2 instance-type ENI/IP-per-ENI limits unless prefix delegation is enabled. This note covers the data plane, the IPAMD warm-pool algorithm, prefix delegation, custom networking, external SNAT, security groups for Pods, and the IPv6 mode. See Pod Networking for the per-Pod sandbox plumbing the plugin participates in, and Container Network Interface for the spec it implements.

Mental Model

flowchart TB
    subgraph AWS["AWS VPC subnet 10.0.1.0/24"]
        subgraph NODE["EC2 worker node (m5.large)"]
            subgraph ENIS["ENIs attached to instance"]
                ENI0[Primary ENI<br/>10.0.1.10 node IP<br/>+ secondary IPs<br/>10.0.1.11, 10.0.1.12, ...]
                ENI1[Secondary ENI<br/>10.0.1.50<br/>+ secondary IPs<br/>10.0.1.51, 10.0.1.52, ...]
            end
            IPAMD[aws-node DaemonSet<br/>ipamd L-IPAM daemon<br/>watches Pod scheduling,<br/>maintains warm pool]
            CNI[/opt/cni/bin/aws-cni<br/>called per Pod by containerd]
            POD1[Pod A<br/>eth0 = 10.0.1.11/24<br/>via veth → ENI0]
            POD2[Pod B<br/>eth0 = 10.0.1.51/24<br/>via veth → ENI1]
            POD3[Pod C<br/>eth0 = 10.0.1.12/24<br/>via veth → ENI0]
        end
        OTHERSVC[Other VPC resource<br/>RDS, EC2, ALB<br/>sees Pod IP directly]
    end
    EC2API["EC2 API<br/>AssignPrivateIpAddresses<br/>AttachNetworkInterface"]
    IPAMD -- "warm pool top-up" --> EC2API
    EC2API -- "secondary IPs / new ENI" --> ENIS
    CNI -- "ADD: allocate from pool" --> IPAMD
    POD1 -. "Pod traffic routed natively<br/>by VPC route table<br/>no overlay, no NAT" .-> OTHERSVC

What this diagram shows. A single EKS worker node with two ENIs attached; each ENI has a primary IP (which the node itself uses) plus a pool of secondary IPs. The aws-node DaemonSet on the node runs ipamd (the L-IPAM daemon), which makes EC2 API calls to keep a warm pool of attached, allocated, but unused IPs ready for the next Pod that lands on this node. When a Pod is created, kubelet → containerd → /opt/cni/bin/aws-cni invokes the local IPAMD over gRPC to pick an IP from the warm pool; the CNI then wires up a veth pair (Pod-side eth0, host-side wired into the node’s route table for the appropriate ENI), and the Pod is online. The crucial property: Pod A’s IP 10.0.1.11 is a real VPC IP, indistinguishable to AWS infrastructure from the node IP 10.0.1.10 — security groups, VPC Flow Logs, VPC endpoints, and Transit Gateway all see it natively. The insight to extract: VPC CNI replaces overlay with VPC-native routing by leasing IPs from the VPC’s own IPAM. There is no VXLAN, no encapsulation, no separate Pod CIDR; the cost is per-node IP scarcity bounded by EC2’s ENI limits.

Mechanical Walk-through

The two components: aws-cni binary and aws-node DaemonSet

Per github.com/aws/amazon-vpc-cni-k8s, the plugin splits along the same lines as the CNI spec’s runtime/plugin division (Container Network Interface):

  1. aws-cni (the CNI plugin binary, dropped at /opt/cni/bin/aws-cni on each node). Invoked by containerd/CRI-O with CNI_COMMAND=ADD for every new Pod. Its job: ask the local ipamd for an IP, create a veth pair, place one end in the Pod namespace, configure eth0 inside, and install per-Pod routes on the host so packets to/from this Pod use the right ENI. It is a thin client to ipamd; it does not call EC2 directly.
  2. ipamd (the long-running daemon, packaged in the aws-node Pod of the aws-node DaemonSet — one Pod per node, hostNetwork: true). This is where the real work happens: it watches the Kubernetes API for Pod scheduling on this node, calls EC2 APIs to attach additional ENIs and assign secondary IPs ahead of demand, maintains a JSON state file at /var/run/aws-node/ipam.json, and serves the local aws-cni over a Unix-domain gRPC socket at /var/run/aws-node/aws-cni.sock.

The DaemonSet runs as a system Pod with hostNetwork: true because it needs IMDSv2 access (to learn the instance’s identity) and the ability to mutate host routes; running in its own namespace would block both.

IP allocation: the warm pool

The naive design — “call EC2 to assign a secondary IP every time a Pod starts” — would put the EC2 control plane in every Pod’s hot path, with ~500 ms latency per assignment, and would rate-limit any cluster doing Pod churn. The VPC CNI avoids this with a warm pool: ipamd allocates IPs ahead of Pod demand, and the per-Pod CNI invocation is a cheap local pool pick.

The defaults (per aws-eks-best-practices — Prefix Mode for Linux):

  • WARM_ENI_TARGET=1 — keep one fully-spare ENI worth of IPs always available. If an instance type allows 10 IPs per ENI, the daemon ensures at least 10 IPs are free; if it falls below, it attaches a new ENI and pre-assigns all its IPs.
  • WARM_IP_TARGET (unset by default) — alternative knob: keep N free IPs total, regardless of ENI count. Conserves IPs at the cost of more EC2 calls.
  • MINIMUM_IP_TARGET (unset by default) — floor on total allocated IPs; useful for Pods-on-startup bursts.
  • WARM_PREFIX_TARGET=1 (when prefix delegation is enabled) — keep one spare /28 prefix.

When WARM_ENI_TARGET=1 and the running Pod count crosses a threshold, ipamd issues AttachNetworkInterface and AssignPrivateIpAddresses API calls in batches to top the pool back up. When Pods exit, the IPs go back to the pool but are not immediately deassigned from the ENI — they stay warm.

EC2 instance-type ENI/IP-per-ENI limits

The fundamental constraint: each EC2 instance type has a fixed max-ENIs and max-IPs-per-ENI. From the EC2 documentation:

Instance typeMax ENIsIPs per ENIPod limit (no prefix delegation)
t3.medium36(3 × 6) − 3 ENI primaries = 15, capped to 17 by EKS calculator
m5.large310(3 × 10) − 3 = 27, EKS reports ~29
m5.xlarge415(4 × 15) − 4 = 56
m5.4xlarge830(8 × 30) − 8 = 232, capped to 110 by K8s scalability default
c5n.18xlarge1550hits Kubernetes scalability ceiling of 110

The formula EKS uses, simplified: pods = (ENIs × (IPs-per-ENI − 1)) — one IP per ENI is reserved as the ENI’s own primary. The Kubernetes scalability working group recommends a 110-Pod-per-node ceiling (github.com/kubernetes/community — sig-scalability thresholds), which EKS managed node groups enforce by default. The cap was raised to 250 Pods per node for instances with > 30 vCPUs (aws.amazon.com/blogs/containers — VPC CNI increases pods-per-node limits).

Prefix delegation — the density multiplier

For Nitro-based instances (aws.amazon.com — VPC CNI announcement, August 2021), the VPC CNI can assign IPv4 /28 prefixes (16 IPs each) or IPv6 /80 prefixes instead of individual secondary IPs. Enabled via ENABLE_PREFIX_DELEGATION=true on the aws-node DaemonSet; requires VPC CNI v1.9.0+.

Math comparison for m5.large (3 ENIs × 10 IPs per ENI):

  • Without prefix delegation: 3 × (10 − 1) = 27 Pod IPs available.
  • With prefix delegation: 3 ENIs × 9 prefix slots × 16 IPs per prefix = 432 IPs, capped to 110 by the Kubernetes scalability default (or 250 for high-vCPU instances).

In practice, prefix delegation lets nearly every Nitro instance hit the 110-Pod ceiling that Kubernetes’ scalability working group recommends (aws.amazon.com/blogs/containers). The tradeoff: prefix delegation requires contiguous /28 blocks in the subnet — heavily fragmented subnets may fail to allocate prefixes even though enough loose IPs exist.

Eligible instance families. AWS itself does not publish an exhaustive enumeration; the EKS Best Practices Prefix Mode guide only states “prefix assignment works with nearly every Nitro instance type.” In current generations, this includes essentially all m5/m6/m7, c5/c6/c7, r5/r6/r7, t3/t3a/t4g families and most special-purpose Nitro types (Inf, Trn, G5+); legacy Xen-based instances (m4, c4, t2, r4 and earlier) are excluded. The authoritative per-instance answer is the EC2 instance-type ENI/IP table cross-referenced against the Nitro-system flag; for production planning, query it via aws ec2 describe-instance-types rather than relying on family-level lists, since AWS continually adds new families.

Custom networking

The default behavior places Pods in the same subnet as the node. This is operationally convenient (one subnet to size) but conflicts with subnet sizing for high-density nodes: a /24 subnet (256 IPs) plus a few m5.4xlarge nodes (each consuming up to 110 IPs) exhausts the subnet in three nodes.

Custom networking (docs.aws.amazon.com/eks — Custom Pod Networking) decouples Pod subnets from node subnets via the ENIConfig CRD:

apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
  name: us-east-1a
spec:
  securityGroups:
    - sg-0a1b2c3d4e5f6g7h8
  subnet: subnet-0fedcba9876543210

Pods on nodes in AZ us-east-1a get IPs from subnet-0fedcba9876543210 instead of the node’s subnet. The CNI is configured with AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true and ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone so that ipamd reads each node’s AZ label and picks the matching ENIConfig.

Trade: the node’s primary ENI still uses the node’s subnet; only secondary ENIs (the ones serving Pods) use the ENIConfig subnet. So a node still consumes one IP from the original subnet plus N IPs from the Pod subnet.

External SNAT

The default VPC CNI behavior SNATs Pod egress traffic to the node’s primary IP when the destination is outside the VPC (Internet, peered VPCs, on-prem). This is convenient because the destination only needs to know about the node’s IPs, not the Pod CIDR. But it loses Pod-IP fidelity — destination services see node IPs in their access logs.

External SNAT (AWS_VPC_K8S_CNI_EXTERNALSNAT=true, docs.aws.amazon.com/eks — external SNAT) disables this in-VPC SNAT. With external SNAT enabled, Pod-egress traffic exits the VPC with the Pod’s IP as source; the VPC’s NAT Gateway (or a custom NAT mechanism) is then expected to do the source NAT. Requires the Pod’s subnet to have a route to a NAT Gateway in the route table; otherwise external destinations can’t reply.

The trade: external SNAT preserves source-IP info but requires NAT Gateway capacity sized for Pod-egress (NAT Gateway is billed per GB and per connection, so this can be expensive).

IPv6 mode and dual-stack rejection

The VPC CNI supports an IPv6-single-stack mode for Pods on Nitro-based EC2 (or Fargate) nodes — but emphatically not dual-stack. The EKS documentation is explicit (docs.aws.amazon.com/eks — cni-ipv6): “Amazon EKS doesn’t support dual-stacked Pods or services, even though Kubernetes does. As a result, you can’t assign both IPv4 and IPv6 addresses to your Pods and services.” This is a cluster-creation-time choice: the IP family is fixed for the lifetime of the cluster and cannot be changed in place. See Dual-Stack Networking for what dual-stack means in upstream Kubernetes and why this is a real constraint, not just a limitation.

When IPv6 mode is enabled, the Pod-IP arithmetic changes radically. AWS gives the VPC a /56 IPv6 CIDR; EKS allocates a /64 per subnet and from each /64 the VPC CNI carves /80 prefixes (each prefix providing 2⁴⁸ ≈ 281 trillion addresses). Pods receive an IPv6 address from one of these prefixes (the /80 is bound to an ENI just as /28 IPv4 prefixes are). The density-vs-ENI-limit problem effectively disappears — the EC2 ENI limits still apply, but each ENI now hosts orders-of-magnitude more addresses than a node could possibly use. In this mode, prefix delegation is mandatory and IPv6 mode requires VPC CNI v1.10.1 or later.

A subtlety in the design: Pods are IPv6-only from the cluster’s point of view, but they still need to reach IPv4-only external services (S3 in IPv4-only regions, the IMDS endpoint, third-party APIs). The VPC CNI handles this with a chained host-local CNI plugin that gives each Pod a node-local (non-routable, not reported to the API server) IPv4 address; when a Pod sends a packet to an external IPv4 destination, the node performs source NAT to the node’s IPv4 primary, exactly like a single-host docker bridge does. This eliminates the need for VPC DNS64/NAT64 (docs.aws.amazon.com/eks — cni-ipv6) but means that from outside the cluster, IPv4-egress flows still appear to come from node IPs (the Pod-IP-in-Flow-Logs benefit is preserved only for IPv6 destinations).

Constraints worth knowing before committing: no Windows IPv6 support, no Outposts IPv6 support, custom networking is mutually exclusive with IPv6, FSx for Lustre is not supported, and the AWS Load Balancer Controller must be ≥v2.3.1 and used in IP-mode (not instance-mode). Most clusters that adopt IPv6 do so primarily for the IP-density relief, not the IPv6-only routing; the cost is the long tail of incompatibilities.

Security Groups for Pods (SGPP)

Without SGPP, all Pods on a node share the node’s security groups — fine for blanket “allow internal VPC” rules but useless for “this Pod should be able to reach RDS, the others shouldn’t.” SGPP (docs.aws.amazon.com/eks — Security Groups for Pods, VPC CNI v1.7.0+) assigns each Pod that opts in a branch ENI — a virtual ENI created on top of a trunk ENI on the node — with its own security group(s).

Selection is via a SecurityGroupPolicy CRD plus Pod labels:

apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata:
  name: my-rds-clients
  namespace: payments
spec:
  podSelector:
    matchLabels:
      app: order-processor
  securityGroups:
    groupIds: ["sg-0123456789abcdef0"]

Pods matching the selector get a branch ENI in sg-0123… instead of a regular secondary IP. The VPC Resource Controller (an EKS add-on running on the control plane) reconciles these requests by issuing the necessary EC2 API calls; the local ipamd then plumbs the branch ENI into the Pod namespace.

Trade: branch ENIs consume EC2 trunk ENI slots (limited per instance type — typically ~10–60 per instance), so SGPP-using Pods compete for a smaller pool than regular secondary-IP Pods.

Configuration / API Surface

A representative aws-node DaemonSet snippet, tuned for a high-density cluster:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: aws-node
  namespace: kube-system
spec:
  template:
    spec:
      hostNetwork: true                           # needs host route table access
      serviceAccountName: aws-node                # IRSA-bound to an IAM role with EC2:*ENI* perms
      containers:
        - name: aws-node
          image: 602401143452.dkr.ecr.us-east-1.amazonaws.com/amazon-k8s-cni:v1.21.1-eksbuild.8
          env:
            - name: ENABLE_PREFIX_DELEGATION       # turn on /28 prefix allocation
              value: "true"                        # required for >29 Pods on m5.large
            - name: WARM_PREFIX_TARGET             # keep 1 spare /28 prefix at all times
              value: "1"
            - name: AWS_VPC_K8S_CNI_EXTERNALSNAT   # don't SNAT Pod->external traffic at the node
              value: "true"                        # NAT Gateway will do it
            - name: AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG  # use ENIConfig subnets, not node subnet
              value: "true"
            - name: ENI_CONFIG_LABEL_DEF
              value: "topology.kubernetes.io/zone"
            - name: ENABLE_POD_ENI                 # enable Security Groups for Pods
              value: "true"
            - name: POD_SECURITY_GROUP_ENFORCING_MODE
              value: "strict"                      # Pod SG takes precedence over node SG
                                                   # even for off-VPC traffic
            - name: AWS_VPC_K8S_CNI_LOGLEVEL
              value: "INFO"

Inside a Pod after the plugin has run:

$ kubectl exec -it order-processor -- ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> ...
   inet 127.0.0.1/8
3: eth0@if42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001                # MTU 9001 = AWS jumbo
   inet 10.0.1.42/32 scope global eth0                                  # /32: no on-link peers;
   link/ether 0a:b1:c2:d3:e4:f5                                         # all traffic via gateway
 
$ ip route
default via 169.254.1.1 dev eth0                                        # link-local gw; node ARPs
10.0.1.42/32 dev eth0 scope link                                        # for it

The /32 mask and link-local gateway pattern is the same as Calico’s; AWS VPC CNI uses it to avoid teaching the Pod about the rest of the subnet (the host’s route table handles all routing decisions).

Failure Modes

  1. Pod stuck in ContainerCreating with “failed to assign an IP address to container”. Cause: ipamd warm pool exhausted because the subnet has no free IPs. Diagnostic: kubectl logs -n kube-system aws-node-XXXXX shows AssignPrivateIpAddresses errors; aws ec2 describe-subnets shows AvailableIpAddressCount: 0. Fix: enable prefix delegation, switch to custom networking with a roomier Pod subnet, or resize the existing subnet.

  2. Pod density caps at instance-type ENI ceiling. Cause: prefix delegation not enabled on a node where it would help. Symptom: nodes run out of IP slots well before they run out of CPU/memory; HPA-scaled workloads stall. Fix: set ENABLE_PREFIX_DELEGATION=true and re-create the node group (cannot retrofit running nodes safely; mixing prefix and non-prefix ENIs on the same node is broken).

  3. AWS API rate limited events in ipamd logs. Cause: many Pods churning rapidly, exhausting the EC2 mutating-API throttle (default ~100 req/s per region per account). Fix: increase WARM_ENI_TARGET or WARM_PREFIX_TARGET so ipamd does less churn; spread node groups across more accounts/regions.

  4. Leaked secondary IPs after a node crash. Cause: kubelet/runtime did not call CNI DEL before the node disappeared; ipamd state file lost. Symptom: VPC IPs remain assigned to a dead ENI; subnet runs out. Fix: VPC CNI v1.11+ ships an introspection endpoint and the IPv4 prefix mode introduced GC to clean stale state; on older versions, run aws ec2 unassign-private-ip-addresses manually based on aws ec2 describe-instances against terminated instances.

  5. Custom networking misconfigured: Pods schedule but can’t reach the cluster’s VPC services. Cause: ENIConfig subnet’s route table doesn’t have a route to the cluster’s Service CIDR (which the node’s subnet does have). Fix: ensure ENIConfig subnet’s route table mirrors the node subnet’s for in-VPC and Service traffic.

  6. SGPP-mode Pod can’t be scheduled because branch ENIs exhausted. Cause: the instance type’s trunk-ENI slot count is lower than the number of Pods requesting unique SGs. Symptom: Pod stuck Pending with Insufficient resource: vpc.amazonaws.com/pod-eni. Fix: spread SGPP-using Pods across more nodes (set anti-affinity), or use larger instance types.

  7. VPC CNI version skew with EKS minor version. The supported VPC CNI version is the one that ships with EKS Auto Mode / managed add-ons for the K8s minor in question (docs.aws.amazon.com/eks — VPC CNI versions). Self-managing the add-on at a much older version produces ENI-attach failures on newer instance types. Fix: keep the add-on up to date through EKS managed add-ons (one minor at a time, never skipping past v1.7.0).

Alternatives and When to Choose Them

  • Calico on EKS. Often run as a policy-only layer alongside VPC CNI (Calico provides NetworkPolicy; VPC CNI provides IPs). The combination — VPC CNI for L3 + Calico for L4/L7 policy enforcement — was the de facto best practice before VPC CNI got native NetworkPolicy support in v1.14+ (mid-2023). Now usable but less compelling for new clusters.
  • Cilium on EKS (ENI mode). Cilium can use VPC ENIs for IPAM, replacing the VPC CNI entirely. Gets you eBPF kube-proxy replacement and Hubble observability while keeping VPC-native IPs. Trades AWS-supported status for eBPF features.
  • EKS Auto Mode. As of 2024-2025, EKS Auto Mode bundles a managed networking stack (still VPC CNI under the hood) and removes the need to install/upgrade it manually. Good default for new clusters; trade is less knob-tuning latitude.
  • VPC CNI in IPv6-single-stack mode (covered in detail above). Eliminates Pod-density pressure entirely; trade is no dual-stack option (cluster must commit to one family at create time) and the long compatibility tail (no Windows, no Outposts, no custom networking, etc.).
  • Overlay CNIs on EC2 (rare). Flannel/Weave on EC2 still work but lose all VPC-native benefits and add encapsulation overhead. Use only in air-gapped scenarios where VPC IP assignment is impossible.

Production Notes

  • Karpenter + prefix delegation is the modern AWS default. The Karpenter cluster autoscaler is bin-packing aware and treats per-node Pod ceilings as a hard scheduling constraint; pairing it with VPC CNI prefix delegation eliminates the most common AWS density complaint. See Karpenter.
  • IRSA (IAM Roles for Service Accounts) is required for SGPP and custom networking. The aws-node ServiceAccount must be IRSA-bound to a role with ec2:*NetworkInterface* and ec2:DescribeInstances permissions. The legacy “EC2 instance profile” approach works but gives every Pod on the node the same IAM blast radius. Cross-link Workload Identity (Kubernetes).
  • VPC Flow Logs see Pod IPs natively. Operational gold: a single VPC Flow Log query can show all traffic from a specific Pod IP, including egress to S3/RDS, without needing in-cluster logging. Overlay CNIs lose this — the underlay only sees node IPs.
  • Subnet sizing math is load-bearing. The Kubernetes capacity planning rule of thumb on EKS: a /24 subnet (256 usable IPs) supports ~2 dense nodes (110 Pods each); a /22 (~1022 IPs) supports ~8; a /19 (~8190 IPs) supports ~70. Cluster operators consistently under-size on first deployment and have to migrate subnets later. See aws-eks-best-practices — Subnet Sizing.
  • The Pod IP is not immutable. Pod restart inside the same node typically re-uses the same IP (because ipamd doesn’t release on restart), but a Pod re-scheduled to a different node gets a new IP from that node’s pool. Applications that cache Pod IPs (e.g., service-discovery clients that pin to an IP) break under this assumption.

See Also