vault

Azure AKS

14 min read

Azure AKS

Azure Kubernetes Service (AKS) is Microsoft Azure’s managed Kubernetes offering: the customer declares a cluster, and Azure provisions and operates the control plane — kube-apiserver, etcd, kube-scheduler, kube-controller-manager — on Microsoft-owned infrastructure that the customer never sees or SSHes into (Azure — AKS core concepts). The customer’s responsibility shrinks to the worker fleet (organized into node pools), the workloads, and the cluster’s configuration knobs. AKS is one of the three hyperscaler managed-Kubernetes services alongside Amazon EKS and Google GKE, and like them it implements the responsibility split catalogued in Managed Kubernetes Services. AKS is distinctive for its tight integration with the surrounding Azure platform — Microsoft Entra ID (formerly Azure Active Directory) for authentication, Azure Virtual Networks for Pod networking via Azure CNI, Azure Monitor for observability — and, since October 2025, for AKS Automatic, an opinionated, heavily-managed cluster mode analogous to GKE Autopilot.

Mental Model

flowchart TB
    subgraph MS["Microsoft-managed (invisible to customer)"]
        CP["Control plane<br/>kube-apiserver · etcd · scheduler · controller-manager"]
    end
    subgraph CUST["Customer Azure subscription"]
        subgraph SYS["System node pool<br/>(taint CriticalAddonsOnly)"]
            SP["CoreDNS · metrics-server · konnectivity"]
        end
        subgraph USR["User node pool(s)<br/>(VMSS-backed)"]
            UP["Application Pods"]
        end
        VNET["Azure VNet / subnets"]
        ENTRA["Microsoft Entra ID"]
    end
    DEV["kubectl / az aks"] -->|"Entra token"| CP
    CP -->|"manages"| SYS
    CP -->|"manages"| USR
    SP --- VNET
    UP --- VNET
    CP -. "validates OIDC" .-> ENTRA
    NAP["Node Auto-Provisioning<br/>(Karpenter-based)"] -. "creates VMs" .-> USR

    classDef ms fill:#def,stroke:#268
    class CP ms

What this shows. AKS draws the same managed/unmanaged boundary as every managed-Kubernetes service: Microsoft owns the control plane (blue), the customer owns the node pools. The insight specific to AKS is the two kinds of node pool — a system pool that hosts critical add-ons (CoreDNS, metrics-server) and at least one user pool for applications — and that every pool is, underneath, an Azure Virtual Machine Scale Set (VMSS). The control plane authenticates humans by validating tokens against Microsoft Entra ID, and node count can be driven either by the classic Cluster Autoscaler or by Node Auto-Provisioning (NAP), Azure’s Karpenter-based provisioner.

Mechanical Walk-through

The managed control plane and pricing tiers

When you run az aks create, Azure provisions the control-plane components on its own fleet. You never get a Node object for a control-plane machine — kubectl get nodes shows only worker nodes. The control plane is billed per cluster-hour, and the per-cluster fee buys a tier (Azure — Free, Standard, Premium pricing tiers):

  • Free tier — no cluster-management fee, no financially-backed SLA on the API server. The control plane still runs (and is still HA on Azure’s side in practice), but Microsoft makes no contractual uptime promise. Recommended only for experimentation and small clusters (Azure’s guidance: under ~10 nodes).
  • Standard tier — $0.10 / cluster-hour. Adds a financially-backed Uptime SLA: 99.95 % API-server availability with Availability Zones, 99.9 % without. Supports clusters up to 5,000 nodes. This is the production default.
  • Premium tier — $0.60 / cluster-hour. Everything in Standard plus AKS Long-Term Support (LTS): Microsoft supports a chosen Kubernetes minor version for up to two years, versus the ~14-month community window. To use it you must also pass --k8s-support-plan AKSLongTermSupport at cluster creation (Azure — Long-Term Support).

The mental trap: “Free” does not mean “non-HA” the way EKS has no free tier and AKS’s old free tier once meant a single-replica control plane. As of 2026 the difference between Free and Standard is the contractual SLA and node-count ceiling, not whether the control plane is replicated.

Node pools — system vs user, VMSS-backed

A node pool is a homogeneous group of worker VMs — same SKU (VM size), same OS image, same labels and taints — that scales as a unit. Every AKS node pool is implemented as an Azure Virtual Machine Scale Set (VMSS), the Azure primitive for identical, autoscalable VM groups. AKS distinguishes two pool modes (Azure — system node pools):

  • System node pools host critical cluster add-ons — CoreDNS, metrics-server, the konnectivity agent. Every cluster needs at least one. AKS taints them CriticalAddonsOnly=true:NoSchedule to discourage application Pods from landing there, and enforces a minimum node count and minimum VM-size constraints (no Bursty/B-series, ≥2 vCPU, ≥4 GiB RAM).
  • User node pools host application workloads. A cluster may have zero (everything runs on system pools) or many — typically segmented by workload class: a general pool, a memory-optimized pool, a GPU pool, a Spot-instance pool for fault-tolerant batch.

This system/user split is AKS’s analog of the labeled-node-group convention described in Kubernetes Cluster Architecture; it just gives the “infrastructure pool” first-class API status.

Networking — Azure CNI modes and Cilium

AKS Pod networking is delegated to Azure CNI, which has evolved several modes:

  • Azure CNI (classic, “Pod Subnet”) — each Pod gets a real VNet IP from a delegated subnet. VNet-native, directly routable, but consumes IP address space aggressively.
  • Azure CNI Overlay — Pods get IPs from a private overlay CIDR; only nodes consume VNet IPs. Scales to far larger clusters without exhausting subnet space; the modern default.
  • Azure CNI Powered by Cilium — Azure CNI for IP-address management combined with Cilium’s eBPF data plane for routing, Service load-balancing (replacing kube-proxy), and NetworkPolicy enforcement (Azure — Azure CNI Powered by Cilium). This is the recommended high-performance configuration and is what AKS Automatic uses by default.

Microsoft Entra ID and Workload Identity

AKS integrates Microsoft Entra ID (the renamed Azure AD) as the cluster’s authentication backend: humans authenticate kubectl with an Entra token (an OIDC bearer token), and the API server validates it. Entra groups map to RBAC subjects, so cluster access is governed by the same directory that governs the rest of the organization.

For workload identity — Pods that need to reach Azure Key Vault, Storage, or Microsoft Graph — AKS offers Microsoft Entra Workload ID. A Kubernetes ServiceAccount is federated with an Entra application or managed identity via the OIDC-federation mechanism: the Pod’s projected, audience-scoped ServiceAccount token is exchanged for an Azure access token, with no long-lived secret stored in the cluster (Azure — Workload ID overview). This is Azure’s instance of the cross-cloud pattern in Workload Identity (Kubernetes) — the direct analog of GKE Workload Identity and EKS IRSA.

Autoscaling — Cluster Autoscaler and Node Auto-Provisioning (NAP)

AKS supports two node-scaling models:

  • The classic Cluster Autoscaler — per-node-pool min/max bounds; adds/removes VMSS instances when Pods are unschedulable or nodes are idle. You still pre-decide the VM SKU per pool.
  • Node Auto-Provisioning (NAP) — Azure’s managed deployment of Karpenter, built on the open-source karpenter-provider-azure (GitHub — Azure/karpenter-provider-azure). Instead of pre-defined node pools with fixed SKUs, NAP looks at pending Pods’ resource requests and provisions the right VM size on demand, consolidating underutilized nodes. NAP requires Azure CNI Overlay and managed identity, and is the autoscaling model baked into AKS Automatic (Azure — NAP overview).

AKS Automatic

AKS Automatic reached general availability in October 2025 (InfoQ — GA announcement) and is exposed in the API as a distinct cluster SKU — --sku automatic rather than --sku base. It is AKS’s opinionated, “production-ready by default” mode — Azure’s answer to GKE Autopilot. An Automatic cluster ships with preselected, best-practice defaults locked in: Azure CNI Overlay Powered by Cilium networking, Azure Linux node images on the system pool, NAP for node management, Azure RBAC for Kubernetes authorization with Microsoft Entra ID, Microsoft Entra Workload ID and the OIDC issuer pre-enabled, managed NGINX ingress via the application routing add-on, a managed NAT gateway for egress, Managed Prometheus, Container Insights, and Deployment Safeguards enforcing Kubernetes best practices via Azure Policy in unconditional enforcement mode (AKS Automatic overview). The Automatic SKU is locked to the Standard pricing tier, so the Uptime SLA is included by default; the Premium tier (with LTS) is not available on Automatic clusters.

Automatic uniquely adds a financially-backed Pod Readiness SLA: Microsoft guarantees that 99.9 % of qualifying Pod-readiness operations complete within five minutes, covering Pod scheduling and node provisioning together. This is the first hyperscaler SLA pegged to workload-level rather than control-plane latency, and it is exclusive to Automatic — standard AKS does not offer it. Cluster upgrades on Automatic run on the stable channel (always N-1 where N is the latest supported minor), node-OS upgrades on the NodeImage channel, and Automatic enforces the node-resource-group-lockdown so users cannot tamper with the MC_* resource group that holds the VMSS and supporting Azure resources.

On the system-pool side, AKS Automatic is moving toward a fully-hosted model. As of late 2025 the option to have Microsoft host the system node pool on Microsoft-owned infrastructure — so the customer is not billed for the system VMs and never sees them — exists as a feature called “AKS Automatic with managed system node pools,” but it remains a preview capability, layered on top of the GA base of AKS Automatic (AKS Engineering Blog, Nov 2025). Without that opt-in, a default Automatic cluster still provisions a system pool of three nodes (sized at 4 vCPU each, dynamically chosen from a small approved D-series allow-list) in the customer subscription, billed at standard VM rates — visible as aks-nodepool1-* in kubectl get nodes. User-pool VMs are always billed to the customer at standard pay-as-you-go VM rates regardless of preview opt-in.

Uncertain

AKS Automatic managed system node pools were still in preview as of the last published AKS-team blog post (26 Nov 2025) and the verified intro-aks-automatic docs page (revised April 2026) lists them as an Optional preview capability. No GA announcement has been published between then and 2026-05-30. Whether Microsoft additionally levies any per-vCPU or per-cluster surcharge specific to AKS Automatic (a model some preview material has alluded to) is not confirmed by the current free-standard-pricing-tiers page, which describes Automatic-SKU costs as standard Standard-tier cluster-management + pay-as-you-go VMs. Resolve by checking the live Azure pricing calculator and the most recent AKS release notes before quoting Automatic-specific surcharges to a customer.

Configuration / API Surface

# 1. Create a Standard-tier AKS cluster with Azure CNI Overlay + Cilium and
#    Microsoft Entra ID + Workload Identity integration.
az aks create \
  --resource-group rg-prod \
  --name aks-prod \
  --tier standard \                       # financially-backed Uptime SLA
  --kubernetes-version 1.32.3 \
  --network-plugin azure \                # Azure CNI...
  --network-plugin-mode overlay \         # ...in Overlay mode (private Pod CIDR)
  --network-dataplane cilium \            # eBPF data plane (replaces kube-proxy)
  --enable-aad \                          # Microsoft Entra ID authentication
  --enable-azure-rbac \                   # Entra-backed RBAC
  --enable-oidc-issuer \                  # required for Workload Identity
  --enable-workload-identity \            # federate ServiceAccounts -> Entra
  --node-count 3 \                        # the initial SYSTEM pool size
  --node-vm-size Standard_D4s_v5
 
# 2. Add a dedicated USER node pool for memory-heavy workloads.
az aks nodepool add \
  --resource-group rg-prod --cluster-name aks-prod \
  --name mempool --mode User \            # User mode = application workloads
  --node-vm-size Standard_E8s_v5 \
  --labels workload=memory \
  --node-taints workload=memory:NoSchedule \
  --enable-cluster-autoscaler --min-count 0 --max-count 10
 
# 3. Alternatively, skip pool sizing entirely — enable Node Auto-Provisioning.
az aks update \
  --resource-group rg-prod --name aks-prod \
  --node-provisioning-mode Auto          # NAP (Karpenter) picks VM sizes itself
 
# 4. Provision an entire AKS Automatic cluster — opinionated defaults, one command.
az aks create \
  --resource-group rg-prod --name aks-auto \
  --sku automatic                         # AKS Automatic: CNI+Cilium, NAP, ingress preset

Line-by-line. --tier standard is the line that buys the SLA — omit it and you silently get the Free tier with no contractual uptime. --network-plugin-mode overlay plus --network-dataplane cilium is the recommended modern stack and a prerequisite for --node-provisioning-mode Auto (NAP needs Overlay). --enable-oidc-issuer exposes the cluster’s OIDC discovery endpoint, without which --enable-workload-identity cannot federate. In step 4, --sku automatic collapses steps 1–3 into the AKS Automatic preset and additionally forces Standard tier, the application-routing ingress add-on, NAT-gateway egress, deployment safeguards, Azure RBAC, and the Pod-readiness SLA — you no longer choose the CNI mode, the data plane, the autoscaler, the ingress controller, or the upgrade channel; AKS picks them.

Failure Modes

Free tier chosen unintentionally. Omitting --tier standard defaults to Free — no API-server SLA. Teams discover this only during an incident review when they have no contractual recourse for control-plane downtime. Always set the tier explicitly for production.

System node pool exhaustion. If the system pool is sized minimally and an add-on (CoreDNS, metrics-server) cannot schedule, cluster DNS and kubectl top break cluster-wide. Symptom: Pods fail name resolution, HPA stops scaling. Keep the system pool with headroom, or use AKS Automatic’s managed system pool.

VNet IP exhaustion with classic Azure CNI. Classic Azure CNI (Pod Subnet) burns one VNet IP per Pod. Large clusters exhaust the delegated subnet and new Pods stay Pending with no node-level error — the failure is in IPAM. Migrating to Azure CNI Overlay is the fix; sizing the subnet generously up front is the prevention.

NAP capacity stranding. NAP requests VM SKUs on demand; if the chosen SKU is unavailable in the region/zone, Pods stay Pending while NAP retries. Azure’s December 2025 NAP capacity guidance recommends configuring multiple acceptable SKU families so NAP has fallbacks.

Entra ID outage blocks kubectl. When the API server delegates authentication to Entra ID, an Entra outage means humans cannot obtain tokens and kubectl access fails — even though workloads keep running. Keep a break-glass non-Entra cluster-admin credential (a client certificate) for emergencies.

Kubernetes version EOL on non-Premium tiers. Standard tier follows the ~14-month community support window; once a minor version ages out, AKS force-upgrades. Teams that need to pin a version longer must be on Premium tier with LTS, or face an unplanned upgrade.

Alternatives and When to Choose Them

Amazon EKS — AWS’s managed Kubernetes. Choose when the organization is AWS-centric; EKS integrates with IAM (IRSA), VPC (AWS VPC CNI), and ALB. EKS has no free tier — every cluster pays the control-plane fee.

Google GKE — the most Kubernetes-native managed service (Google originated Kubernetes). GKE Autopilot is the closest analog to AKS Automatic; both abstract node management entirely. Choose GKE for the deepest upstream-feature fidelity and earliest access to new Kubernetes capabilities.

AKS Automatic vs standard AKS. Choose Automatic when you want Azure to make the platform decisions and you value a production-ready default over fine-grained control. Choose standard AKS when you need to pin specific CNI modes, custom node images, specialized hardware, or non-default add-ons that the opinionated Automatic preset doesn’t expose.

Self-Managed Kubernetes on Azure VMs — running your own control plane on Azure IaaS via kubeadm or kubespray. Choose only for regulatory isolation, exotic component flags, or air-gapped requirements; for almost everyone the per-cluster fee is dwarfed by the operational savings (see the decision framework in Managed Kubernetes Services).

Production Notes

Microsoft’s AKS Engineering Blog publishes operational guidance regularly. The December 2025 post on NAP capacity management documents the real-world failure mode of single-SKU NAP configurations stranding capacity, and recommends broad SKU-family allow-lists — a concrete lesson for anyone adopting NAP or AKS Automatic.

AKS Automatic’s GA in October 2025 (InfoQ) places Azure roughly at parity with GKE Autopilot’s “node-less” managed experience. The strategic read: all three hyperscalers are converging on a model where the cluster is a hosted API surface and the customer thinks in Pods, not nodes — the endpoint of the trajectory described in Kubernetes Cluster Architecture’s discussion of GKE Autopilot.

The Microsoft Entra ID integration is AKS’s strongest differentiator for enterprises already standardized on Azure AD/Entra — cluster access is governed by the same directory, conditional-access policies, and audit trail as the rest of the Microsoft estate, which is an operational and compliance win that EKS (IAM) and GKE (Google IAM) match only for organizations standardized on their clouds.

See Also

Graph View

Backlinks