vault

LoadBalancer Service

17 min read

LoadBalancer Service

A LoadBalancer Service is a Service of type: LoadBalancer that asks the cloud-controller-manager to provision an external load balancer in front of the Service’s NodePort (or, in newer modes, directly in front of the backing Pod IPs). It is the canonical mechanism for exposing a workload to the internet on managed Kubernetes — EKS, GKE, AKS, OKE, DOKS — and the mechanism every “expose this thing” tutorial on the internet ends up touching. Mechanically, a type: LoadBalancer Service is a NodePort plus an asynchronous side effect: when you kubectl apply the manifest, the apiserver allocates the ClusterIP and the NodePort synchronously, but the actual cloud LB is provisioned by the cloud-controller-manager’s service controller in the background, which eventually writes the LB’s public endpoint into status.loadBalancer.ingress[] (kubernetes.io — type=LoadBalancer). The contract is loose: the spec field is type: LoadBalancer, the implementation is delegated entirely to the cloud, and the per-cloud knobs are exposed via annotations — service.beta.kubernetes.io/aws-load-balancer-* on EKS, networking.gke.io/* on GKE, service.beta.kubernetes.io/azure-load-balancer-* on AKS. KEP-1959 added spec.loadBalancerClass (KEP-1959) — GA in K8s 1.24 — so multiple LB implementations can coexist in one cluster (the in-tree cloud LB controller alongside AWS Load Balancer Controller, MetalLB, etc.). Cost reality: each LoadBalancer Service typically provisions its own cloud LB, billed per-hour plus per-GB, which makes “twenty microservices each exposed via type=LoadBalancer” the canonical financial-disaster pattern in managed K8s. The mitigations are Ingress (one LB serves many Services) and Gateway API (the modern successor).

Mental Model

flowchart TB
    USER[Manifest<br/>type: LoadBalancer<br/>+ annotations]
    APISERVER[(kube-apiserver)]
    CCM[cloud-controller-manager<br/>service controller]
    CLOUD[Cloud LB API<br/>AWS ELBv2 / GCLB / Azure LB]
    LB[Provisioned LB<br/>public DNS or static IP]
    subgraph "Worker nodes"
        N1[Node 1<br/>:31234 → kube-proxy DNAT]
        N2[Node 2<br/>:31234 → kube-proxy DNAT]
        N3[Node 3<br/>:31234 → kube-proxy DNAT]
    end
    PODS[Backing Pods<br/>app=foo, port 8080]

    USER -- "kubectl apply" --> APISERVER
    APISERVER -- "1. allocate ClusterIP<br/>2. allocate NodePort 31234<br/>3. expose status as empty" --> APISERVER
    APISERVER -- "watch Service<br/>type=LoadBalancer" --> CCM
    CCM -- "translate annotations" --> CLOUD
    CLOUD -- "provision" --> LB
    CCM -- "patch status.loadBalancer.ingress" --> APISERVER
    EXT[Internet client] -- "DNS → LB endpoint" --> LB
    LB -- "forward instance-mode: :31234 on every node" --> N1
    LB -.-> N2
    LB -.-> N3
    LB == "IP target mode: directly to Pod IPs" ==> PODS
    N1 -- "kube-proxy DNAT" --> PODS

What the diagram shows. Provisioning is a two-phase reconciliation: the apiserver does the synchronous part (allocate ClusterIP + NodePort, leaving status.loadBalancer.ingress empty), and the cloud-controller-manager’s service controller does the asynchronous part (talk to the cloud’s LB API, wait for the LB to come up, patch the status). The cloud LB itself can target the workload in one of two modes: instance mode, where it round-robins among node NodePorts (the original, default behaviour), or IP target mode, where it talks directly to Pod IPs and skips kube-proxy entirely (newer, controller-mediated — AWS NLB-IP, GCP container-native LB). The insight to extract: “LoadBalancer” is not a single mechanism; it’s a pluggable contract. The same Service spec can produce wildly different runtime topologies depending on which controller is reconciling it and which annotations are set. Two clusters in the same cloud can have very different LB behaviour from “identical” manifests.

Mechanical Walk-through

The provisioning sequence

  1. kubectl apply -f svc.yaml writes the Service to etcd.
  2. The apiserver validates type: LoadBalancer, allocates a free ClusterIP, allocates a free NodePort (unless spec.allocateLoadBalancerNodePorts: false), and writes the object back.
  3. The cloud-controller-manager’s service controller sees the new Service via watch. It reads the cluster’s nodes (the LB’s target set will be derived from them), the Service’s annotations (cloud-specific knobs), and translates everything into a cloud-LB API call.
  4. The cloud creates the LB. This is typically the slow step — 30 seconds to several minutes depending on cloud and LB type.
  5. The cloud-controller-manager patches status.loadBalancer.ingress[] with the LB’s public endpoint — either a hostname (AWS ELB/ALB return DNS names) or a static IP (GCP, Azure can return either).
  6. The cloud-controller-manager also keeps the LB’s target set in sync as nodes join/leave the cluster — re-running the reconcile loop on Node events. This is the part you most easily forget exists.

When you kubectl get svc -w, the row for the new Service initially shows EXTERNAL-IP: <pending>; once step 5 completes, the hostname or IP appears.

loadBalancerClass — multiple LB implementations per cluster

Before KEP-1959, a cluster could have exactly one LB implementation — the cloud’s in-tree provider (or whatever you’d configured the cloud-controller-manager to handle). KEP-1959 added spec.loadBalancerClass, GA in K8s 1.24 (KEP-1959), so a cluster can have several LB controllers and route each Service to the appropriate one:

  • loadBalancerClass: "" (or unset) → handled by the cluster’s default LB controller (typically the in-tree cloud provider).
  • loadBalancerClass: service.k8s.aws/nlb → handled by AWS Load Balancer Controller (NLB mode).
  • loadBalancerClass: example.com/internal → handled by a custom controller you wrote.

The matching controller watches Services whose loadBalancerClass it owns; other controllers ignore them. This is essential when running, e.g., a managed cluster with both the cloud provider’s default LB controller and an additional controller that handles a different LB type (MetalLB for internal, AWS LB Controller for external NLBs). loadBalancerClass is immutable after creation and, like type, cannot be added to or removed from an existing Service.

On EKS specifically, service.k8s.aws/nlb is the class the AWS Load Balancer Controller claims (supported since controller v2.4.0 for Kubernetes v1.22+). From controller v2.5.0 onward the controller installs a mutating webhook that stamps spec.loadBalancerClass: service.k8s.aws/nlb onto every new type: LoadBalancer Service automatically, making the controller the default LB provider for the cluster — so on a v2.5+ install you may see the class appear even when you didn’t set it (AWS LB Controller NLB guide). The default class string itself is configurable via the controller’s --load-balancer-class flag.

Per-cloud annotations — the real knobs

Most LB customisation happens via annotations, not Service spec fields. Each cloud’s LB controller defines its own annotation vocabulary; they are not portable across clouds. Examples:

AWS Load Balancer Controller (aws-load-balancer-controller annotations):

metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "external"        # use AWS LB Controller, not in-tree
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"   # talk to Pod IPs directly
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"  # vs "internal"
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:..."
    service.beta.kubernetes.io/aws-load-balancer-subnets: "subnet-a,subnet-b"
    service.beta.kubernetes.io/aws-load-balancer-attributes: "deletion_protection.enabled=true"

The type: "external" value is the most important one: it tells the in-tree cloud provider to ignore the Service and lets AWS Load Balancer Controller reconcile it instead. Combined with nlb-target-type: "ip", traffic goes directly to Pod IPs and skips NodePort entirely.

GKE uses a mix of annotations and the CRDs BackendConfig / FrontendConfig (GKE — LoadBalancer Services):

metadata:
  annotations:
    cloud.google.com/neg: '{"ingress": true}'   # container-native LB (Pod-IP target)
    cloud.google.com/backend-config: '{"default": "my-backend-config"}'
    networking.gke.io/load-balancer-type: "Internal"   # vs External

The cloud.google.com/neg annotation triggers Network Endpoint Group (NEG)-based routing — GKE’s equivalent of AWS NLB-IP target mode.

AKS uses annotations on the standard SKU LB (AKS — public load balancer):

metadata:
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    service.beta.kubernetes.io/azure-load-balancer-resource-group: "rg-aks"

Annotation contracts evolve faster than Kubernetes itself, so the values matter as much as the keys. As of the AWS Load Balancer Controller “latest” docs (v2.x, checked May 2026), service.beta.kubernetes.io/aws-load-balancer-type: "external" is the current value that hands the Service to the controller; the older "nlb-ip" value is documented as deprecated and kept only for backwards compatibility (per the AWS LB Controller annotations reference). aws-load-balancer-nlb-target-type accepts instance (route via NodePort) or ip (route to Pod IPs), defaulting to instance. Still: always check the per-cloud controller’s current docs before pasting these into a production manifest, because the vocabulary genuinely shifts across major versions.

Instance mode vs IP target mode

Instance mode is the original behaviour: the cloud LB targets every node by IP, on the Service’s NodePort. Traffic arrives at a node, hits kube-proxy, gets DNATed to a Pod. externalTrafficPolicy: Cluster is the default, with SNAT loss-of-client-IP; externalTrafficPolicy: Local is opt-in for preserving client IPs at the cost of uneven load. See NodePort Service for the gory detail.

IP target mode is the newer pattern: the LB targets Pod IPs directly. No NodePort, no kube-proxy, no SNAT loss-of-client-IP. The trade-off is that the LB must understand the Pod CIDR and reach Pod IPs — which it does on AWS (because VPC CNI puts Pods on routable secondary ENI IPs), on GCP (because GKE Network Endpoint Groups feed Pod IPs to the LB), and on Azure (CNI-based, Pod IPs on the VNet). Overlay-based CNIs (Calico VXLAN, Flannel) don’t usually support IP target mode because the LB can’t route to the overlay IPs.

IP target mode also means spec.allocateLoadBalancerNodePorts: false is appropriate — there’s nothing for the NodePort to do, so skipping its allocation saves a port and reduces kube-proxy rule churn.

Source IP preservation

In instance mode with externalTrafficPolicy: Cluster, the backend sees the node IP as source. With Local, it sees the client IP (at the cost of possible load imbalance). In IP target mode, the backend always sees the LB’s IP unless the LB supports source-IP preservation natively — AWS NLB does (TCP and UDP), AWS ALB does at the L7 layer via X-Forwarded-For, GCP NEG-based LBs preserve client IP, Azure LB does. So the recipe for true client-IP preservation in 2026 is usually “NLB-IP mode on AWS, NEG on GCP, or externalTrafficPolicy: Local with topology-aware Pod spread on overlay-CNI clusters.”

status.loadBalancer.ingress.ipMode — VIP vs Proxy

A subtler source-IP and traffic-path concern is the ipMode field on each status.loadBalancer.ingress[] entry, added because of a long-standing kube-proxy short-circuit problem. When a Pod inside the cluster connects to its own Service’s external LB IP, kube-proxy historically intercepted that traffic and routed it straight to a backend (a “VIP” assumption), bypassing the cloud LB entirely — which breaks any LB that performs TLS termination, PROXY protocol injection, or source-IP rewriting, because the in-cluster caller never actually traverses the LB. The ipMode field lets the cloud controller declare intent: VIP (the default and historical behaviour — kube-proxy may short-circuit to a backend) or Proxy (kube-proxy must send the packet to the LB IP and let the load balancer handle delivery). ipMode may be set only when ingress.ip is also set (it is meaningless for hostname-only LB endpoints like classic AWS ELB DNS names). The feature (KEP-1860) went alpha in Kubernetes 1.29, beta in 1.30, and GA in 1.32 (Kubernetes 1.29 Load Balancer IP Mode blog; feature-gates reference). If you run a cloud LB that must see in-cluster traffic (PROXY protocol, WAF, mTLS at the LB), confirm the cloud controller sets ipMode: Proxy; otherwise intra-cluster calls to the external IP silently skip the LB.

Bare metal: MetalLB and kube-vip

Managed clouds have a service controller built into their cloud-controller-manager. Bare-metal clusters don’t. Two projects fill the gap:

  • MetalLB (metallb.universe.tf) — a controller plus a speaker DaemonSet that announce Service IPs via BGP (real load balancing across nodes) or L2 ARP/NDP (one node owns the IP at a time, fail-over rather than load balancing). Functionally equivalent to type: LoadBalancer from the Service user’s perspective.
  • kube-vip — similar story; can also handle the control plane VIP for HA kubeadm setups.

Both work alongside kube-proxy: they get traffic to a node, kube-proxy then DNATs to a Pod. They are not eBPF replacements, they are LB controllers.

Configuration / API Surface

apiVersion: v1
kind: Service
metadata:
  name: payments-public
  namespace: prod
  annotations:
    # AWS LB Controller: external NLB targeting Pod IPs
    service.beta.kubernetes.io/aws-load-balancer-type: "external"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:..."
spec:
  type: LoadBalancer
  loadBalancerClass: "service.k8s.aws/nlb"      # opt into AWS LB Controller (post-1.24 GA)
  externalTrafficPolicy: Local                  # preserve client IP (works regardless of target mode
                                                #  on AWS NLB-IP because there's no SNAT step)
  allocateLoadBalancerNodePorts: false          # NLB-IP doesn't need NodePort
  selector:
    app: payments
  ports:
  - name: https
    protocol: TCP
    port: 443
    targetPort: 8443
  - name: http
    protocol: TCP
    port: 80
    targetPort: 8080
  loadBalancerSourceRanges:                     # optional firewall rules at the LB level
  - 203.0.113.0/24
status:
  loadBalancer:
    ingress:
    - hostname: a1b2c3d4e5f6-1234567890.us-east-1.elb.amazonaws.com
      # OR
      # ip: 35.123.45.67

Important fields:

  • loadBalancerClass — GA since K8s 1.24. Routes the Service to a specific LB controller. Leave unset to use the cluster default.
  • loadBalancerSourceRanges — CIDR allow-list applied at the cloud LB layer (not kube-proxy). Honoured by most cloud LBs; on AWS LB Controller it becomes a security-group rule. Don’t confuse with NetworkPolicy.
  • loadBalancerIP — deprecated since K8s 1.24 in favour of cloud-specific annotations. Used to be a way to pin a static IP; cloud-specific annotations are now the portable mechanism.
  • allocateLoadBalancerNodePorts — true by default (allocates a NodePort even if not needed). Set to false for IP target mode LBs.
  • status.loadBalancer.ingress — written by the cloud controller. Returns either hostname (AWS, because ELB DNS names are not stable IPs) or ip (GCP, Azure). Code that consumes this status must handle both. Each entry may also carry ipMode (VIP or Proxy, GA since K8s 1.32, settable only alongside ip) — see the source-IP section above for why Proxy matters when the LB does TLS termination or PROXY-protocol.

Failure Modes

EXTERNAL-IP: <pending> forever. The cloud controller can’t provision the LB. Causes: insufficient cloud IAM permissions (the cloud-controller-manager or AWS LB Controller’s IAM role lacks elasticloadbalancing:CreateLoadBalancer), the cloud’s LB quota exhausted, a missing required annotation (e.g. AWS LB Controller refuses to create NLB-IP without aws-load-balancer-type: external). Diagnose: kubectl describe svc <name> for events; logs of kube-controller-manager / cloud-controller-manager / aws-load-balancer-controller.

Leaked cloud LBs. Changing service.beta.kubernetes.io/aws-load-balancer-type after creation leaves the old LB live (because the old controller no longer reconciles the Service) and creates a new one (because the new controller does). You get two LBs billed per hour for the same Service. AWS LB Controller’s docs explicitly warn: “This annotation should not be modified after service creation. … If you modify the annotation after service creation you will end up with leaked AWS load balancer resources.” Always delete-then-recreate when changing LB type.

externalTrafficPolicy: Local with no local backends on the LB-chosen node. Same trap as bare NodePort, made worse because cloud LB health checks can lag the actual Pod-readiness state. The ProxyTerminatingEndpoints feature — which lets kube-proxy fall back to terminating-but-still-serving endpoints under externalTrafficPolicy: Local so a draining Pod keeps receiving traffic during graceful shutdown — was beta and default-on since K8s 1.26 and went GA in 1.28 (feature-gates reference). It smooths the terminating case; the no-backend-yet case (just-scaled-up new node) is still a window of dropped traffic until the LB health check passes.

Cost surprises. Each type: LoadBalancer provisions a separate cloud LB (25/month minimum each on AWS, similar on GCP/Azure). Twenty microservices each exposed as LoadBalancer is hundreds of dollars per month before any traffic. The “free” kubectl expose command becomes very expensive at scale. Mitigation: Ingress (one LB for many Services) or Gateway API.

Stale status.loadBalancer.ingress after cluster restore. A Velero-restored cluster has Services with the old cluster’s LB hostnames in status.loadBalancer.ingress, which is wrong — the new cluster’s LBs are different. Fix: re-reconcile by changing a benign field on the Service to force the cloud-controller-manager to update the status. Some Velero plugins handle this automatically.

Cross-AZ data charges. Instance-mode LBs cross-zone-load-balance by default, which sounds nice but means a request from a client in AZ-a hits a node in AZ-b and is forwarded to a Pod in AZ-c — two cross-AZ hops, both billed. Mitigation: service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "false" on AWS NLB (then balance ends up uneven if AZ Pod counts differ), or use IP target mode with topology-aware routing.

loadBalancerSourceRanges ignored. Not all cloud LBs honour this field; AWS NLB does (translates to security-group rules), AWS ALB does, GCP does for some LB types, Azure depends on SKU. If the field is silently ignored and you assumed your LB was firewalled, you’ve shipped a public endpoint when you thought it was restricted. Verify by trying to connect from an outside-range IP.

Health checks vs liveness probes. The cloud LB has its own health check, configured separately (or via annotations). A backend Pod with a passing readiness probe but a failing LB health check (because the LB is hitting a different port or path) appears “healthy” inside the cluster but receives no LB traffic. Symptom: LoadBalancer Service shows endpoints, but the LB has zero healthy targets. Always wire the LB health check to the same endpoint as the Pod’s readiness probe.

Alternatives and When to Choose Them

  • Ingress / Gateway API — when you have multiple HTTP/gRPC services to expose. One LB sits in front of an Ingress controller, which dispatches to many ClusterIP Services by host/path. Pays for itself in cloud-LB cost reduction after about three Services on most clouds.
  • AWS Load Balancer Controller in ALB mode for HTTP — even for a single Service, ALB has L7 features (path routing, OIDC integration, WAF) that NLB doesn’t. kind: Ingress plus the ALB controller is usually preferable to type: LoadBalancer for HTTP services on EKS.
  • type: NodePort plus an external HAProxy / nginx — on-prem clusters with an existing hardware LB. Functional, but requires manual LB config; MetalLB or kube-vip is usually a better automated answer.
  • MetalLB / kube-vip on bare metal — implement type: LoadBalancer semantics where there’s no cloud provider. BGP mode for real load balancing; L2 mode for fail-over.
  • hostNetwork: true with an ingress controller deployed as a DaemonSet — historic GKE/AKS pattern for L7 routing without a cloud LB. Rare in modern clusters.
  • Service mesh + east-west traffic — for in-cluster traffic, a service mesh (Istio, Linkerd) handles L7 routing without any LoadBalancer Services at all. LoadBalancer is only for ingress — getting traffic into the cluster from outside.

Production Notes

  • Always pair type: LoadBalancer with cost monitoring. The Kubecost / OpenCost dashboards specifically track per-Service LB cost. Spotify and Shopify engineering blogs have both documented the “spiral of LBs” pattern, where every team independently exposes a Service and the cluster ends up with hundreds.
  • Default to Ingress / Gateway API for HTTP. Reserve type: LoadBalancer for non-HTTP protocols (gRPC at the edge, TCP/UDP services like databases-as-a-service, MQTT brokers) where an L7 Ingress isn’t a fit.
  • Pin the cloud LB by annotation, not by loadBalancerIP. loadBalancerIP is deprecated since K8s 1.24; cloud-specific annotations (e.g. service.beta.kubernetes.io/aws-load-balancer-eip-allocations) are the supported way to attach a known address. The deprecation hasn’t yet been removed but production code shouldn’t add new uses.
  • Don’t share LBs across teams without a contract. A single LoadBalancer Service has one set of annotations, one Service-CIDR allocation, one health-check endpoint, one ACL. Sharing it across teams’ workloads creates change-control conflicts. Use Ingress with hostname-based routing to give each team its own logical surface backed by a shared LB.
  • EKS-specific: install AWS Load Balancer Controller. The in-tree service controller on AWS only supports CLB (classic) and NLB-instance modes; ALB and NLB-IP require the out-of-tree controller. The KEP-1959 loadBalancerClass field is what makes the coexistence safe.
  • GKE container-native LB (NEG) has been the recommended pattern for years; legacy “ingress through NodePort” routes are deprecated. NEG-based LBs are the GCP equivalent of AWS NLB-IP target mode and skip kube-proxy.
  • Annotation churn. AWS LB Controller went through aws-load-balancer-type: "nlb-ip" → "external" plus aws-load-balancer-nlb-target-type between v2.2 and v2.4. Treat any annotation example older than ~12 months as suspect; always validate against the current controller’s docs.
  • Pre-create the cloud LB and use a selectorless Service. For migration scenarios where the LB exists first (e.g. you cut DNS over to a new LB before the cluster exists), you can create the LB out-of-band and then create a selectorless Service with no type (or use loadBalancerClass set to a class no controller claims). This decouples the LB lifecycle from the Service lifecycle but loses the automated update on Pod changes.

See Also

Graph View

Backlinks