vault

NodePort Service

13 min read

NodePort Service

A NodePort Service is a Service that opens the same TCP/UDP/SCTP port on every node in the cluster, forwarding any traffic arriving at <any-node-ip>:<nodePort> to one of the Service’s backing Pods. It is the simplest mechanism for exposing a workload outside the cluster — it requires no cloud provider, no load balancer, no Ingress controller. The port comes from a small reserved range, 30000–32767 by default, configurable on the apiserver via --service-node-port-range (kube-apiserver reference). A NodePort Service is a superset of a ClusterIP Service: it gets a ClusterIP for free, and the NodePort is implemented by an additional set of kube-proxy rules that NAT external traffic into the internal ClusterIP chain. In production, NodePort is rarely used as the front door because (a) clients have to know the IPs of cluster nodes — which are not stable — and (b) the high-port range is awkward to put behind corporate firewalls and TLS terminators. Its real role is as a building block underneath type: LoadBalancer: cloud load balancers do not talk to ClusterIPs directly (the VIPs aren’t externally routable), they talk to NodePorts. This note covers the NodePort mechanics, the externalTrafficPolicy field that fundamentally changes how it behaves, and the alternatives that should usually replace it in front-line production use.

Mental Model

flowchart TB
    EXT["External client<br/>connects to 203.0.113.10:31234"]
    subgraph "Node 1 (203.0.113.10)"
        K1["kube-proxy rules"]
        P1A["Pod A: app=foo<br/>10.244.1.5"]
    end
    subgraph "Node 2 (203.0.113.11)"
        K2["kube-proxy rules"]
    end
    subgraph "Node 3 (203.0.113.12)"
        K3["kube-proxy rules"]
        P3A["Pod B: app=foo<br/>10.244.3.7"]
    end
    EXT -- "1. SYN to :31234" --> K1
    K1 -- "Cluster policy: SNAT src to node IP,<br/>DNAT to any backend, can cross nodes" --> P1A
    K1 -. "or" .-> P3A
    K1 == "Local policy: NO SNAT, only local backends" ==> P1A
    K1 -. "drops if no local backends" .- K2

What the diagram shows. An external client picks any node IP (203.0.113.10) and dials nodePort (31234). The kernel on that node — programmed by kube-proxy — has a rule matching this (local-ip, nodePort) pair. Under the default externalTrafficPolicy: Cluster, the rule SNATs the source to the node’s own IP (so the return packet comes back to this node and can be reverse-NATed) and DNATs to any backend Pod cluster-wide, possibly on a different node. Under externalTrafficPolicy: Local, the rule does not SNAT (the original client IP is preserved) and only DNATs to backends on the same node — dropping the connection if there are none locally. The insight to extract: NodePort opens a port on every node, even nodes that have no backing Pods for the Service. Whether traffic to a back­ends-empty node reaches any Pod depends on externalTrafficPolicy. Under Cluster it does (via cross-node forwarding); under Local it doesn’t. This single field is the source of more NodePort confusion than any other detail in the K8s networking stack.

Mechanical Walk-through

Port allocation

When spec.type: NodePort is set, the apiserver’s port allocator reserves a free port from the configured range. The default range is 30000–32767 — a 2 768-port window in the IANA “registered ports” / “dynamic ports” boundary, deliberately well above privileged ports (≤1023) and well below the typical ephemeral-port range used by node-local clients (/proc/sys/net/ipv4/ip_local_port_range, usually 32768–60999 on Linux). The combination avoids collisions with both (a) host services like SSHD on 22 or kubelet on 10250 and (b) ephemeral source ports of outgoing connections from the node (Baeldung — Why 30000–32767).

You can change the range via the apiserver flag:

kube-apiserver --service-node-port-range=20000-32767

Lower bounds below 1024 are accepted but require kube-proxy to have CAP_NET_BIND_SERVICE and tend to collide with privileged services; almost nobody does this in practice.

If spec.ports[].nodePort is unset, the allocator picks a free port from the range and writes it back. If it’s set to a specific value, the allocator checks that it’s in-range and free, otherwise rejects with Invalid value: provided port is already allocated. Like ClusterIP, the nodePort is immutable once allocated for a given Service (the apiserver rejects edits that change it).

The rules kube-proxy installs

On every node, kube-proxy installs three things (Virtual IPs reference):

  1. A rule matching incoming traffic to <any-local-ip>:<nodePort> (and, if localhostNodePorts: true, to 127.0.0.1:<nodePort> too).
  2. A jump from that rule into the same per-Service chain that the ClusterIP rule jumps into. The two paths converge — the per-Service backend-selection logic is shared.
  3. A masquerade / SNAT rule, conditional on externalTrafficPolicy.

So a NodePort Service is, mechanically, “a ClusterIP Service plus an extra entry rule on every node.” When traffic comes in on the NodePort, kube-proxy DNATs it through the same backend-selection process as ClusterIP traffic.

externalTrafficPolicy: Cluster (default) — load balanced, source IP lost

Under Cluster policy, kube-proxy applies MASQUERADE (SNAT to the node’s own IP) to NodePort-arriving traffic. The reason is mechanical: once kube-proxy DNATs the destination to a Pod that might live on a different node, the reply packet would otherwise return to the original external client via that other node, which knows nothing about the conntrack entry. SNAT’ing to the receiving node’s IP forces the reply to come back to the same node, which still has the conntrack and can reverse both NATs. The cost is that the backend Pod sees the source IP as the node IP, not the real external client. From the app’s perspective, every external connection appears to come from 203.0.113.10, 203.0.113.11, etc. — the node IPs.

externalTrafficPolicy: Local — source IP preserved, uneven balancing

Under Local policy, kube-proxy does not SNAT NodePort traffic and only DNATs to backends on the same node. The reply returns to the original client directly because the original source IP was preserved. The trade-off is twofold:

  • Uneven balancing. If 3 of 10 nodes have backing Pods and 7 do not, traffic arriving at the 7 lacks local backends and is dropped (RST). The receiving cloud LB or external client either gets a hard error or the LB’s health check eventually marks those nodes unhealthy. The 3 nodes with backends then absorb all the load.
  • Healthz on a separate port. kube-proxy exposes a healthCheckNodePort (a separate port in the 30000–32767 range, auto-allocated when externalTrafficPolicy: Local is set) that returns 200 OK only if the node has at least one local ready endpoint. Cloud LBs are supposed to use this for health-check routing, so they stop sending traffic to back­ends-empty nodes. Stand-alone NodePort consumers usually don’t know about this port and the load imbalance hits them.

Local is the right answer when (a) you need the real client IP for security logging, geo, or rate-limiting and (b) you can guarantee adequate replica spread across nodes (a DaemonSet-backed Service, or topologySpreadConstraints ensuring at least one Pod per node).

The K8s 1.26 ProxyTerminatingEndpoints feature (default on) softens a related issue: during a rolling update, a node whose only local backend has just started terminating would, under pre-1.26 behaviour, fail the healthCheckNodePort and lose traffic abruptly. Post-1.26, kube-proxy continues to route to terminating-but-still-ready endpoints so the LB has time to drain (v1.26 blog).

NodePort beneath LoadBalancer

When you create a type: LoadBalancer Service, Kubernetes also allocates a NodePort and a ClusterIP for it. The cloud LB created by the cloud-controller-manager is configured to forward to every node’s <nodePort>. From the cluster’s perspective, the LB is just an external traffic source hitting NodePort. Everything described above — externalTrafficPolicy, SNAT, healthCheckNodePort — applies identically. Setting spec.allocateLoadBalancerNodePorts: false (or the equivalent annotation, depending on cloud) skips the NodePort allocation for cases where the LB talks directly to Pod IPs (AWS NLB in IP target mode, ALB ingress controller — see LoadBalancer Service).

Configuration / API Surface

apiVersion: v1
kind: Service
metadata:
  name: webhook
  namespace: ops
spec:
  type: NodePort
  selector:
    app: webhook
  ports:
  - name: http
    protocol: TCP
    port: 80                                # ClusterIP-side port
    targetPort: 8080                        # Pod port
    nodePort: 31234                         # node-side port (range 30000–32767)
                                            # omit to auto-allocate
  externalTrafficPolicy: Local              # Cluster (default; SNAT, cross-node)
                                            #   vs Local (preserve src IP, local only)
  internalTrafficPolicy: Cluster            # orthogonal; in-cluster Pod-to-VIP traffic
  # healthCheckNodePort: 31235              # auto-assigned when externalTrafficPolicy=Local;
                                            # can be pinned in same range
status:
  loadBalancer: {}                          # empty for plain NodePort

Important details:

  • A NodePort Service still has a ClusterIP. Internal traffic to the ClusterIP works exactly like a ClusterIP Service.
  • Setting nodePort explicitly is a common ops pattern when you want a stable port baked into firewall rules or external monitoring. The downside is that pinning makes it impossible to deploy two copies of the manifest into the same cluster.
  • externalTrafficPolicy defaults to Cluster. Switching to Local is a deliberate trade — make it consciously.
  • healthCheckNodePort is only meaningful with externalTrafficPolicy: Local. It’s auto-allocated and almost never needs pinning; the cloud LB controllers know to query it.
  • On Linux a NodePort listens on 0.0.0.0 (and :: for dual-stack). If a node’s primary NIC is on a private network, the NodePort is still bound on every interface, so a public NIC on the same node would expose it to the internet. Firewall accordingly.

Failure Modes

Port range exhaustion. A cluster with thousands of NodePort Services (rare but possible — preview-env infrastructure, multi-tenant platforms) can run out of the 2 768-port window. Symptom: Invalid value … provided port is already allocated on Service creation. Fix: widen the range via --service-node-port-range and bounce the apiserver. Confirm: kubectl get svc -A -o jsonpath='{range .items[?(@.spec.type=="NodePort")]}{.spec.ports[*].nodePort}{"\n"}{end}' | sort -u | wc -l.

Collision with host services. If the apiserver range is changed to include low ports (e.g. 80, 443), kube-proxy will fail to bind on nodes where another process owns the port. Symptom: log entries like Listen tcp :80: bind: address already in use; kube-proxy will retry indefinitely. Fix: pick a range that doesn’t overlap with anything else running on the nodes.

externalTrafficPolicy: Local with zero local endpoints. A node with no matching Pods receives traffic (because the LB or external client doesn’t know to skip it) and either drops it or, more confusingly, the kube-proxy rule chain has nothing to DNAT it to and emits an RST. Symptom: customers see intermittent connection refusals, especially during rolling updates when Pods are migrating. Diagnose: kubectl get endpointslices -o wide and check the nodeName distribution; if it’s lopsided, either Cluster policy or better Pod spread is the fix.

Asymmetric routing on multi-NIC nodes. A node with two NICs (one cluster-facing, one external) accepts NodePort traffic on both. Under externalTrafficPolicy: Cluster, the SNAT-to-node-IP step uses whatever the kernel decides is the egress IP — which might be the wrong NIC. Symptom: TCP connections half-open or stalls during handshake. Diagnose: tcpdump -i any -nn 'port 31234'. Fix: explicit bindAddress in kube-proxy config or NIC-level routing rules.

Conntrack pinning of long-lived connections. A long-lived gRPC or WebSocket connection through a NodePort stays pinned to its original backend until the connection closes — adding new Pods doesn’t rebalance existing flows. Symptom: scaling up has no immediate effect. Fix: server-side connection age limits (MAX_CONNECTION_AGE in gRPC).

NodePort exposed to the internet by accident. Cloud security groups or node firewall rules sometimes default-open the 30000–32767 range to support cloud LB health checks. If the cluster runs on a public network and these rules are too permissive, NodePort Services are inadvertently exposed to the internet. CIS Kubernetes Benchmark flags this; default to a default-deny security group and let the cloud LB punch holes via tag-based rules.

LB health check timing. With externalTrafficPolicy: Cluster, a cloud LB’s health check sees 200 OK from every node (because kube-proxy can forward to any backend), even nodes with no local Pods. That’s the desired behaviour for Cluster, but it means the LB is blind to per-node Pod presence — fine for Cluster, dangerous for Local.

Alternatives and When to Choose Them

  • LoadBalancer Service — the answer for “expose to the internet” in managed K8s. Don’t use a bare NodePort externally; put a cloud LB in front. NodePort + a cloud LB is a LoadBalancer Service.
  • Ingress / Gateway API — when you have many HTTP services to expose under one or a few hostnames, terminate TLS centrally, and want path / host routing. Massively cheaper than one LoadBalancer per Service on managed clouds.
  • MetalLB / kube-vip (metallb.universe.tf) — on bare metal, MetalLB lets you implement type: LoadBalancer by announcing a virtual IP via BGP or L2. Internally MetalLB still terminates traffic on NodePorts, but the external surface is a single VIP rather than a list of node IPs. The right answer for on-prem clusters that need LB-style exposure.
  • hostNetwork: true Pods — bypasses the Service abstraction entirely; the container shares the node’s network namespace and listens directly on a node port. Use for system DaemonSets (CNI agents, log shippers) where the binding semantics matter. Don’t use for application workloads.
  • hostPort on a Pod container — exposes a single Pod’s port on the node it happens to be scheduled on. Static and fragile; useful only for very specific patterns (legacy GCP load balancer integration, edge-routing daemons).
  • NodePort with a manual external LB (HAProxy, F5) — on-prem clusters with an existing hardware LB sometimes do this: point the LB at all the cluster’s node IPs on a fixed NodePort. Functional and stable, but requires manual update of the LB config when nodes join/leave; MetalLB or kube-vip is usually a better automated answer.

Production Notes

  • Treat NodePort as plumbing, not as an interface. The right mental model is “a port that the cloud LB or your hardware LB talks to.” Engineers who treat the NodePort as the customer-facing surface invariably end up with brittle dependencies on node IPs.
  • externalTrafficPolicy: Local plus topology-aware replica spread is the durable pattern. When source-IP preservation matters (audit logging, geographic rate-limiting, IP allow-listing), use Local, and pair with topologySpreadConstraints to keep at least one Pod per node (or per zone) so the LB never lands on a backend-empty node.
  • Pin nodePort values for stable monitoring scrape paths. A Prometheus deployment that scrapes node-level kube-proxy metrics or a custom DaemonSet’s /metrics endpoint needs a stable port; pinning the NodePort makes this trivial and avoids re-discovering the port on every Service recreation.
  • Don’t widen the NodePort range to include 80/443. Tempting for “I just want to expose this on port 80”, but the consequences (collisions with host services, port-binding races on node restart) are severe. Use an Ingress controller listening on hostPort or hostNetwork, or a real LoadBalancer.
  • MetalLB at scale. Cloudflare and DigitalOcean engineering have written up MetalLB-on-bare-metal deployments; the common gotcha is that MetalLB’s L2 mode uses ARP gratuitous-reply to point an external IP at one specific node — which means it doesn’t load-balance across nodes, it just fails over. For real load balancing on bare metal, BGP mode is required, which means BGP-speaking switches. NodePort is the underlying primitive in both cases.
  • EKS Fargate cannot host NodePort Services backed by Fargate Pods. Fargate nodes don’t open arbitrary host ports. AWS LB Controller in IP target mode (which talks directly to Pod IPs, bypassing NodePort) is the workaround.

See Also

Graph View

Backlinks