vault

Tunnel and Overlay Interfaces

15 min read

Tunnel and Overlay Interfaces

A tunnel interface is a virtual network device that, instead of putting frames on a wire, encapsulates each packet inside an outer header and ships it across an existing IP network to a peer, which strips the outer header and re-injects the inner packet. An overlay network is the virtual L2/L3 topology this builds: a logical network whose nodes can be anywhere on the underlying physical fabric, decoupled from it. The canonical modern overlay encapsulation is VXLAN (Virtual eXtensible Local Area Network, RFC 7348), which carries Ethernet frames over UDP and identifies up to ~16 million isolated segments with a 24-bit VXLAN Network Identifier (VNI); its extensible successor is GENEVE (Generic Network Virtualization Encapsulation, RFC 8926). Linux implements these as kernel drivers (drivers/net/vxlan/, drivers/net/geneve.c) plus the older GRE, IPIP, and SIT IP-in-IP tunnels, all created through ip link add ... type {vxlan|geneve|gre|ipip|sit}. Overlays exist for one reason: to decouple the virtual network from the physical topology — so a container or VM keeps its address and its L2 adjacency no matter which physical host it runs on, and tenants are isolated even though they share one underlay.

This note covers the kernel mechanism and the encapsulation formats, pinned to Linux 6.12 LTS source. The orchestration that drives these — Flannel’s and Calico’s VXLAN modes, Cilium’s/OVN’s GENEVE dataplane — consumes these primitives via a CNI plugin and is owned by Kubernetes MOC; this note stands alone.

Mental Model — A Virtual Network Painted Over a Physical One

The underlay is the real IP network between hosts. The overlay is a set of tunnel endpoints — for VXLAN these are VTEPs (VXLAN Tunnel End Points, RFC 7348 §1) — that each run on a host and present a virtual interface to the local workloads. When a workload sends a frame, its VTEP wraps it (inner Ethernet → VXLAN → UDP → outer IP) and unicasts it across the underlay to the VTEP on the destination host, which unwraps and delivers it. From the workload’s point of view it is on a flat LAN; from the underlay’s point of view it is just carrying ordinary UDP packets between two host IPs.

flowchart LR
  subgraph H1["Host A (underlay IP 10.0.0.1)"]
    C1["Container A1<br/>10.244.1.5"] --> V1["VTEP vxlan0<br/>VNI 100"]
  end
  subgraph UNDER["Physical underlay (any L3 network)"]
    NET["routers / switches<br/>see only UDP:4789 between host IPs"]
  end
  subgraph H2["Host B (underlay IP 10.0.0.2)"]
    V2["VTEP vxlan0<br/>VNI 100"] --> C2["Container B1<br/>10.244.2.7"]
  end
  V1 -->|"encap: [outer IP 10.0.0.1→.2][UDP][VXLAN VNI=100][inner Eth A1→B1]"| NET
  NET --> V2

A VXLAN overlay over an arbitrary IP underlay. What it shows: the inner frame (container A1 → B1, addresses 10.244.x) is wrapped by host A’s VTEP in an outer header addressed host-to-host (10.0.0.1 → 10.0.0.2, UDP port 4789) and routed across the physical network as plain UDP; host B’s VTEP strips it and delivers the original frame. The insight: the underlay needs only IP reachability between hosts and never sees the overlay addresses — so the virtual network’s topology (which container is on which host) is completely decoupled from the physical one, which is exactly what lets pods migrate between nodes without renumbering.

Why Overlays Exist

In a physical datacenter, two machines that must share an L2 broadcast domain have to be on the same VLAN, which means the physical switches between them must trunk that VLAN — the virtual topology is welded to the physical one. 802.1Q VLANs also cap out at 4094 segments (a 12-bit tag), too few for multi-tenant clouds. Overlays break both constraints. By tunneling the inner frame inside an outer IP/UDP packet, any two hosts with mere IP reachability can host endpoints of the same virtual segment — no VLAN trunking, no shared L2 underlay required. And the overlay’s own segment identifier is much larger: VXLAN’s 24-bit VNI gives ~16 million segments (RFC 7348 §1: “Each VXLAN segment is identified through a 24-bit segment ID, termed the VXLAN Network Identifier (VNI). This allows up to 16 M VXLAN segments…”). For containers this is the enabling trick: a pod keeps 10.244.x.y and its neighbours regardless of which node it lands on, because the node-to-node path is a tunnel, not a route the pod can see.

VXLAN — Ethernet over UDP

The encapsulation

VXLAN prepends an 8-byte VXLAN header plus a UDP header plus an outer IP header to the original Ethernet frame. The kernel constants pin the format exactly (include/net/vxlan.h):

#define IANA_VXLAN_UDP_PORT     4789                       // standard dest UDP port
#define VXLAN_N_VID     (1u << 24)                          // 2^24 = 16,777,216 VNIs
#define VXLAN_VID_MASK  (VXLAN_N_VID - 1)                   // 24-bit VNI mask
#define VXLAN_HLEN (sizeof(struct udphdr) + sizeof(struct vxlanhdr))  // 8 + 8 = 16 bytes
#define VXLAN_HF_VNI  cpu_to_be32(BIT(27))                  // "VNI present" flag bit

The struct vxlanhdr is two 32-bit words: a flags word (with the VXLAN_HF_VNI bit set to indicate a valid VNI) and the VNI itself in the top 24 bits of the second word. On transmit, vxlan_build_skb() (vxlan_core.c) reserves headroom (min_headroom = ... + VXLAN_HLEN + iphdr_len), pushes the VXLAN header, and sets the VNI:

vxh = __skb_push(skb, sizeof(*vxh));
vxh->vx_flags = VXLAN_HF_VNI;          // mark VNI valid
vxh->vx_vni = vxlan_vni_field(vni);    // pack the 24-bit VNI into the field

After this the UDP and outer-IP headers are prepended by the IP-tunnel helpers, and the packet is sent across the underlay. The total per-packet overhead for VXLAN over IPv4 is 50 bytes (14 outer Ethernet + 20 outer IPv4 + 8 UDP + 8 VXLAN) — the headline number behind the MTU concern below.

Flood-and-learn vs. control-plane modes

How does a VTEP know which remote host holds a given inner MAC? VXLAN has a per-device FDB (separate from a Linux-bridge FDB) mapping inner MAC → remote VTEP IP. Two ways populate it. The original RFC 7348 model is multicast flood-and-learn: each VNI maps to an IP multicast group; broadcast/unknown-unicast/multicast inner frames are sent to that group (“multicast frames and ‘unknown MAC destination’ frames are also sent using the multicast tree”), and a VTEP learns the inner-MAC→remote-VTEP-IP mapping from the outer source of frames it receives (gated by VXLAN_F_LEARN in the kernel, vxlan_core.c). This requires multicast in the underlay, which most cloud fabrics lack. The modern alternative is a control plane that programs the FDB directly — either a routing protocol (EVPN/BGP) or, in container land, a daemon that watches the cluster and adds static FDB entries (bridge fdb append ... dst <remote-host-IP>). Flannel’s VXLAN backend and Calico-VXLAN do exactly this: no multicast, just unicast tunnels with FDB entries pushed by the control plane. The kernel also supports a head-end-replication / “default remote” mode (remote IPADDR) where unknown traffic is unicast to one peer.

Metadata / external mode

A single VXLAN device per VNI does not scale to thousands of tenants. The kernel’s metadata mode (external flag, also called “collect metadata” / IP_TUNNEL_INFO) creates one VXLAN device that carries all VNIs, with per-packet tunnel metadata (VNI, remote IP) attached out-of-band — set on egress by a eBPF program or conntrack/route, and read on ingress. This is how OVS and Cilium drive a single tunnel device for an entire dataplane. You see it in the man page as [ no ] external for type vxlan (ip-link(8)).

# Unicast (no-multicast) VXLAN device, the container-overlay style:
ip link add vxlan0 type vxlan id 100 dstport 4789 dev eth0 local 10.0.0.1 nolearning
ip link set vxlan0 up
ip addr add 10.244.1.1/24 dev vxlan0
# Control plane pushes one FDB entry per remote host (no multicast group needed):
bridge fdb append 00:00:00:00:00:00 dev vxlan0 dst 10.0.0.2   # default-dest to host B

Line-by-line: id 100 is the 24-bit VNI; dstport 4789 is the IANA UDP port (the kernel default if omitted is also 4789); dev eth0 is the underlay egress device; local 10.0.0.1 is this host’s VTEP source IP; nolearning disables data-plane MAC learning because the control plane owns the FDB. The bridge fdb append line adds an all-zeros (default) entry sending unknown traffic to host B’s underlay IP — repeated per remote host this builds full mesh reachability without multicast.

GENEVE — The Extensible Successor

GENEVE was designed to fix VXLAN’s rigidity: its header carries a base part plus a variable-length list of type-length-value options, so new metadata can be added without inventing a new encapsulation each time. RFC 8926 motivates it directly — fixed 24-bit identifiers “start to look small” for some uses, and the proliferation of encapsulations (VXLAN, NVGRE, STT) showed the need for one extensible format. The kernel struct genevehdr (include/net/geneve.h) has a 2-bit ver, a 6-bit opt_len (length of the options in 4-byte units), oam/critical flags, a 16-bit proto_type, a 3-byte (24-bit) vni, and a variable options[] trailer. Its standard UDP port is 6081 (GENEVE_UDP_PORT, geneve.h), and the fixed base overhead is GENEVE_BASE_HLEN = sizeof(udphdr) + sizeof(genevehdr) = 8 + 8 = 16 bytes before options (geneve.c).

ip link add gnv0 type geneve id 200 remote 10.0.0.2 dstport 6081

Here id 200 is the 24-bit GENEVE VNI and remote is the peer VTEP IP. GENEVE is the encapsulation of choice for OVN (Open Virtual Network) and Cilium’s tunnel mode precisely because the option fields let those dataplanes carry their own metadata (logical port IDs, security identity) inline. The trade-off versus VXLAN is slightly larger and variable header size and marginally less ubiquitous NIC hardware offload, though modern NICs handle both.

GRE, IPIP, and SIT — The Classic IP-in-IP Tunnels

Older and simpler than the UDP overlays, these encapsulate one IP packet directly inside another (no UDP layer):

  • GRE (Generic Routing Encapsulation): a thin protocol-agnostic header (IP protocol 47) that can carry any L3 payload, with optional keys, sequence numbers, and checksums. The kernel module describes itself as “IPv4 GRE tunnels over IP library” and registers the gre, gretap, and erspan link types (ip_gre.c). gre carries L3 (IP-in-GRE-in-IP); gretap carries L2 Ethernet frames (the GRE equivalent of a VXLAN/bridgeable tunnel). GRE is widely used for site-to-site links and as the transport under some VPNs.
  • IPIP (IP-in-IP, RFC 2003): the minimal case — an IPv4 packet inside an IPv4 packet, IP protocol 4. The kernel calls it the “IP/IP protocol decoder library” (ipip.c). Lowest overhead (just a 20-byte outer IPv4 header), but carries only IPv4 and offers no multiplexing identifier. Calico uses IPIP as one of its inter-node encapsulation options.
  • SIT (Simple Internet Transition, “6in4”): IPv6 packets inside IPv4 (IP protocol 41), the classic mechanism for carrying IPv6 across an IPv4-only network, including ISATAP support (sit.c). For IPv6-over-IPv6, the analogous device is ip6tnl (net/ipv6/ip6_tunnel.c).
ip link add gre1 type gre  remote 203.0.113.2 local 198.51.100.1 ttl 64
ip link add ipip1 type ipip remote 203.0.113.2 local 198.51.100.1
ip link add tun6 type sit  remote 203.0.113.2 local 198.51.100.1   # 6in4

Metadata mode and FOU

Like VXLAN/GENEVE, the IP tunnels support collect-metadata (external) mode — a single tunnel device whose remote endpoint is set per-packet by a route or eBPF program (tunnel->collect_md in ip_gre.c/ipip.c) — used for dynamic, scalable tunnel meshes without one device per peer. FOU (Foo-over-UDP) and its variant GUE (Generic UDP Encapsulation) wrap a GRE or IPIP tunnel’s payload inside a UDP header (fou_core.c); you see it in the man page as encap { fou | gue | none } on type gre (ip-link(8)). The point of FOU is that a UDP outer header lets middleboxes and NICs hash on the UDP source port for ECMP load-balancing and RSS, which raw IP-protocol-47/4 packets defeat — and it traverses NAT/firewalls that only pass UDP.

WireGuard — The Encrypted Tunnel

For completeness: WireGuard is an in-kernel encrypted tunnel (drivers/net/wireguard/, module described as “WireGuard secure network tunnel”, main.c, in mainline since 5.6). It is a wg-type virtual interface that encapsulates inner IP packets inside authenticated-encrypted UDP, keyed by peer public keys. Unlike VXLAN/GENEVE it is L3-only, point-to-point-per-peer, and its purpose is confidentiality/integrity rather than tenant multiplexing — but mechanically it is the same shape (a tunnel netdev that encaps over UDP). It is increasingly used as a CNI dataplane for encrypted pod-to-pod traffic (Cilium and Calico both offer WireGuard transparent encryption).

The MTU and Overhead Problem

Every tunnel adds an outer header, so the bytes available for the inner packet shrink. A 1500-byte underlay MTU minus VXLAN’s 50-byte overhead leaves a 1450-byte inner MTU; GENEVE with options or GRE-with-key differ slightly. If the inner stack still thinks it can send 1500-byte packets, the encapsulated frame exceeds the underlay MTU and must be fragmented — and RFC 7348 §4.3 is blunt: “VTEPs MUST NOT fragment VXLAN packets,” and a receiver MAY silently discard fragments. The result is the single most common overlay failure: small packets (pings, TCP handshakes) work, large ones (a file transfer, a TLS certificate) silently black-hole. The fixes are to lower the inner interface MTU to account for the overhead (e.g. set the pod/VXLAN MTU to 1450), enable Path MTU Discovery end-to-end, or raise the underlay MTU to jumbo frames (9000) so the overhead fits comfortably — the standard recommendation for VXLAN fabrics.

Uncertain

Verify: the exact per-encapsulation overhead and resulting recommended inner MTU for GENEVE-with-options and for GRE variants on 6.12, and whether the kernel auto-derives the tunnel device MTU from the parent in each case. Reason: VXLAN’s 50-byte/1450-MTU figure is well established and traced to the header sizes above, but GENEVE’s variable options and GRE’s optional key/seq/csum fields make the overhead configuration-dependent, and I did not exhaustively trace each driver’s MTU-derivation in the 6.12 source. To resolve: read each driver’s *_change_mtu/max_mtu logic (e.g. dev->max_mtu = IP_MAX_MTU - GENEVE_BASE_HLEN - hard_header_len is visible in geneve.c) and confirm against docs.kernel.org/networking. uncertain

Failure Modes

  • Large-packet black-hole (MTU). As above — the defining overlay bug. Diagnose with ping -M do -s 1472 (forces no-fragment at the inner-MTU boundary) and watch for “message too long.”
  • No multicast in the underlay. A flood-and-learn VXLAN device with a multicast group silently fails to learn remote MACs if the cloud fabric blocks multicast (most do). Use nolearning + control-plane FDB entries (unicast) instead.
  • Outer checksum / offload mismatches. UDP-tunnel checksum offload bugs on some NICs corrupt encapsulated traffic; udpcsum/udp6zerocsumtx flags (ip-link(8)) and NIC firmware matter. Symptom: tunneled TCP stalls while ping works.
  • Asymmetric VNI/port config. Both ends must agree on VNI and dstport; a VNI mismatch means frames arrive and are silently dropped because no local device claims that VNI.
  • ECMP polarization with raw GRE/IPIP. Because raw IP-protocol tunnels have no UDP source-port entropy, all traffic between two hosts hashes to one underlay path, wasting parallel links. FOU/GUE or VXLAN/GENEVE (which vary the UDP source port per flow — src_port in vxlan_core.c) fix this.

Alternatives and When to Choose Them

  • VXLAN — choose when you need an L2 overlay with broad hardware/NIC offload support and ecosystem maturity; the default for Flannel and many CNI plugins. 24-bit VNI, fixed header.
  • GENEVE — choose when the dataplane needs to carry its own metadata inline (OVN, Cilium); extensible TLV options at the cost of variable header size.
  • GRE/gretap — choose for simple site-to-site L3 (or L2 via gretap) links, router interop, and where no UDP/tenant multiplexing is needed.
  • IPIP / SIT — choose for minimal-overhead single-protocol tunneling (IPIP for IPv4-in-IPv4, e.g. Calico IPIP mode; SIT for legacy 6in4 transition).
  • WireGuard — choose when the requirement is encryption, not multi-tenancy; pair with another overlay (or use directly) for encrypted pod traffic.
  • No tunnel at all (pure routing). If hosts share L3 reachability and you can program routes, a routed CNI (Calico BGP) avoids encapsulation overhead entirely. Choose an overlay when you need L2 adjacency or cannot inject routes into the underlay.

Production Notes

Overlays are the backbone of multi-host container networking. Flannel’s default backend is VXLAN with a control plane that programs the kernel FDB over unicast (no multicast); Calico offers VXLAN and IPIP encapsulation modes (plus unencapsulated BGP routing when the underlay permits); Cilium and OVN lean on GENEVE for its metadata extensibility (CNI view in Kubernetes MOC). The recurring operational lesson is MTU discipline — nearly every “intermittent connection reset / hung kubectl exec” overlay incident traces to encapsulation overhead not accounted for in the pod MTU. For performance, the per-packet encap/decap and the extra checksum work add CPU cost; production fabrics offload VXLAN/GENEVE to the NIC where possible, use jumbo frames in the underlay, and increasingly move the encap logic into XDP with collect-metadata tunnels so one device serves the whole dataplane.

See Also

Graph View

Backlinks