vault

Anycast Routing

17 min read

Anycast Routing

Anycast is the practice of assigning the same IP address to many physically separate servers and letting the Internet’s own routing system deliver each client’s packets to whichever instance is “nearest” by the routing protocol’s measure of distance. RFC 4786 defines it as “the practice of making a particular Service Address available in multiple, discrete, autonomous locations, such that datagrams sent are routed to one of several available locations” (RFC 4786 §2). Nothing new is added to the packet: the address is an ordinary unicast address that happens to be originated from more than one place, so the routing fabric — BGP between networks, an interior gateway protocol within one — treats the multiple origins as competing routes to the same prefix and installs the best one per router. The payoff is that “the network itself will direct traffic to the nearest CloudFlare data center” (Cloudflare) with no application logic, no DNS trickery, and instant failover. The catch is that anycast delivers stateless request-response beautifully and long-lived stateful connections riskily, and understanding exactly why is the whole subject of this note.

This note is the protocol / design view of anycast. It cross-links but does not duplicate the kernel forwarding mechanism, which lives in IP Routing Decision and Forwarding and The Routing Subsystem and FIB, nor the inter-domain routing protocol itself, which is Border Gateway Protocol.

The Address Taxonomy — Where Anycast Fits

To place anycast you need the four ways an IP address can name a destination. RFC 4291, the IPv6 addressing architecture, gives the cleanest definitions, and they generalize to IPv4:

  • Unicast — “an identifier for a single interface. A packet sent to a unicast address is delivered to the interface identified by that address” (RFC 4291 §2). One address, one destination. This is 99% of Internet traffic.
  • Multicast — “an identifier for a set of interfaces… A packet sent to a multicast address is delivered to all interfaces identified by that address” (RFC 4291 §2). One-to-many, every member gets a copy.
  • Broadcast — one-to-all-on-a-subnet (IPv4 255.255.255.255 or the subnet’s all-ones address). IPv6 abolished it: “there are no broadcast addresses in IPv6, their function being superseded by multicast addresses” (RFC 4291 §2).
  • Anycast — “an identifier for a set of interfaces (typically belonging to different nodes). A packet sent to an anycast address is delivered to one of the interfaces identified by that address (the ‘nearest’ one, according to the routing protocols’ measure of distance)” (RFC 4291 §2.6).

The one-word summary: unicast is one-to-one, multicast is one-to-many-copies, anycast is one-to-one-of-many. Anycast and unicast are indistinguishable on the wire — the same 32-bit or 128-bit number, the same header. The difference is entirely in the control plane: an anycast address is announced into routing from multiple locations, and each router, seeing several equally-valid routes to the prefix, picks the best per its own policy and metrics. The sender never chooses which instance answers; the routing topology does, and it can choose differently for two clients or for the same client at two different moments.

flowchart TB
  subgraph clients["Clients around the world"]
    C1["Client in Tokyo"]
    C2["Client in Paris"]
    C3["Client in São Paulo"]
  end
  ANY(["Anycast address<br/>203.0.113.53<br/>announced from every site"])
  subgraph sites["Identical service instances"]
    S1["Site TYO<br/>announces 203.0.113.53"]
    S2["Site CDG<br/>announces 203.0.113.53"]
    S3["Site GRU<br/>announces 203.0.113.53"]
  end
  C1 --> ANY
  C2 --> ANY
  C3 --> ANY
  ANY -. "BGP picks nearest" .-> S1
  ANY -. "BGP picks nearest" .-> S2
  ANY -. "BGP picks nearest" .-> S3
  C1 ==> S1
  C2 ==> S2
  C3 ==> S3

How one anycast address fans out to many instances. What it shows: all three clients dial the identical address 203.0.113.53, but the routing system silently steers each to a different physical site — Tokyo to TYO, Paris to CDG, São Paulo to GRU — because each site originates the same route and every client’s nearest router prefers its topologically closest origin. The insight to take: anycast pushes the load-balancing and proximity decision into the network layer itself, for free, with no per-request logic — but the corollary is that the client has no say and no guarantee which instance it reaches, which is exactly why the service behind an anycast address must be able to answer correctly from any instance.

How It Works, End to End

The mechanism is deceptively simple because it reuses infrastructure that already exists. There is no “anycast protocol.” Anycast is an operational technique layered on ordinary routing.

Step 1 — Assign one address block to the service. The operator takes a routable prefix, say 203.0.113.0/24, and dedicates it to the anycast service. Every serving location will pretend to be the origin of this prefix.

Step 2 — Originate the prefix from every site. At each data center, a router (or the servers themselves via a routing daemon like BIRD or FRR speaking BGP to a local router) advertises the prefix into the routing system. RFC 4786 distinguishes the two scopes this can happen at. Intra-domain / IGP anycast (RFC 4786 §4.3.1) announces the prefix inside a single administrative domain via an interior gateway protocol (OSPF, IS-IS), where “the routing system is typically under the control of the same organisation that is providing the service.” Inter-domain / BGP anycast (RFC 4786 §4.3.2) announces the same prefix from multiple autonomous systems into global BGP, where “the routing system is, in general, controlled by other people” — i.e., the rest of the Internet. Global CDNs and DNS roots use BGP anycast; a single enterprise might use IGP anycast for an internal service.

Step 3 — Every router in the Internet converges on a best path. BGP’s whole job is to compute, at each router, one best route to each destination prefix. When the same prefix arrives from several origins, BGP applies its usual tie-breaking cascade — local preference, AS-path length, multi-exit discriminator, then IGP metric to the next hop — and installs one winner in that router’s forwarding table. The winner is, loosely, the “closest” instance by BGP’s policy metrics, which correlate with but do not equal geographic distance (BGP has no notion of miles; it counts AS hops and honors business policy). The set of clients whose packets end up at one particular instance is that instance’s catchment: “the topological region of a network within which packets directed at an Anycast Address are routed to one particular node” (RFC 4786 §2).

Step 4 — The client sends; the network delivers to whichever instance it converged on. The client resolves the service name once (via DNS) to the anycast address, opens a socket, and sends. It has no idea, and does not care, which physical box answers. As long as every instance runs the identical service, the answer is correct.

Local versus global nodes. RFC 4786 §4.4.6 refines the deployment with two node roles. A global node advertises its route “such that… reachability information for the service is propagated globally” — it can serve clients from anywhere. A local node advertises “such that their propagation is restricted” — using BGP’s NO_EXPORT community or a stuffed/prepended AS-path so the route stays regional. Local nodes soak up nearby traffic cheaply (say, a node inside a large eyeball ISP serving only that ISP’s customers); global nodes are the backstop that catches everything the local nodes do not. This is how a provider can put a small cache deep inside a partner network without that node accidentally becoming the global best path for the whole planet.

A Concrete BGP Announcement

Here is what the routing configuration looks like on two anycast nodes using the BIRD routing daemon. The point to absorb is that both files announce the identical prefix — that single fact is the entire trick.

# ---- Node in Frankfurt (AS 64500) ----
protocol static anycast_routes {
    ipv4;
    route 203.0.113.0/24 blackhole;   # originate the anycast prefix
}
protocol bgp uplink {
    local as 64500;
    neighbor 198.51.100.1 as 64510;   # the upstream transit provider
    ipv4 { export where proto = "anycast_routes"; };
}
# ---- Node in Singapore (AS 64500, same ASN) ----
protocol static anycast_routes {
    ipv4;
    route 203.0.113.0/24 blackhole;   # THE SAME prefix, originated again
}
protocol bgp uplink {
    local as 64500;
    neighbor 198.51.100.9 as 64520;   # a DIFFERENT upstream, in Asia
    ipv4 { export where proto = "anycast_routes"; };
}

Line-by-line: the static protocol originates 203.0.113.0/24 — the blackhole next-hop is a placeholder that says “this prefix lives here; I am its source.” The bgp protocol peers with a local transit provider and exports that route. Run this unchanged in Frankfurt and Singapore and you have anycast: a European client’s BGP path to 203.0.113.0/24 resolves through the Frankfurt upstream, an Asian client’s through the Singapore upstream, and each is delivered to its nearest node. To make Singapore a local node you would add a community like bgp_community.add((65535,65281)) (NO_EXPORT) or prepend the AS-path several times so the route loses BGP tie-breaks outside Asia. To drain a node for maintenance you simply stop announcing — BGP reconverges within seconds and every client silently shifts to the next-best node, which is the failover story that makes anycast so operationally attractive.

The Stateless Requirement — and Why TCP Is the Sharp Edge

The elegance of anycast hides its one hard constraint: the routing decision is not pinned to a connection. RFC 4786 §4.1 states the governing rule plainly — “the routing system’s node selection decision ought to be stable for substantially longer than the expected transaction time, if the service is to be provided reliably.” Routing is a packet-level function, not a flow-level one. Two packets of the same connection can, in principle, be routed to two different instances if the best path changes between them.

For a stateless single-datagram exchange — a DNS query and its reply, which is one UDP request and one UDP response — this is a non-issue. Even if the catchment shifts a millisecond later, this transaction already completed against one instance. Any instance could have answered; they are interchangeable. This is why anycast and UDP-based request-response are a perfect match, and why the DNS root servers were among the first big anycast deployments.

For a long-lived stateful flow — a TCP connection carrying a large download or a persistent session — anycast is riskier. TCP connection state (sequence numbers, window, the TCP control block) lives only on the instance that completed the handshake. If BGP reconverges mid-connection and the client’s catchment moves to a different instance, that instance has never heard of this connection: it receives a data segment for a connection it has no state for and, per TCP rules, replies with a RST that tears the connection down. RFC 4786 §4.1 warns directly: “for long running flows, there are potential failure modes using anycast that are more complex than a simple ‘destination unreachable’ failure using unicast.” A second, subtler hazard is per-packet load balancing upstream (RFC 4786 §4.4.3): if a router splits a single flow’s packets across equal-cost paths that lead to different anycast nodes, “per-packet load balancing… may cause different packets from a single multi-packet transaction sent by a client to be delivered to different Anycast Nodes, effectively making the anycast service unavailable.” The fix for that specific problem is per-flow (5-tuple-hashed) load balancing, which virtually all modern routers do by default.

How bad is this in practice? Empirical measurement is more reassuring than the RFC’s cautions suggest — but not unconditionally. The USC/ISI study “Does Anycast Hang Up on You?” (Wei & Heidemann, 2018) measured catchment stability from thousands of vantage points and found that clients “flip between anycast sites 0.1–1.0% of the time during TCP connections,” while roughly a quarter of vantage points saw some catchment change over a month of observation (Wei & Heidemann 2018). A sub-1% mid-connection flip rate is tolerable for short flows — the connection is likely finished before any reroute — but unacceptable for a multi-hour flow that will almost certainly outlive some BGP event.

Uncertain

Verify: the exact framing of the ISI paper’s conclusion. My summary of the PDF rendered its headline as the blunt claim “anycast is not safe for TCP in practice,” but the same source’s own numbers (0.1–1% mid-connection flips) show TCP anycast is usually fine for short flows, and it is deployed at planet scale by every major CDN. Reason: the strong wording came from an automated summary of a 1.4 MB PDF, and it sits in tension with the paper’s quantitative results and with observed production reality. To resolve: read the paper’s abstract and conclusion verbatim and reconcile the qualitative claim with the 0.1–1% figure. #uncertain

This is why the modern reconciliation is: anycast is safe for TCP in the way CDNs actually use it. Real deployments lean on three properties (per Catchpoint’s “TCP over IP Anycast” and general CDN practice): (1) BGP is stable enough on the timescale of a typical HTTP request — most web connections live for seconds, and catchments change on the order of minutes to hours, so the overwhelming majority of connections never see a reroute; (2) routers do per-flow hashing, so a stable path keeps a connection pinned even when equal-cost alternatives exist; and (3) when a genuinely long-lived or stateful session is needed, the CDN can hand the client a unicast IP for that specific session after the anycast-served handshake, moving the long flow off anycast entirely. The upshot: anycast HTTP works fine, but you would not run a multi-hour SSH session or a database replication link over a bare anycast address.

Canonical Uses

DNS root and authoritative servers. The 13 named root server “letters” are each, in reality, dozens to hundreds of anycast instances sharing one address. A query to k.root-servers.net (193.0.14.129) reaches whichever instance is nearest. DNS is the ideal anycast workload: every query is a self-contained, stateless UDP transaction, so catchment instability is invisible, and the geographic spread cuts latency for the single most latency-sensitive step of every connection. This is also why authoritative DNS providers (Route 53, Cloudflare, NS1) are built on anycast.

CDN edge and content serving. A CDN announces a handful of anycast prefixes from every point of presence, so each user is pulled to a nearby edge with no DNS-based geo-steering required. Cloudflare’s primer describes exactly this: “there are machines in 12 data centers all listening on those IPs… the network itself will direct traffic to the nearest CloudFlare data center,” cutting last-hop latency to single-digit milliseconds versus ~50 ms for a distant unicast origin (Cloudflare). The connections are short HTTP(S) requests, so the TCP-flip risk is negligible.

DDoS absorption and scrubbing. Anycast is a structural defense against volumetric distributed denial-of-service. Because a botnet is itself geographically distributed, “a distributed botnet will have a portion of its denial of service traffic absorbed by each of our data centers” (Cloudflare) — the attack is sharded across catchments by the same routing that shards legitimate traffic, so no single site takes the whole flood. RFC 4786 §6.1 frames this as “the task of dealing with attack traffic [being] itself distributed across all the nodes that contribute to the service.” A dedicated scrubbing network can also anycast-attract a victim’s traffic, clean it, and forward only the good packets on.

Failure Modes and How to Diagnose Them

  • Mid-connection resets on long flows. Symptom: intermittent, unreproducible TCP RSTs on connections that live longer than a few minutes, clustered around routing events. Diagnosis: correlate resets with BGP churn (looking-glass or your own route monitors). Fix: keep anycast for short flows; move long/stateful sessions to unicast after handshake.
  • Split flows from per-packet load balancing. Symptom: a connection that never completes even on a stable route, because alternating packets hit two nodes. Diagnosis: traceroute showing multiple equal-cost next hops to different sites. Fix: ensure upstream routers use per-flow (5-tuple) hashing, not per-packet round-robin (RFC 4786 §4.4.3).
  • Suboptimal catchment (“my traffic goes to the wrong continent”). BGP proximity is topological, not geographic: a client can be routed to a far site because the AS-path is shorter or a peer’s policy prefers it. Diagnosis: traceroute reveals the surprising site. Fix: adjust announcements (MED, communities, prepending, add a local node in the mispathed region). The Cloudflare primer’s traceroute-to-SJC example is exactly this diagnosis in the happy case (RFC 4786 §4.9 recommends baking node-identity into the service so you can even tell which instance answered).
  • State divergence between instances. If the service does keep state, anycast requires that state be “synchronised between nodes” (RFC 4786 §4.6) or the client gets inconsistent answers depending on catchment. Diagnosis: a client sees answer A, then answer B, with no change on its side. Fix: make the service truly stateless, or replicate the backing store, or accept eventual consistency.

Alternatives and When to Choose Them

DNS-based global load balancing (GSLB) is the main alternative for steering clients to a nearby site. Instead of one anycast address, the service uses many unicast addresses and returns a different one per client based on the resolver’s location (EDNS Client Subnet, geolocation). Trade-offs: GSLB gives you application-level control (health, capacity, cost weighting) and pins a client to one site for the life of a connection — no anycast flip risk — but it steers based on the resolver’s location (often wrong for public resolvers), reacts slowly (DNS TTLs and negative caching delay failover to minutes), and needs an actual health-checking control plane. Anycast fails over in seconds via BGP reconvergence and needs no per-client logic, but gives you no application-level steering knobs and carries the stateful-flow risk. Large systems routinely use both: anycast the DNS layer and the CDN edge for proximity and fast failover, and GSLB or in-cluster balancing behind it for capacity-aware distribution. See Load Balancing Algorithms and Health Checking and Failure Detection for the backend side.

A note on QUIC and connection migration: QUIC is more anycast-tolerant than TCP because a QUIC connection is identified by a connection ID, not by the 4-tuple — “the primary function of a connection ID is to ensure that changes in addressing at lower protocol layers (UDP, IP) do not cause packets… to be delivered to the wrong endpoint” (RFC 9000 §5.1). If an anycast reroute lands a QUIC packet at a different node, that node can, in principle, use the connection ID to recognize and forward (or reconstruct) the flow rather than blindly resetting it, which is one reason anycast + QUIC is an active area of CDN engineering.

Production Notes

Anycast is not exotic — it is the default way the largest networks on Earth serve DNS and static content. The operational discipline it demands is route hygiene: announce the anycast prefix from every node with identical policy, monitor which node each region converges to (catchment monitoring), and treat “stop announcing” as your primary drain/maintenance and failover primitive. The two things that bite operators are (1) forgetting that a “local node” needs its route scoped (NO_EXPORT / prepending), so it does not accidentally win the global best-path and pull the planet into a small cache, and (2) assuming BGP proximity equals geographic proximity — it does not, and a mispathed catchment needs a routing fix (a local node, a MED tweak), not more servers. Because anycast reuses the routing plane you already run, its failover is free and instant in a way DNS-based schemes can never match — but that same reuse means an anycast outage is a routing outage, debugged with looking glasses and BGP monitors, not application logs.

See Also

Graph View

Backlinks