vault

The TCP Protocol in Linux

19 min read

The TCP Protocol in Linux

The Transmission Control Protocol (TCP) is the reliable, ordered, connection-oriented byte-stream transport that carries most of the traffic on the Internet, and Linux implements it as the largest and most heavily tuned protocol in its networking stack. TCP gives the application the illusion of a private, error-free pipe: bytes written with write()/send() on one end emerge in exactly the same order on the other, with losses retransmitted, duplicates discarded, corruption caught by a checksum, and the send rate paced so as not to overrun either the receiver or the network — even though the underlying IP layer is an unreliable, unordered, best-effort datagram service (RFC 9293 §3.1). In the kernel the whole protocol hangs off one structure, struct tcp_sock (defined in include/linux/tcp.h), which embeds struct inet_connection_sock as its first member and therefore is a struct sock at offset zero. The hot paths are tcp_sendmsg() on transmit (in net/ipv4/tcp.c) and tcp_rcv_established() on receive (in net/ipv4/tcp_input.c), the latter built around a clever “header prediction” fast path that handles the common case in a handful of instructions. Source references here are pinned to Linux 6.12 LTS (released 2024-11-17), the current long-term branch.

This note is the orientation map for TCP in Linux: the tcp_sock anatomy, how tcp_sendmsg builds packets, how the receive fast path works, how ACKs drive retransmission, and the modern loss-recovery features (RACK, TLP) and connection-setup optimizations (TCP Fast Open). The connection lifecycle — the state machine, the handshake, TIME-WAIT — is owned by its sibling The TCP State Machine; the rate-control algorithms (CUBIC, BBR) are TCP Congestion Control; the flow-control windows are TCP Send and Receive Windows. This note stitches those together and fills the gaps between them.

Mental Model — A Byte Stream over a Datagram Network

The single most important idea is that TCP is a sequence-number machine layered over unreliable datagrams. Every byte the application sends is conceptually assigned a 32-bit sequence number. The sender remembers which bytes it has sent (snd_nxt), which the receiver has acknowledged (snd_una), and which are still in flight (packets_out). The receiver remembers the next byte it expects (rcv_nxt) and advertises how much more it can buffer (rcv_wnd). The entire protocol is the bookkeeping needed to keep these counters consistent across a network that can drop, duplicate, reorder, and delay segments arbitrarily. Reliability is retransmission driven by missing ACKs; ordering is reassembly keyed on sequence number; flow control is the receive window; congestion control is a second, network-facing window (snd_cwnd) that limits how much the sender may have outstanding regardless of what the receiver allows.

flowchart LR
  subgraph SENDER["Sender side (tcp_sock)"]
    APP1["write()/send()"] --> SQ["send queue<br/>sk_write_queue<br/>(skbs not yet sent)"]
    SQ -->|"tcp_write_xmit"| RTX["rtx queue<br/>tcp_rtx_queue (rb-tree)<br/>(sent, awaiting ACK)"]
    RTX -->|"snd_una advances"| FREE["ACKed → freed"]
  end
  RTX -->|"segments on the wire (IP)"| NET(("IP / network"))
  NET --> RCVR
  subgraph RCVR["Receiver side (tcp_sock)"]
    IN["incoming skb"] --> FP{"in-order?<br/>seq == rcv_nxt"}
    FP -->|"yes (fast path)"| RQ["sk_receive_queue<br/>recv() reads here"]
    FP -->|"no (gap)"| OFO["out_of_order_queue<br/>(rb-tree)"]
    OFO -.->|"gap filled"| RQ
  end
  RQ --> APP2["recv()/read()"]

The two-sided byte-stream model in Linux. What it shows: on the sender, an skb travels from the send queue (sk_write_queue, data accepted from the app but not yet transmitted) into the retransmit queue (tcp_rtx_queue, an rb-tree of segments that have been sent and are awaiting acknowledgement); a cumulative ACK advancing snd_una frees skbs from the front. On the receiver, an in-order segment (seq == rcv_nxt) lands directly in sk_receive_queue where recv() reads it, while an out-of-order segment is parked in out_of_order_queue (also an rb-tree) until the gap ahead of it is filled. The insight to take: TCP maintains two independent queues per direction — “sent but unacked” and “received but unread/ungapped” — and almost all of TCP’s complexity is the rules for moving skbs between them as sequence-number bookkeeping evolves.

The tcp_sock Structure — Anatomy of a Connection

Every established TCP connection is a struct tcp_sock. Because struct inet_connection_sock is its first member (which in turn has struct inet_sock → struct sock as its first members), tcp_sk(sk) is a free downcast and inet_csk(sk) / inet_sk(sk) walk up the layers — the nesting explained in struct socket and struct sock. What makes tcp_sock distinctive in 6.12 is that its fields are explicitly grouped into cacheline groups with __cacheline_group_begin()/end() markers — tcp_sock_read_tx, tcp_sock_read_txrx, tcp_sock_read_rx, tcp_sock_write_tx, tcp_sock_write_txrx, tcp_sock_write_rx — so that the read-mostly and write-mostly hot fields for the transmit and receive paths land on separate cache lines, minimizing false sharing on busy connections (the layout is documented in Documentation/networking/net_cachelines/tcp_sock.rst, referenced by a comment at the top of the struct).

The core sequence-number variables, all u32, are the heart of the protocol (from include/linux/tcp.h, v6.12):

u32 rcv_nxt;     /* What we want to receive next        */
u32 snd_nxt;     /* Next sequence we send               */
u32 snd_una;     /* First byte we want an ack for        */
u32 snd_wnd;     /* The window we expect to receive      */
u32 rcv_wnd;     /* Current receiver window              */
u32 copied_seq;  /* Head of yet unread data             */
u32 write_seq;   /* Tail(+1) of data held in send buffer */
u32 snd_cwnd;    /* Sending congestion window           */
u32 packets_out; /* Packets which are "in flight"        */

Walking the most important ones. rcv_nxt is the sequence number of the next byte the receiver expects — everything below it is already received and acknowledged. snd_una (“send unacknowledged”) is the oldest byte the sender has sent but not yet had acknowledged; the window of outstanding data is [snd_una, snd_nxt). snd_nxt is the next sequence number the sender will assign to fresh data. write_seq is the tail of the send buffer — the sequence number one past the last byte tcp_sendmsg has accepted from the application, which may be ahead of snd_nxt if data is queued but not yet transmitted. copied_seq tracks how far recv() has consumed the receive queue. snd_wnd and rcv_wnd are the flow-control windows (TCP Send and Receive Windows); snd_cwnd is the congestion window (TCP Congestion Control). The amount the sender may transmit is the minimum of what the receiver’s snd_wnd allows and what snd_cwnd allows.

Three loss-tracking counters — sacked_out (segments the receiver has selectively acknowledged), lost_out (segments the kernel has marked lost and will retransmit), and retrans_out (segments currently retransmitted but not yet acked) — let TCP estimate in-flight data precisely under reordering and loss, which is what makes modern recovery (RACK, below) work. The smoothed round-trip-time estimate lives in srtt_us (smoothed RTT, in microseconds, stored << 3), with mdev_us (mean deviation) and rttvar_us feeding the retransmission timeout. out_of_order_queue is an rb-tree of segments received ahead of rcv_nxt.

The send and retransmit queues, by contrast, hang off the embedded struct sock. In include/net/sock.h (v6.12), sk_write_queue is a struct sk_buff_head (the FIFO send queue), and tcp_rtx_queue is a struct rb_root (an rb-tree) sharing a union slot — this is the retransmit queue, keyed by sequence number, replacing the old linear list so that SACK processing and RACK can find arbitrary segments in O(log n).

Transmit — How tcp_sendmsg Builds Packets

A send()/write() on a TCP socket lands, after the socket-layer dispatch, in tcp_sendmsg() (which locks the socket and calls tcp_sendmsg_locked(), both in net/ipv4/tcp.c). The job is to copy user bytes into skbs on the send queue and, when enough has accumulated, push them down to the IP layer. The mechanism, traced from the v6.12 source:

First, tcp_sendmsg_locked checks the socket state. Unless the connection is TCP_ESTABLISHED or TCP_CLOSE_WAIT (or it is a passive TCP-Fast-Open socket allowed to send early), it calls sk_stream_wait_connect() to block until the handshake completes:

if (((1 << sk->sk_state) & ~(TCPF_ESTABLISHED | TCPF_CLOSE_WAIT)) &&
    !tcp_passive_fastopen(sk)) {
        err = sk_stream_wait_connect(sk, &timeo);
        ...
}

Then it enters the main copy loop, while (msg_data_left(msg)). Each iteration looks at the tail of the send queue with tcp_write_queue_tail(sk). If the last skb still has room (its length is below size_goal, the target packet size derived from the MSS and segmentation-offload limits) and can be appended to, the new bytes are coalesced into it. Otherwise the loop jumps to new_segment, where it first checks sk_stream_memory_free(sk) (is there send-buffer budget left? see Socket Buffers and Memory Accounting) and, if so, allocates a fresh skb with tcp_stream_alloc_skb() and appends it with tcp_skb_entail(). The user bytes are then copied in — either into the skb’s linear area, into page fragments via sk_page_frag, or, for MSG_ZEROCOPY, pinned directly from user pages so no copy happens at all.

After data is queued, the loop decides whether to push — actually hand segments to the transmit machinery. The decision honours the Nagle algorithm (tp->nonagle; disabled by TCP_NODELAY), which delays sending a small segment while a previous small segment is still unacknowledged, to avoid flooding the network with tiny packets. When pushing is warranted, tcp_push() → __tcp_push_pending_frames() → tcp_write_xmit() walks the send queue, and for each skb calls tcp_transmit_skb(), which builds the TCP header, sets the checksum (or defers it to hardware — see Checksum Offloads), and calls into the IP layer (ip_queue_xmit). Critically, tcp_write_xmit only sends as many segments as the congestion window and receive window jointly permit (tcp_cwnd_test, tcp_snd_wnd_test); the rest stay in sk_write_queue until an incoming ACK opens the window. As each skb is transmitted it is moved from sk_write_queue onto the tcp_rtx_queue rb-tree (via tcp_event_new_data_sent), where it waits to be acknowledged or retransmitted.

tcp_sendmsg also integrates TSO/GSO (Segmentation Offloads GSO TSO): rather than chop a large user buffer into MSS-sized segments in software, it builds one large “super-skb” (up to gso_segs × MSS) and lets the NIC (TSO) or the lower stack (GSO) do the segmentation just before the wire. This is why size_goal can be far larger than one MSS — fewer, larger skbs traverse the stack, amortizing per-packet cost.

Receive — The Fast Path and Header Prediction

On the receive side, once a segment has climbed the RX path and the IP layer has matched it to an established socket, it enters tcp_rcv_established() (in net/ipv4/tcp_input.c). This function is built around header prediction, an optimization due to a famous Van Jacobson “30-instruction TCP receive” idea that the kernel comment explicitly credits. The premise: in a bulk transfer, the overwhelming majority of received segments are exactly the next in-order segment on a connection where nothing surprising is happening — same window, no options changing, sequence number exactly rcv_nxt, acknowledgement not beyond snd_nxt. For those, almost all of the RFC-793 input processing can be skipped.

The mechanism is a precomputed bitmask, tp->pred_flags. Its layout, per the source comment, is 0xS?10 << 16 + snd_wnd, where S is tcp_header_len >> 2 (the expected header length in 32-bit words) and ? is a nibble that is zero only when the fast path is eligible — the kernel sets pred_flags = 0 (via tcp_fast_path_on() being skipped, or explicitly zeroing it) whenever anything makes the fast path unsafe: a hole opening in the receive sequence space, an unexpected flag, a window-scale change. The fast-path gate is a single compound comparison (v6.12):

if ((tcp_flag_word(th) & TCP_HP_BITS) == tp->pred_flags &&
    TCP_SKB_CB(skb)->seq == tp->rcv_nxt &&
    !after(TCP_SKB_CB(skb)->ack_seq, tp->snd_nxt)) {

This single if checks three things at once: that the packet’s flags-and-header-length word matches the predicted value (so the header is the expected size with the expected flags), that the segment’s starting sequence number is exactly what we expect next (== rcv_nxt, i.e. in order, no gap), and that its acknowledgement number does not acknowledge data we have not sent (ack_seq not after snd_nxt). When all three hold, the code splits into two sub-cases. If len <= tcp_header_len the segment carries no data — it is a pure ACK; the kernel calls tcp_ack() to advance snd_una, frees the skb, and checks whether more data can now be sent. Otherwise it is a pure data segment; after a checksum check the data is appended to the receive queue with tcp_queue_rcv(), LINUX_MIB_TCPHPHITS is bumped (the counter you watch in nstat/netstat -s as “TCPHPHits” to see how often the fast path is taken), and an ACK is scheduled via __tcp_ack_snd_check(). tcp_data_ready() then wakes any recv() blocked or any epoll watcher.

If any fast-path condition fails — out-of-order data, a SYN/RST/FIN flag, a failed timestamp check (PAWS, Protection Against Wrapped Sequences) — control falls through to the slow_path: label, which runs the full RFC-793-style procedure: tcp_validate_incoming() (sequence and RST/SYN validation), tcp_ack() with the slow-path flag, urgent-data handling, and tcp_data_queue(), which is where out-of-order segments get parked in out_of_order_queue and where gaps trigger duplicate ACKs. The split between tcp_rcv_established (for the ESTABLISHED state) and tcp_rcv_state_process (for every other state) is the design choice that keeps the steady-state data path lean while a separate, slower function handles the connection-lifecycle transitions covered in The TCP State Machine.

ACK Processing, Retransmission, and RTT Estimation

The engine of TCP reliability is tcp_ack(). Every incoming ACK runs through it, and it does several jobs in sequence: it advances snd_una over the newly acknowledged sequence range, removes the now-acknowledged skbs from the front of the tcp_rtx_queue rb-tree (tcp_clean_rtx_queue), processes any SACK (Selective Acknowledgement, RFC 2018) blocks in the option to mark non-contiguous acknowledged ranges via sacked_out, feeds the congestion-control algorithm (icsk_ca_ops) so it can grow or shrink snd_cwnd (TCP Congestion Control), updates the RTT estimate, and finally calls tcp_data_snd_check to transmit anything the freshly opened window now permits.

RTT estimation follows RFC 6298. Each acknowledged segment whose RTT can be measured (via the TCP timestamp option, or by timing the first transmission of an unretransmitted segment) updates the smoothed RTT and its deviation:

  • srtt = (1 - 1/8)·srtt + (1/8)·rtt_sample — the smoothed RTT, an exponentially weighted moving average where each new sample contributes 1/8.
  • rttvar = (1 - 1/4)·rttvar + (1/4)·|srtt - rtt_sample| — the variance estimate, weighting deviation by 1/4.
  • RTO = srtt + max(G, 4·rttvar) — the retransmission timeout is the smoothed RTT plus four times the deviation (G is the clock granularity).

In the kernel these live in srtt_us, mdev_us/rttvar_us, and the result is stored in icsk_rto on the inet_connection_sock. The RTO is clamped: TCP_RTO_MIN is HZ/5 (200 ms) and TCP_RTO_MAX is 120*HZ (120 seconds), both #defined in include/net/tcp.h (v6.12). When the retransmission timer fires without an ACK, tcp_retransmit_timer() resends the oldest unacknowledged segment, exponentially backs off the RTO (doubling it, tracked in icsk_backoff), and — per congestion control — collapses snd_cwnd. This timer-driven retransmission is the last-resort recovery; the fast, common-case recovery is RACK and fast retransmit.

Delayed and quick ACKs trade ACK overhead against responsiveness. Rather than acknowledge every segment, TCP delays the ACK (up to TCP_DELACK_MAX, HZ/5 = 200 ms in v6.12) hoping to piggyback it on outgoing data or to acknowledge two segments at once (icsk_ack state on the inet_connection_sock). But at connection start, or when it detects it is in the middle of a bulk transfer, it enters quickack mode (tcp_enter_quickack_mode) and acknowledges promptly, because early prompt ACKs let the sender’s slow-start ramp up faster. The TCP_QUICKACK socket option lets an application force this per-call.

Modern Loss Recovery — RACK and TLP

Two features have largely replaced the classic “three duplicate ACKs” fast-retransmit trigger as the front line of loss detection.

RACK-TLP (Recent ACKnowledgment — Tail Loss Probe), standardized in RFC 8985 and implemented in net/ipv4/tcp_recovery.c, is time-based loss detection. Instead of counting duplicate ACKs, RACK observes: if a segment was sent, and a later-sent segment has since been acknowledged (by SACK), and more than a small reordering window (reo_wnd, derived from the RTT) has elapsed, then the earlier segment is very likely lost — mark it and retransmit. This is far more robust under reordering and works even for the last segments of a flow, which never accumulate three duplicate ACKs because there is nothing after them. The tcp.h flags TCP_RACK_LOSS_DETECTION, TCP_RACK_STATIC_REO_WND, and TCP_RACK_NO_DUPTHRESH configure its behaviour (v6.12). RACK is enabled by default: the net.ipv4.tcp_recovery sysctl defaults to 0x1 (the RACK bit), per Documentation/networking/ip-sysctl.rst at v6.12, which notes that this “also subsumes and disables RFC6675 recovery for SACK connections.”

TLP (Tail Loss Probe), the “TLP” half of RACK-TLP, handles the tail of a transfer specifically. If the sender has sent its last segments and then goes quiet waiting for ACKs, and one of those tail segments was lost, the only recovery would be the slow RTO timer (hundreds of milliseconds). TLP instead arms a short probe timer (roughly 2·RTT, scheduled in tcp_send_loss_probe() in tcp_output.c); if it fires with data still unacknowledged, it retransmits the last segment (or sends new data) to elicit an ACK or SACK, which then triggers fast RACK recovery instead of a full RTO. The tlp_high_seq and tlp_retrans fields in tcp_sock track an outstanding TLP. Together, RACK and TLP turn most tail losses from a multi-hundred-millisecond RTO stall into a fast, RTT-scale recovery.

TCP Fast Open — Data in the Handshake

TCP Fast Open (TFO), RFC 7413, lets a client send application data in the SYN packet of the handshake, saving one round trip for repeat connections to the same server. The first time a client connects, the server hands back a TFO cookie (a server-generated, cryptographically authenticated token). On subsequent connections the client includes that cookie and its request data in the SYN; the server validates the cookie and delivers the data to the application before the handshake completes, which is why tcp_sendmsg has the tcp_passive_fastopen(sk) exception that allows sending before ESTABLISHED. The TFO_CLIENT_ENABLE/TFO_SERVER_ENABLE bits in net/ipv4/tcp.h (and the net.ipv4.tcp_fastopen sysctl) gate it. TFO requires application cooperation (MSG_FASTOPEN on sendto, or TCP_FASTOPEN_CONNECT) and has seen limited deployment because middleboxes sometimes strip the option, but it is fully implemented in Linux.

Configuration and Observability

TCP behaviour is tuned through sysctl (mostly under net.ipv4.tcp_*, applied per network namespace) and per-socket setsockopt (Socket Options and setsockopt). A few high-value knobs, per tcp(7):

# Congestion control algorithm (see TCP Congestion Control)
sysctl net.ipv4.tcp_congestion_control      # e.g. cubic (default) or bbr
 
# Autotuned receive/send buffer min,default,max (bytes) — see TCP Send and Receive Windows
sysctl net.ipv4.tcp_rmem                    # e.g. 4096  131072  6291456
sysctl net.ipv4.tcp_wmem                    # e.g. 4096  16384   4194304
 
# TCP Fast Open: bitmask, 1=client 2=server 3=both
sysctl net.ipv4.tcp_fastopen
 
# Per-socket: disable Nagle for latency-sensitive small writes
setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &one, sizeof(one));

For observability, ss -ti (the -i adds TCP-internal info) prints per-socket cwnd, rtt, rto, retransmit counters, and the congestion-control algorithm — the single most useful command for diagnosing a TCP performance problem. Aggregate counters from nstat or netstat -s expose TCPHPHits (fast-path hits), TCPLostRetransmit, TCPTimeouts (RTO firings), TCPFastRetrans, and the SACK/RACK counters. A high ratio of timeouts to fast retransmits suggests RACK/TLP are not catching losses (or are disabled); a low TCPHPHits ratio on a bulk-transfer socket suggests something is repeatedly knocking the connection off the fast path.

Failure Modes and Common Misunderstandings

“TCP guarantees delivery.” It does not — it guarantees that if bytes are delivered they are in order and uncorrupted, and it retransmits aggressively, but a connection that cannot reach the peer will eventually time out and reset. Reliability is best-effort-with-retransmission, not a magic guarantee.

Confusing the BSD socket state with the TCP state. The coarse struct socket state (SS_CONNECTED, etc.) is not the TCP state machine; the real TCP state is sk->sk_state holding TCP_ESTABLISHED, TCP_TIME_WAIT, and so on — see The TCP State Machine. ss and netstat show the latter.

Head-of-line blocking. Because TCP delivers a strict byte stream, a single lost segment stalls all later data until the gap is retransmitted and filled — even data that has already arrived sits in out_of_order_queue undelivered. This is inherent to the ordered-stream abstraction and is the motivation for QUIC’s per-stream multiplexing over UDP.

Bufferbloat from oversized send buffers. A huge tcp_wmem lets the sender stuff far more than a bandwidth-delay product into the network, inflating latency. Modern Linux mitigates this with TCP Small Queues (TSQ) and TCP pacing (especially with BBR and the fq qdisc), but a misconfigured static buffer can still bloat — see Byte Queue Limits and Buffer Bloat and TCP Send and Receive Windows.

Nagle + delayed ACK interaction. A classic latency bug: Nagle holds a small write waiting for an ACK, while the peer’s delayed-ACK timer holds the ACK waiting for data — a ~200 ms stall on a request/response workload. The fix is TCP_NODELAY on the latency-sensitive side.

Alternatives and When to Choose Them

TCP’s sibling transport is UDP — connectionless, unordered, no retransmission, no congestion control. Choose UDP when you want to manage reliability yourself (real-time media that prefers a fresh frame to a retransmitted stale one), when you need multicast, or when you are building your own transport (QUIC runs over UDP precisely to escape kernel TCP’s head-of-line blocking and to iterate the protocol in userspace). Choose TCP — the default for almost everything — when you want a reliable ordered stream and want the kernel’s mature congestion control and loss recovery to do the hard work. Within TCP, the tuning choices are the congestion-control algorithm (TCP Congestion Control: CUBIC for general use, BBR for high-bandwidth long-fat-pipe links) and the buffer/window sizing (TCP Send and Receive Windows).

Production Notes

The cacheline-group reorganization of tcp_sock (the tcp_sock_read_tx/write_rx groups) is a relatively recent, deliberate performance effort by the networking maintainers to keep the per-packet hot fields cache-resident on high-PPS servers; the documented layout in net_cachelines/tcp_sock.rst is treated as a contract that new fields must respect. On the recovery side, Google’s deployment of RACK-TLP and BBR is the origin of much of this code (RFC 8985 and the BBR drafts are co-authored by the same engineers who wrote the Linux implementations), and the ss -ti delivery_rate/bbr fields exist precisely to expose that machinery. A common production tuning is to raise tcp_rmem/tcp_wmem maxima on bandwidth-delay-product-heavy paths (cross-datacenter, satellite) so window autotuning can open large enough windows — but doing so without fq+pacing or BBR risks bufferbloat.

See Also

Graph View

Backlinks