The Network Transmit Path

The transmit (TX) path is the chain of kernel code that turns a send() of application bytes into electrical (or optical) signals leaving a Network Interface Card (NIC). It is the mirror image of The Network Receive Path, but with a different rhythm: where RX is interrupt-then-poll, TX is synchronous-then-queued — the sending thread runs the packet down the stack itself (copying data into struct sk_buffs, prepending TCP/UDP and IP headers, passing the egress netfilter hooks), and only at the bottom does it hand the packet to a queueing discipline (qdisc) that schedules it onto the wire, possibly on a different timeline. The path is send() → tcp_sendmsg/udp_sendmsg (build skbs) → ip_queue_xmit/ip_output (POSTROUTING netfilter hook) → __dev_queue_xmit → the qdisc enqueue/dequeue → dev_hard_start_xmit → the driver’s ndo_start_xmit → the NIC. Along the way three performance mechanisms intervene: Generic Segmentation Offload (GSO) chops oversized buffers into MTU-sized segments at the last moment, Byte Queue Limits (BQL) throttle how many bytes sit in the NIC’s hardware ring to fight buffer-bloat, and xmit_more batches multiple packets behind a single NIC “doorbell” write. This note traces that path function by function against Linux 6.12 LTS (net/core/dev.c, net/sched/sch_generic.c, net/ipv4/ip_output.c, tcp_output.c; verified 2026-06-05).

---

Mental Model

Picture the TX path as two stacked machines with a queue between them. The upper machine is driven by the sender: when a thread calls send(), that same thread, in process context, builds the packet and pushes it all the way down to __dev_queue_xmit. The lower machine is the qdisc + driver, which decides in what order and how fast packets actually leave — it may transmit immediately (work-conserving, the common case), or hold the packet to shape bandwidth, prioritize, or fight bloat. The handoff between them is the qdisc’s enqueue/dequeue pair. The single most important consequence: by the time send() returns, the bytes are usually queued, not on the wire — completion (and freeing the skb) happens asynchronously when the NIC raises a TX-done interrupt later.

flowchart TB
  APP["Application: send()"] --> SM["tcp_sendmsg / udp_sendmsg<br/>copy user data into sk_buffs"]
  SM -->|"TCP: queue on sk_write_queue"| PUSH["tcp_write_xmit -> __tcp_transmit_skb<br/>build TCP header, checksum"]
  PUSH -->|"icsk->queue_xmit"| IPQ["ip_queue_xmit / __ip_queue_xmit<br/>route lookup, build IP header"]
  IPQ --> LOUT["ip_local_out<br/>NF_INET_LOCAL_OUT (OUTPUT) hook"]
  LOUT --> IPO["ip_output<br/>NF_INET_POST_ROUTING (POSTROUTING) hook, SNAT"]
  IPO --> FIN["ip_finish_output2<br/>neighbour resolves L2 (MAC) header"]
  FIN -->|"neigh_output"| DQX["__dev_queue_xmit<br/>tc egress, pick TX queue"]
  DQX -->|"q->enqueue"| QD["qdisc enqueue<br/>fq_codel / fq / htb ..."]
  QD -->|"__qdisc_run -> qdisc_restart"| DEQ["dequeue + bulk dequeue (xmit_more)"]
  DEQ --> SDX["sch_direct_xmit<br/>validate_xmit_skb_list: GSO segment"]
  SDX --> DHSX["dev_hard_start_xmit -> netdev_start_xmit"]
  DHSX -->|"ndo_start_xmit"| DRV["driver: post to TX ring, BQL accounting"]
  DRV -->|"doorbell"| NIC["NIC DMA reads buffer, transmits"]

The transmit path from socket to wire in Linux 6.12. What it shows: the path is a single descent, but it crosses two distinct execution regimes — the sender’s process context (everything down to q->enqueue) and the qdisc/driver layer (__qdisc_run downward), which may run synchronously on the sender or later under the TX softirq. The insight to take: the qdisc is the pivot. Above it, the packet is fully formed and the sender owns it; below it, the kernel decides scheduling, performs last-moment GSO segmentation in validate_xmit_skb_list, and BQL gates how much reaches the hardware ring. send() returning means “enqueued,” not “transmitted.”

---

Mechanical Walk-through

1. send() → the transport layer builds skbs

A send()/sendmsg()/write() on a connected socket lands in the protocol’s sendmsg handler. For TCP that is tcp_sendmsg → tcp_sendmsg_locked (net/ipv4/tcp.c). It does not transmit immediately. Instead it allocates struct sk_buffs with tcp_stream_alloc_skb, copies user data into them (into the linear area and page fragments), and appends them to the socket’s write queue (sk->sk_write_queue). The amount it can buffer is bounded by the send buffer sk_sndbuf (SO_SNDBUF); when full, a blocking socket sleeps in sk_stream_wait_memory and a non-blocking one returns EAGAIN. Periodically — when enough data has accumulated to fill a segment, or MSG_MORE/Nagle conditions allow — tcp_sendmsg_locked calls tcp_push → __tcp_push_pending_frames → tcp_write_xmit, the function that actually clocks segments out under the control of the congestion and send windows (TCP Congestion Control, TCP Send and Receive Windows). For each segment, tcp_write_xmit calls __tcp_transmit_skb (net/ipv4/tcp_output.c), which builds the TCP header, computes (or defers to hardware) the checksum, and then performs the indirect call into the IP layer:

err = INDIRECT_CALL_INET(icsk->icsk_af_ops->queue_xmit,
                         inet6_csk_xmit, ip_queue_xmit,
                         sk, skb, &inet->cork.fl);

For an IPv4 socket icsk_af_ops->queue_xmit is ip_queue_xmit. UDP is simpler and connectionless: udp_sendmsg (net/ipv4/udp.c) corks the data, builds one datagram skb with ip_make_skb, and calls udp_send_skb → ip_send_skb → ip_local_out, joining the same IP egress path a step lower than TCP. The transport-specific details belong to The TCP Protocol in Linux and The UDP Protocol in Linux; what matters here is that the transport layer produces fully-formed transport segments as skbs and hands them to the IP layer.

2. The IP layer — route, build header, and the egress netfilter hooks

__ip_queue_xmit (net/ipv4/ip_output.c) is where L3 happens. If the skb is not already routed it performs a Forwarding Information Base (FIB) lookup with ip_route_output_ports, caching the resulting dst_entry. Then it pushes the 20-byte IPv4 header into the skb’s headroom (skb_push), fills in version/IHL/TTL/protocol, copies source and destination addresses from the flow, and selects the IP identification field. Finally it calls ip_local_out:

res = ip_local_out(net, sk, skb);

ip_local_out → __ip_local_out fires the first egress netfilter hook:

return nf_hook(NFPROTO_IPV4, NF_INET_LOCAL_OUT,
               net, sk, skb, NULL, skb_dst(skb)->dev, dst_output);

This is the OUTPUT chain — locally-generated packets pass here (iptables -A OUTPUT, nft ... hook output). On NF_ACCEPT, dst_output is an indirect call to the route’s output function, normally ip_output (net/ipv4/ip_output.c), which fires the second and architecturally pivotal egress hook:

int ip_output(struct net *net, struct sock *sk, struct sk_buff *skb)
{
        ...
        return NF_HOOK_COND(NFPROTO_IPV4, NF_INET_POST_ROUTING,
                            net, sk, skb, indev, dev,
                            ip_finish_output,
                            !(IPCB(skb)->flags & IPSKB_REROUTED));
}

This is the POSTROUTING hook — the last point before the packet leaves IP. It is where Source NAT (SNAT) and masquerade rewrite the source address, and where connection tracking confirms the flow (see Network Address Translation NAT, Connection Tracking conntrack, The Netfilter Framework and Hooks). POSTROUTING runs after routing, so the outgoing interface is already chosen and masquerade can pick the right source IP. On accept, ip_finish_output runs.

3. Fragmentation, GSO check, and ip_finish_output2 — the L2 header

ip_finish_output → __ip_finish_output makes the segmentation/fragmentation decision against the path MTU. If the skb is a GSO super-packet (skb_is_gso), it calls ip_finish_output_gso, which in the common case (skb_gso_validate_network_len passes) defers segmentation downward and just calls ip_finish_output2; only if a GSO segment would exceed the MTU does it segment-then-fragment here. A non-GSO skb larger than the MTU goes through ip_fragment. The default and fast case calls ip_finish_output2 (net/ipv4/ip_output.c), which resolves the next-hop’s link-layer (MAC) address through the neighbour subsystem:

neigh = ip_neigh_for_gw(rt, skb, &is_v6gw);
...
res = neigh_output(neigh, skb, is_v6gw);

neigh_output (include/net/neighbour.h) prepends the cached L2 header (neigh_hh_output) if the neighbour is resolved (ARP/NDP complete — see ARP and Neighbour Discovery, The Neighbour Subsystem), and then calls dev_queue_xmit(skb). If the neighbour is not yet resolved, the skb is held while an ARP request goes out. This is the boundary where the packet stops being “an IP packet” and becomes “a fully-framed L2 frame ready for a device.”

4. __dev_queue_xmit — tc egress, pick the TX queue, reach the qdisc

dev_queue_xmit → __dev_queue_xmit (net/core/dev.c) is the device-layer entry. Under rcu_read_lock_bh() (which also disables bottom halves) it:

1. Runs the egress netfilter hook and tc egress (clsact/egress, tc-BPF) if configured (nf_hook_egress, sch_handle_egress) — another eBPF and traffic-control attach point (Linux eBPF MOC, Queueing Disciplines qdisc).
2. Picks the transmit queue: txq = netdev_core_pick_tx(dev, skb, sb_dev) chooses one of the device’s num_tx_queues hardware queues, via the device’s ndo_select_queue, an installed XPS (Transmit Packet Steering) map, or a flow hash. Multiqueue NICs let different CPUs transmit lock-free on different queues.
3. Fetches that queue’s qdisc: q = rcu_dereference_bh(txq->qdisc). If the qdisc has an enqueue method (the normal case), it calls __dev_xmit_skb(skb, q, dev, txq) and is done. The IFF_NOQUEUE software-device case (loopback, tunnels) bypasses the qdisc and calls dev_hard_start_xmit directly.

5. The qdisc — enqueue, then dequeue under __qdisc_run

__dev_xmit_skb (net/core/dev.c) is where the work-conserving fast path and the queueing path diverge. For a classic locked qdisc, if the qdisc is empty and idle (TCQ_F_CAN_BYPASS && !qdisc_qlen(q) && qdisc_run_begin(q)), it skips queueing entirely and transmits the skb directly via sch_direct_xmit. Otherwise it enqueues:

rc = dev_qdisc_enqueue(skb, q, &to_free, txq);   /* q->enqueue() */
if (qdisc_run_begin(q)) {
        __qdisc_run(q);
        qdisc_run_end(q);
}

The qdisc’s enqueue is the policy: pfifo_fast just appends to a band; fq_codel (the common default) classifies into per-flow queues and applies CoDel active-queue management; htb places the skb into a class. (The qdisc taxonomy is Queueing Disciplines qdisc and Classful and Classless Qdiscs.) qdisc_run_begin is a seqcount/lock that ensures exactly one CPU dequeues a given qdisc at a time. The dequeuer runs __qdisc_run (net/sched/sch_generic.c):

void __qdisc_run(struct Qdisc *q)
{
        int quota = READ_ONCE(net_hotdata.dev_tx_weight);
        int packets;
 
        while (qdisc_restart(q, &packets)) {
                quota -= packets;
                if (quota <= 0) {
                        __netif_schedule(q);   /* defer rest to TX softirq */
                        break;
                }
        }
}

qdisc_restart → dequeue_skb pulls the next skb (and, crucially, bulk-dequeues a list of skbs for the same TX queue via try_bulk_dequeue_skb — this is what feeds xmit_more, see below) and calls sch_direct_xmit. If the per-call quota (dev_tx_weight) is exhausted, the rest is punted to __netif_schedule, which raises NET_TX_SOFTIRQ; net_tx_action (the TX softirq, net/core/dev.c) then resumes dequeuing later and also frees completed skbs from the completion_queue. This is the qdisc analogue of the RX budget — no single send monopolizes the CPU.

6. sch_direct_xmit → GSO segmentation → dev_hard_start_xmit

sch_direct_xmit (net/sched/sch_generic.c) is the bridge from qdisc to driver. It drops the qdisc lock, then calls validate_xmit_skb_list outside the lock — this is where last-moment GSO segmentation happens:

if (netif_needs_gso(skb, features)) {
        segs = skb_gso_segment(skb, features);
        ...
        skb = segs;   /* one super-skb becomes a list of MSS-sized skbs */
}

If the NIC supports hardware TSO/TCP-segmentation, the super-packet is left whole and the NIC does the chopping; if not, skb_gso_segment does it in software here, “break[ing] skbuff data across multiple resized buffers matching the MSS value” (kernel docs, Segmentation Offloads). validate_xmit_skb also linearizes if needed and completes the checksum in software when the device cannot offload it. The deeper offload story is Segmentation Offloads GSO TSO and Checksum Offloads. Then sch_direct_xmit takes the device TX lock and calls dev_hard_start_xmit (net/core/dev.c):

struct sk_buff *dev_hard_start_xmit(struct sk_buff *first, ...)
{
        while (skb) {
                struct sk_buff *next = skb->next;
                skb_mark_not_on_list(skb);
                rc = xmit_one(skb, dev, txq, next != NULL);   /* more = next != NULL */
                if (unlikely(!dev_xmit_complete(rc))) { ... goto out; }
                skb = next;
        }
}

The more = (next != NULL) argument is the xmit_more flag: when transmitting a batch, every skb except the last is sent with more = true. xmit_one calls netdev_start_xmit, which sets the per-CPU xmit_more state and calls the driver:

static inline netdev_tx_t netdev_start_xmit(struct sk_buff *skb, ...)
{
        rc = __netdev_start_xmit(ops, skb, dev, more);  /* sets netdev_xmit_more() */
        if (rc == NETDEV_TX_OK)
                txq_trans_update(txq);
        return rc;
}

__netdev_start_xmit finally calls ops->ndo_start_xmit(skb, dev) — the driver’s transmit function. A driver that sees netdev_xmit_more() == true posts the descriptor to its TX ring but defers the doorbell (the MMIO write that tells the NIC “go”), and only rings it on the last packet — coalescing N doorbell writes into one. This is a large win at high packet rates because the doorbell is an expensive PCIe transaction.

7. ndo_start_xmit, netdev_tx_t, and BQL

ndo_start_xmit returns a netdev_tx_t (include/linux/netdevice.h):

enum netdev_tx {
        ...
        NETDEV_TX_OK   = 0x00,   /* driver took care of packet */
        NETDEV_TX_BUSY = 0x10,   /* driver tx path was busy */
};
typedef enum netdev_tx netdev_tx_t;

NETDEV_TX_OK means the driver consumed the skb (it now owns freeing it on TX completion). NETDEV_TX_BUSY means the ring was full and the stack must requeue the skb onto the qdisc (dev_requeue_skb) and retry — a return of NETDEV_TX_BUSY is a soft event, but a driver that returns it when the queue was not stopped triggers the BUG ... code 16 warning seen in sch_direct_xmit. The driver also runs Byte Queue Limits (BQL) accounting here: it calls netdev_tx_sent_queue(txq, bytes) for each packet it posts. BQL tracks the bytes in flight in the hardware ring with a Dynamic Queue Limit (dql); when the in-flight bytes exceed the limit, netdev_tx_sent_queue sets __QUEUE_STATE_STACK_XOFF, which stops the TX queue so the qdisc holds packets instead of overstuffing the ring:

static inline void netdev_tx_sent_queue(struct netdev_queue *dev_queue, unsigned int bytes)
{
        dql_queued(&dev_queue->dql, bytes);
        if (likely(dql_avail(&dev_queue->dql) >= 0))
                return;
        set_bit(__QUEUE_STATE_STACK_XOFF, &dev_queue->state);   /* stop the queue */
        ...
}

When the NIC later raises a TX-completion interrupt, the driver’s TX-clean routine calls netdev_tx_completed_queue(txq, pkts, bytes), which feeds dql_completed and, if room reopened, clears __QUEUE_STATE_STACK_XOFF and calls netif_schedule_queue to wake the qdisc. The point of BQL is to keep the hardware ring just full enough to never starve, but not so full that latency-sensitive packets sit behind a backlog of bulk data — the kernel’s answer to buffer-bloat at the driver ring. The full treatment is Byte Queue Limits and Buffer Bloat.

8. The wire, and asynchronous completion

After the doorbell, the NIC DMA-reads each descriptor’s buffer and serializes the bytes onto the medium. The skb is not freed by send() — it lives until the NIC signals TX completion (via interrupt or NAPI TX poll), at which point the driver frees it (often deferred to net_tx_action’s completion_queue to free in softirq context). For TCP, the skb stays on the retransmission queue even after the NIC transmits it, and is only freed once the peer’s ACK arrives — because TCP may need to retransmit. This asynchrony is why a successful send() says nothing about whether the data reached the peer.

---

Failure Modes and How to Diagnose Them

TX queue stuck / watchdog timeout. If a driver bug leaves the TX queue stopped (XOFF set but never cleared), packets pile up in the qdisc and the kernel’s dev_watchdog timer eventually fires NETDEV transmit queue timed out (tx_timeout), often forcing a NIC reset. BQL’s XOFF/XON dance is a frequent culprit when a driver’s completion path miscounts bytes — the bytes passed to netdev_tx_sent_queue must exactly match those passed to netdev_tx_completed_queue, or the queue stalls forever (the kernel comment in netdev_tx_completed_queue calls this out explicitly).

Buffer-bloat / latency under bulk transfer. Without BQL (or with it disabled), the hardware ring fills with seconds of bulk data and interactive packets queue behind it — high ping times during an upload. Diagnose with tc -s qdisc show dev <dev> (backlog bytes) and the BQL sysfs files under /sys/class/net/<dev>/queues/tx-*/byte_queue_limits/. The cure is BQL (default on for capable drivers) plus a smart qdisc like fq_codel or fq.

NETDEV_TX_BUSY storms. A driver returning NETDEV_TX_BUSY while its queue was not stopped causes the stack to requeue and immediately retry, burning CPU. The BUG %s code 16 ratelimited warning from sch_direct_xmit is the fingerprint.

Qdisc drops. A bandwidth-limiting qdisc (tbf, htb, or fq_codel under overload) drops on enqueue with SKB_DROP_REASON_QDISC_DROP. tc -s qdisc shows the per-qdisc drop counter. This is policy, not error — the qdisc is doing its job.

GSO/checksum mismatches. If a device advertises an offload it does not correctly implement, validate_xmit_skb’s software fallback masks some bugs, but malformed checksums or oversized segments can corrupt traffic. ethtool -K <dev> tso off gso off to isolate, then compare.

Instrumentation. The TX path exposes trace_net_dev_queue, trace_qdisc_enqueue, trace_net_dev_start_xmit, and trace_net_dev_xmit (with the driver’s return code) — perf trace/bpftrace on these pinpoints where a packet stalled or which return code the driver gave.

---

Alternatives and When to Choose Them

• The standard qdisc path (this note) is the default for all socket traffic that wants the kernel’s scheduling, shaping, and fairness. fq paired with BBR, or fq_codel, is the modern recommendation for general egress.
• XDP XDP_TX/XDP_REDIRECT transmits frames from the driver/eBPF layer without ever building a socket-layer skb or touching the qdisc — for line-rate forwarding/load-balancing planes.
• AF_XDP transmits raw frames from a userspace UMEM ring straight to the driver (__dev_direct_xmit / xsk TX), bypassing the protocol stack and qdisc entirely — for DPDK-style userspace data planes.
• io_uring send/sendmsg still uses this exact kernel path below the syscall, but batches submissions from userspace to cut syscall overhead — choose it to reduce per-send syscall cost, not to change the egress mechanics.

---

Production Notes

The xmit_more doorbell-batching mechanism (introduced years ago and refined into the per-CPU softnet_data.xmit.more model used in 6.12) is invisible to applications but central to 100 Gbit/s+ NIC throughput — the kernel networking developers repeatedly cite the PCIe doorbell write as a top per-packet cost that batching amortizes. BQL, similarly, is one of the quieter buffer-bloat wins of the last decade: by sizing the in-flight bytes dynamically to the link’s drain rate rather than to a fixed descriptor count, it keeps the hardware ring shallow on slow links and deep on fast ones automatically. A common operational mistake is conflating the qdisc backlog with the driver ring: shaping with htb/tbf controls the former, BQL controls the latter, and both must be considered to reason about end-to-end latency. Another is assuming send() success means delivery — it means “queued locally”; for delivery semantics you need application-level acknowledgement or TCP’s own ACK accounting.

The dev_tx_weight = 64 default used by __qdisc_run is confirmed in net/core/hotdata.c at v6.12 (runtime-tunable via sysctl net.core.dev_weight, scaled by dev_weight_tx_bias); the netdev_tx_t enum, xmit_more, the BQL helpers, and the full call chain are confirmed directly in v6.12 source (net/sched/sch_generic.c, include/linux/netdevice.h).

Uncertain

Verify: that fq_codel is the effective default qdisc on a given system. Reason: the default is net.core.default_qdisc, whose compiled-in value comes from a Kconfig choice (CONFIG_DEFAULT_NET_SCH) and is commonly overridden by distributions and systemd (which historically sets fq_codel); it is a build/distro policy decision, not a single fixed kernel constant. To resolve: check sysctl net.core.default_qdisc and tc qdisc show on the target system, and the kernel’s CONFIG_DEFAULT_NET_SCH at build time. uncertain

---

See Also

• The Network Receive Path — the mirror-image ingress path (interrupt-then-poll)
• Queueing Disciplines qdisc — the qdisc enqueue/dequeue policy this path drives
• Classful and Classless Qdiscs — fq_codel/fq vs HTB shaping
• Byte Queue Limits and Buffer Bloat — the BQL throttle on the hardware ring
• Segmentation Offloads GSO TSO — the GSO segmentation validate_xmit_skb_list performs
• Checksum Offloads — the checksum completion in validate_xmit_skb
• The IP Layer — ip_queue_xmit / ip_output / ip_finish_output2 at L3
• The Netfilter Framework and Hooks — the OUTPUT and POSTROUTING hooks
• Network Address Translation NAT — SNAT/masquerade at POSTROUTING
• The Neighbour Subsystem / ARP and Neighbour Discovery — L2 header resolution in neigh_output
• NET_RX and NET_TX Softirqs — net_tx_action and NET_TX_SOFTIRQ
• The TCP Protocol in Linux / The UDP Protocol in Linux — the *_sendmsg skb builders
• MOC: Linux Networking Stack MOC