vault

Socket Wait Queues and Wakeups

15 min read

Socket Wait Queues and Wakeups

Blocking socket I/O is built on a single per-socket wait queue. Every struct socket embeds a struct socket_wq whose first field is a wait_queue_head_t; struct sock points at it through sk->sk_wq and reaches it with the sk_sleep() helper. A thread that calls recv() on an empty socket adds itself to this queue and sleeps; when a packet later arrives in softirq context, the protocol invokes the socket’s sk_data_ready callback (by default sock_def_readable), which wakes everyone parked on the queue. The symmetric callbacks sk_write_space (send buffer freed) and sk_state_change (connection state moved) wake writers and connection-waiters the same way. poll/epoll plug into the same queue: a socket’s ->poll method both reports current readiness and registers the poll-waiter on sk->sk_wq, so the very same wakeup that unblocks a sleeping recv() also fires an epoll instance (net/core/sock.c, v6.12 LTS).

This note traces the wakeup machinery end to end: the queue, the three protocol callbacks, how recv() sleeps via sk_wait_data, how poll/epoll register on the queue, and the memory barriers that make it race-free. The general kernel sleep/wake primitives (wait_queue_head_t, prepare_to_wait, wake_up, TASK_INTERRUPTIBLE) are explained in Wait Queues and Task Blocking; this note covers the networking-specific layer built on top of them. The structures are in struct socket and struct sock; the readiness-notification interface this feeds is epoll and Scalable Readiness Notification. All code is pinned to v6.12 LTS (2024-11-17), a current long-term branch in the 2026 kernel era alongside 6.18 LTS.

Mental Model — One Queue, Three Doorbells, Two Audiences

A socket has exactly one wait queue but two kinds of waiter. The first kind is a thread blocked inside a syscall (recv(), send(), connect()) — it adds itself with prepare_to_wait/add_wait_queue, sets itself TASK_INTERRUPTIBLE, and calls schedule(). The second kind is a poll-waiter registered by poll/select/epoll — it does not sleep on this queue itself; instead it leaves a callback (an epoll entry, or a poll_table entry) so that a wakeup notifies the polling subsystem, which then wakes the thread sitting in epoll_wait().

Three “doorbells” can ring the queue, each rung by the protocol when a specific thing becomes true:

flowchart LR
  subgraph SK["struct sock"]
    WQ["sk_wq → socket_wq.wait<br/>(the single wait queue)"]
    DR["sk_data_ready<br/>= sock_def_readable"]
    WS["sk_write_space<br/>= sock_def_write_space"]
    SC["sk_state_change<br/>= sock_def_wakeup"]
  end
  RXIRQ["RX softirq:<br/>skb queued, rcv_nxt advanced"] -->|"calls"| DR
  TXACK["ACK frees retransmit queue<br/>/ TX completion"] -->|"calls"| WS
  STATE["3-way handshake done<br/>/ FIN / RST"] -->|"calls"| SC
  DR -->|"wake_up_interruptible_sync_poll(EPOLLIN)"| WQ
  WS -->|"wake_up_interruptible_sync_poll(EPOLLOUT)"| WQ
  SC -->|"wake_up_interruptible_all"| WQ
  WQ --> SLEEPER["blocked recv()/send()/connect()<br/>(woken_wake_function)"]
  WQ --> POLLER["poll/select/epoll entry<br/>(ep_poll_callback)"]

The wakeup topology of one socket. What it shows: three independent protocol callbacks (sk_data_ready, sk_write_space, sk_state_change) each ring the single per-socket wait queue when their respective condition becomes true, and both blocked syscalls and epoll entries are parked on that one queue. The insight: the kernel does not “notify epoll” separately from “wake a sleeping recv” — it is one wakeup to one queue, and the wakeup is tagged with a poll-mask (EPOLLIN/EPOLLOUT) so an epoll entry can filter for the event it cares about while a plain sleeper just wakes.

The Wait Queue Itself

struct socket_wq is defined in include/linux/net.h (lines 99–105):

struct socket_wq {
	/* Note: wait MUST be first field of socket_wq */
	wait_queue_head_t	wait;
	struct fasync_struct	*fasync_list;   // for SIGIO / async notification
	unsigned long		flags;          // SOCKWQ_ASYNC_NOSPACE, SOCKWQ_ASYNC_WAITDATA
	struct rcu_head		rcu;
} ____cacheline_aligned_in_smp;

The wait field being first is load-bearing: sk_sleep() (sock.h:1989) returns &rcu_dereference_raw(sk->sk_wq)->wait, and a BUILD_BUG_ON(offsetof(struct socket_wq, wait) != 0) enforces the layout so the cast is sound. struct socket embeds the socket_wq directly as its wq member (net.h:128), and sock_init_data_uid() points sk->sk_wq at it: RCU_INIT_POINTER(sk->sk_wq, &sock->wq) (sock.c:3551). The __rcu annotation and rcu_head exist because the socket can be torn down concurrently with a wakeup; the queue is read under rcu_read_lock() so a wakeup in flight cannot dereference freed memory.

sk_sleep() is the canonical way every networking sleep/wake site names the queue — add_wait_queue(sk_sleep(sk), &wait), prepare_to_wait(sk_sleep(sk), ...), wake_up_interruptible(&wq->wait). The two SOCKWQ_* flag bits live in socket_wq.flags and record why someone is waiting: SOCKWQ_ASYNC_WAITDATA (“a reader is blocked”) and SOCKWQ_ASYNC_NOSPACE (“a writer is blocked on a full send buffer”). They are set by sk_set_bit() and gate whether the async (SIGIO/fasync) notification path fires.

The Three Protocol Callbacks

sock_init_data_uid() installs the default handlers (sock.c:3558–3561):

sk->sk_state_change	=	sock_def_wakeup;
sk->sk_data_ready	=	sock_def_readable;
sk->sk_write_space	=	sock_def_write_space;
sk->sk_error_report	=	sock_def_error_report;

A protocol may override any of these (e.g. TCP and stream protocols substitute their own write-space handler in some paths), but the defaults are what most code uses and they show the pattern cleanly.

sk_data_ready → sock_def_readable — “there is data to read”

Called by the receive path after an skb has been queued and the relevant sequence number advanced. It wakes readers and signals EPOLLIN (sock.c:3439):

void sock_def_readable(struct sock *sk)
{
	struct socket_wq *wq;
	trace_sk_data_ready(sk);
	rcu_read_lock();
	wq = rcu_dereference(sk->sk_wq);
	if (skwq_has_sleeper(wq))                                  // (1) any waiter?
		wake_up_interruptible_sync_poll(&wq->wait, EPOLLIN | EPOLLPRI |
						EPOLLRDNORM | EPOLLRDBAND); // (2) wake with read-mask
	sk_wake_async_rcu(sk, SOCK_WAKE_WAITD, POLL_IN);            // (3) SIGIO path
	rcu_read_unlock();
}

Line (1), skwq_has_sleeper(), is the cheap fast path: if nobody is waiting, the whole wakeup is skipped (the common case for a busy socket whose reader is already running). Line (2) uses the poll-aware wake variant: wake_up_interruptible_sync_poll passes a key (EPOLLIN | EPOLLPRI | ...) so that a poll-waiter’s wake function (ep_poll_callback for epoll) can check whether this event matches what the waiter registered for, avoiding a spurious wakeup for an EPOLLOUT-only watcher. The _sync suffix is a scheduler hint: do not preempt the current (softirq) task to run the woken thread immediately — let the softirq finish first. Line (3) drives the older SIGIO/F_SETOWN asynchronous-I/O notification, only relevant if the socket requested it.

For TCP the actual call site is in the receive code: after tcp_data_queue() puts a segment on sk->sk_receive_queue and advances tp->rcv_nxt, it calls sk->sk_data_ready(sk). That is the moment a blocked recv() becomes runnable.

sk_write_space → sock_def_write_space — “there is room to send”

Called when committed send memory drops, e.g. an ACK freed part of the retransmit queue or a TX completion released skbs. It wakes writers and signals EPOLLOUT — but only once a meaningful amount of space is available (sock.c:3454):

static void sock_def_write_space(struct sock *sk)
{
	struct socket_wq *wq;
	rcu_read_lock();
	/* Do not wake up a writer until he can make "significant" progress. --DaveM */
	if (sock_writeable(sk)) {                                  // (1) < half sk_sndbuf?
		wq = rcu_dereference(sk->sk_wq);
		if (skwq_has_sleeper(wq))
			wake_up_interruptible_sync_poll(&wq->wait, EPOLLOUT |
						EPOLLWRNORM | EPOLLWRBAND);
		sk_wake_async_rcu(sk, SOCK_WAKE_SPACE, POLL_OUT);
	}
	rcu_read_unlock();
}

The guard at (1) is the hysteresis discussed in Socket Buffers and Memory Accounting: sock_writeable() returns true only when committed send bytes are below half of sk_sndbuf (sock.h:2509). Waking a writer the instant a single byte frees would thrash — the thread would write a little, refill the buffer, and block again. Requiring half-empty means the woken writer can do real work. There is a fast-path variant sock_def_write_space_wfree() (sock.c:3480) used when freeing send memory under RCU from the skb destructor, with an explicit smp_mb__after_atomic() to order the refcount decrement before the queue-empty check.

sk_state_change → sock_def_wakeup — “the connection state moved”

Called whenever the protocol state machine transitions — the SYN/SYN-ACK handshake completes (so a blocked connect() returns), or a FIN/RST arrives (so blocked readers learn the peer hung up). It is the bluntest of the three (sock.c:3416):

static void sock_def_wakeup(struct sock *sk)
{
	struct socket_wq *wq;
	rcu_read_lock();
	wq = rcu_dereference(sk->sk_wq);
	if (skwq_has_sleeper(wq))
		wake_up_interruptible_all(&wq->wait);   // wake EVERYONE, no poll-mask filter
	rcu_read_unlock();
}

It uses wake_up_interruptible_all with no poll key — a state change can make a socket simultaneously readable, writable, and error-ready, so every waiter of every kind is woken to re-evaluate. (sock_def_error_report, sock.c:3427, is the fourth callback, waking with EPOLLERR when sk_err is set.)

How recv() Blocks: sk_wait_data

When tcp_recvmsg() finds the receive queue empty and the socket is blocking, it sleeps in sk_wait_data() (sock.c:3113). The TCP call site (net/ipv4/tcp.c:2698–2705):

} else {
	tcp_cleanup_rbuf(sk, copied);             // send any pending ACK first
	err = sk_wait_data(sk, &timeo, last);     // sleep until data or timeout
	...
}

And sk_wait_data itself:

int sk_wait_data(struct sock *sk, long *timeo, const struct sk_buff *skb)
{
	DEFINE_WAIT_FUNC(wait, woken_wake_function);
	int rc;
	add_wait_queue(sk_sleep(sk), &wait);                       // (1) enqueue on socket wq
	sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);                     // (2) "a reader is waiting"
	rc = sk_wait_event(sk, timeo,
		skb_peek_tail(&sk->sk_receive_queue) != skb, &wait);   // (3) sleep on condition
	sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk);
	remove_wait_queue(sk_sleep(sk), &wait);
	return rc;
}

Step (1) adds the waiter to the socket’s queue using woken_wake_function (the wake function that pairs with wait_woken(), the modern memory-safe sleep pattern from Wait Queues and Task Blocking). Step (2) sets SOCKWQ_ASYNC_WAITDATA so the data-ready path knows a reader is parked. Step (3), the sk_wait_event macro (sock.h:1150), is the heart of it:

#define sk_wait_event(__sk, __timeo, __condition, __wait)        \
	({  int __rc, __dis = __sk->sk_disconnects;              \
	    release_sock(__sk);                                  \  // (a) drop socket lock
	    __rc = __condition;                                  \  // (b) re-check condition
	    if (!__rc) {                                          \
		*(__timeo) = wait_woken(__wait, TASK_INTERRUPTIBLE, *(__timeo)); \ // (c) sleep
	    }                                                    \
	    sched_annotate_sleep();                              \
	    lock_sock(__sk);                                     \  // (d) re-acquire lock
	    __rc = __dis == __sk->sk_disconnects ? __condition : -EPIPE; \
	    __rc;                                                \
	})

The subtlety: the socket lock must be dropped before sleeping (a), or no incoming packet could ever advance the receive queue (the softirq receive path needs the lock to enqueue). The condition is re-checked after arming the waiter but before sleeping (b) to close the wait/wakeup race, then wait_woken() sleeps (c) until the wake function flips the woken flag. On wake the lock is re-taken (d). This drop-recheck-sleep-relock dance is the canonical lockless wakeup pattern; the sk_disconnects check detects a disconnect that raced with the sleep and returns -EPIPE.

The datagram equivalent is __skb_wait_for_more_packets() (net/core/datagram.c:87): it uses prepare_to_wait_exclusive(sk_sleep(sk), ...) — exclusive so a single arriving datagram wakes only one of several blocked readers (thundering-herd avoidance), checks sock_error(sk), sk->sk_shutdown & RCV_SHUTDOWN, and signal_pending(current), then schedule_timeout(). The send-blocking counterpart sock_wait_for_wmem() (sock.c:2830) is the symmetric loop for a full sk_sndbuf, sleeping until sk_wmem_alloc < sk_sndbuf (detailed in Socket Buffers and Memory Accounting).

How poll/epoll Plug Into the Same Queue

poll, select, and epoll all call the socket’s ->poll method (proto_ops->poll), which for TCP is tcp_poll() (net/ipv4/tcp.c:510). Every ->poll implementation does two things in one call: register the caller on the wait queue, then compute the current readiness mask.

Registration is sock_poll_wait() (sock.h:2285), called first thing in tcp_poll (tcp.c:518):

static inline void sock_poll_wait(struct file *filp, struct socket *sock, poll_table *p)
{
	if (!poll_does_not_wait(p)) {
		poll_wait(filp, &sock->wq.wait, p);  // add poll-waiter to THIS socket's wq
		smp_mb();                            // pair with wq_has_sleeper barrier
	}
}

poll_wait() adds an entry to sock->wq.wait — the same queue a blocked recv() uses. For epoll, the registered callback is ep_poll_callback, so a later wake_up_..._poll(&wq->wait, EPOLLIN) invokes it, which moves the file to the epoll ready-list and wakes any thread in epoll_wait(). The poll_does_not_wait(p) guard skips registration on a pure readiness query (p == NULL), e.g. the second ->poll call epoll makes just to read the mask.

Uncertain

Verify: the epoll-internal names and mechanics — that the callback registered via poll_wait for epoll is ep_poll_callback, that it moves the file to a “ready-list”, and the precise poll_does_not_wait / two-call epoll readiness-query behavior. Reason: this note traced the socket-side registration (sock_poll_wait/poll_wait in include/net/sock.h) from primary source, but the epoll consumer side was not read from fs/eventpoll.c during this task. To resolve: read fs/eventpoll.c (ep_ptable_queue_proc, ep_poll_callback, ep_item_poll) at the v6.12 tag; the authoritative write-up is epoll and Scalable Readiness Notification. uncertain

Computing the mask is the rest of tcp_poll (tcp.c:558–606): it sets EPOLLIN | EPOLLRDNORM when tcp_stream_is_readable() (enough bytes past sk_rcvlowat), EPOLLOUT | EPOLLWRNORM when __sk_stream_is_writeable() (send space below the half-buffer threshold), EPOLLHUP when fully shut down, EPOLLERR on sk_err, and EPOLLPRI on urgent data. The critical race-breaker is at tcp.c:581–592: if the socket is not currently writable, tcp_poll sets SOCKWQ_ASYNC_NOSPACE / SOCK_NOSPACE, issues smp_mb__after_atomic(), then re-checks writability:

} else {  /* send SIGIO later */
	sk_set_bit(SOCKWQ_ASYNC_NOSPACE, sk);
	set_bit(SOCK_NOSPACE, &sk->sk_socket->flags);
	/* Race breaker. If space is freed after wspace test but before the
	 * flags are set, IO signal will be lost. Memory barrier pairs with
	 * the input side. */
	smp_mb__after_atomic();
	if (__sk_stream_is_writeable(sk, 1))
		mask |= EPOLLOUT | EPOLLWRNORM;
}

This is the classic poll/wakeup race: between “I tested not-writable” and “I registered to be told when writable”, the ACK that frees space might arrive on another CPU. Without the barrier and re-check, that wakeup would be lost and the writer would hang forever despite space being available. The same NOSPACE flag is what the write-space wakeup path checks to decide whether to ring the queue.

Memory Barriers — the wq_has_sleeper Race

The deepest correctness issue in this machinery is a documented races between adding to the wait queue and the wakeup test. The source spells it out in the skwq_has_sleeper comment (sock.h:2247–2270):

  CPU1                       CPU2
  sys_select                 receive packet
  __add_wait_queue           update tp->rcv_nxt
  tp->rcv_nxt check          sock_def_readable {
  schedule                       wq = rcu_dereference(sk->sk_wq);
                                 if (wq && waitqueue_active(&wq->wait))
                                     wake_up_interruptible(&wq->wait)
                             }

The danger: CPU1’s __add_wait_queue (making the queue non-empty) may sit in CPU1’s store buffer, and CPU2’s rcv_nxt update may sit in CPU2’s store buffer. If CPU2 checks waitqueue_active() and sees the queue still empty (CPU1’s add not yet visible) while CPU1 checks rcv_nxt and sees it not yet advanced (CPU2’s update not yet visible), CPU1 sleeps and CPU2 declines to wake it — a lost wakeup, and CPU1 “could then end up calling schedule and sleep forever”. The fix is a pair of full memory barriers: sock_poll_wait()/the wait-add side issues smp_mb() after adding (so the add is globally visible before the readiness test), and skwq_has_sleeper() → wq_has_sleeper() issues a barrier before reading waitqueue_active() (so the rcv_nxt update is visible before the sleeper check). This is why the wrappers skwq_has_sleeper and sock_poll_wait exist at all — they “wrap the memory barrier call. They were added due to the race found within the tcp code” (sock.h:2247–2248). The general memory-ordering rules are in Wait Queues and Task Blocking and the kernel’s memory-barriers.txt.

Failure Modes

Lost wakeup / hung writer. If a ->poll implementation tests writability, finds it false, and registers without the smp_mb__after_atomic() re-check (tcp.c:585), a concurrent space-freeing ACK can be missed and the writer never wakes. This is exactly the race the NOSPACE-flag re-check defends against; custom protocol code that omits it hangs under load.

Thundering herd on a shared datagram socket. Multiple processes blocked in recvfrom() on one UDP socket: because __skb_wait_for_more_packets uses prepare_to_wait_exclusive, a single datagram wakes only one waiter — good. But a state-change wake (sock_def_wakeup → wake_up_interruptible_all) wakes all of them, since a state change is not direction-specific. Designs that fan many readers onto one socket should expect occasional all-wake storms on connection events.

Spurious epoll wakeups. Because state changes wake every poll-waiter unconditionally, an epoll watching only EPOLLIN can be woken by an EPOLLOUT-causing event and must re-check via a non-blocking read returning EAGAIN. This is correct behavior, not a bug — but level-triggered code that assumes “woken ⇒ readable” will spin. Edge-triggered epoll (EPOLLET) is more sensitive: a wakeup tagged with the wrong mask that ep_poll_callback filters out will not re-arm the edge, so the poll-mask passed to wake_up_..._poll must be accurate, which is why each callback passes its specific EPOLLIN/EPOLLOUT key.

Sleeping with the socket lock held. Forgetting the release_sock() before schedule() (as sk_wait_event does) deadlocks: the softirq receive path cannot take the socket lock to enqueue the data that would wake the sleeper, so the wakeup never comes. Every blocking socket sleep must drop the lock.

Alternatives and When to Choose Them

The blocking model (one thread per connection sleeping in recv()) is simplest and is fine at low connection counts, but does not scale — a thread per socket is expensive. The readiness model — poll/select and especially epoll — lets one thread watch thousands of sockets by registering once on each queue and waiting centrally; this is the dominant server model and the surface the Go runtime netpoller integrates with. The completion model — io_uring — inverts it again: the kernel does the I/O and posts a completion, so there is no per-socket wait-queue dance in the hot path at all. For raw line-rate frames with no socket wait queue whatsoever, AF_XDP Zero-Copy Sockets bypasses the model entirely. Choose blocking for simplicity at small scale, epoll for C10K-style readiness multiplexing, io_uring for batched completion-driven async, and AF_XDP for a userspace dataplane.

Production Notes

The poll/wakeup memory-barrier race documented in skwq_has_sleeper is one of the most-cited cautionary tales in kernel networking — it is a real bug that bit TCP and motivated the wrapper functions, and it recurs whenever someone writes a new ->poll implementation and forgets the barrier or the post-registration re-check. The _sync variants of the wake calls (wake_up_interruptible_sync_poll) are a deliberate latency-versus-throughput tuning: they avoid preempting the waking (softirq) context so the network bottom-half can finish a batch of packets before the scheduler runs the woken thread, trading a little wakeup latency for fewer context switches under high packet rates. The half-buffer write-space hysteresis (“—DaveM”) is likewise a throughput optimization that occasionally surprises latency-sensitive applications: a writer is not woken the instant the smallest amount of space frees, so a tiny SO_SNDBUF plus large writes can see longer-than-expected blocking. Tooling-wise, a thread stuck in sk_wait_data shows up in /proc/<pid>/stack and in ss -tp as a process blocked on the socket; ss -tm (see Socket Buffers and Memory Accounting) reveals whether the buffer counters explain the block.

See Also

Graph View

Backlinks