vault

Pipe Capacity Atomicity and PIPE_BUF

15 min read

Pipe Capacity Atomicity and PIPE_BUF

A pipe is governed by two numbers that every correct multi-writer protocol depends on. The first is its capacity — by default 16 pages = 65,536 bytes on a 4 KiB-page system — which can be queried and grown with fcntl(fd, F_GETPIPE_SZ) / F_SETPIPE_SZ, bounded for unprivileged callers by /proc/sys/fs/pipe-max-size (default 1 MiB). The second, and the one that matters for correctness, is PIPE_BUF = 4096 bytes on Linux: POSIX guarantees that a write() of PIPE_BUF bytes or fewer is atomic — it lands in the pipe as one contiguous run, never interleaved with bytes from a concurrent writer — whereas a write() of more than PIPE_BUF bytes may be split and interleaved with other writers’ data (pipe(7)). That single guarantee is what lets many independent processes append log lines to one shared pipe without garbling each other’s records, provided each line is ≤ PIPE_BUF. This note walks the capacity-resizing machinery (F_SETPIPE_SZ, round_pipe_size, the rlimit ceilings), the exact atomicity contract and why PIPE_BUF is the threshold, and the subtle interaction between atomicity and O_NONBLOCK. The pipe’s internal ring structure that underlies all of this is in Pipes and the Pipe Buffer.

This note is pinned to Linux 6.12 LTS, with constants cross-checked against 6.18 LTS (fs/pipe.c @ v6.12, @ v6.18); PIPE_BUF, PIPE_DEF_BUFFERS, and the default pipe_max_size are identical in both.

Two Different Sizes — Don’t Conflate Them

The most common confusion is treating “the pipe holds 64 KiB” and “writes up to 4 KiB are atomic” as the same fact. They are not, and the kernel header says so explicitly:

/* Differs from PIPE_BUF in that PIPE_SIZE is the length of the actual
   memory allocation, whereas PIPE_BUF makes atomicity guarantees.  */
#define PIPE_SIZE        PAGE_SIZE

— include/linux/pipe_fs_i.h lines 257-259.

  • Capacity (ring_size × PAGE_SIZE, default 16 × 4096 = 65536) is how much unread data the pipe can hold before a writer blocks. It is tunable.
  • PIPE_BUF (a fixed 4096) is the largest single write the kernel promises to deliver atomically. It is a constant of the ABI, not tunable, and on Linux it equals one page.

Growing the capacity to 1 MiB does not raise the atomic-write threshold — PIPE_BUF stays 4096 regardless of how big you make the pipe. Atomicity and capacity are orthogonal. The rest of this note treats them separately.

flowchart TB
  CAP["Pipe CAPACITY<br/>ring_size × PAGE_SIZE<br/>default 16 × 4096 = 65536 B"]
  PB["PIPE_BUF = 4096 B<br/>(one page, fixed)"]
  CAP --> Q1["governs: when does a writer BLOCK?<br/>(pipe full → block / EAGAIN)"]
  PB --> Q2["governs: is a single write() ATOMIC?<br/>(≤ PIPE_BUF → never interleaved)"]
  CAP -.->|"tunable via F_SETPIPE_SZ<br/>up to pipe-max-size (1 MiB)"| TUNE["resizable"]
  PB -.->|"compile-time constant<br/>NOT changed by resizing"| FIXED["fixed by ABI"]

Capacity vs PIPE_BUF. What it shows: capacity controls flow (when writers block) and is tunable; PIPE_BUF controls atomicity (whether a write can be torn apart by a concurrent writer) and is fixed at one page. The insight to take: enlarging a pipe buys you more buffering and fewer blocks, but it does not let you write a larger atomic record — keep each message that must stay intact under multiple writers at or below 4096 bytes.

The Atomicity Guarantee, Precisely

POSIX.1 states, and Linux honors, the following contract (quoting pipe(7)):

“POSIX.1 says that writes of less than PIPE_BUF bytes must be atomic: the output data is written to the pipe as a contiguous sequence. Writes of more than PIPE_BUF bytes may be nonatomic.”

“Atomic” here means non-interleaved: if process A and process B both write to the same pipe concurrently, and each write is ≤ PIPE_BUF, then A’s bytes appear in the stream as one unbroken run and B’s as another — the reader never sees a fragment of A’s write sandwiched inside B’s. The kernel achieves this trivially: pipe_write() holds pipe->mutex for the entire duration of a write that fits, so no other writer can splice its bytes in between (see the locking in Pipes and the Pipe Buffer).

For writes larger than PIPE_BUF, no such promise holds. A large write may block partway when the pipe fills, the mutex is released while the writer sleeps on wr_wait, and another writer can run and deposit its own bytes — so the two writes can end up interleaved in the stream. From pipe(7):

“The write is nonatomic: the data given to write(2) may be interleaved with write(2)s by other process[es]; the write(2) blocks until n bytes have been written.”

Note the boundary subtlety in the standard’s wording: it says writes of less than PIPE_BUF must be atomic. A write of exactly PIPE_BUF bytes is the edge case — on Linux a write of exactly 4096 bytes fits in a single page and is in practice handled atomically, but portable code treats PIPE_BUF as the exclusive upper bound and keeps atomic records strictly under it (i.e. n < PIPE_BUF).

Why multi-writer logging is the canonical use

The textbook application is many cooperating processes writing to one shared pipe — for example several worker processes that all inherited the write end of a pipe and each emit log lines to a single collector reading the other end. As long as each write() is one complete line of at most PIPE_BUF bytes, the collector reads cleanly separated lines even though the writers never coordinate with each other. There is no need for an external lock around the pipe: the kernel’s per-write mutex is the lock, and PIPE_BUF is the size budget within which that lock guarantees a clean record. This is precisely why syslog-style fan-in over a pipe works, and why log lines longer than 4 KiB can suddenly start interleaving and corrupting under load — the line crossed PIPE_BUF and lost its atomicity.

The O_NONBLOCK Interaction

Atomicity and blocking interact in a way that trips people up, because non-blocking mode changes what a partial write means. The four cases, straight from pipe(7), with n the requested byte count:

O_NONBLOCK disabled, n ≤ PIPE_BUF: “All n bytes are written atomically; write(2) may block if there is not room for n bytes to be written immediately.” The write either completes in full (atomically) or sleeps until it can — it never returns a partial count for a small write.

O_NONBLOCK enabled, n ≤ PIPE_BUF: “If there is room to write n bytes to the pipe, then write(2) succeeds immediately, writing all n bytes; otherwise write(2) fails, with errno set to EAGAIN.” This is the crucial property: for a small write, non-blocking mode is all-or-nothing — you get the full atomic write or EAGAIN, never a torn partial. So a non-blocking multi-writer logger can safely retry on EAGAIN knowing it has not emitted a half-line.

O_NONBLOCK disabled, n > PIPE_BUF: “The write is nonatomic … the write(2) blocks until n bytes have been written.” The big write eventually delivers all n bytes but may interleave with other writers along the way.

O_NONBLOCK enabled, n > PIPE_BUF: “If the pipe is full, then write(2) fails, with errno set to EAGAIN. Otherwise, from 1 to n bytes may be written (i.e., a ‘partial write’ may occur).” Here you can get a partial write — a return value between 1 and n. The caller must loop, and because the write is non-atomic it may also be interleaved.

The takeaway: keep records ≤ PIPE_BUF and you get atomicity in both blocking and non-blocking modes (all-or-nothing or EAGAIN). Cross PIPE_BUF and you must handle partial writes and accept possible interleaving. The kernel’s pipe_write() returns the partial count by setting ret = ret ?: -EAGAIN only when nothing was written, otherwise returning the bytes copied so far (fs/pipe.c lines 557-562).

Resizing a Pipe: F_SETPIPE_SZ

The capacity is read and changed with fcntl() (F_SETPIPE_SZ(2const)):

long cur = fcntl(fd, F_GETPIPE_SZ);          /* current capacity in bytes */
long got = fcntl(fd, F_SETPIPE_SZ, 1 << 20); /* request 1 MiB; returns actual size */

F_GETPIPE_SZ returns the capacity in bytes (pipe->max_usage * PAGE_SIZE). F_SETPIPE_SZ takes the desired size in bytes as arg, sets the capacity to at least that, and returns the actual capacity used, which may be larger because the kernel rounds up. Per the man page: “When allocating the buffer for the pipe, the kernel may use a capacity larger than arg … the allocation is the next higher power-of-two page-size multiple of the requested size.”

In the kernel, F_SETPIPE_SZ lands in pipe_fcntl() → pipe_set_size() → round_pipe_size() (fs/pipe.c lines 1397-1422):

unsigned int round_pipe_size(unsigned int size)
{
    if (size > (1U << 31))          return 0;        /* hard upper bound: 2 GiB */
    if (size < PAGE_SIZE)           return PAGE_SIZE;/* floor: one page */
    return roundup_pow_of_two(size);                 /* round up to power-of-two */
}

So three rules are baked in: (1) a request below one page is silently raised to one page (the POSIX minimum); (2) every request is rounded up to the next power-of-two multiple of the page size — the ring must be a power of two so the & (ring_size - 1) masking in the read/write paths works; (3) anything above 2 GiB is rejected. Ask for 10000 bytes on a 4 KiB-page box and you get roundup_pow_of_two(10000) = 16384 (4 pages); F_SETPIPE_SZ returns 16384.

pipe_set_size() then enforces the limits and performs the resize (lines 1335-1380):

static long pipe_set_size(struct pipe_inode_info *pipe, unsigned int arg)
{
    size = round_pipe_size(arg);
    nr_slots = size >> PAGE_SHIFT;
    if (!nr_slots) return -EINVAL;
    /* growing past the ceiling needs privilege */
    if (nr_slots > pipe->max_usage && size > pipe_max_size && !capable(CAP_SYS_RESOURCE))
        return -EPERM;
    ...
    ret = pipe_resize_ring(pipe, nr_slots);   /* re-allocate the slot array */
    ...
    return pipe->max_usage * PAGE_SIZE;        /* the actual capacity */
}

pipe_resize_ring() allocates a fresh slot array, copies the existing unread buffers into it, and swaps it in under a spinlock. It can also shrink a pipe, but only if the current unread data fits in the smaller ring — otherwise it returns -EBUSY (lines 1281-1286):

n = pipe_occupancy(head, tail);
if (nr_slots < n) {           /* shrinking below current occupancy is illegal */
    spin_unlock_irq(&pipe->rd_wait.lock);
    kfree(bufs);
    return -EBUSY;
}

So F_SETPIPE_SZ’s error surface is: -EBUSY (cannot shrink below the data currently buffered), -EPERM (unprivileged caller asking to exceed pipe-max-size), and -EINVAL (degenerate zero size). The man page confirms each (F_SETPIPE_SZ(2const)).

The Ceilings: pipe-max-size and the Per-User Page Limits

Three /proc knobs bound how big and how many pipes an unprivileged user can have (pipe(7), /proc files):

static unsigned int pipe_max_size = 1048576;                          /* 1 MiB */
static unsigned long pipe_user_pages_hard;                            /* 0 = unset */
static unsigned long pipe_user_pages_soft = PIPE_DEF_BUFFERS * INR_OPEN_CUR; /* 16 * 1024 */

— fs/pipe.c lines 54-60.

  • /proc/sys/fs/pipe-max-size (since Linux 2.6.35; default 1048576 = 1 MiB) — the largest capacity an unprivileged process may set via F_SETPIPE_SZ. A process with CAP_SYS_RESOURCE may exceed it (up to the 2 GiB hard cap in round_pipe_size). This is the per-pipe ceiling enforced by the -EPERM check above.
  • /proc/sys/fs/pipe-user-pages-soft (since Linux 4.5; default 16384 pages, “which permits creating up to 1024 pipes with the default capacity”) — once a single unprivileged user’s total pipe pages cross this soft limit, new pipes are shrunk to two pages (PIPE_MIN_DEF_BUFFERS; it was one page before Linux 5.14), and attempts to grow a pipe are denied. This is the logic at the top of alloc_pipe_info() and the soft-limit branch in pipe_set_size().
  • /proc/sys/fs/pipe-user-pages-hard (since Linux 4.5; default 0 = no limit) — a hard cap; once a user crosses it, creating new pipes and growing existing ones both fail outright.

The unit confusion to watch: pipe-max-size is in bytes, the two pipe-user-pages-* knobs are in pages. The accounting is per-user via account_pipe_buffers(), which atomically adjusts user->pipe_bufs, and unprivileged is defined as lacking both CAP_SYS_RESOURCE and CAP_SYS_ADMIN (pipe_is_unprivileged_user(), lines 785-788). These limits exist to stop one user from pinning unbounded kernel memory in pipe pages — a denial-of-service vector, since pipe pages are unswappable kernel allocations.

Worked Example: Atomic Multi-Writer Logging

Two processes appending lines to one pipe, relying on PIPE_BUF atomicity:

#include <unistd.h>
#include <limits.h>     /* PIPE_BUF */
#include <stdio.h>
#include <string.h>
 
/* Each process calls this with the SAME inherited write fd `wfd`.
   No lock between processes — PIPE_BUF atomicity is the lock. */
void log_line(int wfd, const char *line) {
    char buf[PIPE_BUF];                 /* cap the record at PIPE_BUF */
    int n = snprintf(buf, sizeof buf, "%d: %s\n", getpid(), line);
    if (n > PIPE_BUF) n = PIPE_BUF;     /* never exceed the atomic threshold */
    write(wfd, buf, n);                 /* single write ≤ PIPE_BUF → atomic, non-interleaved */
}

Because every write() here is ≤ PIPE_BUF, the kernel guarantees each line lands contiguously: the reader on the other end sees whole lines from each PID, never a fragment of one PID’s line spliced inside another’s — even with the two processes racing and no mutex between them. The moment a line could exceed PIPE_BUF, this breaks and you would need an explicit lock or a length-prefixed framing protocol. Raising the pipe capacity with F_SETPIPE_SZ would not help, because — as established above — capacity does not change PIPE_BUF.

A second example, sizing a pipe for throughput rather than atomicity:

int fds[2];
pipe(fds);
long cur = fcntl(fds[0], F_GETPIPE_SZ);          /* e.g. 65536 */
long got = fcntl(fds[0], F_SETPIPE_SZ, 1 << 20); /* ask for 1 MiB */
/* got == 1048576 on success; -EPERM if unprivileged and > pipe-max-size; */
/* setting on either fd resizes the shared pipe — both ends see the new capacity */

Enlarging the pipe reduces how often a bursty producer blocks (more in-flight buffering), which can meaningfully raise throughput when producer and consumer run at uneven rates — but it costs pinned kernel memory and counts against the per-user page limits.

Failure Modes and Common Misunderstandings

  • “My log lines are interleaving/corrupting under load.” The lines grew past PIPE_BUF (4096 bytes), losing atomicity. Shorten the records, or serialize writers with an explicit lock, or switch to a framed protocol. Enlarging the pipe will not fix it.
  • “F_SETPIPE_SZ returned a bigger number than I asked for.” Expected — the kernel rounds up to the next power-of-two page multiple. Always use the return value, not your requested size, as the true capacity.
  • “F_SETPIPE_SZ failed with EBUSY.” You tried to shrink the pipe below the amount of data currently buffered in it. Drain the pipe first, or shrink to a size that still holds the unread bytes.
  • “F_SETPIPE_SZ failed with EPERM even though I’m under 1 MiB.” Either the request exceeded pipe-max-size (lowered by an admin), or your user crossed pipe-user-pages-soft/hard, which forbids growing pipes. Check the /proc knobs and your total pipe page usage.
  • Assuming a partial write can’t happen. For writes > PIPE_BUF in non-blocking mode, a partial write (return value < n) is normal and you must loop. Only writes ≤ PIPE_BUF are all-or-nothing.
  • Confusing PIPE_BUF with the buffer size for read(). PIPE_BUF constrains write atomicity only. Reads have no atomicity guarantee and freely return less than requested.

Alternatives and When to Choose Them

When PIPE_BUF atomicity is not enough for your record size or coordination needs:

  • Records larger than PIPE_BUF that must stay intact under multiple writers → put an explicit lock around the writes (a futex-backed mutex in shared memory, or a file lock), or use a datagram Unix socket (SOCK_DGRAM), where each send() is a discrete message preserved whole regardless of size (up to the socket buffer).
  • Discrete, prioritized messages with kernel queuing → POSIX Message Queues, which deliver whole messages and never interleave by design.
  • Preserving message boundaries on a pipe → O_DIRECT packet mode (see Pipes and the Pipe Buffer), where each write is one packet read whole — though a packet still cannot exceed the pipe’s per-buffer page size cleanly and is split past PIPE_BUF.

For the common case — short log lines or single-token jobserver writes — plain pipe atomicity under PIPE_BUF is the simplest correct answer and needs no extra synchronization.

Production Notes

The PIPE_BUF guarantee is load-bearing in real systems: container log collectors, syslog fan-in, and the GNU make jobserver all rely on it. The jobserver passes single-byte tokens through a shared pipe to throttle parallel jobs; because one byte is trivially ≤ PIPE_BUF, token writes never tear, and the pipe acts as a clean counting semaphore. The kernel’s PIPE_MIN_DEF_BUFFERS == 2 floor (rather than one page) exists specifically so a jobserver pipe under memory pressure does not deadlock when many make processes block trying to write tokens back before any reads one (source comment, fs/pipe.c lines 35-48).

A frequent real-world tuning move is bumping pipe capacity for high-throughput data plumbing — e.g. piping large tar/dd/compression streams — where the default 64 KiB ring forces frequent context switches between producer and consumer. Tools like pv and mbuffer exist partly to add a larger user-space buffer for exactly this reason; F_SETPIPE_SZ is the in-kernel equivalent, capped by pipe-max-size. Administrators sometimes raise pipe-max-size system-wide for such workloads, weighing it against the unswappable kernel memory each large pipe pins.

See Also

Graph View