vault

sendfile and File-to-Socket Transfer

13 min read

sendfile and File-to-Socket Transfer

sendfile() is the classic “ship a file straight out a socket” system call: it “copies data between one file descriptor and another” entirely inside the kernel, so the bytes never make the round trip into a user-space buffer and back (sendfile(2)). The input descriptor in_fd must be something that “supports mmap(2)-like operations (i.e., it cannot be a socket)” — in practice a regular file — and the data flows page cache → socket send buffer with no user-space bounce, and on capable hardware (scatter-gather DMA plus checksum offload) with no CPU copy at all down to the network interface card (NIC). That elimination of two copies and two context switches per chunk is why sendfile powers web servers, video streamers, and fast file-copy paths. The original restriction that out_fd had to be a socket was lifted in Linux 2.6.33, so out_fd can now be any file. On a modern kernel sendfile is no longer a bespoke code path: it is splice under the hood — do_sendfile() calls do_splice_direct(), which runs splice_direct_to_actor() over an internal kernel pipe. This note traces that path at Linux 6.12 LTS (fs/read_write.c, fs/splice.c).

This note is pinned to Linux 6.12 LTS (released 2024-11-17), with the man-pages as the stable-ABI reference. The general zero-copy primitive it builds on lives in splice and Zero-Copy Data Transfer; where the data lands on the network side is Socket Buffers and Memory Accounting.

Mental Model — Cut Out the User-Space Bounce

To appreciate sendfile you must first picture the naive way to send a file over a socket and count its copies. A server that does read(file, buf, n); write(sock, buf, n); in a loop moves each chunk of data four times and crosses the user/kernel boundary four times (kernel-internals: splice/sendfile):

  1. disk → page cache (DMA, unavoidable),
  2. page cache → user buffer (CPU copy, on the read),
  3. user buffer → socket buffer (CPU copy, on the write),
  4. socket buffer → NIC (DMA, unavoidable).

Steps 2 and 3 are pure waste: the bytes go up into user space and immediately back down, untouched. sendfile deletes them. The data goes page cache → socket buffer directly inside the kernel, and the application never sees a buffer at all. With a NIC that supports scatter-gather DMA, even the “page cache → socket buffer” step becomes a copy of references, not bytes: the socket’s sk_buff simply points at the page-cache page, and the NIC DMA-reads that page in place — zero CPU copies on the whole path.

flowchart TB
  subgraph NAIVE["read() + write()  — 4 copies, 4 boundary crossings"]
    direction LR
    D1["disk"] -->|DMA| PC1["page cache"]
    PC1 -->|"CPU copy<br/>(read)"| UB["user buffer"]
    UB -->|"CPU copy<br/>(write)"| SB1["socket buffer"]
    SB1 -->|DMA| N1["NIC"]
  end
  subgraph SF["sendfile() with scatter-gather + checksum offload — 0 CPU copies"]
    direction LR
    D2["disk"] -->|DMA| PC2["page cache"]
    PC2 -->|"reference<br/>(no byte copy)"| SB2["sk_buff frag<br/>→ same page"]
    SB2 -->|"DMA reads<br/>page cache directly"| N2["NIC"]
  end

Naive copy loop versus sendfile. What it shows: the read+write path pays two CPU copies (page cache↔user buffer) and four user/kernel transitions per chunk; sendfile removes the user-space bounce and, with scatter-gather DMA, lets the NIC read the page-cache page directly so no CPU copy happens at all. The insight to take: the win is not “fewer syscalls” so much as “the bytes never touch a CPU register on the way out” — sendfile turns a memory-bandwidth-bound operation into a reference-passing operation.

Mechanical Walk-through — do_sendfile() Is a Thin Shell Over splice

The syscall surface

There are two entry points in fs/read_write.c (v6.12), differing only in the width of the offset they accept:

SYSCALL_DEFINE4(sendfile,   int out_fd, int in_fd, off_t  __user *offset, size_t count)
SYSCALL_DEFINE4(sendfile64, int out_fd, int in_fd, loff_t __user *offset, size_t count)

Both marshal the user offset and call the internal do_sendfile(). The 32-bit off_t form exists for historical ABI reasons; sendfile64 (added in Linux 2.4) takes the 64-bit loff_t so large files work on 32-bit userlands (sendfile(2)).

The user-visible prototype is:

ssize_t sendfile(int out_fd, int in_fd, off_t *offset, size_t count);

offset, if non-NULL, points to the starting position in in_fd and is updated on return to the new position — and crucially, when offset is non-NULL the file’s own seek position is not moved (which is what lets concurrent senders each carry their own offset over one shared fd). If offset is NULL, the kernel reads from the file’s current position and advances it.

What do_sendfile() actually does

Walking the v6.12 body, do_sendfile():

  1. Resolves and validates both fds. It fetches in_fd and requires FMODE_READ:

    if (!(fd_file(in)->f_mode & FMODE_READ))  goto fput_in;

    and fetches out_fd and requires FMODE_WRITE:

    if (!(fd_file(out)->f_mode & FMODE_WRITE)) goto fput_out;

    A non-readable input or non-writable output yields EBADF.

  2. Verifies the I/O regions. It calls rw_verify_area(READ, fd_file(in), &pos, count) and rw_verify_area(WRITE, fd_file(out), &out_pos, count) — these run the mandatory-lock and LSM/permission checks on the byte ranges.

  3. Clamps the count. if (count > MAX_RW_COUNT) count = MAX_RW_COUNT; — this is why the man page documents the cap precisely: “sendfile() will transfer at most 0x7ffff000 (2,147,479,552) bytes, returning the number of bytes actually transferred.” MAX_RW_COUNT is INT_MAX & PAGE_MASK, i.e. INT_MAX rounded down to a page boundary — 0x7ffff000. A larger request is silently truncated; the caller must loop.

  4. Splices. For the normal (non-pipe) destination it calls into the splice machinery:

    retval = do_splice_direct(fd_file(in), &pos, fd_file(out), &out_pos,
                          count, fl);

    For a pipe destination it instead calls splice_file_to_pipe(...). That pipe branch corresponds to the man page note that “If out_fd refers to a pipe, then sendfile() internally calls splice(2)” (since Linux 5.12) — but as we will see, every sendfile is really splice underneath.

do_splice_direct → splice_direct_to_actor: the internal pipe

do_splice_direct() and splice_direct_to_actor() live in fs/splice.c. The comment on splice_direct_to_actor reveals the whole trick:

/*
 * This is a special case helper to splice directly between two
 * points, without requiring an explicit pipe. Internally an allocated
 * pipe is cached in the process, and reused during the lifetime of
 * that process.
 */

sendfile’s problem is that ordinary splice() requires one side to be a pipe, but sendfile’s two sides are a file and a socket — neither is a pipe. The solution is a hidden, per-task cached pipe. splice_direct_to_actor borrows a small kernel-internal pipe (allocated lazily and stashed on the task so it can be reused across calls), splices the file’s page-cache pages into that pipe (installing page references, exactly as in splice and Zero-Copy Data Transfer), then drains the pipe into the destination by running an actor, direct_splice_actor:

static int direct_splice_actor(struct pipe_inode_info *pipe,
                               struct splice_desc *sd)
{
    struct file *file = sd->u.file;
    file_start_write(file);
    long ret = do_splice_from(pipe, file, sd->opos, sd->total_len, sd->flags);
    file_end_write(file);
    return ret;
}

So the data path is: file page cache → (references into) cached internal pipe → destination’s ->splice_write. The pipe never holds copies of the bytes; it is a queue of page references, and the file’s pages are passed by reference the whole way. This is why sendfile and splice share their performance characteristics — they are the same machinery with sendfile supplying a built-in pipe so the caller does not have to.

Down to the socket: splice_to_socket and MSG_SPLICE_PAGES

When the destination is a socket, do_splice_from reaches the socket’s ->splice_write, which in v6.12 is splice_to_socket(). It assembles the pipe’s pages into a bio_vec[] and sends them with a single sendmsg, asking the protocol to splice rather than copy:

msg.msg_flags = MSG_SPLICE_PAGES;
if (flags & SPLICE_F_MORE)
    msg.msg_flags |= MSG_MORE;
iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, bvec, bc, len - remain);
ret = sock_sendmsg(sock, &msg);

This is a 6.12-era detail worth getting right: older write-ups describe sendfile calling the socket’s ->sendpage operation. ->sendpage no longer exists. It was removed across the network stack in Linux 6.5 and replaced by sendmsg() carrying the internal flag MSG_SPLICE_PAGES — a hint that “the protocol sendmsg() instance will attempt to splice the pages out of the buffer, copying into individual fragments those that it can’t” (David Howells’ series, LWN). So at 6.12 the splice-to-socket loop hands up to 16 pages at a time to sock_sendmsg with MSG_SPLICE_PAGES; TCP attaches them as sk_buff fragments by reference where it can. The net effect is identical to the old sendpage zero-copy, with cleaner plumbing. Note also that SPLICE_F_MORE propagates into MSG_MORE, the cork hint that tells TCP to wait for more data before transmitting a small segment.

The hardware floor: scatter-gather and checksum offload

Whether the last CPU copy disappears depends on the NIC. When the driver advertises NETIF_F_SG (scatter-gather), the stack can build a packet from a header sk_buff plus paged fragments pointing straight at page-cache pages, and the NIC DMA-gathers them; “the NIC reads the page cache pages directly, achieving zero CPU copies” (Mark McLoughlin, scatter-gather/offload). Two hardware features make this possible: scatter-gather lets the descriptor reference non-contiguous pages, and checksum offload (the NIC computes the TCP/IP checksum in hardware) means the kernel never has to read the payload bytes to checksum them — “if the network interface cannot generate checksums, the kernel will have to perform a pass over the data to calculate it itself,” re-introducing a copy. Without scatter-gather, the kernel falls back to a single copy from page cache into one contiguous sk_buff — still one copy fewer than the read+write path. So the “true zero-copy” claim is hardware-conditional; on a feature-complete server NIC it holds, on a minimal NIC sendfile degrades gracefully to one copy.

A Worked Example

A minimal static-file server transmit loop:

off_t off = 0;
struct stat st;
fstat(in_fd, &st);                      /* learn the file size */
size_t remaining = st.st_size;
 
while (remaining > 0) {
    ssize_t n = sendfile(client_sock, in_fd, &off, remaining);
    if (n == -1) {
        if (errno == EAGAIN) continue;  /* non-blocking socket not ready */
        if (errno == EINTR)  continue;  /* interrupted; retry */
        break;                          /* real error */
    }
    if (n == 0) break;                  /* unexpected EOF */
    remaining -= n;                     /* off was advanced by the kernel */
}

Line by line: off starts at the file beginning; because we pass a non-NULL &off, the kernel advances it (not the file’s seek pointer) so the loop resumes where it left off and the same in_fd could be served to many clients concurrently with independent offsets. remaining is the count, re-clamped each iteration. The loop is mandatory for three reasons — sendfile caps any single call at 0x7ffff000 bytes, it may short-transfer against a socket whose send buffer fills, and on a non-blocking socket it returns EAGAIN when the socket would block (the socket side, not a sendfile flag, controls blocking). When the file fits and the NIC is capable, the entire body of this loop moves zero payload bytes through any CPU.

Failure Modes and Common Misunderstandings

  • in_fd cannot be a socket or pipe. The input “must correspond to a file which supports mmap(2)-like operations (i.e., it cannot be a socket)” (sendfile(2)). Trying to sendfile from a socket yields EINVAL. To move data from a pipe or socket, use splice — the man page explicitly recommends it: “Applications may wish to fall back to read(2)/write(2) … If out_fd refers to a socket or pipe … use splice(2) instead.”
  • The out_fd-must-be-a-socket myth. This was true before Linux 2.6.33 — “out_fd must refer to a socket” — but since 2.6.33 it can be any file; if out_fd is seekable, sendfile updates its offset. Code or documentation asserting the socket-only restriction is stale by over a decade.
  • EOVERFLOW is returned when count is too large for the file’s offset limits — a real possibility on 32-bit off_t; prefer sendfile64/the 64-bit loff_t form.
  • mmap-then-write is not equivalent. A common “optimization” is mmap the file and write the mapping to the socket. That still copies the mapped pages into the socket buffer (one CPU copy) and risks a SIGBUS if the file is truncated under the mapping. sendfile avoids the copy and the SIGBUS class of bugs.
  • It is not magic for tiny files. The setup cost (two fd lookups, region verification, internal-pipe management) means for a few-hundred-byte response the saved copy is in the noise; sendfile shines on bulk transfers.
  • SIGPIPE still applies. Writing to a socket whose peer has closed delivers SIGPIPE/EPIPE just as write would — see SIGPIPE and Broken Pipes.

Alternatives and When to Choose Them

  • splice — the general primitive sendfile is built on. Choose raw splice when you need more than file→socket: splicing socket→file (a download to disk), interposing a tee to mirror the stream, or pumping through a pipe-connected helper. sendfile is the ergonomic shortcut for the one case where neither side is a pipe.
  • io_uring zero-copy send (IORING_OP_SEND_ZC) — the modern successor for sending application-generated data (not a file): it pins user pages, hands them to the network stack, and posts a completion when the buffer is reusable. For file→socket, io_uring can also issue sendfile/splice-equivalent operations asynchronously; see io_uring and the File Path. Where sendfile is one blocking (or EAGAIN-polling) syscall per chunk, io_uring batches many transfers and reaps completions, eliminating the per-chunk syscall.
  • MSG_ZEROCOPY on send() — copy-avoidance for user-buffer transmits without io_uring, via a SO_ZEROCOPY socket and an error-queue notification when the buffer is free (kernel MSG_ZEROCOPY doc). It targets the same user-data case as IORING_OP_SEND_ZC, not the file case.
  • copy_file_range(2) — for file→file copies (not sockets); can use filesystem reflinks or NFS server-side copy to avoid moving data at all.
  • Plain read+write — still the right choice when you must transform the bytes (compress, encrypt in user space, transcode) — zero-copy is impossible when the CPU must touch every byte anyway.

Production Notes

sendfile is one of the most battle-tested syscalls in the server world. nginx serves static files with it by default (sendfile on;), Apache’s EnableSendfile, Netflix’s FreeBSD edge servers and Linux CDNs, and Kafka’s log-segment fan-out to consumers all lean on it to keep multi-gigabit transfers off the CPU. The recurring operational gotcha is the network-filesystem caveat: sendfile over NFS or other network-backed in_fd has historically been unreliable or disabled (nginx ships warnings about sendfile with certain remote filesystems), because the “page cache page the NIC reads directly” assumption breaks when the file’s pages are not locally cache-resident in the simple way. A second subtlety is TLS: a plain sendfile cannot encrypt, so HTTPS static serving needs either kernel TLS (kTLS, where the encryption happens in the kernel/NIC and sendfile can still feed it) or a fallback to in-user-space encryption that forfeits zero-copy. Finally, on the kernel-internals side, the 6.5 sendpage→MSG_SPLICE_PAGES rewrite is invisible to applications (the sendfile/splice ABI is unchanged) but is why any deep-dive written before 2023 describing ->sendpage is describing a code path that no longer exists — when reading older zero-copy material, mentally substitute sendmsg(MSG_SPLICE_PAGES) for sendpage.

See Also

Graph View