vault

Why System Calls Are Expensive

15 min read

Why System Calls Are Expensive

A system call is the only legitimate way for unprivileged user code to ask the kernel to do something the hardware forbids it to do itself — read a file, send a packet, map memory. Mechanically it is “just” a single instruction (syscall on x86-64) that flips the CPU from ring 3 (user) to ring 0 (kernel) and back, so it is tempting to think of it as a slightly fancy function call. It is not. A plain function call is a few cycles — push a return address, jump, return. A system call costs hundreds of nanoseconds even on a fast modern core, one to three orders of magnitude more, and on a CPU running the post-Meltdown page-table-isolation defence it costs more still. The cost decomposes into four layers: the hardware mode switch itself (privilege change, pipeline serialization, register save/restore), the cache and TLB pollution the kernel inflicts on the way through, the Page Table Isolation (PTI/KPTI) address-space switch forced by the 2018 Meltdown vulnerability — the dominant modern cost, but only on affected CPUs — and the Spectre branch-prediction barriers sprinkled along the entry path. Understanding this breakdown is why “minimize syscalls” (batch I/O, use the vDSO, use io_uring) is real, measurable advice rather than folklore.

This note dissects the cost layer by layer, pins each magnitude to a primary source where one exists (and flags the rest), and explains why the conventional wisdom to avoid syscalls is correct. The mechanism of the trap instruction itself lives in The Syscall Instruction and Trap Mechanism; the way the vDSO sidesteps the trap entirely is in Raw Syscalls vs vsyscall vs vDSO Performance. Version-specific claims are pinned to Linux 6.12 LTS (released 2024-11-17) source, with as-of dating where the cost depends on hardware era.

Uncertain

Absolute nanosecond figures for a bare syscall (commonly cited as “~100–500 ns” depending on CPU, mitigations, and what the call does) and the clean “100×-vs-a-function-call” ratio are not taken from a primary benchmark fetched for this note. The percentage slowdowns below (Gregg’s 2%/5%/~30%, the “few hundred cycles” CR3 cost) are well-sourced; the absolute ns ratio is an order-of-magnitude estimate. Reason: no primary microbenchmark of a single getpid() was fetched. To resolve: run perf bench syscall basic or an lmbench lat_syscall on a known CPU+kernel and record the number with its hardware/mitigation context. uncertain

Mental Model — Four Taxes Stacked on One Instruction

Think of a syscall not as a function call but as crossing a guarded border. The guard post itself takes time (the mode switch); crossing disturbs everything you were carrying (cache and TLB state); since 2018 the border has a second checkpoint where your entire map of the country is swapped for a different one (PTI’s page-table switch); and at the gate every signpost is deliberately scrambled so an attacker cannot trick the guard’s “anticipation” (Spectre barriers). Each is additive, and each is fixed cost per crossing — independent of how much work the call actually does. That fixed cost is the whole reason batching wins: amortize one border-crossing over a thousand operations.

flowchart TB
  FC["Plain function call<br/>(~1 ns: jump + return,<br/>stays in cache & TLB)"]
  subgraph SC["A system call — fixed cost per crossing"]
    direction TB
    L1["1. Mode switch<br/>syscall/sysret: privilege change,<br/>pipeline drain, save/restore regs<br/>(tens of cycles)"]
    L2["2. Cache + TLB pollution<br/>kernel code/data evict yours;<br/>i-cache, d-cache, branch predictors<br/>cold on return"]
    L3["3. PTI page-table switch<br/>CR3 reload on entry AND exit<br/>(~hundreds of cycles each)<br/>ONLY on Meltdown-vulnerable CPUs"]
    L4["4. Spectre barriers<br/>array_index_nospec on syscall nr,<br/>LFENCE in uaccess,<br/>IBPB/STIBP/retpoline overhead"]
    L1 --> L2 --> L3 --> L4
  end
  FC -.->|"100× to 1000× cheaper"| SC

The cost of a syscall as four stacked, additive taxes versus a plain function call. What it shows: the bare mode switch (layer 1) is only the floor; layers 2–4 are what make a modern syscall hurt, and layer 3 (PTI) is conditional on the CPU being Meltdown-vulnerable. The insight to take: every layer is fixed per crossing, so the cure is fewer crossings — which is exactly why the vDSO (zero crossings) and io_uring (one crossing for thousands of ops) exist.

Layer 1 — The Mode Switch Itself

The cheapest part, and the part everyone thinks is the whole story. On x86-64 the userspace libc wrapper loads the syscall number into rax and the arguments into rdi, rsi, rdx, r10, r8, r9 and executes the syscall instruction (syscall(2)). The CPU does several things that a plain call never does:

  • Privilege escalation. The current privilege level (CPL) goes from 3 to 0. The processor loads a new code segment and stack from model-specific registers (MSR_LSTAR, MSR_STAR) configured at boot. This is not a memory jump the branch predictor can speculate freely past — it is an architectural mode change.
  • Serialization. The syscall/sysret pair forces a degree of pipeline serialization: the deep out-of-order machinery of a modern core cannot freely reorder instructions across the privilege boundary. The pipeline effectively drains and refills, costing the latency of however many instructions were in flight.
  • Register save/restore. The kernel entry assembly (entry_SYSCALL_64 on x86-64) pushes the full user register set into a pt_regs frame on the kernel stack so the handler can run C code and so the registers can be restored on return. The general-purpose registers, flags, and instruction pointer all have to be spilled and reloaded — work a leaf function call never does. The mechanics of this frame are covered in The pt_regs Register Frame and Per-Architecture Syscall Entry Assembly.

By itself this layer is on the order of tens of cycles — the reason the syscall instruction was introduced in the first place was that it is much cheaper than the legacy int 0x80 software interrupt it replaced (see The Syscall Instruction and Trap Mechanism). If layer 1 were the whole cost, syscalls would be merely “a bit slower than a function call.” It is layers 2–4 that turn “a bit” into “orders of magnitude.”

Layer 2 — Cache and TLB Pollution

A function call keeps running your code on your data; the instruction cache, data cache, and branch predictors stay warm with your working set. A system call is a detour into a completely different body of code — the kernel — touching kernel code pages, kernel data structures, the syscall table, lock words, and per-CPU variables. Every kernel cache line that gets pulled in evicts one of yours. When control returns to userspace, the instruction cache, data cache, branch target buffer, and the Translation Lookaside Buffer (TLB) are all partly cold with respect to your program. The next several hundred userspace instructions then run slower because they re-incur cache and TLB misses that would not have happened without the detour.

This pollution is invisible in a naive “time the syscall” microbenchmark that only measures the call itself — the cost is paid afterward, in the userspace code that resumes. It is also why syscall cost is workload-dependent: a program with a large working set (say >10 MB) is hurt far more than a tight loop, because the kernel detour evicts more of a large set’s hot lines. Brendan Gregg’s KPTI analysis makes this concrete: in one heavily-affected configuration, instructions-per-cycle (IPC) collapsed from 0.86 to 0.10 and “half the CPU cycles have page walks active” — i.e. the core spent half its time walking page tables servicing TLB misses caused by the syscall traffic (Gregg 2018). That is layers 2 and 3 compounding: PTI flushes the TLB, and a large working set then pays to refill it.

Layer 3 — Page Table Isolation (PTI/KPTI), the Dominant Modern Cost

This is the layer that changed the economics of syscalls in January 2018, and on an affected CPU it is usually the single largest contributor.

Why it exists: Meltdown

Meltdown (CVE-2017-5754) is a hardware vulnerability in which a userspace instruction speculatively reads a kernel memory address — which is mapped into the process’s address space but marked supervisor-only — and, although the architectural read is squashed when the permission check fails, the speculative read has already left a measurable footprint in the cache that an attacker can read out via a timing side channel. The flaw exists because, historically, the kernel’s entire address space was mapped into every process (with supervisor-only protection), so kernel data was physically reachable by speculation even though it was architecturally forbidden.

What PTI does

Page Table Isolation (PTI), originally the academic KAISER patch, removes that reachability by giving each process two sets of page tables (pti.html, kernel docs; LWN KAISER, 2017):

  1. A kernel page table with the full mapping (kernel + user), used while running in ring 0.
  2. A userspace (“shadow”) page table that maps only userspace plus a tiny stub of kernel code/data — primarily the cpu_entry_area needed to perform the entry/exit transition. The bulk of the kernel is simply not mapped while in user mode, so speculation cannot reach it. As the docs put it, “the bulk of the kernel’s address space will thus be completely hidden from the process, defeating the known hardware-based attacks” (LWN KAISER).

Where the cost comes from

The two tables mean the CR3 register (which points at the active page-table root on x86) must be reloaded on every syscall entry and again on every exit — two extra CR3 writes per syscall that did not exist before 2018. The kernel docs state plainly: “Moves to CR3 are on the order of a hundred cycles, and are required at every entry and exit” (pti.html). The KAISER LWN coverage agrees: “Just the new instructions (CR3 manipulation) add a few hundred cycles to a syscall or interrupt” (LWN).

Worse, a CR3 reload historically flushes the entire TLB (the cache of virtual→physical translations; see The Translation Lookaside Buffer and TLB Shootdowns). A flushed TLB means the next userspace memory accesses must re-walk the page tables from RAM — the very page-walk storm Gregg measured. The mitigation for that is PCID (Process-Context Identifier): tagging TLB entries with an address-space ID so that switching CR3 need not flush, only re-select. Linux gained PCID-aware PTI in the 4.14 cycle, and it “reduces the cost of switching page tables during system calls considerably” (LWN KAISER). With PCID the data-TLB walk cost returns roughly to pre-KPTI levels (Gregg 2018).

The measured magnitude

Gregg’s analysis is the canonical primary-ish benchmark write-up. The headline findings (Gregg 2018):

  • Overhead scales with syscall rate. At ~50k syscalls/sec/CPU, ~2%. A MySQL OLTP workload at ~75k syscalls/sec/CPU lost 5%.
  • A stress test at ~210k syscalls/sec/CPU plus ~27k context-switches/sec/CPU lost ~25%.
  • The five factors that drive cost: syscall rate, context-switch rate, page-fault rate, working-set size (the >10 MB “jump”), and cache-access patterns.
  • His practical expectation for typical (Netflix) cloud workloads: 0.1% to 6% overhead.

The original Linux KPTI patch notes corroborate: “Most workloads … show single-digit regressions. 5% is a good round number for what is typical. The worst we have seen is a roughly 30% regression on a loopback networking test that did a ton of syscalls and context switches” (LWN KAISER).

Note the gap between these microbenchmark-level numbers and the kernel doc’s “never exceeding 1%” claim (pti.html): the doc is describing typical mixed desktop/server workloads where syscalls are a small fraction of cycles, while the 5–30% figures are syscall-bound workloads. Both are true; they measure different things. For a syscall-heavy program — a proxy, a database, an event loop hammering epoll_wait/read/write — the syscall-bound figure is the one that bites.

The crucial caveat: PTI is conditional on the CPU

PTI is not unconditionally on. In pti_check_boottime_disable() (verified in arch/x86/mm/pti.c at the v6.12 tag), the auto path is:

if (pti_mode == PTI_AUTO && !boot_cpu_has_bug(X86_BUG_CPU_MELTDOWN))
    return;                 /* CPU not vulnerable → no PTI, no cost */
setup_force_cpu_cap(X86_FEATURE_PTI);

That is, in the default pti=auto mode PTI is enabled only on CPUs flagged with X86_BUG_CPU_MELTDOWN. All AMD processors are Meltdown-immune, and Intel parts with the hardware fix (roughly Coffee Lake refresh / Whiskey Lake onward, ~late 2018) report “Not affected” — on those machines PTI is off by default and layer 3 costs nothing (source: arch/x86/mm/pti.c v6.12). You can check on any running kernel with cat /sys/devices/system/cpu/vulnerabilities/meltdown (Mitigation: PTI vs Not affected). pti=on/nopti (alias pti=off) override the auto decision. So “PTI is the dominant cost” is accurate on a Meltdown-vulnerable CPU and false on a modern AMD or post-fix Intel.

Layer 4 — Spectre Branch-Prediction Barriers

Spectre (variants 1 and 2, CVE-2017-5753 and CVE-2017-5715) tricks the CPU’s branch predictor into speculatively executing the wrong path, leaking data through the same cache side channel. Unlike Meltdown, Spectre is not fixed by hiding the kernel’s pages; it is mitigated by inserting barriers that constrain speculation, and several of those barriers sit directly on the syscall path (spectre.html, kernel docs):

  • Spectre v1 — array_index_nospec on the syscall number. When the kernel uses the userspace-supplied syscall number to index sys_call_table[], an attacker could speculatively force an out-of-bounds index. The actual v6.12 dispatch clamps it: in do_syscall_x64() the kernel does unr = array_index_nospec(unr, NR_syscalls) before indexing (verified in arch/x86/entry/common.c at v6.12). This masks the index to a safe range even under misspeculation — a handful of extra ALU ops on every single syscall.
  • Spectre v1 — LFENCE in copy_from_user. The docs note “copy-from-user code has an LFENCE barrier to prevent the access_ok() check from being mis-speculated” (spectre.html). Any syscall that copies in arguments (see copy_to_user and copy_from_user and The access_ok Check and User Pointer Validation) eats a serializing fence.
  • Spectre v2 — retpolines or IBRS/eIBRS. Indirect branches in the kernel are converted to retpolines (return trampolines that trap misspeculation in a harmless loop) on CPUs without hardware help; on CPUs with enhanced IBRS (eIBRS) the retpolines are disabled at runtime and the hardware mode is used instead (spectre.html). The syscall dispatch is full of indirect calls (the handler pointer in the table), so this overhead lands on the hot path.
  • IBPB / STIBP — context-switch and SMT barriers. The Indirect Branch Prediction Barrier (IBPB) is issued when switching to/from a process to clear the branch target buffer; STIBP prevents a sibling hyperthread from poisoning predictions. These are context-switch-time, not per-syscall, but a context switch almost always rides on the back of a syscall, so syscall-heavy + context-switch-heavy workloads pay both (this is why Gregg’s worst-case mixed syscalls and context switches).

Like PTI, the v2 mitigations are CPU-dependent: the kernel “selects reasonable default mitigations for the current CPU” (spectre.html), so the exact cost varies by microarchitecture and microcode. The speculation-barrier story at the boundary is developed further in Speculation Barriers and Spectre Hardening at the Syscall Boundary.

There is also a small, unrelated entry-path cost worth naming: do_syscall_64() calls add_random_kstack_offset() on every syscall (verified in arch/x86/entry/common.c v6.12), randomizing the kernel stack offset per call as a hardening measure against stack-based attacks — a few more cycles on the floor.

Why “Minimize Syscalls” Is Real Advice

Put the layers together and the conclusion is forced. The cost of a syscall is fixed per crossing and largely independent of the work done — getpid() (which does almost nothing) and a 4 KB read() pay almost the same border tax. So a program that issues a million tiny syscalls pays the tax a million times. The strategies that exist in Linux specifically to dodge this are not micro-optimizations; they are architectural:

  • The vDSO turns a few hot, harmless “syscalls” (clock reads, getcpu) into ordinary function calls against a shared memory page — zero border crossings. The measured gap is large enough to matter; see Raw Syscalls vs vsyscall vs vDSO Performance.
  • io_uring lets a process submit thousands of I/O operations through a shared-memory ring with one (or, in polled mode, zero) io_uring_enter syscall — amortizing one border tax over thousands of operations.
  • Vectored and batched calls — writev, recvmmsg — pack many buffers or messages into one crossing.
  • Buffering in userspace — stdio’s BUFSIZ buffering, a database’s page cache — is the oldest version of the same idea: do many logical operations between physical syscalls.

This is also why the standard diagnostic for “my I/O-bound program is slow” is strace -c (see How strace Works): it counts syscalls and shows where the crossings are, so you know what to batch.

Common Misunderstandings

  • “A syscall is basically a function call.” No — it is a privilege transition with serialization, register spills, cache/TLB pollution, and (on affected CPUs) two CR3 reloads plus speculation barriers. Orders of magnitude apart.
  • “KPTI made everything 30% slower.” Only syscall-and-context-switch-bound workloads on Meltdown-vulnerable CPUs approach that; typical mixed workloads are single-digit, and modern AMD/post-fix Intel pay nothing because PTI is off by default (arch/x86/mm/pti.c v6.12).
  • “The cost is in the call; once it returns I’m fine.” The cache/TLB pollution is paid after return, in your own code — invisible to a microbenchmark that times only the call.
  • “getpid() is cheap because it does so little.” It is cheap relative to a blocking read, but it still pays the full fixed border tax; the work it does is dwarfed by the crossing. (This is exactly why getpid-class calls were candidates for the vDSO historically — though on Linux only the clock/cpu calls actually live there today; see Raw Syscalls vs vsyscall vs vDSO Performance.)

See Also

Graph View

Backlinks