Why System Calls Are Expensive

A system call is the only legitimate way for unprivileged user code to ask the kernel to do something the hardware forbids it to do itself — read a file, send a packet, map memory. Mechanically it is “just” a single instruction (syscall on x86-64) that flips the CPU from ring 3 (user) to ring 0 (kernel) and back, so it is tempting to think of it as a slightly fancy function call. It is not. A plain function call is a few cycles — push a return address, jump, return. A null system call costs tens of nanoseconds and a few hundred core cycles on a fast modern core with no mitigations patched in, and several hundred nanoseconds on a Meltdown-vulnerable CPU running the full post-2018 mitigation stack — in either case one to three orders of magnitude more than a function call. (The measured figure on the machine this note was written on is 34.3 ns / 173 core cycles, against 0.80 ns / 4.0 cycles for a cross-translation-unit function call: a ratio of 43×. See Measuring It below for the full methodology.) The cost decomposes into four layers: the hardware mode switch itself (privilege change, pipeline serialization, register save/restore), the cache and TLB pollution the kernel inflicts on the way through, the Page Table Isolation (PTI/KPTI) address-space switch forced by the 2018 Meltdown vulnerability — the dominant modern cost, but only on affected CPUs — and the Spectre branch-prediction barriers sprinkled along the entry path. Understanding this breakdown is why “minimize syscalls” (batch I/O, use the vDSO, use io_uring) is real, measurable advice rather than folklore.

This note dissects the cost layer by layer, pins each magnitude to a primary source where one exists (and flags the rest), and explains why the conventional wisdom to avoid syscalls is correct. The mechanism of the trap instruction itself lives in The Syscall Instruction and Trap Mechanism; the way the vDSO sidesteps the trap entirely is in Raw Syscalls vs vsyscall vs vDSO Performance. Version-specific claims are pinned to Linux 6.12, this vault’s reference tag and one of the currently-maintained longterm series, with as-of dating where the cost depends on hardware era and an explicit note wherever the code moved after 6.12. For orientation as of 2026-09-11, verified against kernel.org/releases.json: mainline is 7.3-rc2, latest stable is 7.2.5, and the longterm series are 6.18.51, 6.12.109, 6.6.156, 6.1.187, 5.15.220 and 5.10.269 — so 6.12 is a pin, not the only maintained LTS. The benchmark in this note was run on 7.1.13, which releases.json already marks end-of-life.

This note contains a benchmark that was run for it

Every nanosecond and cycle figure attributed to “this machine” in this note was measured on 2026-09-11 on an AMD Ryzen AI MAX+ 395 (Zen 5, Fedora 44, Linux 7.1.13) by a program written for this note; the source, methodology, and raw output are described in Measuring It — A Benchmark Run for This Note. Numbers attributed to Brendan Gregg, LWN, or the kernel documentation are cited to those sources and were not re-measured here.

Uncertain

Verify: the mitigated cost figures. This machine reports Meltdown: Not affected and spectre_v2: Enhanced / Automatic IBRS, which means Page Table Isolation, retpolines, legacy IBRS-on-entry, UNTRAIN_RET, the BHI clear loop and the MDS VERW are all patched out of the entry path at boot (see The Mitigation Matrix). So the 34.3 ns measured here is the unmitigated floor, not what a Meltdown-vulnerable Skylake-era Intel part pays. The per-mitigation costs in the matrix below are therefore taken from published sources rather than measured here. Reason: disabling or enabling these mitigations requires a reboot with different kernel command-line parameters, which was out of scope for this note, and no Meltdown-vulnerable machine was available. To resolve: re-run the same benchmark on a pre-Coffee-Lake-refresh Intel part, and again with pti=off mitigations=off, and record the delta per mitigation. uncertain

Mental Model — Four Taxes Stacked on One Instruction

Think of a syscall not as a function call but as crossing a guarded border. The guard post itself takes time (the mode switch); crossing disturbs everything you were carrying (cache and TLB state); since 2018 the border has a second checkpoint where your entire map of the country is swapped for a different one (PTI’s page-table switch); and at the gate every signpost is deliberately scrambled so an attacker cannot trick the guard’s “anticipation” (Spectre barriers). Each is additive, and each is fixed cost per crossing — independent of how much work the call actually does. That fixed cost is the whole reason batching wins: amortize one border-crossing over a thousand operations.

flowchart TB
  FC["Plain function call<br/>(~1 ns: jump + return,<br/>stays in cache & TLB)"]
  subgraph SC["A system call — fixed cost per crossing"]
    direction TB
    L1["1. Mode switch<br/>syscall/sysret: privilege change,<br/>pipeline drain, save/restore regs<br/>(tens of cycles)"]
    L2["2. Cache + TLB pollution<br/>kernel code/data evict yours;<br/>i-cache, d-cache, branch predictors<br/>cold on return"]
    L3["3. PTI page-table switch<br/>CR3 reload on entry AND exit<br/>(~hundreds of cycles each)<br/>ONLY on Meltdown-vulnerable CPUs"]
    L4["4. Spectre barriers<br/>array_index_nospec on syscall nr,<br/>LFENCE in uaccess,<br/>IBPB/STIBP/retpoline overhead"]
    L1 --> L2 --> L3 --> L4
  end
  FC -.->|"100× to 1000× cheaper"| SC

The cost of a syscall as four stacked, additive taxes versus a plain function call. What it shows: the bare mode switch (layer 1) is only the floor; layers 2–4 are what make a modern syscall hurt, and layer 3 (PTI) is conditional on the CPU being Meltdown-vulnerable. The insight to take: every layer is fixed per crossing, so the cure is fewer crossings — which is exactly why the vDSO (zero crossings) and io_uring (one crossing for thousands of ops) exist.

Measuring It — A Benchmark Run for This Note

A note about a cost that contains no measurement is an opinion. So this section reports one, taken on the machine this note was written on, with the methodology stated in full so that the numbers can be checked, reproduced, and — most importantly — correctly scoped.

The machine and its mitigation state

Property	Value
CPU	AMD Ryzen AI MAX+ 395 w/ Radeon 8060S (“Strix Halo”, Zen 5), family 26 / model 112 / stepping 0
Topology	16 cores / 32 threads, 1 socket, 1 NUMA node
Caches	L1d 16 × 48 KiB, L1i 16 × 32 KiB, L2 16 × 1 MiB, L3 2 × 32 MiB (64 MiB total)
Clocks	min 625 MHz, max 5187.5 MHz, boost enabled; governor `performance`, driver `amd-pstate-epp`
Time-stamp counter	`constant_tsc`, `nonstop_tsc`; clocksource `tsc`; kernel-refined TSC rate 2994.382 MHz
Kernel	Linux 7.1.13-200.fc44.x86_64 (Fedora 44), `PREEMPT_DYNAMIC`
Compiler	GCC 16.2.1, `-O2`
Date measured	2026-09-11

The mitigation state is not a footnote — it is the single most important piece of context for any syscall-cost number, because it determines which of the four layers are actually present in the instruction stream. Read verbatim from /sys/devices/system/cpu/vulnerabilities/:

sysfs file	Value on this machine	Effect on the syscall entry path
`meltdown`	`Not affected`	PTI off. `SWITCH_TO_KERNEL_CR3` / `SWITCH_TO_USER_CR3_STACK` are no-ops — layer 3 costs zero
`spectre_v1`	`Mitigation: usercopy/swapgs barriers and __user pointer sanitization`	`array_index_nospec` on the syscall number and `LFENCE` in `copy_from_user` are live
`spectre_v2`	`Mitigation: Enhanced / Automatic IBRS; IBPB: conditional; STIBP: always-on; PBRSB-eIBRS: Not affected; BHI: Not affected`	eIBRS is a mode bit, not per-entry work: no retpolines, no `IBRS_ENTER` MSR write on entry, no BHI clear loop
`spec_rstack_overflow`	`Mitigation: IBPB on VMEXIT only`	`UNTRAIN_RET` on the syscall path is not armed (`X86_FEATURE_ENTRY_IBPB` clear)
`spec_store_bypass`	`Mitigation: Speculative Store Bypass disabled via prctl`	Opt-in per process; off by default, and not on the entry path
`mds`, `l1tf`, `retbleed`, `srbds`, `tsx_async_abort`, `itlb_multihit`, `mmio_stale_data`, `gather_data_sampling`, `reg_file_data_sampling`, `indirect_target_selection`, `ghostwrite`, `tsa`	`Not affected`	`CLEAR_CPU_BUFFERS` (the `VERW`) is a no-op on the return path
`vmscape`	`Mitigation: IBPB on VMEXIT`	Virtualization exit only; not on the syscall path

So this machine measures the unmitigated floor. That is a legitimate and useful thing to measure — it is what 2026-vintage AMD silicon actually costs — but it is not what a 2017 Skylake Xeon pays, and the numbers below must never be quoted without this context.

Methodology

Four methodological problems had to be solved, and each solution is worth stating because getting any of them wrong produces a plausible-looking wrong number.

Defeating the vDSO. clock_gettime() called through libc does not perform a system call at all on a normal Linux system — it is serviced entirely in userspace by the vDSO (see How the vDSO Reads Kernel Time Without Trapping). Timing it and calling the result “a syscall” is the classic way to under-report syscall cost by 3×. The benchmark therefore measures both: clock_gettime() through libc (the vDSO path) and the same operation issued as a raw syscall instruction with rax = 228 through inline assembly, which forces the kernel to service it. Similarly, getpid is issued through raw inline assembly so that no libc caching or PLT indirection is in the measurement:
```
static inline long rawsys0(long n) {
    long r;
    __asm__ __volatile__("syscall" : "=a"(r) : "a"(n) : "rcx", "r11", "memory");
    return r;
}
```
The clobber list is the whole x86-64 syscall ABI in one line — rcx and r11 are destroyed by the instruction itself, and memory stops the compiler from caching loads across the call. See The x86-64 Syscall Calling Convention for why those two registers and no others.
Separating the TSC from the core clock. This is the trap that silently corrupts most “cycles per syscall” figures on modern parts. The time-stamp counter read by RDTSC on any constant_tsc CPU ticks at a fixed reference rate that is not the core clock — here 2.994382 GHz, as the kernel itself determined at boot (“Refined TSC clocksource calibration: 2994.382 MHz”). The core, meanwhile, boosts to over 5 GHz. Dividing TSC deltas by the TSC rate gives correct nanoseconds; calling those TSC ticks “cycles” under-reports the true core-cycle count by about 40% on this machine. The benchmark therefore self-calibrates the core clock by timing a chain of dependent addq instructions, each of which has exactly one cycle of latency and cannot be reordered or executed in parallel with its successor:
```
__asm__ __volatile__(
    "1:\n\t"
    "addq $1, %0\n\t addq $1, %0\n\t addq $1, %0\n\t addq $1, %0\n\t"
    "subq $4, %1\n\t"
    "jg 1b\n\t"
    : "+r"(x), "+r"(n) :: "cc");
```
400 million dependent adds take 400 million core cycles by construction; dividing that by the elapsed wall time (from the TSC) yields the core frequency directly. It came out at 5.03–5.07 GHz across runs, consistent with the 5.1875 GHz boost ceiling minus a little. Every “core-cyc” figure below is nanoseconds × measured core GHz.
Frequency scaling and warmup. The governor was already performance and boost enabled. The process is pinned to a single logical CPU with both sched_setaffinity(2) and taskset, so it cannot migrate between cores of different clock states. Before any timed batch, every code path under test is executed 200,000 times, which faults in every page it touches, warms the instruction cache and branch predictors, and — critically — drives the core to its boosted P-state before the clock is calibrated. The calibration itself is run twice and the second result used.
Noise rejection. Each operation is timed as a batch of 2–200 million iterations (the batch size scaled so that every batch runs for tens of milliseconds), the batch is repeated nine times, and the minimum is taken. The minimum is the right statistic here: every source of noise on this measurement — an interrupt, a migration, a scheduler tick, an SMT sibling stealing issue slots — can only add time. The whole program was then run three times end to end; run-to-run spread was under 1% on every row except write(), where it was 8%.

Results

All figures are per call, best of nine batches, median of three whole-program runs. SYSCALL insn rows bypass libc entirely.

Operation	ns/call	core cycles	vs. function call
Empty loop (the `RDTSC` measurement floor)	0.20	1.0	—
Userspace function call, cross-translation-unit, not inlinable	0.80	4.0	1×
vDSO `clock_gettime(CLOCK_MONOTONIC)` — no kernel entry	17.1	86	21×
`lseek(0, 0, SEEK_CUR)` — raw `SYSCALL`	33.8	171	42×
`getpid()` — raw `SYSCALL`, the null-syscall floor	34.2	173	43×
`getppid()` — raw `SYSCALL`	34.5	174	43×
`getpid()` through the glibc wrapper	34.9	176	44×
`close(-1)` → `EBADF` — raw `SYSCALL`, error return path	35.0	176	44×
`getpid()` with a trivial seccomp-BPF `RET_ALLOW` filter installed	38.3	194	48×
`write(/dev/null, buf, 64)` — raw `SYSCALL`	49.4	248	62×
`clock_gettime(CLOCK_MONOTONIC)` — raw `SYSCALL`, vDSO defeated	58.6	296	73×
`getpid()` via legacy `int $0x80` (IA-32 compat entry)	126.1	637	158×

xychart-beta
    title "Cost of one operation, AMD Zen 5, no mitigations patched in (ns)"
    x-axis ["fn call", "vDSO gettime", "getpid SYSCALL", "+seccomp", "write 64B", "gettime SYSCALL", "getpid int 0x80"]
    y-axis "nanoseconds per call" 0 --> 135
    bar [0.8, 17.1, 34.2, 38.3, 49.4, 58.6, 126.1]

The measured ladder from a userspace function call to a legacy software-interrupt syscall, on one Zen 5 core at ~5.05 GHz. What it shows: the leftmost bar — the plain function call — is 0.80 ns and is effectively invisible at a scale that fits the rightmost bar. The insight to take: that invisibility is the finding. Even on the most favourable possible hardware — a CPU immune to Meltdown, with eIBRS instead of retpolines, and every entry-path mitigation patched out at boot — the cheapest possible system call still costs 43 times a function call, and the legacy int $0x80 entry costs 158 times. There is no such thing as a cheap border crossing.

What the five headline ratios actually say

43× — a null syscall versus a function call. getpid() does essentially no work: it reads a field out of the current task structure and returns. Its 173 cycles are almost entirely the crossing itself. This is the fixed border tax, isolated.
3.4× — clock_gettime through the kernel (58.6 ns) versus through the vDSO (17.1 ns). This is the cleanest possible demonstration of the vDSO’s value, because it is the same operation, computing the same answer, differing only in whether a privilege boundary is crossed. Note that the vDSO path is not free either: 86 cycles, most of which is the RDTSC read and the seqlock dance against the kernel’s shared timekeeping page. It is 21× a function call — just 3.4× cheaper than trapping. See Raw Syscalls vs vsyscall vs vDSO Performance.
3.7× — int $0x80 (126.1 ns) versus SYSCALL (34.2 ns). The legacy software-interrupt entry is still present and functional on this 64-bit kernel via CONFIG_IA32_EMULATION, and measuring it quantifies exactly what AMD bought when it added the SYSCALL instruction: an int gate does a full IDT lookup, a privilege-level stack switch through the TSS, and pushes a hardware interrupt frame, where SYSCALL reads three MSRs and jumps. 463 cycles of pure architectural overhead, avoided. (Note that int $0x80 also lands in the 32-bit compat handler and pays argument translation; the comparison is of entry mechanisms as a user would actually use them, not a perfectly controlled A/B.)
+4.1 ns / ~20 cycles — installing a trivial seccomp filter. A seccomp-BPF program that does nothing but check the architecture and return SECCOMP_RET_ALLOW raises the null-syscall cost from 34.2 to 38.3 ns, a 12% tax on every syscall in the process, forever. The mechanism is that SYSCALL_WORK_SECCOMP in the thread’s syscall-work mask forces do_syscall_64 off the fast path and into syscall_trace_enter(), which then runs the filter. This is a real, measurable, and often-unbudgeted cost of container sandboxing — see Seccomp and seccomp-BPF and seccomp and Syscall Filtering for the mechanism and the filter language.
+15 ns — write() of 64 bytes to /dev/null versus getpid(). The extra 75 cycles are the only part of that call that is actual work: vfs_write, the file-descriptor table lookup, copy_from_user of 64 bytes (with its Spectre v1 LFENCE), and the null device’s write handler. Everything else it pays is identical to getpid. This is the ratio that makes batching obviously correct: for a small write, roughly 70% of the cost is the crossing, not the I/O.

Layer 2, measured — and the surprise

The conventional claim (repeated in the section below, and widely elsewhere) is that a syscall’s cache and TLB pollution is a major hidden cost paid after the call returns, invisible to a naive microbenchmark. That claim is testable, so it was tested.

The experiment: build a randomly-permuted pointer-chase through a buffer of a given size, one node per 64-byte cache line, so that each dereference is a dependent load that defeats the prefetcher. Then, in a loop, perform an event and immediately time a fixed 16-node chase segment with RDTSC. The event is one of: nothing, a userspace function call, getpid(), or write(/dev/null, 64). Because the RDTSC pair brackets only the chase and is identical in every condition, its overhead cancels exactly in the differences. Any pollution the syscall inflicts must show up as a longer chase segment immediately afterwards.

Working set	after nothing	after fn call	after `getpid()`	after `write(64B)`	Δ from `getpid()`
32 KiB (fits L1d)	18.2 ns	18.4 ns	19.4 ns	20.4 ns	+1.2 ns
256 KiB (fits L2)	49.9	49.9	49.9	49.9	−0.0
2 MiB (fits L2)	141.4	143.9	149.2	143.7	+7.8
16 MiB (fits L3)	2064.5	2015.4	2077.9	2094.0	+13.4
64 MiB (= L3)	2470.0	2498.1	2459.9	2467.7	−10.1
256 MiB (> L3)	2587.2	2589.8	2583.8	2573.5	−3.5

The pollution from a null syscall on this machine is about 1 ns, and it is only visible when the working set fits in L1. At every larger size the delta is under 1% of the baseline and changes sign between rows — it is noise, not signal. This directly contradicts the strong form of the conventional claim, and the reason is instructive: the page-walk storm that Gregg measured was a PTI effect, not an intrinsic property of system calls. Writing CR3 on a pre-PCID kernel flushed the entire TLB, so every syscall destroyed the process’s address translations and the next memory accesses paid to rebuild them. With PTI off — as it is on every AMD part and on Intel silicon from roughly the Coffee Lake refresh onward — no CR3 write happens on the syscall path at all, the TLB is never flushed, and a null syscall touches so little kernel state that it evicts barely more than a function call would.

The honest conclusion: layer 2 is real, but on unmitigated 2026 hardware it is a rounding error for small syscalls, and the note’s original framing of it as a co-equal “tax” overstated it. It becomes significant again in exactly two circumstances: when PTI is enabled without PCID support (the 2018 scenario), and when the syscall does enough work to walk a lot of kernel data structures — a read() through a filesystem, a sendmsg() through the network stack, a page fault. For those, the pollution is a function of the work, not of the crossing.

Uncertain

Verify: whether the ~1 ns L1-only pollution figure holds for syscalls that touch substantially more kernel state than getpid/write-to-null — for example openat + close of a real file, or a recvmsg on a loaded socket. Reason: only two syscalls were tested in the pollution experiment, both deliberately trivial, and the experiment was not run on a PTI-enabled machine where the expected result is very different. To resolve: extend the same chase-segment method across a range of syscalls of increasing kernel-side footprint, and repeat on a machine booted with pti=on. uncertain

What these numbers do not include

Three costs are deliberately outside this measurement and must not be assumed absent in production:

Blocking. Every syscall here returns immediately. A read() that sleeps pays a context switch, and a context switch is a different and much larger cost — on the order of a microsecond — that also drags in IBPB, a possible CR3 change to a genuinely different address space, and a cold cache on the far side. See System Calls and the Scheduler.
Contention. These are single-threaded measurements on an idle machine with a dedicated core. A syscall that takes a contended lock, or runs on a core whose SMT sibling is busy, costs more.
The mitigated case. As stated at the top: this machine has essentially every entry-path mitigation patched out. The next two sections explain what each one would add.

Where The 173 Cycles Go — The v6.12 Entry Path, Annotated

The measurement says a null syscall is ~173 core cycles. This section walks the actual code that spends them, in order, at the Linux 6.12 LTS tag, and attributes cost to each stage.

First, a measured split: user-mode versus kernel-mode

Before attributing anything, it helps to know how much of the cost is even inside the kernel. The perf_event_open(2) interface can count PERF_COUNT_HW_CPU_CYCLES with exclude_kernel = 1, which an unprivileged process is permitted to do at the default kernel.perf_event_paranoid = 2. Counting a getpid() loop that way, and subtracting from the independently-measured total, gives:

Loop body	User-mode (CPL 3) cycles/iter	Total cycles/iter	Kernel-mode (CPL 0) cycles/iter
`sink += i` (control)	1.0	1.0	0.0
`getpid()` via raw `SYSCALL`	40–44	~180	~136–151
`getppid()` via raw `SYSCALL`	~43	~185	~143–145

The control row is the validation: an empty loop attributes essentially zero cycles to ring 0, which it should, so the arithmetic is sound. (Two caveats on the middle column: having the PMU counter attached and running inflates the measured wall time slightly — 34.8–38.6 ns across runs, against 34.2 ns with no counter — so the totals here run a little high, and the split is best read as a ratio rather than as absolute cycles.) The real finding is that ratio — roughly 22–24% of a null syscall’s cycles retire in user mode and 76–78% at CPL 0 — and the more surprising half of it: the user-mode figure is forty cycles, not one. Those forty cycles are the SYSCALL instruction’s own execution before the privilege level changes, plus the pipeline restart on the userspace side of SYSRET. Roughly a quarter of the cost of a system call is paid in userspace, by the two instructions that bracket it, before the kernel executes anything at all. That is the “serialization” of layer 1, isolated and measured.

The path, stage by stage

sequenceDiagram
    autonumber
    participant U as User code<br/>(CPL 3)
    participant HW as CPU<br/>(microcode + MSRs)
    participant A as entry_SYSCALL_64<br/>(entry_64.S)
    participant C as do_syscall_64<br/>(common.c)
    participant H as __x64_sys_getpid<br/>(generated stub)

    U->>HW: syscall  (rax=39, args in rdi/rsi/rdx/r10/r8/r9)
    Note over HW: rip -> rcx, rflags -> r11,<br/>rflags &= ~MSR_SYSCALL_MASK,<br/>cs/ss/rip from MSR_STAR/MSR_LSTAR.<br/>NOTHING pushed, rsp unchanged.<br/>~40 cyc attributed to CPL 3
    HW->>A: enter at MSR_LSTAR, CPL 0, user rsp still live
    Note over A: swapgs  (L91) - swap in per-CPU GS base
    Note over A: stash user rsp in tss.sp2 (L93)
    Note over A: SWITCH_TO_KERNEL_CR3 (L94)<br/>PTI only - NOP on this machine
    Note over A: load kernel stack from<br/>pcpu_hot.top_of_stack (L95)
    Note over A: build pt_regs: 6 pushes for the<br/>hw frame + orig_ax (L101-107),<br/>then PUSH_AND_CLEAR_REGS (L109)<br/>= 15 more pushes + 13 xors
    Note over A: IBRS_ENTER / UNTRAIN_RET /<br/>CLEAR_BRANCH_HISTORY (L117-119)<br/>all ALTERNATIVE - NOPs here
    A->>C: call do_syscall_64(regs, nr)  (L121)
    Note over C: add_random_kstack_offset() (L78)
    Note over C: syscall_enter_from_user_mode() (L79)<br/>- fast path unless SYSCALL_WORK is set
    Note over C: unr < NR_syscalls, then<br/>array_index_nospec(unr, NR_syscalls) (L51)
    C->>H: x64_sys_call(regs, 39) - a switch,<br/>NOT an indirect call (since v6.9)
    Note over H: stub loads args out of pt_regs,<br/>zeroes unused arg registers,<br/>calls the real handler
    H-->>C: return value -> regs->ax (L52)
    Note over C: syscall_exit_to_user_mode() (L89)
    Note over C: five SYSRET-eligibility checks (L98-129)
    C-->>A: returns true (use SYSRET)
    Note over A: testb al/al then jz to the<br/>slow IRET path (L130)
    Note over A: IBRS_EXIT, then POP_REGS (L138-139)<br/>= 14 pops, rdi left on the stack
    Note over A: switch to the trampoline stack<br/>tss.sp0, re-push rsp and rdi (L145-150)
    Note over A: SWITCH_TO_USER_CR3_STACK (L158)<br/>PTI only - NOP on this machine
    Note over A: swapgs (L164) then<br/>CLEAR_CPU_BUFFERS verw (L165) - NOP here
    A->>HW: sysretq  (L166)
    HW->>U: rip from rcx, rflags from r11, CPL 3

The complete v6.12 SYSCALL round trip for getpid(), with line numbers into arch/x86/entry/entry_64.S and arch/x86/entry/common.c. What it shows: every instruction that executes between a userspace syscall and the resumption of the next userspace instruction, including the four mitigation hooks that are compiled in but patched out on this machine. The insight to take: the hardware does almost nothing — SYSCALL does not touch memory at all, pushes nothing, and does not even switch the stack pointer. Everything else, including finding a kernel stack and saving the user’s registers, is software the kernel has to write, and that software is where most of the instruction count lives.

The instruction budget

Counting from the v6.12 source, and marking which stages are present on the measured machine:

Stage	What it does	Instructions	Present here?	Attributed cost
`SYSCALL` (hardware)	`rip`→`rcx`, `rflags`→`r11`, mask `rflags` from `MSR_SYSCALL_MASK`, load `cs`/`ss`/`rip` from `MSR_STAR`/`MSR_LSTAR`, CPL 3→0	1	always	part of the ~40 measured CPL-3 cycles
`swapgs`	swap `MSR_GS_BASE` with `MSR_KERNEL_GS_BASE` so per-CPU data is reachable	1	always	an MSR-class operation; tens of cycles is the usual estimate
Stack acquisition	stash user `rsp` in `tss.sp2`, load `pcpu_hot.top_of_stack`	2	always	2 dependent loads/stores, one of them to per-CPU memory
`SWITCH_TO_KERNEL_CR3`	PTI: write `CR3` to the kernel page-table root	~4	no (PTI off)	~100+ cycles when present (pti.html)
`pt_regs` construction	5 hardware-frame pushes + `orig_ax` + `PUSH_REGS`’s 15	21 pushes	always	21 stores into a cold-ish kernel stack
`CLEAR_REGS`	zero `rsi rdx rcx r8 r9 r10 r11 rbx rbp r12 r13 r14 r15` so nothing user-controlled survives into speculative gadgets	13 `xorl`	always	cheap (zeroing idiom, often renamed away) but 13 slots of front-end bandwidth
`IBRS_ENTER`	write `MSR_IA32_SPEC_CTRL` to set IBRS for kernel execution	~6 + `wrmsr`	no (eIBRS)	a `wrmsr` is serializing and expensive — this is the legacy-IBRS penalty
`UNTRAIN_RET`	AMD Zen RETBleed/SRSO mitigation; may `call entry_ibpb`	~1–5	no	an `IBPB` is hundreds of cycles
`CLEAR_BRANCH_HISTORY`	`call clear_bhb_loop` — Intel BHI mitigation	1 + loop	no (BHI: Not affected)	a loop of dozens of branches
`do_syscall_64` C prologue	`add_random_kstack_offset()`, `syscall_enter_from_user_mode()`	tens	always	fast path unless `SYSCALL_WORK` bits are set — see below
Dispatch	bounds check, `array_index_nospec`, `switch (nr)`	~5	always	no indirect branch since v6.9
Handler stub	`__x64_sys_getpid(regs)` — unpack args from `pt_regs`, zero unused arg registers, tail into the real function	~10	always	trivial for `getpid`
Five SYSRET checks	`rcx==rip`, `r11==flags`, `cs`/`ss` match, `rip < TASK_SIZE_MAX`, `RF`/`TF` clear	~10	always	all predictable branches
`POP_REGS` + trampoline	14 pops, stack switch to `tss.sp0`, 2 re-pushes, 2 pops	~19	always	14 loads from the (now warm) kernel stack
`SWITCH_TO_USER_CR3_STACK`	PTI: write `CR3` back to the shadow root	~4	no (PTI off)	~100+ cycles when present
`CLEAR_CPU_BUFFERS`	`verw` — MDS/TAA buffer clear	1	no (Not affected)	a `verw` with microcode overload is expensive
`swapgs` + `sysretq`	restore user GS, `rcx`→`rip`, `r11`→`rflags`, CPL 0→3	2	always	the other half of the ~40 CPL-3 cycles

Two things jump out of that table. First, the mandatory part of the path is around 60–70 instructions of assembly, of which 35 are pushes and pops of the user register set — the register save/restore that a leaf function call simply never does. Second, eight of the eighteen stages are absent on this machine, every one of them a security mitigation, and every one of them implemented as an ALTERNATIVE — which is the mechanism the next section is about.

Uncertain

Verify: the per-stage cycle attributions in the right-hand column. Only two numbers in that column are measured here (the ~40 CPL-3 cycles and the ~140 CPL-0 total); the rest are instruction counts read from source plus order-of-magnitude characterisations of wrmsr, swapgs, IBPB and verw. Reason: attributing cycles to individual stages inside the kernel requires either a kernel module taking RDTSC samples at each label or a cycle-accurate simulator, neither of which was used. To resolve: instrument entry_64.S with RDPMC reads at the stage boundaries on a machine where kernel.perf_event_paranoid permits it, or run the path under a tool such as gem5. uncertain

The fast path is conditional: `SYSCALL_WORK`

One stage above is marked “fast path unless SYSCALL_WORK bits are set”, and that conditional is worth a paragraph because it is where a great deal of real-world syscall cost hides. syscall_enter_from_user_mode() checks a per-thread bitmask; if any bit is set the call detours into syscall_trace_enter(), which runs ptrace stops, seccomp filters, syscall tracepoints, and audit. Anything that observes or filters syscalls sets one of those bits and moves every syscall in that thread onto the slow path, permanently. The seccomp measurement above quantifies the cheapest possible case: a filter that does nothing but return SECCOMP_RET_ALLOW costs +4.1 ns / ~20 cycles, a 12% tax on every syscall, and a realistic container filter with a hundred rules costs more. The generic machinery that implements this is shared across architectures — see The Generic Syscall Entry and Exit Layer — and the filtering itself is covered in Seccomp and seccomp-BPF and seccomp and Syscall Filtering.

Version note — this code moved after v6.12

Everything cited above is at the v6.12 tag. Verified by existence-checking the same paths across tags: arch/x86/entry/common.c returns HTTP 200 at v6.12, v6.13 and v6.14, and HTTP 404 at v6.15 through v7.0 — the file was deleted in the 6.15 merge window, and do_syscall_64 together with the five SYSRET checks moved into arch/x86/entry/syscall_64.c (which grew from 995 bytes at v6.12 to 3,787 bytes at v6.15). The logic is unchanged: the same five checks with the same comments appear at syscall_64.c lines 103–139 at v6.15, v6.18 and v7.2. Beware that a different, unrelated arch/x86/entry/common.c exists again at v7.2 (1,891 bytes, containing only x86_entry_from_kvm() for KVM event forwarding) — the filename was reused for something else entirely, which is a good reason never to cite a path without also checking what is in it. v7.2 also carries FRED (Flexible Return and Event Delivery), Intel’s replacement event-delivery architecture, which changes the exception and interrupt entry paths — though SYSCALL still enters through entry_SYSCALL_64.

Layer 1 — The Mode Switch Itself

The cheapest part, and the part everyone thinks is the whole story. On x86-64 the userspace libc wrapper loads the syscall number into rax and the arguments into rdi, rsi, rdx, r10, r8, r9 and executes the syscall instruction (syscall(2)). The CPU does several things that a plain call never does:

Privilege escalation. The current privilege level (CPL) goes from 3 to 0. The processor loads a new code segment and stack from model-specific registers (MSR_LSTAR, MSR_STAR) configured at boot. This is not a memory jump the branch predictor can speculate freely past — it is an architectural mode change.
Serialization. The syscall/sysret pair forces a degree of pipeline serialization: the deep out-of-order machinery of a modern core cannot freely reorder instructions across the privilege boundary. The pipeline effectively drains and refills, costing the latency of however many instructions were in flight.
Register save/restore. The kernel entry assembly (entry_SYSCALL_64 on x86-64) pushes the full user register set into a pt_regs frame on the kernel stack so the handler can run C code and so the registers can be restored on return. The general-purpose registers, flags, and instruction pointer all have to be spilled and reloaded — work a leaf function call never does. The mechanics of this frame are covered in The pt_regs Register Frame and Per-Architecture Syscall Entry Assembly.

By itself this layer is on the order of tens of cycles — the reason the syscall instruction was introduced in the first place was that it is much cheaper than the legacy int 0x80 software interrupt it replaced (see The Syscall Instruction and Trap Mechanism). If layer 1 were the whole cost, syscalls would be merely “a bit slower than a function call.” It is layers 2–4 that turn “a bit” into “orders of magnitude.”

Layer 2 — Cache and TLB Pollution

A function call keeps running your code on your data; the instruction cache, data cache, and branch predictors stay warm with your working set. A system call is a detour into a completely different body of code — the kernel — touching kernel code pages, kernel data structures, the syscall table, lock words, and per-CPU variables. Every kernel cache line that gets pulled in evicts one of yours. When control returns to userspace, the instruction cache, data cache, branch target buffer, and the Translation Lookaside Buffer (TLB) are all partly cold with respect to your program. The next several hundred userspace instructions then run slower because they re-incur cache and TLB misses that would not have happened without the detour.

This pollution is invisible in a naive “time the syscall” microbenchmark that only measures the call itself — the cost is paid afterward, in the userspace code that resumes. Brendan Gregg’s KPTI analysis makes the severe case concrete: in one heavily-affected configuration, instructions-per-cycle (IPC) collapsed from 0.86 to 0.10 and “half the CPU cycles have page walks active” — i.e. the core spent half its time walking page tables servicing TLB misses caused by the syscall traffic (Gregg 2018).

But that severe case is layer 3 wearing layer 2’s clothes, and it is important to separate them. The page-walk storm Gregg measured happened because PTI writes CR3 on every entry and exit, and on a pre-PCID kernel a CR3 write flushes the whole TLB. The flush is what destroys the working set’s translations; the syscall is merely what triggers the flush. Remove PTI — as is the case on all AMD parts and on Intel silicon from roughly the Coffee Lake refresh onward — and there is no CR3 write on the syscall path at all.

The measurement reported in Layer 2, measured — and the surprise above tests exactly this on a PTI-off machine, by timing a dependent pointer-chase segment immediately after a syscall and comparing against the same segment after a function call. The result: about 1 ns of extra chase time after a null getpid(), and only when the working set fits in L1; at 256 KiB and above the difference is under 1% of baseline and changes sign between runs. So on unmitigated 2026 hardware, the intrinsic cache/TLB pollution of a small syscall is close to a rounding error — roughly 3% of the call’s own cost, not a co-equal tax.

Layer 2 is nonetheless real, and matters in two regimes. The first is PTI-enabled machines without PCID, which is the 2018 scenario Gregg measured and where layers 2 and 3 genuinely compound. The second is syscalls that do enough work to have a large kernel-side footprint: a read() that walks a filesystem’s page cache and inode structures, a sendmsg() through the network stack, a fork() that copies page tables. For those, the pollution scales with the work the kernel does, not with the fact that a boundary was crossed — which means it is not really a border tax at all, and batching does not amortize it the way it amortizes the crossing.

Layer 3 — Page Table Isolation (PTI/KPTI), the Dominant Cost When It Is Present

This is the layer that changed the economics of syscalls in January 2018, and on an affected CPU it is usually the single largest contributor. The qualifier is doing real work: on the machine benchmarked above it contributes nothing at all, because the CPU is not affected and the CR3 writes were patched out at boot. Whether layer 3 is the dominant cost or a non-cost is decided entirely by one line of sysfs.

Why it exists: Meltdown

Meltdown (CVE-2017-5754) is a hardware vulnerability in which a userspace instruction speculatively reads a kernel memory address — which is mapped into the process’s address space but marked supervisor-only — and, although the architectural read is squashed when the permission check fails, the speculative read has already left a measurable footprint in the cache that an attacker can read out via a timing side channel. The flaw exists because, historically, the kernel’s entire address space was mapped into every process (with supervisor-only protection), so kernel data was physically reachable by speculation even though it was architecturally forbidden.

What PTI does

Page Table Isolation (PTI), originally the academic KAISER patch, removes that reachability by giving each process two sets of page tables (pti.html, kernel docs; LWN KAISER, 2017):

A kernel page table with the full mapping (kernel + user), used while running in ring 0.
A userspace (“shadow”) page table that maps only userspace plus a tiny stub of kernel code/data — primarily the cpu_entry_area needed to perform the entry/exit transition. The bulk of the kernel is simply not mapped while in user mode, so speculation cannot reach it. As the docs put it, “the bulk of the kernel’s address space will thus be completely hidden from the process, defeating the known hardware-based attacks” (LWN KAISER).

Where the cost comes from

The two tables mean the CR3 register (which points at the active page-table root on x86) must be reloaded on every syscall entry and again on every exit — two extra CR3 writes per syscall that did not exist before 2018. The kernel docs state plainly: “Moves to CR3 are on the order of a hundred cycles, and are required at every entry and exit” (pti.html). The KAISER LWN coverage agrees: “Just the new instructions (CR3 manipulation) add a few hundred cycles to a syscall or interrupt” (LWN).

Worse, a CR3 reload historically flushes the entire TLB (the cache of virtual→physical translations; see The Translation Lookaside Buffer and TLB Shootdowns). A flushed TLB means the next userspace memory accesses must re-walk the page tables from RAM — the very page-walk storm Gregg measured. The mitigation for that is PCID (Process-Context Identifier): tagging TLB entries with an address-space ID so that switching CR3 need not flush, only re-select. Linux gained PCID-aware PTI in the 4.14 cycle, and it “reduces the cost of switching page tables during system calls considerably” (LWN KAISER). With PCID the data-TLB walk cost returns roughly to pre-KPTI levels (Gregg 2018).

The measured magnitude

Gregg’s analysis is the canonical primary-ish benchmark write-up. The headline findings (Gregg 2018):

Overhead scales with syscall rate. At ~50k syscalls/sec/CPU, ~2%. A MySQL OLTP workload at ~75k syscalls/sec/CPU lost 5%.
A stress test at ~210k syscalls/sec/CPU plus ~27k context-switches/sec/CPU lost ~25%.
The five factors that drive cost: syscall rate, context-switch rate, page-fault rate, working-set size (the >10 MB “jump”), and cache-access patterns.
His practical expectation for typical (Netflix) cloud workloads: 0.1% to 6% overhead.

The original Linux KPTI patch notes corroborate: “Most workloads … show single-digit regressions. 5% is a good round number for what is typical. The worst we have seen is a roughly 30% regression on a loopback networking test that did a ton of syscalls and context switches” (LWN KAISER).

A widely-repeated counter-claim is that the kernel’s own documentation says PTI costs “never exceeding 1%”. Read in context, that sentence does not say what it is usually quoted as saying. The PTI document’s “Overhead” section is a list of separate costs, and the 1% figure belongs to exactly one bullet in that list — the one about global pages: “Global pages are disabled for all kernel structures not mapped into both kernel and userspace page tables. This feature of the MMU allows different processes to share TLB entries mapping the kernel. Losing the feature means more TLB misses after a context switch. The actual loss of performance is very small, however, never exceeding 1%” (pti.html §23.3). The preceding bullet in the same list is the one about the CR3 writes, and it carries no percentage at all — only “Moves to CR3 are on the order of a hundred cycles, and are required at every entry and exit.” So the document bounds the global-pages sub-cost at 1% and declines to bound the total. There is no contradiction with Gregg’s 5–30% for syscall-bound workloads; the two statements are about different bullets. For a syscall-heavy program — a proxy, a database, an event loop hammering epoll_wait/read/write — the syscall-bound figure is the one that bites.

On the machine benchmarked for this note, the conditional turns out the favourable way: cat /sys/devices/system/cpu/vulnerabilities/meltdown prints Not affected, so none of layer 3 is present, and the 34.2 ns null-syscall figure contains zero CR3 writes. This is the single largest reason a 2026 AMD part can do a null syscall in 173 cycles.

The crucial caveat: PTI is conditional on the CPU

PTI is not unconditionally on. In pti_check_boottime_disable() (verified in arch/x86/mm/pti.c at the v6.12 tag), the auto path is:

if (pti_mode == PTI_AUTO && !boot_cpu_has_bug(X86_BUG_CPU_MELTDOWN))
    return;                 /* CPU not vulnerable → no PTI, no cost */
setup_force_cpu_cap(X86_FEATURE_PTI);

That is, in the default pti=auto mode PTI is enabled only on CPUs flagged with X86_BUG_CPU_MELTDOWN. All AMD processors are Meltdown-immune, and Intel parts with the hardware fix (roughly Coffee Lake refresh / Whiskey Lake onward, ~late 2018) report “Not affected” — on those machines PTI is off by default and layer 3 costs nothing (source: arch/x86/mm/pti.c v6.12). You can check on any running kernel with cat /sys/devices/system/cpu/vulnerabilities/meltdown (Mitigation: PTI vs Not affected). pti=on/nopti (alias pti=off) override the auto decision. So “PTI is the dominant cost” is accurate on a Meltdown-vulnerable CPU and false on a modern AMD or post-fix Intel.

Layer 4 — Spectre Branch-Prediction Barriers

Spectre (variants 1 and 2, CVE-2017-5753 and CVE-2017-5715) tricks the CPU’s branch predictor into speculatively executing the wrong path, leaking data through the same cache side channel. Unlike Meltdown, Spectre is not fixed by hiding the kernel’s pages; it is mitigated by inserting barriers that constrain speculation, and several of those barriers sit directly on the syscall path (spectre.html, kernel docs):

Spectre v1 — array_index_nospec on the syscall number. When the kernel uses the userspace-supplied syscall number to index sys_call_table[], an attacker could speculatively force an out-of-bounds index. The actual v6.12 dispatch clamps it: in do_syscall_x64() the kernel does unr = array_index_nospec(unr, NR_syscalls) before indexing (verified in arch/x86/entry/common.c at v6.12). This masks the index to a safe range even under misspeculation — a handful of extra ALU ops on every single syscall.
Spectre v1 — LFENCE in copy_from_user. The docs note “copy-from-user code has an LFENCE barrier to prevent the access_ok() check from being mis-speculated” (spectre.html). Any syscall that copies in arguments (see copy_to_user and copy_from_user and The access_ok Check and User Pointer Validation) eats a serializing fence.
Spectre v2 — retpolines or IBRS/eIBRS. Indirect branches in the kernel are converted to retpolines (return trampolines that trap misspeculation in a harmless loop) on CPUs without hardware help; on CPUs with enhanced IBRS (eIBRS) the retpolines are disabled at runtime and the hardware mode is used instead (spectre.html).

It is often said that syscall dispatch is itself full of indirect calls and therefore eats a retpoline on the hot path. That was true once and is no longer true. At v6.12, do_syscall_x64() does not index a table of function pointers at all — it calls x64_sys_call(regs, unr), which is a plain switch (nr) over direct calls, generated by including asm/syscalls_64.h with #define __SYSCALL(nr, sym) case nr: return __x64_##sym(regs);. The sys_call_table[] array still exists but its comment says plainly: “The sys_call_table[] is no longer used for system calls, but kernel/trace/trace_syscalls.c still wants to know the system call address” (arch/x86/entry/syscall_64.c v6.12). Converting the dispatch from an indirect call to a jump table removed a retpoline from the hottest path in the kernel. Existence-checking the same file across tags dates the change precisely: x64_sys_call is absent at v6.8 and present at v6.9, so the switch-based dispatch landed in Linux 6.9 (2024). On pre-6.9 kernels — and on any architecture that still dispatches through a pointer table — the retpoline cost is real.
IBPB / STIBP — context-switch and SMT barriers. The Indirect Branch Prediction Barrier (IBPB) is issued when switching to/from a process to clear the branch target buffer; STIBP prevents a sibling hyperthread from poisoning predictions. These are context-switch-time, not per-syscall, but a context switch almost always rides on the back of a syscall, so syscall-heavy + context-switch-heavy workloads pay both (this is why Gregg’s worst-case mixed syscalls and context switches).

Like PTI, the v2 mitigations are CPU-dependent: the kernel “selects reasonable default mitigations for the current CPU” (spectre.html), so the exact cost varies by microarchitecture and microcode. The speculation-barrier story at the boundary is developed further in Speculation Barriers and Spectre Hardening at the Syscall Boundary.

There is also a small, unrelated entry-path cost worth naming: do_syscall_64() calls add_random_kstack_offset() on every syscall (verified in arch/x86/entry/common.c v6.12), randomizing the kernel stack offset per call as a hardening measure against stack-based attacks — a few more cycles on the floor.

The Mitigation Matrix — What Is On, How To Tell, What It Costs

The four layers above describe costs that may be present. Which ones actually are is decided once, at boot, by binary-patching the kernel’s own instruction stream — and knowing how to read that state off a running machine is the difference between reasoning about syscall cost and guessing at it.

Mitigations are patched in, not branched over

Every mitigation on the entry path is written as an ALTERNATIVE macro. At boot, apply_alternatives() walks a table of these sites and rewrites each one in place: if the CPU feature bit is set, the site becomes the mitigation instructions; if not, it becomes NOPs of the same length. There is no runtime if. Looking at the actual v6.12 definitions makes this concrete — this is the MDS buffer clear (asm/nospec-branch.h):

.macro CLEAR_CPU_BUFFERS
	ALTERNATIVE "", "verw mds_verw_sel(%rip)", X86_FEATURE_CLEAR_CPU_BUF
.endm

and this is the branch-history clear:

.macro CLEAR_BRANCH_HISTORY
	ALTERNATIVE "", "call clear_bhb_loop", X86_FEATURE_CLEAR_BHB_LOOP
.endm

The first operand is the “not affected” text — empty — and the second is the mitigation. Even the SYSRET/IRET choice is patched this way: entry_64.S line 130 uses ALTERNATIVE "testb %al, %al; jz swapgs_restore_regs_and_return_to_usermode", "jmp swapgs_restore_regs_and_return_to_usermode", X86_FEATURE_XENPV, so a Xen PV guest gets an unconditional jump to the slow return path with no test at all.

flowchart TD
    B["Boot: check_bugs() /<br/>cpu_select_mitigations()"] --> D{"CPU has<br/>X86_BUG_CPU_MELTDOWN<br/>and pti=auto?"}
    D -->|yes| P1["setup_force_cpu_cap(X86_FEATURE_PTI)"]
    D -->|no| P0["leave X86_FEATURE_PTI clear"]
    P1 --> A
    P0 --> A
    A["apply_alternatives():<br/>walk every ALTERNATIVE site"]
    A --> E{"feature bit set<br/>for this site?"}
    E -->|yes| ON["patch in the mitigation<br/>e.g. mov CR3 / verw /<br/>wrmsr SPEC_CTRL / call clear_bhb_loop"]
    E -->|no| OFF["patch in NOPs of<br/>equal length"]
    ON --> R["entry_64.S text is now fixed<br/>for the life of the boot"]
    OFF --> R
    R --> S["/sys/devices/system/cpu/vulnerabilities/*<br/>reports what was chosen"]

How a mitigation gets onto (or off) the syscall entry path. What it shows: the decision is made once at boot and expressed as self-modifying code, not as a branch evaluated per syscall. The insight to take: there is no per-syscall cost for a mitigation your CPU does not need — the instructions are literally not there — which is why the same kernel binary can cost 173 cycles per syscall on one machine and several hundred on another, and why the only honest way to quote a syscall cost is alongside the sysfs readout of the machine it was measured on.

The matrix

Mitigation	Defends against	Where it lands on the syscall path	sysfs file (under `/sys/devices/system/cpu/vulnerabilities/`)	“On” looks like	Cost
PTI / KPTI	Meltdown (CVE-2017-5754)	`SWITCH_TO_KERNEL_CR3` on entry (`entry_64.S` L94) and `SWITCH_TO_USER_CR3_STACK` on exit (L158) — two `CR3` writes per syscall	`meltdown`	`Mitigation: PTI`	“Moves to CR3 are on the order of a hundred cycles, and are required at every entry and exit” (pti.html); “a few hundred cycles to a syscall or interrupt” (LWN 2017); “KPTI is an expensive mitigation, with overheads of 30% or more reported for some specific workloads” (LWN 2018). Without PCID it also flushes the whole TLB
Spectre v1 index masking	Bounds-check bypass (CVE-2017-5753)	`array_index_nospec(unr, NR_syscalls)` in `do_syscall_x64()` (`common.c` L51)	`spectre_v1`	`Mitigation: usercopy/swapgs barriers and __user pointer sanitization`	A handful of ALU ops; effectively free, and always on — there is no CPU that opts out
Spectre v1 `LFENCE` in usercopy	Same, via `access_ok()` misspeculation	inside `copy_from_user()`, so every syscall that reads user memory	`spectre_v1`	same string	A serializing fence per usercopy
*`FENCE_SWAPGS_`**	Spectre v1 swapgs (CVE-2019-1125)	`LFENCE` around conditional `swapgs` on the interrupt/exception paths (not the `SYSCALL` path, which always swaps)	`spectre_v1`	same string	One `LFENCE` on affected entries
Retpoline	Spectre v2 / branch target injection (CVE-2017-5715)	every indirect branch in the kernel. Note: syscall dispatch itself stopped being an indirect branch in v6.9	`spectre_v2`	`Mitigation: Retpolines`	A `call`/`ret` trampoline per indirect branch. LWN’s contemporaneous estimate was 0–1.5% overall (LWN 2018) — but concentrated in indirect-call-heavy paths, where it hurt badly enough that the networking and DMA layers were rewritten to avoid indirect calls entirely
Legacy IBRS on entry	Spectre v2 on CPUs without eIBRS	`IBRS_ENTER` (L117) and `IBRS_EXIT` (L138) — a `wrmsr` to `MSR_IA32_SPEC_CTRL` on both entry and exit	`spectre_v2`	`Mitigation: IBRS`	Two serializing `wrmsr`s per syscall. LWN: “the performance cost of IBRS is evidently quite high. So it seems that, as a general rule, retpolines are preferable to IBRS” (LWN 2018) — the most expensive entry-path mitigation after PTI
eIBRS / Automatic IBRS	Spectre v2, in hardware	nothing — it is a mode bit set once, so `IBRS_ENTER`/`IBRS_EXIT` are NOP’d	`spectre_v2`	`Mitigation: Enhanced / Automatic IBRS`	Zero per-syscall. This is what the benchmarked machine has
IBPB	Cross-process BTB poisoning	context switch (`switch_mm`), not syscall entry — but a blocking syscall usually causes a context switch	`spectre_v2` (`IBPB: conditional` / `always-on`)	`IBPB: always-on`	An `IBPB` flushes the branch predictors; the docs call the always-on mode higher cost than conditional (spectre.html)
STIBP	SMT sibling poisoning	a `SPEC_CTRL` bit, set per task at context switch	`spectre_v2` (`STIBP: ...`)	`STIBP: always-on`	Degrades the sibling thread’s prediction quality rather than adding instructions; “This will add overhead as indirect branch speculations for all programs will be restricted” (spectre.html)
SSBD	Speculative Store Bypass (Spectre v4)	not the entry path — a `SPEC_CTRL` bit affecting store-to-load forwarding in the task	`spec_store_bypass`	`Mitigation: Speculative Store Bypass disabled via prctl` means opt-in and off by default; `disabled` under `seccomp` means seccomp’d tasks get it automatically	Slows the task’s own stores, not its syscalls. Note the seccomp interaction: a container that installs a seccomp filter may silently acquire SSBD too
`UNTRAIN_RET` / Safe RET	RETBleed (Zen 1–2) and SRSO / Inception (Zen 3–4)	`UNTRAIN_RET` at `entry_64.S` L118 — may `call entry_ibpb` per entry	`retbleed`, `spec_rstack_overflow`	`Mitigation: safe RET` or `Mitigation: IBPB`	“does come at the cost of some performance depending on the workload” (srso.html); the `IBPB` variant is far worse
BHI clear loop	Branch History Injection	`CLEAR_BRANCH_HISTORY` at L119 — `call clear_bhb_loop`	`spectre_v2` (`BHI: ...`)	`BHI: BHI_DIS_S` or `BHI: SW loop`	A loop of dozens of branches per entry when the software loop is used
`VERW` buffer clear	MDS, TAA, MMIO stale data, RFDS	`CLEAR_CPU_BUFFERS` at L165 — one `verw` immediately before `sysretq`	`mds`, `tsx_async_abort`, `mmio_stale_data`, `reg_file_data_sampling`	`Mitigation: Clear CPU buffers`	A microcode-overloaded `verw` on every return to userspace
seccomp-BPF (not a CPU mitigation, but the same shape of cost)	untrusted syscalls	sets `SYSCALL_WORK_SECCOMP`, forcing every syscall off the fast path into `syscall_trace_enter()`	n/a — check `Seccomp:` in `/proc/<pid>/status`	`Seccomp: 2`	Measured here: +4.1 ns / ~20 cycles / +12% on a null syscall, for a filter that does nothing

Reading the state off a running machine

The whole matrix collapses into one command:

# What is my kernel actually doing on every syscall?
grep . /sys/devices/system/cpu/vulnerabilities/*

Each file prints one line: Not affected (the CPU does not have the flaw, nothing is patched in), Vulnerable (the flaw is present and nothing is mitigating it), or Mitigation: <name> (patched in, and the name tells you which technique). Two habits are worth forming. First, read spectre_v2 in full — it is a compound line with four or five semicolon-separated clauses (Mitigation: Enhanced / Automatic IBRS; IBPB: conditional; STIBP: always-on; PBRSB-eIBRS: Not affected; BHI: Not affected), and the expensive part may be in any of them. Second, never infer the state from the CPU model; microcode revisions change it, and so do the mitigations=, pti=, spectre_v2=, spec_rstack_overflow= and nosmt command-line knobs. The blunt instrument mitigations=off disables the lot and is the correct thing to benchmark against when you want to know what the mitigations are costing you — and the wrong thing to run in production on a machine that executes anyone else’s code.

For a per-process view, /proc/<pid>/status carries a Speculation_Store_Bypass: line (thread vulnerable vs thread force mitigated) and a Seccomp: line (0 none, 1 strict, 2 filter), both of which change the cost of that process’s syscalls specifically.

Why “Minimize Syscalls” Is Real Advice

Put the layers together and the conclusion is forced. The cost of a syscall is fixed per crossing and largely independent of the work done — getpid() (which does almost nothing) and a 4 KB read() pay almost the same border tax. So a program that issues a million tiny syscalls pays the tax a million times. The strategies that exist in Linux specifically to dodge this are not micro-optimizations; they are architectural:

The vDSO turns a few hot, harmless “syscalls” (clock reads, getcpu) into ordinary function calls against a shared memory page — zero border crossings. The measured gap is large enough to matter; see Raw Syscalls vs vsyscall vs vDSO Performance.
io_uring lets a process submit thousands of I/O operations through a shared-memory ring with one (or, in polled mode, zero) io_uring_enter syscall — amortizing one border tax over thousands of operations.
Vectored and batched calls — writev, recvmmsg — pack many buffers or messages into one crossing.
Buffering in userspace — stdio’s BUFSIZ buffering, a database’s page cache — is the oldest version of the same idea: do many logical operations between physical syscalls.

This is also why the standard diagnostic for “my I/O-bound program is slow” is strace -c (see How strace Works): it counts syscalls and shows where the crossings are, so you know what to batch.

Common Misunderstandings

“A syscall is basically a function call.” No — it is a privilege transition with serialization, register spills, cache/TLB pollution, and (on affected CPUs) two CR3 reloads plus speculation barriers. Orders of magnitude apart.
“KPTI made everything 30% slower.” Only syscall-and-context-switch-bound workloads on Meltdown-vulnerable CPUs approach that; typical mixed workloads are single-digit, and modern AMD/post-fix Intel pay nothing because PTI is off by default (arch/x86/mm/pti.c v6.12).
“The cost is in the call; once it returns I’m fine.” The cache/TLB pollution is paid after return, in your own code — invisible to a microbenchmark that times only the call.
“getpid() is cheap because it does so little.” It is cheap relative to a blocking read, but it still pays the full fixed border tax; the work it does is dwarfed by the crossing. Measured here, getpid() (34.2 ns) and lseek() (33.8 ns) and close(-1) (35.0 ns) are within 4% of each other, because in all three the crossing is the cost. (This is exactly why getpid-class calls were candidates for the vDSO historically — though on Linux only the clock/cpu calls actually live there today; see Raw Syscalls vs vsyscall vs vDSO Performance.)
“A syscall costs hundreds of nanoseconds.” Not on unmitigated 2026 hardware. The floor measured here is 34.2 ns. “Hundreds of nanoseconds” describes a Meltdown-vulnerable CPU carrying the full mitigation stack, and repeating it as a universal figure over-states modern cost by roughly an order of magnitude — which then justifies optimisations that do not pay for themselves. Always attach the mitigation state to the number.
“Syscall dispatch eats a retpoline because it is an indirect call through sys_call_table[].” True before Linux 6.9, false since. do_syscall_x64() calls x64_sys_call(), which is a switch over direct calls; the table survives only for the syscall tracepoints (syscall_64.c v6.12).
“The kernel docs say PTI costs under 1%, so KPTI panic was overblown.” The 1% in pti.html §23.3 bounds one specific sub-cost — the loss of global pages — and the document deliberately does not put a percentage on the CR3 writes. See the discussion in layer 3.
“Mitigations are checked at runtime, so an unaffected CPU pays a predictable branch.” It pays nothing: the sites are ALTERNATIVEs, rewritten to NOPs at boot. There is no branch to predict.

Alternatives and When to Choose Them

“Minimize syscalls” is a direction, not a technique. There are six distinct techniques hiding under it, they are not interchangeable, and choosing the wrong one is a common way to spend a week making something no faster. The measurements above give a rough exchange rate: one avoided crossing is worth about 34 ns / 173 cycles on unmitigated 2026 AMD hardware, and several hundred nanoseconds on a Meltdown-vulnerable Intel part — so an optimisation is worth doing when it removes crossings in proportion to that.

Technique	What it removes	Applies to	Measured / expected gain	Cost of adopting it
vDSO	the crossing entirely, for a handful of calls	`clock_gettime`, `gettimeofday`, `time`, `getcpu` only	58.6 ns → 17.1 ns measured here (3.4×)	none — it is already the default through libc. The work is not accidentally defeating it
Bigger buffers / userspace buffering	crossings in proportion to buffer size	any streaming read/write	linear: 4 KiB→64 KiB reads is 16× fewer crossings	memory, and latency if you buffer writes that someone is waiting on
Vectored I/O — writev, `preadv2`	one crossing per scatter/gather set instead of per buffer	code that already has several buffers in hand	n buffers → 1 crossing	restructuring call sites; no ordering change
Batched socket calls — recvmmsg	one crossing per message batch	UDP and packet-rate-bound network code	up to `vlen` messages per crossing	error handling gets harder: a partial batch reports how many succeeded, not which
io_uring	almost all crossings, via shared-memory submission and completion rings	arbitrary I/O, including file, network, and `openat`/`close`	one `io_uring_enter` for thousands of operations; zero in `SQPOLL` mode	by far the largest: a new asynchronous programming model, a kernel-version floor, a real security-surface history, and it is frequently disabled by container seccomp policy
Kernel bypass — DPDK, SPDK, AF_XDP	the kernel from the data path altogether	specialised packet/storage processing at millions of ops/sec	crossings go to zero on the data path	you lose the kernel’s networking or storage stack, and usually a whole CPU core to a busy-poll loop
`mitigations=off`	layers 3 and 4, not the crossing	any workload on hardware you fully trust	restores the unmitigated floor — potentially several hundred ns per syscall on affected parts	you are turning off Meltdown and Spectre defences. Defensible on a single-tenant machine running only your own code; indefensible anywhere that runs someone else’s

flowchart TD
    S["Profile says syscall overhead<br/>is actually significant<br/>(perf stat -e raw_syscalls:sys_enter)"] --> Q1{"Is it clock or<br/>CPU-id reads?"}
    Q1 -->|yes| V["Use the vDSO.<br/>Check you have not defeated it:<br/>no raw syscall, no vsyscall=emulate,<br/>clocksource is tsc not hpet/acpi_pm"]
    Q1 -->|no| Q2{"Streaming a byte<br/>or record stream?"}
    Q2 -->|yes| B["Buffer bigger in userspace.<br/>Cheapest possible change,<br/>usually one constant"]
    Q2 -->|no| Q3{"Several buffers or<br/>messages ready at once?"}
    Q3 -->|yes| VEC["readv / writev /<br/>sendmmsg / recvmmsg"]
    Q3 -->|no| Q4{"Many independent<br/>in-flight operations,<br/>and you can restructure?"}
    Q4 -->|yes| IOU["io_uring<br/>(check the container's<br/>seccomp policy first)"]
    Q4 -->|no| Q5{"Millions of ops/sec,<br/>dedicated hardware,<br/>willing to lose the<br/>kernel stack?"}
    Q5 -->|yes| BY["Kernel bypass:<br/>DPDK / SPDK / AF_XDP"]
    Q5 -->|no| M{"Single-tenant machine<br/>running only trusted code?"}
    M -->|yes| MIT["Consider selectively relaxing<br/>mitigations, measure the delta,<br/>document the decision"]
    M -->|no| ACC["Accept the cost.<br/>Optimise elsewhere"]

A decision tree for removing syscall cost, ordered by cost-of-adoption rather than by size of win. What it shows: each branch eliminates a different kind of crossing, and the cheap branches are at the top. The insight to take: the first box is not optional. Gregg’s guidance applies directly here — many services run below 10,000 syscalls/sec/CPU, where syscall overhead is under half a percent and every box below the first one is wasted effort (Gregg 2018). Measure the rate before choosing a technique.

A seventh option deserves naming only to be dismissed: caching syscall results in userspace. It is tempting (getpid() is constant! cache it!) and it is how glibc used to behave, but it is a correctness hazard — glibc removed its getpid() cache precisely because it went wrong across clone() — and it buys 34 ns. Do not re-invent it.

Production Notes

Know your syscall rate before optimising anything. The single most useful number is syscalls per second per CPU, and it is one command: perf stat -e raw_syscalls:sys_enter -a -I 1000 gives the system-wide rate, divided by the CPU count for the per-CPU figure (Gregg 2018). Gregg’s calibration for PTI-era overhead maps rate to impact directly: below ~10k/sec/CPU the overhead is negligible (he quotes <0.5% for most Netflix services); at 50k/sec/CPU it is around 2%; and the workload classes that actually run high are, in his words, “proxies, databases, and others that do lots of tiny I/O”, plus microbenchmarks. For a per-process breakdown of which calls, strace -c -p <pid> (see How strace Works) is the blunt tool and perf trace the sharper one, because strace itself uses ptrace and therefore adds two stops per syscall — it will distort exactly what you are measuring.

The 2018 rollout is the best-documented instance of syscall cost mattering at scale, and its lessons are about tuning, not avoidance. Gregg’s production validation is worth reading as a methodology example: a MySQL OLTP benchmark at 75k syscalls/sec/CPU was predicted by his model to lose about 4% and measured at 5%; the same workload on a 64-CPU system measured 3%. The one case that badly missed the model was a stress test at 210k syscalls/sec/CPU plus 27k context-switches/sec/CPU with a ~25 MB working set — predicted under 12%, measured 25% — and the excess was entirely TLB page-walk cycles, i.e. that application was more sensitive to TLB flushing than the microbenchmark. The fix in every case was the same two knobs: PCID and huge pages. His tlbstat output quantifies PCID precisely — data-TLB walks per second went from 291,125 without PTI, to 6,546,554 with pti, nopcid (a 22× increase), and back to 313,171–359,117 with pti, pcid, with IPC recovering from 0.81 to 0.86–0.87. In the worst configuration he measured, IPC fell from 0.86 to 0.10 and “half the CPU cycles have page walks active” — his comment, “I’ve never seen TLB pain this bad, ever.” The practical lesson: if you are running PTI, confirm PCID is in use before you conclude anything about syscall cost, because the difference between the two configurations is larger than almost any application-level optimisation you could make.

Container sandboxing is an unbudgeted syscall tax. Every containerised process on a modern runtime runs under a seccomp-BPF filter, and the measurement in this note puts a floor under what that costs: +4.1 ns / ~20 cycles / +12% on a null syscall for a filter that allows everything unconditionally. A realistic Docker or Kubernetes default profile evaluates a long list of rules, and the cost scales with filter complexity and with the number of filters stacked (each seccomp(2) call adds another program, and all of them run). This shows up in exactly the workloads that are already syscall-bound, and it is invisible to application profiling because it is charged to kernel entry rather than to any function. If a service is syscall-bound and containerised, the filter is a legitimate thing to measure — perf stat the same binary with and without the profile — before concluding the application is at fault.

Syscall cost is a hardware-generation property, and the generation gap is enormous. The same kernel binary, the same application, and the same syscall rate can differ by a factor of several purely on the mitigation state of the CPU. A 2017 Skylake Xeon pays PTI’s two CR3 writes, retpolines, legacy IBRS wrmsrs on entry and exit, and an MDS verw on every return; the Zen 5 part measured here pays none of them, because every one of those sites was patched to NOP at boot. This has a direct consequence for capacity planning and for benchmark hygiene: a syscall-cost number is meaningless without the /sys/devices/system/cpu/vulnerabilities/ readout of the machine it came from, and a fleet with mixed CPU generations will have genuinely different per-node syscall costs that no amount of application tuning will equalise. When comparing two machines, diff those files first.

The retpoline era rewrote kernel data paths, not just entry code. The most under-appreciated production consequence of Spectre v2 is that retpolines made indirect calls expensive everywhere, and the kernel’s response was to remove them from hot paths rather than absorb the cost. LWN’s contemporaneous estimate put retpoline overhead at “0–1.5%” overall (LWN, January 2018) — a figure that is true on average and badly misleading in the places that mattered. The DMA-mapping layer was reworked to test for the direct-mapping case and make a direct call instead of going through ops->map_page(); Jesper Dangaard Brouer, who had reported the regression, summarised the fix as “my XDP performance is back”. Later work on optimising indirect calls generally reported “a 10% improvement in an nginx benchmark”, and — the detail worth remembering — “even on a system with retpolines disabled, simply optimizing the indirect calls yields a 2% improvement for nginx” (LWN, December 2018). Syscall dispatch itself was eventually part of this wave: the switch from sys_call_table[] to a switch statement in Linux 6.9 removed an indirect branch from the single hottest path in the kernel. The general lesson for production is that a mitigation’s headline percentage is an average over a codebase, and the right question is always where in your code the affected construct is concentrated.

Watch for accidental vDSO defeat. The vDSO is the largest free win available, and it is easy to lose. Three ways it goes: a language runtime or library that issues clock_gettime as a raw syscall instruction rather than through the vDSO; a clocksource other than tsc (check /sys/devices/system/clocksource/clocksource0/current_clocksource — if it reads hpet or acpi_pm, the vDSO’s fast path cannot be used and every clock read falls back to a real syscall); and virtualised environments where the hypervisor has not exposed a stable TSC. The measurement in this note — 17.1 ns through the vDSO versus 58.6 ns through the kernel for the identical operation — is the size of what is lost. See How the vDSO Reads Kernel Time Without Trapping for why the fast path depends on the clocksource.

vault

Explorer

Why System Calls Are Expensive

Why System Calls Are Expensive

Mental Model — Four Taxes Stacked on One Instruction

Measuring It — A Benchmark Run for This Note

The machine and its mitigation state

Methodology

Results

What the five headline ratios actually say

Layer 2, measured — and the surprise

What these numbers do not include

Where The 173 Cycles Go — The v6.12 Entry Path, Annotated

First, a measured split: user-mode versus kernel-mode

The path, stage by stage

The instruction budget

The fast path is conditional: `SYSCALL_WORK`

Layer 1 — The Mode Switch Itself

Layer 2 — Cache and TLB Pollution

Layer 3 — Page Table Isolation (PTI/KPTI), the Dominant Cost When It Is Present

Why it exists: Meltdown

What PTI does

Where the cost comes from

The measured magnitude

The crucial caveat: PTI is conditional on the CPU

Layer 4 — Spectre Branch-Prediction Barriers

The Mitigation Matrix — What Is On, How To Tell, What It Costs

Mitigations are patched in, not branched over

The matrix

Reading the state off a running machine

Why “Minimize Syscalls” Is Real Advice

Common Misunderstandings

Alternatives and When to Choose Them

Production Notes

See Also

Graph View

Table of Contents

Backlinks

vault

Explorer

Why System Calls Are Expensive

Why System Calls Are Expensive

Mental Model — Four Taxes Stacked on One Instruction

Measuring It — A Benchmark Run for This Note

The machine and its mitigation state

Methodology

Results

What the five headline ratios actually say

Layer 2, measured — and the surprise

What these numbers do not include

Where The 173 Cycles Go — The v6.12 Entry Path, Annotated

First, a measured split: user-mode versus kernel-mode

The path, stage by stage

The instruction budget

The fast path is conditional: SYSCALL_WORK

Layer 1 — The Mode Switch Itself

Layer 2 — Cache and TLB Pollution

Layer 3 — Page Table Isolation (PTI/KPTI), the Dominant Cost When It Is Present

Why it exists: Meltdown

What PTI does

Where the cost comes from

The measured magnitude

The crucial caveat: PTI is conditional on the CPU

Layer 4 — Spectre Branch-Prediction Barriers

The Mitigation Matrix — What Is On, How To Tell, What It Costs

Mitigations are patched in, not branched over

The matrix

Reading the state off a running machine

Why “Minimize Syscalls” Is Real Advice

Common Misunderstandings

Alternatives and When to Choose Them

Production Notes

See Also

Graph View

Table of Contents

Backlinks

The fast path is conditional: `SYSCALL_WORK`