vault

Speculation Barriers and Spectre Hardening at the Syscall Boundary

15 min read

Speculation Barriers and Spectre Hardening at the Syscall Boundary

A system call is the moment a CPU running attacker-controlled userspace code transitions, in the space of one instruction, into the most privileged code in the system — with attacker-chosen values sitting in registers and an attacker-chosen number selecting which kernel function runs. That makes the syscall boundary the single richest target for transient-execution attacks: speculation that runs past a bounds check, past a privilege transition, or that leaks the contents of microarchitectural buffers filled by kernel work. Since the 2018 Spectre/Meltdown disclosures, the Linux entry path has been peppered with defenses that mostly cost nothing on unaffected hardware (they are ALTERNATIVE no-ops patched in only when the CPU is vulnerable): array_index_nospec() masks the syscall number before it can index anything, barrier_nospec() (an LFENCE) fences off speculative loads, the swapgs fences stop the GS-base from being speculatively wrong, IBRS/retpoline/UNTRAIN_RET/CLEAR_BRANCH_HISTORY blunt indirect-branch and return prediction, and a VERW instruction on the return path clears CPU buffers to defeat MDS. This note traces each of those at the x86-64 syscall entry/exit, as of Linux 6.12 LTS (arch/x86/entry/common.c, entry_64.S).

Mental Model

Speculative execution is the CPU guessing what comes next and running it before it knows whether the guess is right. If the guess is wrong, the architectural results are discarded — but the microarchitectural side effects (which cache lines got loaded, which buffers got read) persist, and a side-channel can read them back. Spectre-class attacks are about steering that speculation into a “gadget” that touches secret data, then recovering the secret from the cache footprint.

The syscall boundary concentrates three flavors of this:

  1. Bounds-check bypass (Spectre v1): the kernel checks nr < NR_syscalls, but the CPU can speculate past the branch and use a too-large nr to index a table — reading out of bounds. The fix is to mask the index itself so that under misspeculation it is forced to a safe value, not just to add a branch the CPU can ignore.
  2. Branch/return target injection (Spectre v2 and relatives — RETBleed, BHI, SRSO): the indirect calls and returns inside the kernel can be steered by attacker-poisoned predictors. The fix is to constrain or flush those predictors at the user→kernel transition.
  3. Microarchitectural data sampling (MDS, and the conditional-swapgs v1 variant): speculation can sample stale data out of internal CPU buffers (store/fill/load-port buffers), or run kernel code with a user GS base. The fix is a fence around swapgs and a buffer-clearing VERW before returning to userspace.
flowchart TB
  US["Userspace<br/>(attacker-controlled regs, nr in RAX)"]
  US -->|"syscall instruction"| SG["swapgs<br/>(switch to kernel GS)"]
  subgraph ENTRY["entry_SYSCALL_64 (entry_64.S)"]
    SG --> SAVE["PUSH_AND_CLEAR_REGS<br/>build pt_regs, zero scratch regs"]
    SAVE --> IB["IBRS_ENTER<br/>(set SPEC_CTRL.IBRS)"]
    IB --> UR["UNTRAIN_RET<br/>(IBPB / return-thunk)"]
    UR --> BHB["CLEAR_BRANCH_HISTORY<br/>(BHB clear loop)"]
    BHB --> DO["call do_syscall_64"]
  end
  subgraph C["do_syscall_64 → do_syscall_x64 (common.c)"]
    DO --> CHK{"unr &lt; NR_syscalls ?"}
    CHK -->|"no"| NI["__x64_sys_ni_syscall (-ENOSYS)"]
    CHK -->|"yes"| MASK["unr = array_index_nospec(unr, NR_syscalls)"]
    MASK --> DISP["x64_sys_call(regs, unr)<br/>dispatch handler"]
  end
  DISP --> EXIT["syscall return path"]
  EXIT --> VERW["CLEAR_CPU_BUFFERS (VERW)<br/>then swapgs; sysretq"]
  VERW --> US

The speculation hardening applied across one x86-64 syscall, entry to exit. What it shows: branch/return-predictor scrubbing (IBRS_ENTER, UNTRAIN_RET, CLEAR_BRANCH_HISTORY) happens immediately after the register frame is built; the syscall-number bounds check is followed by array_index_nospec() masking before any dispatch; buffer-clearing VERW happens last, right before returning. The insight: these are layered defenses against distinct attacks — masking for v1, predictor scrubbing for v2/RETBleed/BHI/SRSO, VERW for MDS — and on unaffected CPUs each is an ALTERNATIVE that patches to nothing, so the cost is paid only where the hardware needs it.

The Spectre-v1 Defense: Masking the Syscall Number

Why the number is a gadget

The classic Spectre v1 gadget is if (i < n) x = array[i];: a CPU that mispredicts the branch as taken will speculatively load array[i] for an out-of-bounds i, leaving a cache footprint that leaks array’s neighbours or whatever array[i] pointed at (spectre.rst; the original Spectre paper, Kocher et al. 2018). The syscall dispatch is exactly this shape: userspace puts an arbitrary value in RAX, the kernel checks nr < NR_syscalls, and on the fast path uses nr to reach a handler. A mispredicted bounds check would let nr run far out of range during speculation.

array_index_nospec — masking, not branching

The kernel’s tool for this is array_index_nospec() from include/linux/nospec.h. The crucial idea is that it does not add another branch (the CPU can speculate past branches); it computes a mask with straight-line arithmetic and ANDs it into the index, so that even under misspeculation the index is clamped:

/* include/linux/nospec.h */
static inline unsigned long array_index_mask_nospec(unsigned long index,
                                                    unsigned long size)
{
    OPTIMIZER_HIDE_VAR(index);  /* stop the compiler reasoning about index */
    return ~(long)(index | (size - 1UL - index)) >> (BITS_PER_LONG - 1);
}
#define array_index_nospec(index, size) ({ ... (typeof(_i))(_i & _mask); })

Walking the arithmetic symbol by symbol for the in-range case index < size: size - 1 - index is non-negative, and index is non-negative, so their bitwise OR has its top (sign) bit clear; casting to signed long and arithmetic-shifting right by BITS_PER_LONG - 1 (63 on a 64-bit machine) sign-extends that clear sign bit to all zeros, then ~ flips it to all ones — a mask of ~0. For the out-of-range case index >= size, size - 1 - index is negative (its top bit set), the OR’s sign bit is set, the arithmetic shift sign-extends to all ones, and ~ flips it to 0 — a zero mask. ANDing the index with ~0 leaves it unchanged when in range; ANDing with 0 forces it to 0 (a safe slot) when out of range. OPTIMIZER_HIDE_VAR() prevents the compiler from “helpfully” optimizing the mask away based on a value range it thinks it has proven — because the compiler reasons about architectural values, not the speculative ones the attacker controls (nospec.h).

Where it sits in the syscall path

On x86-64 in 6.12, the dispatch is no longer a single sys_call_table[nr] indexed load — arch/x86/entry/syscall_64.c turns it into a generated switch (x64_sys_call()), and the array sys_call_table[] is kept only so kernel/trace/trace_syscalls.c can resolve handler addresses (syscall_64.c). The masking happens in do_syscall_x64() before the dispatch:

/* arch/x86/entry/common.c */
static __always_inline bool do_syscall_x64(struct pt_regs *regs, int nr)
{
    unsigned int unr = nr;                 /* negative nr -> huge unsigned -> fails check */
    if (likely(unr < NR_syscalls)) {
        unr = array_index_nospec(unr, NR_syscalls);  /* clamp under misspeculation */
        regs->ax = x64_sys_call(regs, unr);          /* dispatch */
        return true;
    }
    return false;
}

Treating nr as unsigned int is itself part of the defense: a negative syscall number becomes a huge unsigned value that fails < NR_syscalls, so one comparison covers both ends of the range. The array_index_nospec() line then guarantees that even if the CPU speculates the bounds check as passing for an out-of-range unr, the value handed to x64_sys_call() is masked to a safe in-range slot, so the speculative dispatch cannot reach an attacker-chosen address or read out-of-bounds metadata. The x32 and IA-32-emulation paths (do_syscall_x32, do_syscall_32_irqs_on) apply the identical pattern with their own table sizes (common.c).

Uncertain

Verify: the precise residual speculation risk of a jump-table-compiled switch (x64_sys_call) versus the old array-indexed sys_call_table[nr]. Reason: the v1 concern classically targets an indexed load; the kernel still masks unr before the switch, which suggests the maintainers consider a compiler-generated jump table to carry equivalent risk, but the comment in syscall_64.c only explains the table is retained for tracing, not the speculation rationale. To resolve: read the commit that converted x86-64 dispatch from sys_call_table[] indexing to the x64_sys_call() switch and its changelog. uncertain

barrier_nospec — the heavier LFENCE hammer

Where masking the index is not enough — for example when the speculative load itself must be fenced rather than the index clamped — the kernel uses barrier_nospec(). On x86 it is an LFENCE patched in via ALTERNATIVE:

/* arch/x86/include/asm/barrier.h */
#define barrier_nospec() alternative("", "lfence", X86_FEATURE_LFENCE_RDTSC)

LFENCE serializes the instruction stream so that loads after it cannot execute speculatively before it. It is heavier than array_index_nospec() (which is branchless arithmetic with no pipeline stall), so the kernel prefers masking and reserves barrier_nospec() for places like copy_from_user’s pointer checks and __get_user speculation fences. The generic stub in nospec.h (#define barrier_nospec() do {} while (0)) means architectures without a speculation problem compile it away entirely (nospec.h). See The access_ok Check and User Pointer Validation and copy_to_user and copy_from_user for the uaccess use of this fence.

The Conditional-swapgs Spectre-v1 Variant

The Spectre-v1 documentation explicitly calls out the entry code’s swapgs: the user→kernel transition conditionally executes swapgs to switch the GS base from the user value to the kernel per-CPU base, and a mispredicted condition could leave kernel code running speculatively with the wrong GS base, dereferencing a user-controlled per-CPU pointer (spectre.rst, “kernel entry code … conditional swapgs checks”). The defense is a pair of LFENCEs gated by CPU-feature ALTERNATIVEs:

/* arch/x86/entry/calling.h */
.macro FENCE_SWAPGS_USER_ENTRY   /* in the path that DID swapgs from user */
    ALTERNATIVE "", "lfence", X86_FEATURE_FENCE_SWAPGS_USER
.endm
.macro FENCE_SWAPGS_KERNEL_ENTRY /* in the path that did NOT swapgs (from kernel) */
    ALTERNATIVE "", "lfence", X86_FEATURE_FENCE_SWAPGS_KERNEL
.endm

These appear in the interrupt/exception entry paths in entry_64.S (error_entry, paranoid_entry), not in the entry_SYSCALL_64 fast path — the syscall instruction’s transition is structured so the GS swap is unconditional there. But they are part of the same boundary-hardening story and are why the Spectre-v1 doc lists “conditional swapgs” as a v1 vector (entry_64.S, calling.h).

Branch- and Return-Predictor Scrubbing (Spectre v2, RETBleed, BHI, SRSO)

Spectre v2 and its descendants steer indirect branches and returns inside the kernel using predictors the attacker poisoned from userspace. The entry_SYSCALL_64 fast path scrubs these immediately after building the register frame:

; arch/x86/entry/entry_64.S, entry_SYSCALL_64
PUSH_AND_CLEAR_REGS rax=$-ENOSYS   ; save user regs, zero scratch regs
movq    %rsp, %rdi
movslq  %eax, %rsi                 ; sign-extend syscall nr as int
IBRS_ENTER                         ; raise SPEC_CTRL.IBRS for kernel
UNTRAIN_RET                        ; flush return predictor (IBPB / return-thunk)
CLEAR_BRANCH_HISTORY               ; clear Branch History Buffer
call    do_syscall_64

Each line maps to a distinct vulnerability:

  • IBRS_ENTER writes the Indirect Branch Restricted Speculation bit into the IA32_SPEC_CTRL MSR so that, while in the kernel, indirect-branch predictions cannot be controlled by less-privileged code. It is gated on X86_FEATURE_KERNEL_IBRS and CONFIG_MITIGATION_IBRS_ENTRY; on retpoline-mitigated kernels it patches to nothing because retpolines (in the indirect-call thunks) handle v2 instead (calling.h).
  • UNTRAIN_RET addresses return-predictor attacks (RETBleed, CVE-2022-29900/29901, and AMD’s SRSO / “Inception”, CVE-2023-20569). It expands to an Indirect Branch Prediction Barrier (IBPB) or a return-thunk untraining sequence depending on X86_FEATURE_ENTRY_IBPB / the SRSO config — overwriting poisoned return-stack-buffer state before the kernel executes any RET (nospec-branch.h).
  • CLEAR_BRANCH_HISTORY mitigates Branch History Injection (BHI, CVE-2022-0001), where the attacker poisons the Branch History Buffer to steer an indirect branch even under Enhanced IBRS. It patches in a call clear_bhb_loop (a sequence engineered to overwrite the BHB) under X86_FEATURE_CLEAR_BHB_LOOP; the Spectre doc notes BHI was first practical via unprivileged eBPF and that full mitigation needs BHI_DIS_S or the clearing sequence (spectre.rst, nospec-branch.h).

On return, IBRS_EXIT clears the kernel IBRS bit, and FILL_RETURN_BUFFER (in __switch_to_asm) stuffs the Return Stack Buffer on context switch to stop cross-task RSB poisoning (entry_64.S). These are the v2-family analogues of the v1 number-masking: instead of clamping a value, they sanitize the predictor state that the boundary crossing exposes.

MDS Buffer Clearing on the Return Path (VERW)

Microarchitectural Data Sampling (MDS — RIDL/Fallout/ZombieLoad, CVEs 2018-12126/12127/12130 and 2019-11091) lets speculation sample stale data out of the store buffer, fill buffers, and load ports — buffers that kernel work fills with secret data (mds.rst). The mitigation reuses the otherwise-obsolete VERW instruction: affected CPUs received a microcode update that overloads the memory-operand form of VERW to also clear those buffers (nospec-branch.h).

In 6.12 the clearing happens on the return-to-user path, immediately before sysretq (and the IRET path’s swapgs), via the CLEAR_CPU_BUFFERS macro:

; entry_64.S, syscall return via SYSRET
swapgs
CLEAR_CPU_BUFFERS    ; -> "verw mds_verw_sel(%rip)" under X86_FEATURE_CLEAR_CPU_BUF
sysretq
/* arch/x86/include/asm/nospec-branch.h */
.macro CLEAR_CPU_BUFFERS
    ALTERNATIVE "", "verw mds_verw_sel(%rip)", X86_FEATURE_CLEAR_CPU_BUF
.endm

The placement is deliberate: clearing buffers as the kernel leaves ensures no secret the kernel just touched lingers in a buffer that a sibling SMT thread or the returning userspace could sample. Only the memory-operand form of VERW clears buffers (the register form does not), which is why the operand is the static mds_verw_sel selector value in memory (nospec-branch.h). The whole thing is an ALTERNATIVE gated on X86_FEATURE_CLEAR_CPU_BUF, so an unaffected CPU runs a no-op.

Uncertain

Verify: that the historical mds_user_clear static key (named in the task brief) has been replaced by the X86_FEATURE_CLEAR_CPU_BUF ALTERNATIVE-patched CLEAR_CPU_BUFFERS macro, and that as of 6.12 the clear is on the return-to-user path rather than at entry. Reason: older write-ups and the brief refer to mds_user_clear; the 6.12 source uses the feature-bit ALTERNATIVE form and places CLEAR_CPU_BUFFERS before sysretq/IRET, which is the post-rework arrangement. To resolve: confirmed in entry_64.S and nospec-branch.h at v6.12 (X86_FEATURE_CLEAR_CPU_BUF, no mds_user_clear static-branch in the entry path) — but the exact kernel version where mds_user_clear was converted to the feature bit should be pinned from the changelog. uncertain

The C helper mds_clear_cpu_buffers() (a plain verw on the kernel DS selector) still exists for the idle path (mds_idle_clear_cpu_buffers()), which clears buffers before entering deeper idle states where an SMT sibling might sample them (nospec-branch.h).

Why the Syscall Boundary Specifically

Three properties make this boundary the prime gadget site, all of which compound the per-call cost discussed in Why System Calls Are Expensive:

  1. Attacker controls the inputs. The syscall ABI (The x86-64 Syscall Calling Convention) hands userspace direct control of RAX (the number that indexes The System Call Table) and six argument registers — precisely the “attacker-controlled input parameter” the Spectre doc says makes a gadget useful (spectre.rst).
  2. The transition crosses a privilege and address-space boundary. swapgs, the CR3 switch (KPTI), and the predictor state are all things that change at the boundary and can be speculatively wrong, which is unique to the entry/exit code.
  3. It runs on every privileged request. Because every resource access funnels through it, a single gadget here is reachable from any process — and a mitigation here protects the entire kernel surface, which is why the entry code is where the most defenses are concentrated.

Failure Modes and Common Misunderstandings

  • “array_index_nospec is just a bounds check.” It is the opposite: the bounds check is the branch the CPU ignores under speculation; array_index_nospec is branchless masking that survives misspeculation. Adding more ifs does nothing.
  • “Mitigations always slow my syscalls.” Only on affected hardware. Every defense above is an ALTERNATIVE/static-key/feature-bit construct that patches to a no-op when the booting CPU is not vulnerable; the cost is paid selectively. The exceptions are the always-present software constructs like retpoline thunks where built in.
  • “VERW at entry.” As of 6.12 the MDS buffer-clear (CLEAR_CPU_BUFFERS) is on the return-to-user path, not entry — clearing what the kernel touched as it leaves. Putting it at entry would not protect data the kernel reads during the call.
  • Disabling mitigations. mitigations=off (or per-bug knobs like nospectre_v1, mds=off, spectre_v2=off) patches these out. The /sys/devices/system/cpu/vulnerabilities/ files report the live state per bug; trusting a benchmark run with mitigations off as representative of production is a classic mistake.
  • SMT makes some of this leak anyway. MDS and BTB/L1 attacks can cross hyperthreads; the kernel’s per-boundary clearing helps, but the strongest mitigation for cross-sibling leakage is disabling SMT (nosmt), which the MDS doc discusses (mds.rst).

Alternatives and When to Choose Them

  • array_index_nospec() vs. barrier_nospec() — prefer masking (branchless, no pipeline stall) when you can clamp an index; use the LFENCE of barrier_nospec() only when the load itself must be serialized.
  • Retpoline vs. IBRS for Spectre v2 — retpolines (software, the indirect-call thunks) avoid the MSR-write cost of IBRS on older parts; eIBRS (enhanced IBRS in hardware) is preferred on CPUs that support it. The kernel selects automatically and patches the entry IBRS_ENTER accordingly.
  • VERW clearing vs. disabling SMT — buffer-clearing handles same-thread sampling cheaply; only nosmt fully closes cross-sibling MDS, at a large throughput cost.

Production Notes

The cumulative effect of these defenses is the dominant reason a bare getpid() got measurably slower across the 2018–2023 vulnerability cycle, and a major motivation for batching the boundary entirely with io_uring and serving hot reads from the vDSO without trapping at all. Operators tuning latency-sensitive, fully-trusted workloads sometimes set mitigations=off, accepting the security trade-off for the syscall-path speedup; the per-bug state is always auditable under /sys/devices/system/cpu/vulnerabilities/. The kernel’s hw-vuln admin guide is the authoritative, per-CPU reference for which mitigations apply (hw-vuln index).

See Also

Graph View

Backlinks