vault

The Syscall Instruction and Trap Mechanism

15 min read

The Syscall Instruction and Trap Mechanism

A system call is the only sanctioned way for unprivileged code to ask the kernel to do something the hardware forbids userspace to do directly. On x86-64 the entire userspace-to-kernel transition is one instruction — syscall — whose behaviour is defined by the CPU, not the operating system. Executing it does a small, fixed dance in silicon: it saves the return address, masks the flags register, switches the privilege level to ring 0, and jumps to a kernel entry point whose address the kernel programmed into a model-specific register at boot. Crucially, syscall saves nothing on the stack and does not change the stack pointer — the kernel must manually swap to a trusted stack and per-CPU data area itself, which is the first and most security-critical thing entry_SYSCALL_64 does (per arch/x86/entry/entry_64.S, v6.12 LTS). This note traces that dance step by step and contrasts it with the older int 0x80 software interrupt and sysenter, and with the very different way the CPU enters the kernel on an interrupt or exception.

Throughout, “ring 3” / CPL=3 (Current Privilege Level 3) means unprivileged userspace, and “ring 0” / CPL=0 means the privileged kernel — see User Mode and Kernel Mode for the protection-ring model this note assumes. The register ABI (which register carries the syscall number and arguments) is the subject of the sibling The x86-64 Syscall Calling Convention; this note covers the control-flow transition, not the register contract.

Mental Model

Think of syscall as a trapdoor with a pre-wired destination. Userspace cannot jump into arbitrary kernel code — that would defeat protection entirely. Instead, at boot the kernel writes the address of its single entry point (entry_SYSCALL_64) into a special CPU register (the LSTAR model-specific register). From then on, executing syscall always lands at exactly that address in ring 0, no matter what userspace intended. The CPU also stashes the userspace return address in RCX and the flags in R11 so the kernel can return precisely, and it forces a clean flags state by masking off dangerous bits. Everything else — saving the rest of the registers, finding a safe stack, switching page tables — is software’s job, done by the assembly at the entry point.

sequenceDiagram
    participant U as Userspace (ring 3)
    participant CPU as CPU (silicon)
    participant K as Kernel entry (ring 0)
    U->>CPU: execute `syscall`<br/>(RAX=nr, args in regs)
    Note over CPU: RCX := RIP (return addr)<br/>R11 := RFLAGS<br/>RFLAGS &= ~SYSCALL_MASK<br/>RIP := MSR_LSTAR<br/>CS/SS := from MSR_STAR<br/>CPL 3 to 0
    CPU->>K: jump to entry_SYSCALL_64
    Note over K: swapgs (user GS to kernel GS)<br/>save RSP, switch to kernel CR3<br/>load kernel stack top<br/>build pt_regs, clear regs
    K->>K: call do_syscall_64(pt_regs, nr)
    K-->>CPU: sysretq (or iretq)
    Note over CPU: swapgs back<br/>RIP := RCX, RFLAGS := R11<br/>CPL 0 to 3
    CPU-->>U: resume at return address

The syscall/sysret round trip on x86-64. What it shows: the CPU performs only the boxed register dance — return address into RCX, flags into R11, flags masked, RIP loaded from a kernel-programmed MSR, privilege dropped to ring 0; everything else (the GS swap, stack switch, page-table switch, register save) is kernel software. The insight: the hardware part is deliberately minimal and fixed-destination, so the kernel is in complete control of where execution lands and on what stack — the entry assembly, not the CPU, establishes a trustworthy execution environment.

What the syscall Instruction Does in Silicon

The syscall instruction (introduced by AMD for the AMD64/long-mode architecture, and the only fast syscall path used by 64-bit Linux) performs a fixed sequence with no memory accesses. The authoritative description is the processor vendor manual (AMD64 Architecture Programmer’s Manual Vol. 2/3, Intel SDM Vol. 2); the kernel’s own summary in the header comment of entry_SYSCALL_64 corroborates it exactly (entry_64.S, v6.12):

“64-bit SYSCALL saves rip to rcx, clears rflags.RF, then saves rflags to r11, then loads new ss, cs, and rip from previously programmed MSRs. rflags gets masked by a value from another MSR (so CLD and CLAC are not needed). SYSCALL does not save anything on the stack and does not change rsp.”

Decomposed step by step:

  1. Return address saved. The address of the instruction following syscall (the value of RIP at that point) is copied into RCX. This is why RCX is unavailable as a syscall argument register and why the kernel’s argument ABI substitutes R10 for the C convention’s RCX — explained in full in The x86-64 Syscall Calling Convention.
  2. Flags saved and masked. RFLAGS is copied into R11, then RFLAGS is masked: every bit set in the MSR_SYSCALL_MASK model-specific register (address 0xC0000084, per msr-index.h, v6.12) is cleared. Linux programs this mask aggressively in idt_syscall_init() to clear, among others, IF (the interrupt-enable flag — so the kernel begins with interrupts disabled), DF (the direction flag — so string instructions count upward, making the explicit CLD unnecessary), AC (alignment-check / SMAP access flag — equivalent to a CLAC), TF (trap flag), and the arithmetic flags (cpu/common.c, v6.12, idt_syscall_init). Clearing these “as much as possible to minimize user space-kernel interference” (the comment’s words) means the kernel does not inherit a hostile or surprising flags state from userspace.
  3. New code and stack segments loaded. CS and SS are loaded from fixed fields of the MSR_STAR register (0xC0000081), which Linux sets to __KERNEL_CS (and __USER32_CS for the sysret half). Loading CS with a ring-0 selector is what drops the Current Privilege Level from 3 to 0 — the actual privilege escalation. Note that SS is loaded but the stack pointer RSP is not touched.
  4. RIP loaded from MSR_LSTAR. The instruction pointer jumps to whatever address sits in MSR_LSTAR (0xC0000082, the “Long mode SYSCALL TARget”). Linux writes entry_SYSCALL_64 here in idt_syscall_init(): wrmsrl(MSR_LSTAR, (unsigned long)entry_SYSCALL_64). This is the trapdoor’s pre-wired destination.

For syscall to work at all, the System Call Extensions bit (EFER_SCE, bit 0 of the Extended Feature Enable Register MSR_EFER at 0xC0000080) must be set; Linux enables it during CPU bring-up.

The Kernel Entry: entry_SYSCALL_64 Step by Step

The CPU has now jumped to entry_SYSCALL_64 in ring 0, but on the user stack and with user per-CPU base — a dangerous state. The first instructions establish a trustworthy environment (entry_64.S, v6.12):

SYM_CODE_START(entry_SYSCALL_64)
	swapgs                                       ; (1) GS base: user -> kernel per-CPU
	movq	%rsp, PER_CPU_VAR(cpu_tss_rw + TSS_sp2)  ; (2) stash user RSP in scratch slot
	SWITCH_TO_KERNEL_CR3 scratch_reg=%rsp        ; (3) switch page tables (PTI/Meltdown)
	movq	PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rsp  ; (4) load kernel stack top
	; --- now on the kernel stack, build struct pt_regs ---
	pushq	$__USER_DS                           ; pt_regs->ss
	pushq	PER_CPU_VAR(cpu_tss_rw + TSS_sp2)    ; pt_regs->sp  (the saved user RSP)
	pushq	%r11                                 ; pt_regs->flags (saved RFLAGS)
	pushq	$__USER_CS                           ; pt_regs->cs
	pushq	%rcx                                 ; pt_regs->ip  (the return address)
	pushq	%rax                                 ; pt_regs->orig_ax (the syscall number)
	PUSH_AND_CLEAR_REGS rax=$-ENOSYS             ; save & zero the rest; default ret = -ENOSYS
	movq	%rsp, %rdi                           ; arg0 = pointer to pt_regs
	movslq	%eax, %rsi                           ; arg1 = sign-extended syscall number
	IBRS_ENTER                                   ; Spectre v2 mitigation
	UNTRAIN_RET                                  ; retbleed mitigation
	CLEAR_BRANCH_HISTORY
	call	do_syscall_64                        ; C dispatcher

Line-by-line, the load-bearing parts:

  • swapgs (line 1). The GS segment base is used by the kernel to reach per-CPU data (the current task, the kernel stack pointer, etc.). In userspace, GS may point at thread-local storage. swapgs atomically exchanges the current GS base with the value cached in MSR_KERNEL_GS_BASE (0xC0000102) — so after swapgs, GS points at the kernel’s per-CPU area and the user’s GS base is parked in the shadow MSR until exit. This single instruction exists precisely because there is no other cheap way to recover the kernel’s per-CPU pointer after a userspace entry. On the way out, entry_SYSRETQ does swapgs again to restore the user value.
  • Stash user RSP and switch stacks (lines 2 and 4). Because syscall did not change RSP, the kernel is momentarily running on the user stack. It saves the user RSP into a per-CPU scratch slot (cpu_tss_rw + TSS_sp2), then loads the top of the kernel stack for the current task. Only now is it safe to push anything.
  • SWITCH_TO_KERNEL_CR3 (line 3). With Kernel Page-Table Isolation (PTI), the Meltdown mitigation, userspace runs with a restricted page-table tree that does not map most of the kernel. CR3 (the page-table base register) must be switched to the full kernel tree on entry. This is one of the largest fixed costs of a syscall on PTI-enabled machines — see Why System Calls Are Expensive.
  • Building pt_regs (the pushq block). The kernel manufactures a struct pt_regs on the kernel stack by pushing the hardware-frame fields (ss, sp, flags, cs, ip) plus orig_ax (the original syscall number) and then all general registers via PUSH_AND_CLEAR_REGS. This frame is what every syscall handler, signal-delivery path, ptrace, and core-dump reads. Its full layout is the subject of The pt_regs Register Frame; here the point is simply that syscall builds it in software, unlike an interrupt where the CPU pushes part of it.
  • The mitigations (IBRS_ENTER, UNTRAIN_RET, CLEAR_BRANCH_HISTORY). Spectre-class CPU vulnerabilities mean the kernel must scrub speculative state on entry. These are real, measurable costs added since 2018; they are part of why a modern syscall is slower than the bare instruction would suggest.

do_syscall_64(regs, nr) then runs the generic enter-from-user-mode bookkeeping, dispatches through the syscall table, and runs exit work; it returns a boolean indicating whether the fast sysret exit is safe (true) or whether the slow iret path is required (false) (common.c, v6.12, do_syscall_64). See The Generic Syscall Entry and Exit Layer and Returning to Userspace and exit_to_user_mode.

Returning: sysretq vs iretq

The mirror instruction sysretq is fast: it restores RIP from RCX, RFLAGS from R11, reloads the user CS/SS from MSR_STAR, and drops back to ring 3. But it is fragile — both AMD and Intel CPUs mishandle non-canonical return addresses with sysret, which is a security hazard. So the kernel only uses sysretq when the saved state is a “completely clean 64-bit userspace context”; otherwise it falls back to the slower but fully general iretq (the swapgs_restore_regs_and_return_to_usermode path). The entry comment notes: “When user can change pt_regs→foo always force IRET … because it deals with uncanonical addresses better. SYSRET has trouble with them due to bugs in both AMD and Intel CPUs.” On the way out, the kernel also switches CR3 back to the user tree and runs CLEAR_CPU_BUFFERS (an MDS/microarchitectural-data-sampling mitigation) before swapgs; sysretq.

Configuration: How the MSRs Are Programmed at Boot

All of the above only works because the kernel sets up the MSRs once per CPU. From idt_syscall_init() (cpu/common.c, v6.12):

static inline void idt_syscall_init(void)
{
	wrmsrl(MSR_LSTAR, (unsigned long)entry_SYSCALL_64);   /* 64-bit entry point  */
	/* ... ia32/compat CSTAR + SYSENTER MSRs set here if 32-bit ABI enabled ... */
	wrmsrl(MSR_SYSCALL_MASK,                              /* RFLAGS bits to CLEAR */
	       X86_EFLAGS_CF|X86_EFLAGS_PF|X86_EFLAGS_AF|
	       X86_EFLAGS_ZF|X86_EFLAGS_SF|X86_EFLAGS_TF|
	       X86_EFLAGS_IF|X86_EFLAGS_DF|X86_EFLAGS_OF|
	       X86_EFLAGS_IOPL|X86_EFLAGS_NT|X86_EFLAGS_RF|
	       X86_EFLAGS_AC|X86_EFLAGS_ID);
}
 
void syscall_init(void)
{
	wrmsr(MSR_STAR, 0, (__USER32_CS << 16) | __KERNEL_CS); /* CS/SS selectors */
	if (!cpu_feature_enabled(X86_FEATURE_FRED))
		idt_syscall_init();
}
  • MSR_STAR (0xC0000081) packs the kernel CS (which implies SS = CS+8) for entry and the user 32-bit CS for exit. This is what the syscall/sysret instructions read to reload segment registers.
  • MSR_LSTAR (0xC0000082) holds the entry RIP — the single 64-bit entry point.
  • MSR_SYSCALL_MASK (0xC0000084) is the flags-clear mask applied by the hardware on every syscall. Note that IF is in the list: the kernel begins each syscall with interrupts disabled, re-enabling them later in C once the stack and state are sound.

The FRED branch is a point-in-time fact worth dating: on CPUs with FRED (Flexible Return and Event Delivery, a newer Intel event-delivery architecture), the kernel sets only MSR_STAR and skips idt_syscall_init() entirely, because under FRED syscall/sysenter are routed through a ring-3 FRED entrypoint and sysret/sysexit raise #UD — ERETU becomes the only legal return-to-ring-3 instruction (per the comment in syscall_init, cpu/common.c, v6.12). On the LSTAR-based path described in this note (the overwhelmingly common case as of v6.12 LTS), FRED is not in play.

Uncertain

Verify: the exact micro-architectural ordering of the syscall instruction’s effects (e.g. whether flags are masked before or after CS/SS load) and any corner cases. Reason: the AMD64 Architecture Programmer’s Manual (Vol. 3) and the Intel SDM (Vol. 2) are the canonical authority and the AMD PDF fetch was blocked during research; the description here rests on the entry_SYSCALL_64 header comment in entry_64.S, v6.12, which is reliable kernel-source corroboration but not the ISA spec itself. To resolve: read AMD APM Vol. 3 “SYSCALL/SYSRET” and Intel SDM Vol. 2 “SYSCALL—Fast System Call”. uncertain

How This Differs From an Interrupt or Exception Trap

It is easy to lump “syscall” together with “interrupt” as “things that enter the kernel,” but on x86-64 the two paths are mechanically quite different:

  • Vector vs fixed destination. An interrupt or exception is dispatched through the Interrupt Descriptor Table (IDT): the CPU uses the interrupt vector number (0–255) to index the IDT and find the handler’s gate descriptor. syscall uses no table — it jumps unconditionally to the address in MSR_LSTAR. (The legacy int 0x80 is an IDT-based path, vector 0x80; see below.)
  • Who builds the stack frame. On an interrupt/exception, the CPU automatically pushes SS, RSP, RFLAGS, CS, RIP (and an error code for some exceptions) onto the kernel stack, and it can switch to a known-good stack via the Interrupt Stack Table (IST) or TSS.RSP0 mechanism. syscall pushes nothing and does not switch the stack — the kernel does both manually, as shown above. This is the single most important contrast: syscall is faster precisely because it offloads less to the CPU, at the cost of more careful kernel software.
  • Flags handling. An interrupt clears IF automatically (for interrupt gates) by virtue of the gate type; syscall clears IF only because Linux put it in MSR_SYSCALL_MASK. Either way the kernel begins with interrupts off, but by different mechanisms.
  • Synchrony. A syscall is synchronous and voluntary — userspace chose to execute the instruction. A hardware interrupt is asynchronous — it can preempt any instruction. An exception (page fault, divide error) is synchronous but involuntary. The kernel’s entry code for asynchronous interrupts must therefore be re-entrant and able to nest in ways the syscall path need not be.

Alternatives: int 0x80, sysenter, and Why syscall Won

Before syscall, 32-bit x86 Linux entered the kernel with int 0x80 — a software interrupt through IDT vector 0x80. It works on every x86 CPU and is still supported for ancient 32-bit binaries via the compat path, but it is slow: a software interrupt does the full IDT lookup, privilege-check, and automatic stack-frame push, costing far more cycles than syscall. Intel introduced sysenter/sysexit as a fast-path for 32-bit code (configured via the MSR_IA32_SYSENTER_* MSRs, also set in idt_syscall_init() when the 32-bit ABI is enabled), but its semantics were awkward (it does not save the return address or stack pointer, forcing the kernel/vDSO to reconstruct them). AMD’s syscall/sysret, designed for long mode, became the 64-bit mechanism: it is the only path 64-bit Linux uses, and it is what the LSTAR MSR points at. The man page syscall(2) lists syscall as the x86-64 transition instruction and int $0x80 as the i386 one, confirming the split.

The practical upshot for a 64-bit program: glibc’s wrappers and the vDSO emit the syscall instruction; you will only see int 0x80 if you run a 32-bit binary or hand-write 32-bit assembly. The vDSO itself even keeps a syscall instruction inline as a fallback for clock_gettime (per the entry comment), tying this note to The vDSO Virtual Dynamic Shared Object.

Failure Modes and Pitfalls

  • Assuming syscall preserves RCX/R11. It does not — the instruction overwrites both. Code that hand-issues syscall and expects RCX/R11 to survive is buggy. This is the deep reason behind the R10-not-RCX argument rule in The x86-64 Syscall Calling Convention.
  • sysret and non-canonical addresses. A return address that is not canonical (the high bits not a sign-extension of bit 47) can cause a general-protection fault in ring 0 on the sysret, historically exploitable. The kernel’s “force IRET when userspace can change the frame” rule exists exactly to dodge this CPU erratum.
  • Forgetting that interrupts are off on entry. Because MSR_SYSCALL_MASK clears IF, the very start of entry_SYSCALL_64 runs with interrupts disabled; any long-running work before the kernel re-enables them stalls interrupt servicing. The entry path is deliberately short before it gets to C.
  • PTI cost surprise. On Meltdown-affected CPUs with PTI on, the CR3 switch on every entry/exit measurably raises syscall latency. Benchmarks that pre-date 2018 understate real syscall cost — see Why System Calls Are Expensive.

Production Notes

The syscall entry path is one of the hottest pieces of code in the entire kernel — it runs on every system call on the machine — so it is heavily optimized and heavily scrutinized. The cluster of mitigations layered into it since 2018 (PTI for Meltdown, IBRS_ENTER/UNTRAIN_RET for Spectre v2 and Retbleed, CLEAR_CPU_BUFFERS for MDS) is a vivid record of the speculative-execution vulnerability era: each line of mitigation in entry_SYSCALL_64 corresponds to a real CVE and a real performance regression that operators measured in production. The introduction of PTI in particular (LWN coverage and the kernel PTI docs) caused widely reported syscall-heavy workload slowdowns. The longer-term escape is architectural: FRED (above) streamlines event delivery, and userspace batching via io_uring as a Syscall Batching Mechanism sidesteps the per-call cost entirely by crossing the boundary far less often.

See Also

Graph View

Backlinks