vault

Per-Architecture Syscall Entry Assembly

13 min read

Per-Architecture Syscall Entry Assembly

When a userspace program executes the syscall instruction on x86-64, the CPU does not land in C code — it lands in a hand-written assembly stub named entry_SYSCALL_64, defined in arch/x86/entry/entry_64.S (entry_64.S, v6.12). This stub is the architecture-specific glue between the bare hardware transition (which leaves registers and the stack in a fragile, half-switched state) and the architecture-neutral C dispatcher do_syscall_64. Its job is narrow but absolutely critical: flip the per-CPU GS base, switch from the user stack to the kernel stack, build a struct pt_regs snapshot of every user register on that stack, apply the Spectre/Meltdown mitigations, call into C, and then carefully reverse all of it on the way out. Get one step wrong and you either crash the kernel or hand userspace a privilege-escalation primitive. This note walks entry_SYSCALL_64 line by line and contrasts it with arm64’s el0t_64_sync path, which solves the same problem with a different instruction-set toolkit.

Mental Model

The syscall instruction is deliberately minimal hardware. Per the Intel/AMD architecture, syscall saves the return address into rcx, saves the flags into r11, masks flags with a mask MSR, and loads a new cs/ss/rip from MSRs — and that is all (SYSCALL, felixcloutier). Critically, it does not change rsp: immediately after syscall, the kernel is running at CPL=0 (ring 0, privileged) but still on the user’s stack pointer, with the user’s GS base, and with the user’s page tables mapped. The entry stub’s entire reason for existing is to repair this half-transitioned state into a well-defined kernel execution context before any C code runs.

The same conceptual sequence — land, switch GS, switch stack, switch page tables, save registers, call C — recurs on every architecture; only the instructions differ. The kernel comment in entry_64.S notes the x86-64 register-to-argument mapping “fits well with the registers that are available when SYSCALL is used,” which is why Linux picked rdi, rsi, rdx, r10, r8, r9 for the six arguments (see The x86-64 Syscall Calling Convention).

flowchart TB
  U["Userspace<br/>syscall instruction<br/>(rax=nr, args in regs)"]
  U -->|"CPU: save rip→rcx, flags→r11,<br/>load kernel cs/ss/rip,<br/>rsp UNCHANGED"| L["entry_SYSCALL_64<br/>(CPL=0, user rsp/gs/CR3)"]
  L --> SG["swapgs<br/>(user GS → kernel per-CPU GS)"]
  SG --> ST["save user rsp to TSS.sp2,<br/>SWITCH_TO_KERNEL_CR3,<br/>load kernel stack top"]
  ST --> PR["build pt_regs frame:<br/>push ss/sp/flags/cs/ip/orig_ax,<br/>PUSH_AND_CLEAR_REGS"]
  PR --> MIT["IBRS_ENTER, UNTRAIN_RET,<br/>CLEAR_BRANCH_HISTORY"]
  MIT --> C["call do_syscall_64(regs, nr)"]
  C --> RET{"clean 64-bit<br/>context?"}
  RET -->|"yes"| SR["sysret path:<br/>POP_REGS, switch to trampoline,<br/>SWITCH_TO_USER_CR3, swapgs, sysretq"]
  RET -->|"no (TF/RF set,<br/>non-canonical rip, Xen…)"| IR["swapgs_restore_regs_and_<br/>return_to_usermode (IRET path)"]
  SR --> U
  IR --> U

The x86-64 syscall entry/exit assembly path at v6.12. What it shows: the hardware drops the kernel into entry_SYSCALL_64 on the user’s own stack with user GS and user page tables still active; the stub must swapgs, switch CR3, switch stacks, and snapshot registers into pt_regs before any C runs. The insight: the fast sysretq exit is only taken when the register state is provably safe to restore via sysret; any anomaly (trap flag set, non-canonical return address, Xen PV) falls back to the slower but fully general iret path.

Mechanical Walk-through — entry_SYSCALL_64 on x86-64

The stub begins with swapgs (entry_64.S, v6.12). swapgs is a privileged instruction that atomically exchanges the current GS base register with the value in the IA32_KERNEL_GS_BASE MSR (SWAPGS, felixcloutier). In userspace, the GS base points at thread-local storage; the kernel uses GS to reach its per-CPU data area. After swapgs, %gs:offset addressing reaches per-CPU variables — which the very next instructions need.

SYM_CODE_START(entry_SYSCALL_64)
	UNWIND_HINT_ENTRY
	ENDBR
 
	swapgs
	/* tss.sp2 is scratch space. */
	movq	%rsp, PER_CPU_VAR(cpu_tss_rw + TSS_sp2)
	SWITCH_TO_KERNEL_CR3 scratch_reg=%rsp
	movq	PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rsp

Line by line: ENDBR is the Intel CET (Control-flow Enforcement Technology) indirect-branch landing pad — it marks this as a legal target of an indirect jump so a hijacked branch cannot land mid-stub. After swapgs, the stub stashes the user stack pointer into tss.sp2, a scratch slot in the per-CPU Task State Segment (TSS), because the next step clobbers %rsp and the user rsp must be preserved for the eventual return. SWITCH_TO_KERNEL_CR3 swaps the page-table root register CR3 to the kernel’s page tables — this is the Kernel Page-Table Isolation (KPTI) switch, the Meltdown mitigation that keeps kernel page tables unmapped while userspace runs (KPTI docs). Until this instruction executes, only a tiny “trampoline”/entry mapping of the kernel is present; afterward the full kernel address space is reachable. Finally %rsp is loaded with pcpu_hot + X86_top_of_stack, the top of this CPU’s kernel stack for the current task. (pcpu_hot is a per-CPU “hot data” aggregation struct whose top_of_stack field is reached via the X86_top_of_stack offset emitted by asm-offsets.c, verified at v6.12 in arch/x86/include/asm/current.h and arch/x86/kernel/asm-offsets.c.) From this instruction onward the kernel is on its own stack.

Building the pt_regs frame

With the kernel stack live, the stub manufactures a struct pt_regs (see The pt_regs Register Frame) by pushing values in a precise order so the in-memory layout matches the C struct field order:

	/* Construct struct pt_regs on stack */
	pushq	$__USER_DS				/* pt_regs->ss */
	pushq	PER_CPU_VAR(cpu_tss_rw + TSS_sp2)	/* pt_regs->sp */
	pushq	%r11					/* pt_regs->flags */
	pushq	$__USER_CS				/* pt_regs->cs */
	pushq	%rcx					/* pt_regs->ip */
	pushq	%rax					/* pt_regs->orig_ax */
 
	PUSH_AND_CLEAR_REGS rax=$-ENOSYS

The first five pushes hand-build the part of the frame the hardware would normally push on an iret-style interrupt — a “synthetic hardware frame.” ss and cs are the constant user segment selectors; sp is the user stack pointer recovered from tss.sp2; flags came from r11 (where syscall saved them); ip came from rcx (where syscall saved the return address). Then orig_ax is set to rax, preserving the original syscall number — this matters because pt_regs->ax will be overwritten by the return value, but orig_ax keeps the number around for ptrace, seccomp, and syscall restart (see Restartable System Calls and ERESTARTSYS).

PUSH_AND_CLEAR_REGS (from arch/x86/entry/calling.h) expands into PUSH_REGS followed by CLEAR_REGS (calling.h, v6.12). PUSH_REGS pushes the general-purpose registers in the exact order rdi, rsi, rdx, rcx, rax, r8, r9, r10, r11, rbx, rbp, r12, r13, r14, r15. Because the stack grows downward and pt_regs is read with the lowest field at the lowest address, this push order produces a struct whose first field (r15) sits at the lowest address and last GP field (rdi) at the highest — precisely the kernel struct pt_regs layout. The rax=$-ENOSYS argument means the slot that will hold the return value is pre-seeded with -ENOSYS so that an unrecognised syscall number naturally returns “not implemented” if nothing overwrites it. CLEAR_REGS then zeroes rsi, rdx, rcx, r8, r9, r10, r11, rbx, rbp, r12–r15 with xorl — a Spectre hardening measure so that leftover user-controlled register values cannot be used as gadgets in speculative execution (see Speculation Barriers and Spectre Hardening at the Syscall Boundary).

Calling into C and the mitigation barriers

	movq	%rsp, %rdi
	movslq	%eax, %rsi	/* sign-extend syscall nr as int */
	IBRS_ENTER
	UNTRAIN_RET
	CLEAR_BRANCH_HISTORY
	call	do_syscall_64

%rsp (the address of the freshly built pt_regs) becomes the first C argument (regs), and the syscall number — sign-extended from the 32-bit eax, because syscall numbers are treated as int — becomes the second (nr). IBRS_ENTER, UNTRAIN_RET, and CLEAR_BRANCH_HISTORY are the indirect-branch and return-stack mitigations (IBRS = Indirect Branch Restricted Speculation; the RET untraining defends against retbleed-class attacks). Only then is do_syscall_64(regs, nr) called. That C function (in arch/x86/entry/common.c) runs the generic enter-from-user work, dispatches through x64_sys_call()/the syscall table, and runs the generic exit work — all of which belongs to The Generic Syscall Entry and Exit Layer, not the assembly.

The exit path: sysret fast path vs. iret slow path

do_syscall_64 returns a bool saying whether the fast sysretq exit is safe. The stub tests it:

	ALTERNATIVE "testb %al, %al; jz swapgs_restore_regs_and_return_to_usermode", \
		"jmp swapgs_restore_regs_and_return_to_usermode", X86_FEATURE_XENPV

If the C code returned true (and we are not a Xen PV guest), control falls through to syscall_return_via_sysret, which runs IBRS_EXIT, POP_REGS (restoring all GP registers except rdi), switches onto the per-CPU trampoline stack (tss.sp0), runs STACKLEAK_ERASE_NOCLOBBER to wipe the kernel stack residue, performs SWITCH_TO_USER_CR3_STACK (the KPTI switch back to user page tables), restores rdi and the user rsp, then swapgs, CLEAR_CPU_BUFFERS (the MDS/microarchitectural-buffer flush), and finally sysretq. The trampoline-stack dance exists because of KPTI: once CR3 is switched back to the user tables, the full kernel stack is no longer mapped, so the last few instructions must run on a stack that is mapped in the user page tables — the trampoline.

The reason the kernel cannot always use sysret is spelled out in do_syscall_64: sysret requires rcx == rip and r11 == flags, requires cs/ss to match the canonical user selectors, and — most importantly — on Intel CPUs a sysret to a non-canonical rip faults in kernel mode, which “essentially lets the user take over the kernel” since userspace controls rsp (common.c, v6.12). When any of these conditions fail — a ptrace tracer changed the registers, the trap flag TF or resume flag RF is set, the return address is non-canonical — the kernel jumps to swapgs_restore_regs_and_return_to_usermode, the IRET path, which POP_REGS, skips orig_ax, and ends in a hardware iret. iret is slower but tolerates these states correctly.

arm64 Contrast — el0t_64_sync and kernel_entry

On ARM64 (AArch64), a system call is issued with the svc #0 (“supervisor call”) instruction, which raises a synchronous exception from EL0 (the userspace exception level) to EL1 (the kernel). The vector table routes this to el0t_64_sync, whose assembly stub runs the kernel_entry 0 macro in arch/arm64/kernel/entry.S and then branches to the C handler el0t_64_sync_handler (entry.S / entry-common.c, v6.12).

The kernel_entry macro is the structural twin of x86-64’s PUSH_REGS, but the register file is regular, so it saves registers in pairs with stp (store-pair):

	.macro	kernel_entry, el, regsize = 64
	...
	stp	x0, x1, [sp, #16 * 0]
	stp	x2, x3, [sp, #16 * 1]
	...
	stp	x28, x29, [sp, #16 * 14]
	.if	\el == 0
	clear_gp_regs
	mrs	x21, sp_el0
	ldr_this_cpu	tsk, __entry_task, x20
	msr	sp_el0, tsk
	...
	mrs	x22, elr_el1
	mrs	x23, spsr_el1
	...
	stp	x22, x23, [sp, #S_PC]

The differences from x86-64 are instructive. There is no swapgs and no sysret/iret fast-path split: AArch64 banks the stack pointer per exception level, and Linux runs EL1 (kernel) code on SP_EL0, freeing it to repurpose the sp_el0 system register to hold the current task_struct pointer (the mrs x21, sp_el0 / msr sp_el0, tsk pair below does exactly this swap). The return address and processor state live in dedicated system registers elr_el1 (Exception Link Register) and spsr_el1 (Saved Program Status Register), read with mrs. The macro reads those and stores them into the S_PC/S_PSTATE slots of pt_regs. The userspace stack pointer is read from sp_el0 and saved; sp_el0 is then repurposed to hold the current task_struct pointer (tsk). clear_gp_regs zeroes the registers as a speculation-hardening measure, mirroring x86-64’s CLEAR_REGS. There is no KPTI CR3 swap here: arm64 uses two separate translation-table base registers, ttbr0_el1 (user) and ttbr1_el1 (kernel), so the kernel mapping is always present in ttbr1 and only the optional CONFIG_ARM64_SW_TTBR0_PAN/KPTI-equivalent path touches ttbr0. The C side mirrors x86-64 closely: el0_svc() calls enter_from_user_mode(), do_el0_svc(), then exit_to_user_mode() — the same generic entry layer (entry-common.c, v6.12).

asmlinkage void noinstr el0t_64_sync_handler(struct pt_regs *regs)
{
	...
	switch (ESR_ELx_EC(esr)) {
	case ESR_ELx_EC_SVC64:
		el0_svc(regs);
		break;
	...

The single arm64 sync vector demultiplexes on the Exception Syndrome Register (ESR) exception class: ESR_ELx_EC_SVC64 means “this was an AArch64 svc,” routing to el0_svc; data aborts, instruction aborts, and floating-point traps share the same vector and are dispatched by the same switch. x86-64, by contrast, has a dedicated syscall entry MSR-programmed target, so its syscall stub does not have to demultiplex against faults.

Failure Modes and Subtleties

  • The swapgs window. Between hardware entry and the swapgs, GS still points at user TLS, yet the kernel is at CPL=0. If an NMI or machine check fires here, the handler must detect “did we already swapgs?” — this is why x86-64 NMI handling is famously intricate. A bug that does swapgs twice, or reads per-CPU data before it, corrupts kernel state silently.
  • KPTI ordering. If SWITCH_TO_KERNEL_CR3 ran after touching full kernel memory, the access would fault (kernel not mapped). Conversely, returning to userspace without SWITCH_TO_USER_CR3 would leave the kernel mapped under user CR3, re-opening Meltdown. The trampoline-stack choreography on exit exists precisely so the final instructions execute on a stack mapped in both page-table sets.
  • sysret non-canonical trap. The historical AMD/Intel divergence on sysret with a non-canonical return address is a real CVE-class issue; the kernel’s defensive regs->ip >= TASK_SIZE_MAX check in do_syscall_64 is what forces the iret path for any address a tracer might have set out of range.
  • No IST for syscalls. The Interrupt Stack Table (IST) is a per-vector mechanism selected through the Interrupt Descriptor Table (IDT) for events that cannot trust the current stack — non-maskable interrupts (NMI), double faults (#DF), machine checks. The syscall fast path does not use an IST stack: it deliberately switches to the per-task kernel stack (top_of_stack) because a syscall arrives in a well-defined context where that stack is known-good. IST matters here only as a contrast — the entry stub hand-switches stacks precisely because syscall, unlike an IST-backed exception, gets no hardware stack switch.
  • STACKLEAK and CLEAR_CPU_BUFFERS. Omitting STACKLEAK_ERASE leaves kernel data on the stack for the next syscall to potentially leak; omitting CLEAR_CPU_BUFFERS (the verw-based MDS flush) re-exposes the Microarchitectural Data Sampling side channel. These are not optional cleanups — they are mitigations gated by ALTERNATIVE patching on affected CPUs.

Alternatives and Evolution — FRED

Intel’s FRED (Flexible Return and Event Delivery) is a newer event-delivery architecture intended to replace the syscall/sysret and IDT machinery with a cleaner stack-based frame that the hardware itself manages, removing much of the swapgs/stack-switch hand-choreography. The v6.12 struct pt_regs already carries FRED-aware union fields (csx/fred_cs, ssx/fred_ss) and an orig_ax-anchored “FRED stack frame starts here” comment (ptrace.h, v6.12).

Uncertain

Verify: the exact kernel version in which FRED-based syscall entry became the default on supporting Intel hardware, and whether v6.12/v6.18 select a distinct FRED entry stub instead of entry_SYSCALL_64. Reason: FRED support landed incrementally across recent releases and the precise “enabled by default” milestone is version- and hardware-specific; not directly verified against a primary changelog here. To resolve: read arch/x86/entry/entry_fred.c and the FRED Kconfig at the v6.12 and v6.18 tags. uncertain

See Also

Graph View

Backlinks