vault

Architecture Setup and Mode Switching

15 min read

Architecture Setup and Mode Switching

Between the firmware handing off and the kernel’s architecture-independent C code (start_kernel) lies a thin, almost entirely assembly layer whose job is to drag the CPU out of whatever state it booted in and into the precise environment the kernel expects: 64-bit long mode, paging on with the kernel’s own page tables, a kernel-controlled Global Descriptor Table (GDT) and Interrupt Descriptor Table (IDT), a real stack, and the per-CPU register (%gs) pointed at kernel data. On x86-64 the historical full sequence is real mode → protected mode → long mode, walking the CPU up through three decades of architectural compatibility layers; on a modern UEFI machine using the kernel’s EFI stub the firmware already runs the CPU in long mode, so the kernel “only” has to install its page tables and descriptors rather than enable the mode from scratch. The canonical code is arch/x86/kernel/head_64.S (the decompressed kernel’s entry, reached from extract_kernel’s final jump) and, for the legacy mode-switch, arch/x86/boot/compressed/head_64.S (per Linux v6.12). The single most important idea: every privileged “world” the kernel relies on — paging, descriptors, the stack, per-CPU storage — must be built by hand in assembly here, because none of it exists yet and there is no C runtime to lean on.

This note is about the CPU bringup: the mode transitions, the early page tables, the descriptor tables, and reaching the first C function. The decompression that runs just before this is Kernel Decompression; the C subsystem initialization that runs just after is start_kernel and Early Initialization.

Why So Many Modes Exist

An x86 CPU powers on in real mode, the 16-bit 1978 8086 environment: 20-bit segmented addressing (segment << 4 + offset), no memory protection, no paging, ~1 MiB reachable. This is a backward-compatibility relic — the firmware and the very first kernel bytes must cope with it. Protected mode (the 32-bit 80386 world) adds the GDT-based segmentation with privilege rings, 32-bit flat addressing, and optional paging. Long mode (the 64-bit AMD64 environment) adds 64-bit registers and a flat 64-bit address space, but with a hard requirement: long mode cannot be entered without paging enabled, and paging on x86-64 requires PAE (Physical Address Extension). So the CPU literally cannot jump straight from real mode to 64-bit code; it must pass through 32-bit protected mode, turn on PAE and a page table, set the long-mode-enable bit, and only then far-jump into a 64-bit code segment. Understanding this staircase is the key to reading the boot assembly.

Mental Model

flowchart TB
  RM["Real mode (16-bit)<br/>firmware / legacy entry<br/>1 MiB, segmented"] --> PM["Protected mode (32-bit)<br/>load GDT, flat segments,<br/>enable PAE (CR4.PAE)"]
  PM --> PT["Build early identity<br/>page tables<br/>(L4/L3/L2 in pgtable buf)"]
  PT --> LME["Set EFER.LME<br/>(long-mode enable in MSR)"]
  LME --> PG["Set CR0.PG | CR0.PE<br/>(paging + protection on)"]
  PG --> FJ["Far jump (lret) into<br/>64-bit __KERNEL_CS<br/>CS.L = 1 -> long mode active"]
  FJ --> S64["startup_64 (kernel head_64.S)<br/>__startup_64() fixes page tables,<br/>detects LA57 (5-level)"]
  S64 --> COMMON["common_startup_64<br/>load CR4/CR3, GSBASE,<br/>EFER.NX/SCE, CR0, IDT"]
  COMMON --> C["callq *initial_code<br/>= x86_64_start_kernel()<br/>-> start_kernel()"]
  EFI["EFI stub boot:<br/>firmware already in long mode"] -.skips RM/PM.-> S64

The mode staircase and the two entry routes. What it shows: the legacy path climbs real → protected → long mode by enabling PAE, building page tables, setting EFER.LME, then CR0.PG, then far-jumping into a 64-bit segment; the modern EFI-stub path enters already in long mode and rejoins at startup_64. The insight to take: the order is forced by hardware — paging and PAE must be on before the long-mode far jump, and the kernel still rebuilds its own page tables and descriptors at startup_64 regardless of which route delivered it there.

The Legacy Mode Switch (compressed/head_64.S)

When a legacy bootloader enters at the 32-bit startup_32 in arch/x86/boot/compressed/head_64.S, the CPU is already in protected mode (the bootloader’s real-mode setup code switched it), but with the loader’s GDT and no paging. The stub performs the climb to long mode in this order:

1. Load a kernel-controlled GDT and flat segments. It computes its own load address with the classic call 1f; popl %ebp trick (the only way on x86 to learn where you are actually running), loads a GDT with 32- and 64-bit descriptors, and reloads every segment register with flat descriptors:

	leal	rva(gdt)(%ebp), %eax
	movl	%eax, 2(%eax)
	lgdt	(%eax)
	movl	$__BOOT_DS, %eax
	movl	%eax, %ds
	movl	%eax, %es
	...

2. Verify long mode is available (call verify_cpu; jump to .Lno_longmode and halt if not — a 64-bit kernel on a 32-bit-only CPU dies here with a clear message).

3. Enable PAE. Long mode mandates it:

	movl	%cr4, %eax
	orl	$X86_CR4_PAE, %eax
	movl	%eax, %cr4

4. Build early identity-mapped page tables by hand. The code zeros a pgtable buffer and fills three levels — one Level-4 (PML4) entry pointing at a Level-3 (PDPT) table, four Level-3 entries (covering 4 GiB) pointing at Level-2 (PD) tables, and 2048 Level-2 entries each mapping a 2 MiB page (movl $0x00000183, %eax — the 0x183 flags are Present | Writable | Page-Size(2 MiB) | Global). “Identity-mapped” means virtual address X maps to physical address X, so that the very instruction stream keeps working at the instant paging turns on. It then points CR3 at this table.

5. Set EFER.LME. The Extended Feature Enable Register is a Model-Specific Register (MSR); the Long Mode Enable bit is set via the read-modify-write MSR dance:

	movl	$MSR_EFER, %ecx
	rdmsr
	btsl	$_EFER_LME, %eax
	wrmsr

At this point EFER.LME = 1 but the CPU is still in 32-bit compatibility mode (CS.L = 0) — long mode is enabled but not active.

6. Enable paging and protection (CR0), then far-jump. Setting CR0.PG with EFER.LME already on transitions the CPU to long mode; the actual switch to 64-bit code happens with a far return (lret) into the __KERNEL_CS descriptor whose L bit is set:

	leal	rva(startup_64)(%ebp), %eax
	pushl	$__KERNEL_CS
	pushl	%eax
	movl	$CR0_STATE, %eax	/* PG | PE | ... */
	movl	%eax, %cr0
	lret			/* far jump into 64-bit startup_64 */

The comment in the source spells out the subtlety exactly: “When the jump is performed we will be in long mode but in 32bit compatibility mode with EFER.LME = 1, CS.L = 0, CS.D = 1 (and in turn EFER.LMA = 1). To jump into 64bit mode we use the new gdt/idt that has __KERNEL_CS with CS.L = 1.” After lret, the CPU executes 64-bit instructions at startup_64. (On a 64-bit bootloader or EFI-stub boot, control arrives at startup_64 directly at offset 0x200, skipping all of the above; the comment notes “We come here either from startup_32 or directly from a 64bit bootloader.”)

The decompressor’s startup_64 does one more architecture-level thing relevant here: it may need to switch the number of paging levels (4 vs 5). Because CR4.LA57 cannot be toggled while in long mode (it would #GP), the code uses a small 32-bit trampoline_32bit_src in low memory to drop out of long mode, flip CR4.LA57, and re-enter — handled by the C helper configure_5level_paging(). The source comment: “Setting or clearing CR4.LA57 in long mode would trigger GP. So we need to switch off long mode and paging first… we need a trampoline in lower memory.”

Setting Up the Kernel’s World (kernel/head_64.S)

After extract_kernel returns the entry point and the decompressor jumps to it, control lands at startup_64 in the decompressed kernel’s arch/x86/kernel/head_64.S. The entry comment states the contract precisely: “At this point the CPU runs in 64bit mode CS.L = 1 CS.D = 0, and someone has loaded an identity mapped page table for us… %RSI holds the physical address of the boot_params structure.” So the CPU is already in long mode; the kernel now installs its own permanent structures.

1. Preserve boot_params and set a stack. mov %rsi, %r15 stashes the boot_params pointer in a callee-saved register so it survives the upcoming C calls, then:

	leaq	__top_init_kernel_stack(%rip), %rsp

This __top_init_kernel_stack is the top of a statically allocated boot stack (a v6.12-era symbol; earlier kernels used initial_stack). Without a stack, no call instruction works, so this is necessarily one of the first steps.

2. Set the per-CPU base (%gs) early. The kernel stores per-CPU data and the stack-protector canary relative to %gs, and C code emitted with stack protection dereferences %gs:40 on entry — so %gs must be valid before the first C call:

	movl	$MSR_GS_BASE, %ecx
	leaq	INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx
	movl	%edx, %eax
	shrq	$32,  %rdx
	wrmsr

MSR_GS_BASE is the MSR holding the 64-bit base address that %gs:offset accesses resolve against; it is written as a (low, high) pair in (%eax, %edx). The source comment: “The base of %gs always points to fixed_percpu_data.”

3. Build the kernel GDT/IDT and reload CS. call startup_64_setup_gdt_idt installs the kernel’s descriptor tables, then a pushq $__KERNEL_CS; ... lretq reloads the code segment so that interrupt-return (IRET) works against a CS that genuinely exists in the new GDT.

4. Fix up the page tables for the actual load address (C: __startup_64). Because KASLR may have placed the kernel anywhere, the statically built page-table entries in the image hold compile-time addresses that must be biased by the real load delta. head_64.S calls into C:

	leaq	_text(%rip), %rdi
	movq	%r15, %rsi
	call	__startup_64
	leaq	early_top_pgt(%rip), %rcx
	addq	%rcx, %rax
	movq	%rax, %cr3

__startup_64() in head64.c computes load_delta = physaddr - (_text - __START_KERNEL_map) and adds it into every PGD/P4D/PUD/PMD entry, sets phys_base, and crucially calls check_la57_support() to decide 4- vs 5-level paging at runtime — writing __pgtable_l5_enabled, pgdir_shift, and ptrs_per_p4d. The result is loaded into CR3, switching from the decompressor’s temporary identity map to the kernel’s early_top_pgt, which has both the identity mapping (so the current physical-address code keeps running) and the kernel’s high-half virtual mapping.

5. Common bringup — CR4, CR3, EFER, CR0, IDT (common_startup_64). The boot CPU and every secondary (AP) CPU funnel through common_startup_64 (also entered as secondary_startup_64 from the SMP trampoline). Here the architectural control registers get their final values. CR4 is rebuilt preserving only the bits that must persist, with LA57 included when 5-level paging is in use:

	movl	$(X86_CR4_PAE | X86_CR4_LA57), %edx
	...
	movq	%rcx, %cr4

%gs is set again (for APs), the IDT is loaded (call early_setup_idt), and EFER gets two more bits — SCE (System Call Enable, so the SYSCALL instruction works) and, if the CPU advertises it via CPUID, NX (No-Execute page protection):

	movl	$MSR_EFER, %ecx
	rdmsr
	btsl	$_EFER_SCE, %eax	/* Enable System Call */
	btl	$20,%edi		/* No Execute supported? */
	jnc     1f
	btsl	$_EFER_NX, %eax
	...
	wrmsr
	movl	$CR0_STATE, %eax
	movq	%rax, %cr0

6. Call the first C entry. Finally the frame pointer is zeroed and the kernel jumps into C through an indirect call:

	movq	%r15, %rdi		/* boot_params as first arg */
	xorl	%ebp, %ebp
	callq	*initial_code(%rip)
	ud2

where SYM_DATA(initial_code, .quad x86_64_start_kernel). The ud2 (undefined instruction) after the call is a tripwire — x86_64_start_kernel never returns, and if it somehow did, ud2 faults loudly rather than running into garbage.

Reaching start_kernel

x86_64_start_kernel() in head64.c is the first real C function. Its signature is asmlinkage __visible void __init __noreturn x86_64_start_kernel(char *real_mode_data). It tears down the now-unneeded identity-map trampoline (reset_early_page_tables()), zeros .bss/.brk (clear_bss()), copies the boot parameters into the kernel’s own boot_params (copy_bootdata(__va(real_mode_data))), initializes memory-encryption (sme_early_init()) and KASAN (kasan_early_init()), installs the early exception handlers (idt_setup_early_handler()), and then calls x86_64_start_reservations(), which in turn calls the architecture-independent start_kernel(). From that point the boot becomes portable C — covered in start_kernel and Early Initialization.

KASLR’s Two Randomizations, Revisited Here

KASLR’s decision is made during decompression (choose_random_location), but its consequences are realized in this layer. The physical randomization means the kernel can be loaded at an unpredictable RAM address; the page-table fixups in __startup_64 (the load_delta arithmetic) exist precisely to make a randomly-placed image run correctly. The virtual randomization shifts the kernel’s high-half mapping within KERNEL_IMAGE_SIZE; the relocation pass applied by the decompressor’s handle_relocations rewrote every absolute reference to match. So “KASLR” is not a single act — it is a physical choice, a virtual choice, page-table biasing here, and relocation patching there, all of which must agree. Disabling it (nokaslr) simply pins the load address to LOAD_PHYSICAL_ADDR and skips the relocation/bias work.

5-Level Paging — A Dated, Moving Fact

Standard x86-64 long mode uses 4-level paging: a 48-bit virtual address split across PML4 → PDPT → PD → PT, giving 256 TiB of virtual and 64 TiB of physical space. Intel’s 5-level paging (LA57) adds a fifth table level (PML5/P4D), extending virtual addressing to 57 bits (128 PiB) and physical to 52 bits (4 PiB) (per the 5-level paging LWN coverage). Because not all CPUs implement LA57, the kernel detects it at runtime (check_la57_support) and the early code can switch levels via the low-memory trampoline described above.

The build-time story changed within the 6.x series and must be dated carefully:

  • At v6.12 (our pin): 5-level support is a separate Kconfig option, CONFIG_X86_5LEVEL, declared bool "Enable 5-level page tables support" with default y (verified directly in arch/x86/Kconfig at v6.12). A kernel built without it cannot use LA57 even on a capable CPU.
  • From v6.16: CONFIG_X86_5LEVEL was removed and 5-level support became unconditional for x86-64. Verified by primary source: the config X86_5LEVEL block is present through v6.15 and absent from v6.16, v6.17, and v6.18 (grep -c "config X86_5LEVEL" returns 1 for v6.13–v6.15 and 0 for v6.16+ in arch/x86/Kconfig). The change matches the proposal Kirill Shutemov posted in June 2024 (then an unmerged patch series per LWN), which landed later in the 6.x window.

So: a v6.12 LTS kernel still carries the CONFIG_X86_5LEVEL toggle (defaulting on); a v6.16-or-newer kernel always builds 5-level support and the #ifdef CONFIG_X86_5LEVEL scaffolding is gone. Runtime behavior is unchanged either way — LA57 is still used only on CPUs that have it. This corrects the common (and the task-brief’s) belief that 5-level became unconditional earlier in the series; the exact landing version is v6.16, not v6.10.

Failure Modes

  • .Lno_longmode halt. A 64-bit kernel started on a CPU (or VM) lacking long-mode support: verify_cpu fails and the stub prints a message and stops. The fix is a 32-bit kernel or a capable CPU/VM CPU model.
  • Triple fault right at the long-mode switch. A bug in the hand-built page tables (wrong flags, unmapped instruction page) makes the very first paged instruction fault, with no handler installed yet → triple fault → CPU reset → boot loop. These are brutal to debug because no console output survives; bisecting the assembly or using a hardware/VM debugger is usually required.
  • Stack-protector fault before the first C call. If %gs/MSR_GS_BASE is wrong, the canary read at %gs:40 on C-function entry faults. This is why GSBASE is set so early; reordering it later breaks the build’s stack-protected entry.
  • 5-level mismatch on kexec. Booting a 4-level kernel via kexec from a 5-level kernel (or vice versa) requires the level-switching trampoline; the source explicitly handles “starting 4-level paging kernel via kexec() when original kernel worked in 5-level paging mode.” A kernel built without 5-level support (pre-v6.16, CONFIG_X86_5LEVEL=n) cannot kexec into a 5-level world.
  • no5lvl on the command line forces 4-level even on LA57 hardware — occasionally needed to work around early firmware/hypervisor bugs in 5-level mode, and a useful diagnostic when a machine boots 4-level-only kernels fine but hangs with a 5-level one.

Alternatives and Boundaries

The legacy real → protected → long climb is increasingly vestigial: on a UEFI machine the EFI stub is entered by firmware already in long mode, so the kernel skips the mode switch and arrives near startup_64 with paging and a GDT already up — it still rebuilds its own page tables and descriptors, but it never touches EFER.LME or CR0.PG to enable the mode. Other architectures replace this layer wholesale: arm64’s arch/arm64/kernel/head.S brings the CPU from EL2/EL1 with its own page-table and exception-vector setup, sharing the concept (build paging and a stack by hand, then call C) but none of the x86 mode mechanics. The boundary with siblings: the contract delivering the CPU here is The Linux Boot Protocol; the decompression just before is Kernel Decompression; the portable C after is start_kernel and Early Initialization.

Production Notes

For day-to-day operators this layer is invisible until it breaks, and when it breaks it breaks hard (triple faults, boot loops, no logs). The practical levers are command-line flags — nokaslr, no5lvl, and earlyprintk=serial to get any output past the mode switch. The most consequential real-world change in the recent 6.x window is the v6.16 removal of CONFIG_X86_5LEVEL: distributions building 6.16+ kernels no longer need to choose, and the kernel binary universally supports 128 PiB virtual address spaces on capable hardware while still running correctly on 4-level CPUs. Confidential-computing platforms (AMD SEV/SEV-SNP, Intel TDX) bolt extra steps onto this layer — sme_enable/sev_enable in the decompressor head and sme_early_init in x86_64_start_kernel — to bring up memory encryption before the kernel writes secrets to RAM; on bare metal these are no-ops. SMP secondary-CPU bringup re-uses this exact code (secondary_startup_64/common_startup_64), which is why a single bug here can manifest as “the box boots but only on one core.”

Uncertain

Verify: the exact 4-level vs 5-level address-space sizes quoted (256 TiB / 64 TiB virtual/physical for 4-level; 128 PiB / 4 PiB for 5-level). Reason: these are summarized from LWN secondary coverage and the Kconfig help text, not walked field-by-field from the Intel SDM or Documentation/arch/x86/x86_64/mm.rst. To resolve: cross-check Documentation/arch/x86/x86_64/5level-paging.rst and mm.rst at v6.12. uncertain

See Also

  • Kernel Decompression — the self-extracting head whose final jmp *%rax delivers control to this layer’s startup_64
  • The Linux Boot Protocol — the entry-point contract (startup_32 at 0, startup_64 at 0x200) and the state the CPU is in on arrival
  • start_kernel and Early Initialization — the architecture-independent C the kernel reaches via x86_64_start_kernel → start_kernel
  • The EFI Stub — the modern path that enters in long mode and skips most of the mode staircase
  • kexec — re-runs a variant of this bringup to boot a new kernel from a running one, including 4-/5-level transitions
  • The Kernel Command Line — nokaslr, no5lvl, earlyprintk are parsed and acted on around this stage
  • Linux Boot and Init MOC — the parent map (§3 Kernel Self-Setup)

Graph View