Context Switching in a Microkernel

A context switch is the moment a single processor stops being one task and starts being another. On a bare RV32 hart running a microkernel entirely in machine mode (M-mode), with no memory management unit and no operating system underneath, the whole trick reduces to one sentence: the kernel returns to a different mepc than the one it arrived with. Everything else — the thirty-one general-purpose registers that must be saved because x0 is hardwired to zero, the mstatus privilege-and-interrupt stack, the mscratch register that gives the trap handler its first free pointer, the PMP entries that must be reprogrammed so the incoming task cannot read the outgoing one’s memory — is bookkeeping arranged around that single substitution. This note builds the switch from the register file upward, in real RISC-V assembly that was assembled and disassembled for this note (GNU assembler 2.46, -march=rv32imc_zicsr -mabi=ilp32), and counts what it actually costs: 79 instructions and 178 bytes for the preemptive, interrupt-driven path, against 31 instructions and 64 bytes for a cooperative yield. That 2.5× gap is not an accident of coding style; it is the RISC-V calling convention doing work for you, and it is the most consequential design choice in this stage of the definitely-not-esp32 build.

This is the missing mechanism in Stage 8 of definitely-not-esp32 MOC. That stage’s program is “two tasks that alternate, each printing its own name — which requires a real context switch, not a loop.” Round-Robin Scheduling decides which task runs next and Synchronous IPC decides when a task stops being runnable; this note is the part in between — the code that makes the decision physically true. It assumes the hardware side of RISC-V Trap Handling and the register semantics of Control and Status Registers, and links downward into Bare-Metal Rust for the no_std half.

This is not the Linux note

The vault already contains Context Switch Mechanics switch_to and switch_mm, which traces context_switch() → switch_mm_irqs_off() → __switch_to_asm in Linux 6.12 on x86-64. That note and this one describe genuinely different machines. Linux has an MMU, so half of its switch is an address-space swap (CR3 reload, lazy-TLB borrowing, active_mm refcounting) that has no analogue here — there are no page tables in an M-mode/U-mode RV32 system, only PMP entries. Linux has a scheduler that runs from __schedule() under rq->lock with a task struct, cgroups, and per-task FPU state. And Linux’s switch_to is deliberately minimal — it saves only callee-saved registers, because the entry code has already spilled everything else. Read the two together and the contrast is the lesson: the Linux switch is small because a large amount of machinery around it is doing the saving; the microkernel switch is large because there is nothing else. Where the concepts genuinely coincide — the “one function returns as a different task” trick, and the cost accounting — this note cross-links rather than restating.

Specification version. CSR semantics here follow The RISC-V Instruction Set Manual, Volume II: Privileged Architecture. Read from the manual’s own preface on 2026-09-04, the current main branch documents the Machine and Supervisor privilege levels at version 1.13, and states that the module versions it contains “have been ratified”. The most recent numbered document version the preface names is 20260120; main carries further changes on top of it and is therefore a working draft, not a ratified snapshot (src/priv/preface.adoc). Nothing quoted below is new or contentious material — mepc, mstatus, mscratch, and mret have been stable since Privileged 1.10 — but quotations are from src/priv/machine.adoc and src/unpriv/zicsr.adoc at main on that date, and a reader implementing to sign-off should check them against the numbered release rather than main.

Mental Model — Return to a Different mepc

Start from what the hardware already does for free. When a trap fires, the hart performs a fixed, atomic sequence before a single instruction of your handler runs: it writes the interrupted program counter into mepc, a cause code into mcause, sets mstatus.MPIE to the old mstatus.MIE, clears MIE, records the interrupted privilege mode in mstatus.MPP, and jumps to the address in mtvec (see RISC-V Trap Handling for the full sequence, and Control and Status Registers for the register encodings). The mret instruction is the exact inverse: per the privileged manual, “When executing an xRET instruction, supposing xPP holds the value y, xIE is set to xPIE; the privilege mode is changed to y; xPIE is set to 1; and xPP is set to the least-privileged supported mode” (machine.adoc) — and then the program counter is loaded from mepc.

So the hardware gives you a return address register that software is allowed to overwrite. A trap handler that does nothing returns exactly where it came from. A trap handler that changes mepc returns somewhere else. That is the entire mechanism. Everything a context switch does beyond writing mepc exists to make the destination survivable: the registers the destination expects, the stack it expects, and the memory permissions it is entitled to.

flowchart TB
    subgraph IN["Arrival — hardware does this"]
        A1["task A executing at PC = 0x8001_0244"] --> A2["timer interrupt<br/>(CLINT raises mtip)"]
        A2 --> A3["hardware: mepc &lt;- 0x8001_0244<br/>mcause &lt;- 0x8000_0007<br/>MPIE &lt;- MIE, MIE &lt;- 0, MPP &lt;- U"]
        A3 --> A4["PC &lt;- mtvec"]
    end
    subgraph KERN["Kernel — software does this"]
        A4 --> K1["save A's 31 registers<br/>+ mepc + mstatus into A's frame"]
        K1 --> K2["scheduler picks B"]
        K2 --> K3["reprogram PMP for B"]
        K3 --> K4["restore B's 31 registers<br/>write mepc &lt;- 0x8002_00c8<br/>write mstatus &lt;- B's saved value"]
    end
    subgraph OUT["Departure — hardware does this"]
        K4 --> O1["mret"]
        O1 --> O2["MIE &lt;- MPIE, privilege &lt;- MPP<br/>PC &lt;- mepc"]
        O2 --> O3["task B executing at PC = 0x8002_00c8"]
    end
    A3 -. "the ONE value<br/>that changes<br/>the outcome" .-> K4

The switch as a substitution on one register. What it shows: the arrival path and the departure path are hardware-defined mirror images; the kernel sits between them and the only edit that changes which task runs next is the write to mepc. The insight to take: if you delete every sw/lw from the handler and keep only the mepc write, task B still starts running — it just runs with task A’s registers and immediately corrupts itself. Saving and restoring registers is not what makes the switch happen; it is what makes the switch correct. Getting that ordering right in your head is what stops the classic bug where a kernel switches perfectly for two tasks that happen to use disjoint registers and falls apart on the third.

The consequence worth internalizing early: a context switch is a trap that returns to the wrong place on purpose. In a system with an operating system underneath, a switch is a function call that mysteriously returns as somebody else. Here it is more honest and more legible — there is a literal csrw mepc, t1 in the code, and you can watch the value change in a waveform.

What Must Be Saved: Thirty-One Registers, mepc, and Two mstatus Bits

RV32I defines thirty-two integer registers, x0 through x31. Thirty-one of them need saving, not thirty-two, because x0 is hardwired to zero: reads always return 0 and writes are discarded. Storing it wastes a word and four bytes of code; restoring it is a no-op the hardware ignores. Off-by-one here is a genuinely common bug in first kernels, and it is silent — you get a frame that is four bytes bigger than it needs to be and a sw x0 that appears to work.

The architectural state that a resumed task can observe, and therefore the state a switch must preserve, is:

State	Width	Why it must be saved
x1–x31	31 × 32 bits	The task’s live values. x0 is excluded — hardwired zero.
mepc	32 bits	The resume address. Overwritten by the next trap the moment interrupts are re-enabled.
mstatus.MPP	2 bits	Which privilege mode to mret into (U for a user task, M for a kernel thread).
mstatus.MPIE	1 bit	Whether the task had interrupts enabled when it was interrupted.
mcause, mtval	32 bits each	Not task state — they describe the trap, are consumed by the handler, and are dead by the time you mret. Do not save them per task.
mscratch	32 bits	Not task state either. It is a kernel-owned slot; see below.

Why mepc is the one people forget. Every other item on that list lives in a register the programmer is already thinking about. mepc does not: it is written by the hardware, invisibly, at trap entry, and the code that reads it is the same code that wrote it, so during single-task bring-up it looks like it takes care of itself. The failure only appears once traps can nest — or, far more commonly, the first time the handler makes a nested call that itself takes a trap, or the first time you enable interrupts inside the handler. The privileged manual states the hazard directly: “Trap handlers must be designed to neither enable interrupts nor cause exceptions during the phase of handling where the trap handler preserves the critical state information required to handle and resume from the trap. An exception or interrupt in this critical phase of trap handling may lead to a trap that can overwrite such critical state” (machine.adoc). mepc is that critical state. There is exactly one of it per hart, and the second trap clobbers the first trap’s return address.

The symptom, when it happens, is memorably confusing: the kernel runs correctly for hundreds of switches and then returns into the middle of its own trap handler, or into address zero, and the UART stops mid-word. The fix is one instruction — csrr t1, mepc before anything can trap, sw t1, TF_MEPC(sp) immediately after — but you have to know it belongs there.

mstatus is saved whole, restored whole. In principle only MPP and MPIE are per-task. In practice a 32-bit csrr/csrw pair costs the same as a bitfield extraction and is harder to get wrong, so the frame stores the whole register. The one thing to be careful about is that mstatus also carries global policy bits — on a larger system MPRV, SUM, MXR, and the floating-point FS field — which you do not want a user task to be able to influence. On the RV32IMC target here, with no S-mode, no floating point, and no address translation, those fields are hardwired to zero or absent, so the whole-register save is safe. On a design that later adds them, the restore must mask.

flowchart LR
    subgraph MUST["Must be saved per task"]
        R["x1..x31<br/>31 words"]
        E["mepc<br/>the resume address"]
        S["mstatus<br/>MPP + MPIE"]
    end
    subgraph NOT["Must NOT be saved per task"]
        Z["x0<br/>hardwired zero"]
        C["mcause / mtval<br/>describe the trap,<br/>dead after handling"]
        M["mscratch<br/>kernel-owned,<br/>per-hart not per-task"]
    end
    subgraph MAYBE["Saved only if the design has them"]
        P["PMP config<br/>pmpaddr* / pmpcfg*"]
        F["f0..f31 + fcsr<br/>absent on RV32IMC"]
    end
    R --> FR["144-byte trap frame"]
    E --> FR
    S --> FR
    P -.->|"held in the task<br/>struct, not the frame"| TS["task control block"]

The three-way split of processor state. What it shows: only the left column belongs in the trap frame; the middle column is state that either cannot be saved (x0) or is not per-task (mcause, mscratch); the right column is state that exists only if you built it. The insight to take: the mcause/mtval column is where beginners over-save. They are trap metadata, valid for exactly the duration of one handler invocation, and copying them into a per-task frame implies a lifetime they do not have. The PMP entries are the opposite mistake — they are per-task but they do not belong in the trap frame, because they are reloaded from the task control block rather than pushed and popped.

The Trap Frame: Layout, and Where It Lives

A trap frame is the block of memory that holds one task’s saved processor state. Its layout is a contract between two languages — the assembly that writes it and the Rust that reads it — so it wants a rule simple enough to hold in your head while debugging at 3 a.m. The rule used here is:

The offset of xn is 4 × (n − 1). So x1 (ra) is at 0, x2 (sp) at 4, x31 (t6) at 120. mepc follows at 124, mstatus at 128. The structure is padded to 144 bytes so that stack alignment stays 16-byte, as the RISC-V calling convention requires.

 offset   +0            +4            +8            +12
        +-------------+-------------+-------------+-------------+
   0    | x1   ra     | x2   sp     | x3   gp     | x4   tp     |
        +-------------+-------------+-------------+-------------+
  16    | x5   t0     | x6   t1     | x7   t2     | x8   s0/fp  |
        +-------------+-------------+-------------+-------------+
  32    | x9   s1     | x10  a0     | x11  a1     | x12  a2     |
        +-------------+-------------+-------------+-------------+
  48    | x13  a3     | x14  a4     | x15  a5     | x16  a6     |
        +-------------+-------------+-------------+-------------+
  64    | x17  a7     | x18  s2     | x19  s3     | x20  s4     |
        +-------------+-------------+-------------+-------------+
  80    | x21  s5     | x22  s6     | x23  s7     | x24  s8     |
        +-------------+-------------+-------------+-------------+
  96    | x25  s9     | x26  s10    | x27  s11    | x28  t3     |
        +-------------+-------------+-------------+-------------+
 112    | x29  t4     | x30  t5     | x31  t6     |  mepc       |
        +-------------+-------------+-------------+-------------+
 128    |  mstatus    |   (pad)     |   (pad)     |   (pad)     |
        +-------------+-------------+-------------+-------------+
                                                    total: 144 B

The 144-byte RV32 trap frame, at word accuracy. What it shows: thirty-one register slots in numeric order, then the two CSRs, then twelve bytes of padding that exist purely so the frame is a multiple of 16. The insight to take: there is no x0 slot — the frame starts at x1, which is why the offset formula has the − 1 in it. Note also that sp occupies a slot like any other register even though the frame is addressed through sp; that forces the restore of x2 to be the very last load, because the base register dies with it. Diagram medium: this is an ASCII box diagram rather than the vault’s preferred mermaid packet-beta, because the structure is word-addressed (36 words, 1,152 bits) and packet-beta labels ranges in bits — a bit-accurate rendering would number this 0-31, 32-63, … 1120-1151 and obscure the one thing the reader needs, the byte offsets. See Drawing Wire Formats with Mermaid Packet Diagrams for where packet-beta is the right call instead.

The layout is checkable, and was checked. Because the frame is shared between assembly and Rust, the safest thing you can do is make the compiler assert the offsets rather than trusting a comment. Declaring the frame as a #[repr(C)] struct and asking core::mem::offset_of! gives a machine-verified answer:

#[repr(C)]
pub struct TrapFrame {
    pub x: [u32; 31],   // x1..x31; x0 is hardwired zero and is NOT stored
    pub mepc: u32,      // must land at 124
    pub mstatus: u32,   // must land at 128
    _pad: [u32; 3],     // pad to 144 = 16-byte aligned
}

Compiled with rustc 1.98.0 for riscv32imc-unknown-none-elf and disassembled with llvm-objdump, the accessor functions constant-fold to exactly the expected literals — frame_size returns 0x90 (144), off_mepc returns 0x7c (124), off_mstatus returns 0x80 (128) — confirming that the Rust view and the assembly .equ constants agree. In a real kernel these become const _: () = assert!(offset_of!(TrapFrame, mepc) == 124); so that a mismatch is a build failure rather than a 3 a.m. debugging session. This is one of the practical payoffs of writing the kernel in Rust; see Bare-Metal Rust.

Per-task kernel stack versus a fixed save area

Where the frame physically lives is a real design decision with three defensible answers.

One fixed save area per task, in a static array. The simplest possible scheme: static mut FRAMES: [TrapFrame; NTASKS], and the trap handler indexes by the current task ID. It needs no stack discipline at all, the addresses are known at link time, and it is trivially inspectable in a waveform or a memory dump. Its limitation is that it cannot nest: a second trap taken while the handler is running would overwrite the one frame that task owns. For a kernel whose handler never re-enables interrupts, that is fine, and for a first working switch it is the right starting point.

A per-task kernel stack, frame pushed on entry. This is what the code below does and what xv6, Linux, and essentially every production kernel do. Each task owns a small kernel stack (1–2 KiB is plenty for a microkernel that does no deep recursion); the trap handler carves a frame off the top of it with addi sp, sp, -144. Nesting works naturally — a second trap pushes a second frame — and the handler gets a stack to make ordinary function calls on, which matters the moment any of the handler is written in Rust rather than assembly. The cost is memory: NTASKS × stack_size, which on a Tang Nano 20K with a few tens of kilobytes of usable block RAM is a real budget line, and a stack overflow silently corrupts whatever is below it unless you place a guard region with PMP.

A single shared kernel stack for all tasks. Used by kernels that never block inside the kernel — the handler runs to completion on one stack, and the frame is copied out to the task’s control block before the stack unwinds. This is the seL4-style event kernel posture and it is genuinely the cheapest in RAM, but it forbids a handler from sleeping mid-way, which constrains how Synchronous IPC can be written.

Scheme	RAM cost	Nesting	Handler may call Rust freely	Notes
Fixed per-task save area	NTASKS × 144 B	No	Needs a separate stack anyway	Best for bring-up; simplest to inspect
Per-task kernel stack	NTASKS × (1–2 KiB)	Yes	Yes	The default; what the code below implements
Single shared kernel stack	one stack	Yes, within one trap	Yes, but must not block	seL4-style event kernel; cheapest in RAM

Three homes for the trap frame. What it shows: the cost and capability trade of each placement. The insight to take: the choice is really a question about your handler, not about your tasks — if the handler is allowed to block or to take a nested trap, it needs a stack per task; if it always runs to completion, it does not. Deciding this before writing the handler saves rewriting it.

The mscratch Trick: Where the Handler Gets Its First Register

Here is the bootstrapping problem that every trap handler on every architecture must solve, stated as sharply as possible: the first instruction of the handler must save a register, but saving a register requires an address, and computing an address requires a register. At the instant mtvec is entered, all thirty-one registers hold the interrupted task’s values. Touching any of them destroys state. There is nowhere to put a pointer.

RISC-V solves this with a dedicated machine-mode scratch CSR and one instruction. The privileged manual says what it is for: “The mscratch register is an MXLEN-bit read/write register dedicated for use by machine mode. … Typically, it is used to hold a pointer to a machine-mode hart-local context space and swapped with a user register upon entry to an M-mode trap handler” (machine.adoc).

The swap is a single csrrw, the atomic read-write CSR instruction from the Zicsr Extension. Written with sp as both source and destination:

csrrw sp, mscratch, sp    # sp <- mscratch ; mscratch <- old sp

csrrw rd, csr, rs1 reads the CSR into rd and writes rs1 into the CSR, in one atomic operation. With rd == rs1 == sp it is a pure exchange. After it executes, sp points at the kernel stack (because that is what the kernel had parked in mscratch) and mscratch holds the interrupted task’s stack pointer (safe, because nothing else writes mscratch). One instruction, and the handler now owns a usable pointer without having destroyed anything.

sequenceDiagram
    autonumber
    participant U as Task A registers
    participant SP as sp (x2)
    participant MS as mscratch CSR
    participant KS as A's kernel stack
    Note over SP,MS: before the trap
    SP->>SP: sp = 0x8001_9F00 (A's user stack)
    MS->>MS: mscratch = 0x8000_2400 (A's kernel stack top)
    Note over U,KS: trap fires — hardware sets mepc/mcause/mstatus, jumps to mtvec
    SP->>MS: csrrw sp, mscratch, sp
    MS-->>SP: sp = 0x8000_2400 (kernel stack)
    SP-->>MS: mscratch = 0x8001_9F00 (A's user sp, parked)
    SP->>KS: addi sp, sp, -144 — carve the frame
    KS->>KS: sw ra, 0(sp) ... sw t6, 120(sp) — 30 stores
    MS->>KS: csrr t0, mscratch then sw t0, 4(sp)<br/>— recover the parked user sp
    Note over KS: the frame is now complete — the handler may call Rust
    KS->>MS: before mret: csrw mscratch, (sp + 144)<br/>= the INCOMING task's kernel stack top

The mscratch swap, step by step. What it shows: the exchange gives the handler a stack pointer without spending a register, and the displaced user sp is retrieved from mscratch a few instructions later and written into the frame’s x2 slot like any other register. The insight to take: step 9 is the step people forget. mscratch must be reloaded before mret with the incoming task’s kernel stack top, not the outgoing one’s — otherwise the next trap taken by task B carves its frame on top of task A’s kernel stack and the two tasks quietly share saved state. The bug survives a two-task test if both tasks trap at the same rate, and shows up as impossible register values much later.

Two variants you will see in real kernels. xv6-riscv, which runs in S-mode under an MMU, uses sscratch slightly differently: uservec does csrw sscratch, a0 to park one register, then loads a fixed trapframe virtual address into a0 and saves everything relative to that (trampoline.S). That works because xv6 maps each process’s trapframe at the same virtual address in every address space — a trick available only with paging. In an M-mode system with no MMU there is no fixed address that means something different per task, so the swap-into-sp form is the natural one.

The second variant is a nesting discriminator: some kernels keep mscratch = 0 while executing in kernel mode and the kernel stack top while executing user code. The handler then does csrrw sp, mscratch, sp; bnez sp, from_user; ... and can tell a user-mode trap from a kernel-mode one in two instructions. That is worth adding the moment your handler re-enables interrupts; it is unnecessary while it does not.

The aliased-register case is architecturally guaranteed, not a lucky idiom. The unprivileged manual is explicit: “The CSRRW (Atomic Read/Write CSR) instruction atomically swaps values in the CSRs and integer registers. CSRRW reads the old value of the CSR, zero-extends the value to XLEN bits, then writes it to integer register rd. The initial value in rs1 is written to the CSR” (src/unpriv/zicsr.adoc, read 2026-09-04, emphasis added). The word initial is the guarantee: the value written to the CSR is the one rs1 held before rd was updated, so csrrw sp, mscratch, sp is a well-defined exchange rather than a race. One caveat from the same section: csrrw with rd = x0 “shall not read the CSR”, so the swap form genuinely requires a real destination register — you cannot write it as csrrw x0, mscratch, sp and expect a read.

The Assembly, Line by Line

Everything below was assembled and disassembled while writing this note with GNU assembler 2.46 from riscv64-linux-gnu-gcc 16.1.1, targeting -march=rv32imc_zicsr -mabi=ilp32. The byte counts and instruction counts quoted later are read out of objdump, not estimated.

Toolchain gotcha, measured 2026-09-04

-march=rv32imc does not include the CSR instructions. The first assembly attempt failed with seven copies of Error: unrecognized opcode `csrrw sp,mscratch,sp', extension `zicsr' required. Zicsr was split out of the base I extension when the unprivileged ISA was modularized, and modern binutils enforces the split. The target string must be rv32imc_zicsr. This is precisely the point definitely-not-esp32 MOC Stage 1 makes about Zicsr Extension — it is not optional decoration, it is the prerequisite for having traps at all, and the toolchain will tell you so.

The preemptive path: trap_entry

        .equ TF_MEPC,    124
        .equ TF_MSTATUS, 128
        .equ TF_SIZE,    144
 
        .section .text.trap, "ax", @progbits
        .globl  trap_entry
        .align  2                       # GAS RISC-V: .align n == 2^n bytes
trap_entry:
        csrrw   sp, mscratch, sp        # sp <- kernel stack top; mscratch <- task sp
        addi    sp, sp, -TF_SIZE        # carve the trap frame
 
        sw      ra,   0(sp)             # x1   (x2/sp is handled below)
        sw      gp,   8(sp)             # x3
        sw      tp,  12(sp)             # x4
        sw      t0,  16(sp)             # x5
        ...                             # 30 stores in total, x1 and x3..x31
        sw      t6, 120(sp)             # x31
 
        csrr    t0, mscratch            # t0 = the interrupted task's sp
        sw      t0,   4(sp)             # into the x2 slot
        csrr    t1, mepc                # THE register everyone forgets
        sw      t1, TF_MEPC(sp)
        csrr    t2, mstatus             # MPP/MPIE: which mode to return to
        sw      t2, TF_MSTATUS(sp)
 
        mv      a0, sp                  # arg0 = &frame of the outgoing task
        jal     ra, kernel_trap         # Rust; returns &frame of the task to resume
 
        mv      sp, a0                  # switch to the incoming task's frame
        addi    t0, sp, TF_SIZE         # its kernel stack top ...
        csrw    mscratch, t0            # ... is what the NEXT trap will swap in
 
        lw      t2, TF_MSTATUS(sp)
        csrw    mstatus, t2
        lw      t1, TF_MEPC(sp)
        csrw    mepc, t1                # return to a DIFFERENT mepc than we arrived with
 
        lw      ra,   0(sp)             # 30 loads, x1 and x3..x31
        ...
        lw      t6, 120(sp)
        lw      sp,   4(sp)             # x2 LAST: it is the base register
        mret

Three lines carry all the subtlety and deserve individual attention.

csrrw sp, mscratch, sp — the swap discussed above. Note it is the first instruction; anything before it would clobber a task register.

lw sp, 4(sp) — the restore of x2 must be the final load, because the frame is addressed through sp. Restore it early and every subsequent lw reads from the task’s user stack instead of the frame. On a design where the frame is addressed through a different base register (a0, say, as in xv6’s trampoline) this constraint disappears — but then that base register itself has to be restored last, and it needs somewhere to be parked. The constraint does not go away; it moves.

csrw mscratch, t0, where t0 = sp + 144 — the reload of mscratch with the incoming task’s kernel stack top, computed from the frame pointer the scheduler returned. Because the frame sits at the bottom of the region carved by addi sp, sp, -144, adding the frame size back gets you the stack top. If the scheduler returned the same frame it was given (no switch this tick), this is a no-op that costs two instructions; if it returned a different task’s frame, this is what keeps the next trap from landing on the wrong stack.

The scheduler contract is deliberately thin: kernel_trap(frame: *mut TrapFrame) -> *mut TrapFrame. It receives the outgoing task’s saved state and returns the incoming task’s. A round-robin implementation (Round-Robin Scheduling) is a handful of lines; a rendezvous implementation (Synchronous IPC) blocks the caller and returns a different task’s frame. Neither of them touches a register directly — the assembly above is the only code in the kernel that knows what a register is.

The cooperative path: switch_context

When a task gives up the processor voluntarily — a yield(), a blocking send — the switch does not need to be a trap at all, and it does not need to save most of the registers. The RISC-V calling convention already did that work. Per the psABI’s integer register convention, x1 (ra), x5–x7 (t0–t2), x10–x17 (a0–a7) and x28–x31 (t3–t6) are not preserved across calls, while x2 (sp) and x8–x9, x18–x27 (s0–s11) are (riscv-cc.adoc). If switch_context is reached by an ordinary call, the compiler has already spilled everything the caller cared about in the caller-saved set. Saving it again is pure waste.

        .equ CTX_SIZE, 64               # 13 words -> 52 bytes, rounded to 16-byte alignment
 
        .globl  switch_context
switch_context:                         # void switch_context(u32 **old_sp, u32 *new_sp)
        addi    sp, sp, -CTX_SIZE
        sw      ra,   0(sp)             # x1  — where to resume
        sw      s0,   4(sp)             # x8
        sw      s1,   8(sp)             # x9
        sw      s2,  12(sp)             # x18 ...
        ...
        sw      s11, 48(sp)             # x27
        sw      sp,   0(a0)             # *old_sp = sp   <-- the handoff
        mv      sp, a1                  # sp = new_sp    <-- now we are the other task
        lw      ra,   0(sp)
        ...
        lw      s11, 48(sp)
        addi    sp, sp, CTX_SIZE
        ret

Thirteen registers, not thirty-one. sw sp, 0(a0) followed by mv sp, a1 is the entire switch: the outgoing task’s stack pointer is published where the scheduler can find it, and the incoming task’s is installed. The ret at the end returns into a different function than the one that called — specifically, into whichever switch_context call site the incoming task last suspended at, because ra was reloaded from the incoming task’s saved context. This is the same conceptual trick that makes Linux’s switch_to(prev, next, prev) “return as next” (Context Switch Mechanics switch_to and switch_mm), reduced to its minimum.

The reference implementation to compare against is xv6-riscv’s swtch, which is the same routine at RV64 width and stores into a caller-provided struct context rather than onto the stack: fourteen sds (ra, sp, s0–s11), fourteen lds, and ret (swtch.S). The register set is identical; only the addressing differs.

Counting the Cost

The following are measured, by assembling the routines above and reading objdump -h and objdump -d. They are static counts of the code as emitted, not simulated cycle counts.

Routine	Instructions	Bytes (RV32IMC)	Bytes (RV32IM, no compression)	Registers touched
trap_entry (preemptive)	79	178 (69 × 16-bit + 10 × 32-bit)	316	31 GPRs + mepc + mstatus + mscratch
switch_context (cooperative)	31	64 (30 × 16-bit + 1 × 32-bit)	124	13 (ra, s0–s11) + sp
Ratio	2.55×	2.78×	2.55×	2.4×

Static cost of the two switch styles. What it shows: the preemptive path costs about two and a half times the instructions and nearly three times the code bytes. The insight to take: the extra 138 bytes are not overhead you can optimize away — they are the price of not knowing what the interrupted task was doing. The cooperative path is cheaper for exactly one reason: the compiler already spilled the caller-saved half, so the switch only has to preserve what the ABI promises. This is a design decision with a number attached, which is the kind of decision Stage 9 of definitely-not-esp32 MOC exists to teach.

Two things fall out of the byte column that are worth pausing on. First, the compressed extension pays for itself dramatically here: 178 bytes versus 316 is a 44% reduction, and it happens because almost every instruction in the routine is a stack-relative sw/lw of a register, which is exactly the case C.SWSP and C.LWSP exist for. Of trap_entry’s 79 instructions, 69 compress; the ten that do not are the CSR accesses (csrrw, csrr ×3, csrw ×3, mret), the jal, and the addi t0, sp, 144 whose immediate is out of compressed range. On a Tang Nano 20K where the whole kernel must live in block RAM, that 44% is the difference between fitting and not. Second, the CSR instructions are structurally incompressible — there is no compressed CSR form in the C extension — so as a kernel does more CSR work its compression ratio gets worse.

From instructions to cycles

Instruction count is not cycle count, and this machine does not exist yet, so the honest thing is to state the model rather than a number.

Uncertain

Verify: the cycle counts in the table below. Reason: they are calculated from a pipeline model, not measured. No RTL for the target core was simulated for this note — there is no synthesized or simulated definitely-not-esp32 core available here, and no RISC-V instruction-set simulator (spike, qemu-riscv32) is installed on this machine, so nothing dynamic was executed. The instruction and byte counts above are measured; the cycle counts are the instruction counts plus a hazard model whose assumptions are listed explicitly. To resolve: run trap_entry on the actual core under Verilator with mcycle read before and after, and replace this callout with the measured figure and the commit of the RTL it was measured on. uncertain

Assume the Classic Five-Stage Pipeline the project is building: IF · ID · EX · MEM · WB, full forwarding, single-cycle instruction and data memory, branches resolved in EX, and no branch prediction yet. Then:

Cost component	Cycles	Why
79 instructions at ideal CPI = 1	79	The baseline; see Cycles Per Instruction
Trap entry pipeline flush	+3	The interrupt redirects fetch; IF/ID/EX are squashed
jal kernel_trap	+2	Unconditional jump resolved in EX with no predictor
lw t2, 128(sp) → csrw mstatus, t2	+1	Load-Use Hazard: the CSR write needs its operand in EX
lw t1, 124(sp) → csrw mepc, t1	+1	Same hazard, second occurrence
mret redirect	+3	Another full flush
Modelled subtotal for trap_entry	≈ 89	Plus whatever kernel_trap itself costs

The equivalent model for switch_context is 31 instructions, one jal/ret pair charged to the caller, and no flushes at all — call it ≈ 33 cycles. So a cooperative yield is roughly a third the cost of an interrupt-driven preemption on this core, before the scheduler body is counted.

Two corrections that will move these numbers on real hardware, both upward. Instruction fetch from block RAM is not free. On the FPGA the instruction memory is a block RAM with a registered output, which adds at least one cycle of fetch latency; whether that shows up as an extra cycle per instruction or is hidden by the pipeline depends on how the fetch stage is built (see Timing Closure and Fmax for why that register is there in the first place). And the CSR instructions may not be single-cycle. A design that implements CSR access as a microcoded or multi-cycle operation — a common simplification, since CSRs live outside the main register file and do not need the datapath’s forwarding network — will charge two or three cycles each for the eight CSR accesses in trap_entry, adding 8–16 cycles.

The two corrections point at the same practical advice: read mcycle around the switch on the real core and publish that number, exactly as Stage 9 demands for CPI. A modelled 89 and a measured 130 are both useful; a modelled 89 presented as measured is not.

The Two Switch Styles, End to End

The full preemptive path, from task A running to task B running, involves the timer peripheral, the hardware trap logic, the assembly stub, and the scheduler — four components that each own one contiguous stretch of the sequence.

sequenceDiagram
    autonumber
    participant A as Task A (U-mode)
    participant HW as Hart trap logic
    participant TE as trap_entry (asm)
    participant K as kernel_trap (Rust)
    participant B as Task B (U-mode)

    A->>A: executing — mtip asserted by the CLINT
    HW->>HW: mepc <- A's PC, mcause <- 0x8000_0007<br/>MPIE <- MIE, MIE <- 0, MPP <- U
    HW->>TE: PC <- mtvec
    TE->>TE: csrrw sp, mscratch, sp
    TE->>TE: addi sp, sp, -144<br/>30 stores of x1, x3..x31
    TE->>TE: csrr t0, mscratch into frame x2 slot
    TE->>TE: csrr mepc and mstatus into frame +124 / +128
    TE->>K: jal kernel_trap(a0 = &frame_A)
    K->>K: clear mtip (write mtimecmp)<br/>pick next runnable task
    K->>K: reprogram PMP for B (2 CSR writes)
    K-->>TE: return a0 = &frame_B
    TE->>TE: mv sp, a0 then csrw mscratch, sp+144
    TE->>TE: csrw mstatus, B's<br/>csrw mepc, B's PC
    TE->>TE: 30 loads, then lw sp, 4(sp) last
    TE->>HW: mret
    HW->>B: MIE <- MPIE, privilege <- MPP<br/>PC <- mepc
    B->>B: executing, believing it never stopped

One complete preemptive switch. What it shows: the four owners of the sequence and exactly where each hands off — hardware does steps 2 and 18 for free, the assembly stub does the register work, and the Rust does policy and nothing else. The insight to take: step 9 is easy to overlook and will hang the system if omitted. The timer interrupt is level-sensitive on the CLINT: mtip stays asserted until software writes a new mtimecmp. Forget it and mret immediately re-traps, and the kernel spins in trap_entry forever with the UART silent. This is the single most common “my scheduler locked up” bug at this stage, and it is a peripheral bug, not a context-switch bug — which is why it is worth drawing them on the same diagram.

The cooperative style collapses most of that. There is no trap, so steps 2, 3, 18 and 19 do not happen; there is no mscratch swap, because the task was already running on a stack the kernel is willing to use; and there is no mepc, because the resume address is ra on the stack like any ordinary function return. A yield is a function call that comes back later.

	Cooperative (switch_context)	Preemptive (trap_entry)
Entered by	an ordinary call	a trap (timer, external, ecall)
Registers saved	13 (ra, s0–s11) + sp	31 + mepc + mstatus
Why fewer	the psABI already spilled the caller-saved set	nothing spilled anything; the task did not consent
Resume address held in	ra, on the task’s own stack	mepc, in the trap frame
Measured cost	31 instructions / 64 bytes	79 instructions / 178 bytes
A runaway task	never yields → starves everyone	is preempted on the next timer tick
Needs mscratch	no	yes
Needs a per-task kernel stack	no (reuses the task’s stack)	yes, or a fixed save area

The design choice, with its price. What it shows: cooperative switching is cheaper on every axis except the one that matters for robustness. The insight to take: the two are not mutually exclusive and real kernels ship both — a fast cooperative path for yield and blocking Synchronous IPC, and the full trap path for the timer tick and external interrupts. Building both is not gold-plating; the cooperative path is 31 instructions and it is what the IPC rendezvous will use.

A task’s states

stateDiagram-v2
    [*] --> Ready: task created;<br/>frame initialised with<br/>mepc = entry point,<br/>MPP = U, MPIE = 1
    Ready --> Running: scheduler returns<br/>this task's frame;<br/>mret
    Running --> Ready: timer tick preempts<br/>(quantum expired)
    Running --> Ready: voluntary yield()
    Running --> Blocked: send/recv with<br/>no partner ready
    Blocked --> Ready: partner arrives;<br/>rendezvous completes
    Running --> Zombie: task returns or faults
    Zombie --> [*]: reaped
    note right of Ready
      frame is complete and
      valid in memory; the
      task is resumable by a
      single mret
    end note
    note right of Running
      frame is STALE — the live
      values are in the register
      file, not in memory
    end note

A task’s lifecycle, annotated with where the truth lives. What it shows: the standard ready/running/blocked cycle, with the two edges out of Running that a microkernel actually implements — preemption by the timer and blocking on IPC. The insight to take: the two notes are the important part. A task’s trap frame is authoritative in every state except Running, where the register file holds the live values and the frame is stale. Any kernel code that inspects task[i].frame for the currently running task is reading garbage — a mistake that becomes possible the moment you write a debugger or a ps-like dump, and which is invisible while there is only one task.

Initialising a frame for a task that has never run. The elegance of the trap-frame design is that starting a task and resuming one are the same operation. To create a task you allocate its stack, zero the 31 register slots, write the entry point into the mepc slot, write the top of its user stack into the x2 slot, and set MPP = 0b00 (U-mode) and MPIE = 1 in the mstatus slot. Then hand that frame to trap_entry’s restore path and mret. There is no special “first run” code path. Linux does need one — ret_from_fork exists precisely because a forked task has never been through switch_to before (Context Switch Mechanics switch_to and switch_mm) — and avoiding it is a genuine simplification of the M-mode design.

Switching Memory Protection as Part of the Switch

A switch that restores registers but leaves the memory permissions alone is not isolation — task B resumes with task A’s memory still readable. On a system with no MMU, the protection state is the PMP configuration, and reprogramming it is part of the switch.

The RISC-V designers anticipated this exactly. The privileged manual says of the configuration registers: “The PMP configuration registers are densely packed into CSRs to minimize context-switch time” (machine.adoc, emphasis added). Four eight-bit pmpXcfg fields per RV32 CSR means four regions’ permissions can be installed with one csrw. The address registers are not packed — one pmpaddr CSR per entry — so the cost of a switch is dominated by how many regions change, not how many entries exist.

For the two-region-per-task layout that Physical Memory Protection describes for this project (one locked kernel entry that never changes, plus per-task data and code regions), the incremental cost is small enough to write out:

#[repr(C)]
pub struct Task {
    pub frame:   *mut TrapFrame,
    pub pmp_lo:  u32,   // pmpaddr0: NAPOT-encoded base+size of this task's region
    pub pmp_cfg: u32,   // the pmpcfg0 byte pattern for this task
}
 
#[inline(always)]
pub fn load_pmp(t: &Task) {
    unsafe {
        core::arch::asm!(
            "csrw pmpaddr0, {a}",
            "csrw pmpcfg0,  {c}",
            a = in(reg) t.pmp_lo,
            c = in(reg) t.pmp_cfg,
            options(nostack)
        );
    }
}

Compiled for riscv32imc-unknown-none-elf with rustc 1.98.0 and disassembled, the whole thing is five instructions and fourteen bytes: two lw to fetch the fields out of the task struct, two csrw, and ret. Reprogramming one NAPOT region costs two CSR writes. Against trap_entry’s 79 instructions, adding PMP switching is roughly a 5% increase, and it buys the entire isolation property.

No sfence.vma is needed on this design — and this refines the sibling note

Physical Memory Protection’s context-switch walk-through lists sfence.vma x0, x0 after the PMP writes, citing the manual’s requirement that “when the PMP settings are modified, M-mode software must synchronize the PMP settings with the virtual memory system and any PMP or address-translation caches.” That requirement is real, but the very next sentence of the same section scopes it: “If page-based virtual memory is not implemented, memory accesses check the PMP settings synchronously, so no SFENCE.VMA is needed” (machine.adoc, read 2026-09-04). The definitely-not-esp32 core implements no MMU and no Sv32 translation, so the fence is unnecessary — and on a core that does not implement Zifencei/Svinval at all, emitting it may be an illegal instruction. Keep the fence in the design only when the MMU stretch goal lands.

Where in the sequence the PMP writes go. After the scheduler has chosen the incoming task and before mret — anywhere in that window is correct, because M-mode is exempt from PMP enforcement for the un-locked entries and the kernel is not touching user memory in between. Putting them inside kernel_trap (Rust) rather than in the assembly stub is the better factoring: the assembly stub should know about registers and nothing else, and PMP encoding is fiddly enough to want the type system.

The one ordering constraint that does bite: if the kernel uses locked PMP entries (L = 1) for its own image, those entries cannot be rewritten until the next reset, so the per-task entries must occupy different, higher-numbered slots. PMP matching is priority-ordered lowest-number-first, so a locked low entry protecting the kernel also shadows any higher-numbered entry that overlaps it — which is the desired behaviour, but it means a task’s region must not overlap the kernel’s or the task silently loses access to its own memory. Physical Memory Protection covers the matching rules in full.

Failure Modes and Common Misunderstandings

These are ordered roughly by how early they bite.

Saving x0. Harmless but revealing: a 32-slot frame with sw x0, 0(sp) at the front means the offset formula is 4 × n, not 4 × (n − 1), and the mismatch will surface the first time Rust and assembly disagree about where mepc lives. Fix the frame, not the constant.

Forgetting mepc. Discussed above. The tell is a kernel that works for a long time and then returns to a nonsensical PC. If you can read mepc from the UART in your fault handler, print it — a mepc that points into trap_entry itself is the signature.

Restoring sp too early. Every load after it reads from the wrong memory. Because the task’s user stack usually contains plausible values, the symptom is not a crash but a task that resumes with subtly wrong registers. Put lw sp, 4(sp) immediately before mret, always.

Not reloading mscratch before mret. The next trap lands on the previous task’s kernel stack. With two tasks that trap at different rates, this produces a frame that is half task A and half task B. Extremely hard to see in a debugger; trivial to prevent with the two-instruction addi/csrw pair.

Not clearing the interrupt source. For the CLINT timer this means writing a new mtimecmp; mtip is level-driven and stays asserted until you do. The system appears to hang with no output. See Core Local Interruptor.

Enabling interrupts inside the handler before the frame is complete. The manual warns about this explicitly (quoted earlier): a nested trap overwrites mepc and mcause before the first handler has copied them out. Either keep MIE = 0 for the whole handler — which is fine for a microkernel whose handler is 79 instructions plus a small scheduler — or complete the save first and accept the need for nested frames.

Assuming mret returns to M-mode. It returns to whatever mstatus.MPP says. If the frame for a user task was initialised with MPP = 0b11 (M) instead of 0b00 (U), the task runs with full privilege and PMP does not apply to it, so every isolation test passes vacuously. A protection mechanism you have not seen fault is a protection mechanism you have not tested — the same discipline definitely-not-esp32 MOC states for Stage 7.

Believing the running task’s frame is current. Restated because it is worth repeating: while a task is Running, its saved frame is stale.

Under-aligning mtvec. mtvec’s low two bits are the MODE field, so the base address must be at least 4-byte aligned; in vectored mode the requirement is stricter. And a GAS-specific trap that cost a rebuild while writing this note: on RISC-V, .align n means align to 2ⁿ bytes, so .align 4 requests 16-byte alignment, not 4-byte. The first build of trap_entry carried fourteen bytes of leading padding for that reason. .align 2 is what you want.

Alternatives and When to Choose Them

Save nothing; run tasks as coroutines in one address space. Rust’s async/await compiles a task into a state machine whose live state is a struct, so “switching” is just calling a different future’s poll. There is no trap frame at all and the cost is a handful of instructions. This is a genuinely good answer for a firmware-shaped workload, and it is what Embassy does on real RV32 parts. It is not a microkernel: there is no preemption, one task that loops forever hangs the system, and there is no privilege boundary, so PMP buys nothing. Choose it when the tasks are cooperating parts of one program; choose a real switch when they are not.

Hardware-assisted stacking. ARM’s Cortex-M pushes eight registers (R0–R3, R12, LR, PC, xPSR) onto the active stack in hardware at exception entry and pops them on return, so a Cortex-M handler’s save code is roughly half the length of the RISC-V equivalent. RISC-V deliberately does not do this — trap entry writes only CSRs and touches no memory, which keeps the trap-entry latency fixed and independent of memory-system behaviour, and keeps the hardware simpler. The trade is real and it is visible in the 79-instruction count: RISC-V moves the cost from silicon into software. For a project whose point is to build the silicon, that is the correct side of the trade.

A dedicated shadow register bank. Some embedded architectures (and RISC-V’s own Smrnmi, plus various vendor extensions) provide a second copy of some registers for the handler to use, eliminating the save entirely for short handlers. Cheap in cycles, expensive in area — every shadowed register is another flip-flop bank in The Register File, and register-file area and its read path are already candidates for the critical path (see Timing Closure and Fmax). Not worth it at this scale.

Saving fewer registers by constraining the handler’s ABI. If the trap handler is written entirely in assembly and provably touches only t0–t2, a preemptive switch that does not switch tasks needs to save only those three. This is the classic “fast path / slow path” split: check whether a switch is actually required, and if not, restore three registers and mret. On a round-robin scheduler with a short quantum most ticks do switch, so the win is smaller than it looks; on an interrupt handler that services a UART and returns to the same task, the win is large. Worth building only after measuring which case dominates.

Production Notes

xv6-riscv is the reference worth reading in full, because it is small enough to read in an afternoon and its two halves map exactly onto the two styles here: kernel/swtch.S is the cooperative switch (fourteen sds, fourteen lds, ret) and kernel/trampoline.S is the preemptive one. The instructive difference is that xv6 runs in S-mode with paging, so its trap frame lives at a fixed virtual address mapped identically in every process, and uservec parks a single register in sscratch rather than swapping the stack pointer. Without an MMU that option does not exist, which is why this note’s stub swaps sp.

seL4 takes the opposite architectural position and is worth knowing about for contrast: it is an event kernel with a single kernel stack, no kernel threads, and no blocking inside the kernel — every system call runs to completion. That makes the switch cheaper and the worst-case execution time analysable, which is why seL4 can be formally verified and has published WCET bounds. It also makes Synchronous IPC the only IPC that fits, which is not a coincidence.

The number to publish. Once the core runs, the deliverable from this stage is a measured switch cost: read mcycle at the top of trap_entry and again after mret lands, on a fixed workload, and report it alongside the CPI numbers Stage 9 demands. Compared against an ESP32-C3 running the same two-task alternation under its vendor RTOS, that is one of the four numbers definitely-not-esp32 MOC Stage 10 asks for — and unlike raw clock speed, it is a number a from-scratch design can plausibly win, because a 79-instruction switch with no MMU, no cache, and no address-space change has very little to do.

See Also

• definitely-not-esp32 MOC — the project ladder; this note is Stage 8’s missing mechanism
• Computer Architecture MOC — the concept companion for the whole RISC-V/SoC cluster
• Round-Robin Scheduling — decides which frame kernel_trap returns
• Synchronous IPC — decides when a task stops being runnable; uses the cooperative path
• Microkernel — why the kernel is this small in the first place
• RISC-V Trap Handling — the hardware half of arrival and departure
• Control and Status Registers — mepc, mstatus, mscratch, mtvec encodings
• Zicsr Extension — the six instructions that reach the CSRs, and the -march string that enables them
• Physical Memory Protection — the isolation state that must switch with the registers
• Bare-Metal Rust — no_std, #[repr(C)] frames, and core::arch::asm!
• Core Local Interruptor — the timer whose mtip drives preemption
• Classic Five-Stage Pipeline · Load-Use Hazard · Cycles Per Instruction — the model behind the cycle estimate
• Timing Closure and Fmax — the other half of “how fast is this really”
• Context Switch Mechanics switch_to and switch_mm — the contrast: the same problem on Linux/x86-64 with an MMU, a scheduler, and a task struct
• Context Switch Cost — what a switch costs on a large machine, where cache and TLB pollution dominate the register save