vault

BPF Interpreter vs JIT

15 min read

BPF Interpreter vs JIT

A verified extended BPF (eBPF) program can be executed by the kernel in one of two ways. The interpreter is a software loop — ___bpf_prog_run() in kernel/bpf/core.c — that fetches each BPF instruction and dispatches to a handler via a computed goto; it is portable to any architecture but slow. The Just-In-Time (JIT) compiler instead translates the verified bytecode into native machine code once at load time, so the program runs as ordinary kernel text at near-native speed (see BPF JIT Compiler). On the dominant 64-bit architectures (x86-64, arm64) the JIT is the real execution path, and on security-hardened kernels the interpreter is compiled out of the kernel entirely via CONFIG_BPF_JIT_ALWAYS_ON — a change that first shipped in Linux 4.15 (commit 290af86, Jan 2018 — the tag v4.15 contains it while v4.14 does not, confirmed via the GitHub compare API) specifically to deny Spectre v2 attackers the interpreter’s instructions as speculative-execution gadgets. The bpf(2) man page states it directly: “Since Linux 4.15, the kernel may be configured with the CONFIG_BPF_JIT_ALWAYS_ON option. In this case, the JIT compiler is always enabled, and the bpf_jit_enable is initialized to 1 and is immutable” (man7 bpf(2)). The interpreter survives only because some architectures still lack a JIT backend. (Verified against the Linux 6.12 LTS source tree.)

Mental Model

There is exactly one frontend to BPF execution — prog->bpf_func(ctx, insn) — and two possible implementations behind that function pointer. At load time, bpf_prog_select_runtime() decides which one this program will use. If a JIT backend exists and succeeds, bpf_func points at freshly emitted native code. If not (no JIT for this arch, or JIT disabled), bpf_func points at one of the interpreter trampolines that eventually call ___bpf_prog_run(). On a kernel built with CONFIG_BPF_JIT_ALWAYS_ON, the second branch does not exist at all — the interpreter source is #ifdef’d out and a JIT failure is a hard load error.

flowchart TB
  LOAD["bpf() BPF_PROG_LOAD<br/>verifier approves program"]
  SEL["bpf_prog_select_runtime()"]
  LOAD --> SEL
  SEL --> TRYJIT{"bpf_int_jit_compile()<br/>succeeds?"}
  TRYJIT -->|"yes"| NATIVE["bpf_func = native code<br/>(per-arch JIT output)<br/>fast path"]
  TRYJIT -->|"no, JIT disabled<br/>or no backend"| ALWAYSON{"CONFIG_BPF_JIT<br/>_ALWAYS_ON?"}
  ALWAYSON -->|"yes"| FAIL["load fails: -ENOTSUPP<br/>(interpreter removed)"]
  ALWAYSON -->|"no"| INTERP["bpf_func = interpreter<br/>(___bpf_prog_run)<br/>slow fallback"]
  NATIVE --> RUN["program runs"]
  INTERP --> RUN

The runtime-selection decision at program load. What it shows: after the verifier approves a program, bpf_prog_select_runtime tries the per-architecture JIT first; on success the program’s bpf_func is native code; on failure it falls back to the interpreter — unless CONFIG_BPF_JIT_ALWAYS_ON is set, in which case there is no interpreter to fall back to and the load fails with -ENOTSUPP. The insight to take: the JIT is the intended path and the interpreter is a portability fallback; modern security-hardened kernels delete the fallback entirely, so “the interpreter” is increasingly a thing that exists only on niche architectures.

The Interpreter: ___bpf_prog_run and Computed-Goto Dispatch

The interpreter lives in kernel/bpf/core.c, guarded by #ifndef CONFIG_BPF_JIT_ALWAYS_ON — meaning the whole function is absent when always-on is configured. Its signature is static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn): it takes the BPF pseudo-register array and a pointer to the instruction stream, and returns “whatever value is in %BPF_R0 at program exit” (per the kernel doc comment, 6.12).

Dispatch is a computed goto over a jump table indexed by the 8-bit opcode, not a plain switch. The kernel builds a 256-entry table of label addresses:

static const void * const jumptable[256] __annotate_jump_table = {
    [0 ... 255] = &&default_label,        /* every opcode defaults to "unknown" */
    BPF_INSN_MAP(BPF_INSN_2_LBL, BPF_INSN_3_LBL),  /* fill in the real handlers */
    ...
};
...
select_insn:
    goto *jumptable[insn->code];          /* jump straight to this opcode's handler */

Each instruction handler ends with CONT, defined as ({ insn++; goto select_insn; }) — advance to the next instruction and dispatch again. This goto *table[code] (“threaded code”) is faster than a switch because it skips the bounds-check and range-compare a compiled switch would emit, and it gives the branch predictor a separate indirect branch site per opcode. The handlers themselves are the obvious thing: ALU64_ADD_X: DST = DST + SRC; CONT;, and so on, where DST, SRC, IMM, OFF are macros decoding the instruction’s fields (core.c, 6.12). An unknown opcode lands at default_label, which warns: pr_warn("BPF interpreter: unknown opcode %02x ...").

Even within the interpreter the kernel takes care with one Spectre-relevant detail: register-based shift amounts are explicitly masked (SRC & 63 for 64-bit, & 31 for 32-bit) to avoid undefined behavior, and a long comment notes this AND must not be emitted by JIT backends (the hardware shift already behaves correctly there). That asymmetry is a small window into the interpreter-vs-JIT split: the same BPF semantics, realized two different ways.

How a Program Reaches the Interpreter

The interpreter is not called directly; there is a thin layer of trampolines parameterized by stack size. bpf_prog_select_func() sets fp->bpf_func = interpreters[(round_up(stack_depth, 32) / 32) - 1] — picking one of a family of generated functions (__bpf_prog_run32, __bpf_prog_run64, … up to …512), each of which allocates a stack frame of the right size and then tail-calls ___bpf_prog_run. There are sixteen of them, one per 32-byte stack-size bucket from 32 to 512, matching the 512-byte stack ceiling (MAX_BPF_STACK; see The BPF Calling Convention and Stack). This indirection lets the kernel allocate exactly the stack a program needs rather than always reserving the maximum.

The JIT: Native Code Instead of a Loop

When a JIT backend is present, bpf_prog_select_runtime() calls bpf_int_jit_compile(fp), the per-architecture compiler that walks the verified bytecode and emits native instructions into an executable buffer; fp->bpf_func is then set to that buffer and fp->jited becomes true. Because the BPF calling convention was designed to map one-to-one onto native ABIs, the translation is largely mechanical — a BPF_CALL becomes a single native call, BPF registers map straight onto hardware registers, and a BPF_ADD becomes an add. The full mechanism (prologue/epilogue, register allocation, tail-call patching, kallsyms integration) is the subject of BPF JIT Compiler; what matters here is that it replaces the dispatch loop with straight-line native code, eliminating the per-instruction goto *table[code] overhead. The performance gap is large: a JIT-compiled XDP program can process tens of millions of packets per second, which an interpreted program cannot approach.

The kernel doc states the supported eBPF-JIT architectures (6.12): x86-64, x86-32, arm64, arm32, ppc64, ppc32, sparc64, mips64, s390x, riscv64, riscv32, loongarch64, and arc (net.rst). Architectures outside that list have no eBPF JIT and must use the interpreter — which is the entire reason the interpreter still exists.

Selecting the Runtime: bpf_prog_select_runtime

The decision logic is worth reading exactly, because it encodes the always-on policy. From kernel/bpf/core.c (6.12):

struct bpf_prog *bpf_prog_select_runtime(struct bpf_prog *fp, int *err)
{
    bool jit_needed = false;
    ...
    if (IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON) ||
        bpf_prog_has_kfunc_call(fp))
        jit_needed = true;
 
    bpf_prog_select_func(fp);          /* sets bpf_func to interpreter, or to the warn-stub */
    ...
    fp = bpf_int_jit_compile(fp);      /* try to JIT; may leave fp->jited == false */
    if (!fp->jited && jit_needed) {
        *err = -ENOTSUPP;              /* JIT was mandatory and failed -> hard error */
        return fp;
    }
    ...
}

Two things to read out of this. First, jit_needed is forced true under CONFIG_BPF_JIT_ALWAYS_ON — so if the JIT fails (or there is none), the load fails with -ENOTSUPP rather than silently interpreting. Second, jit_needed is also forced true for any program that calls a kfunc — the interpreter cannot resolve kfunc calls, so kfunc-using programs require a JIT even on a non-always-on kernel.

And bpf_prog_select_func() shows what happens to the fallback when the interpreter is gone:

static void bpf_prog_select_func(struct bpf_prog *fp)
{
#ifndef CONFIG_BPF_JIT_ALWAYS_ON
    u32 stack_depth = max_t(u32, fp->aux->stack_depth, 1);
    fp->bpf_func = interpreters[(round_up(stack_depth, 32) / 32) - 1];
#else
    fp->bpf_func = __bpf_prog_ret0_warn;   /* should be overwritten by the JIT */
#endif
}

When always-on is set, bpf_func is initialized to __bpf_prog_ret0_warn, a stub whose body is literally WARN_ON_ONCE(1); return 0; with the comment “If this handler ever gets executed, then BPF_JIT_ALWAYS_ON is not working properly, so warn about it!” The JIT is expected to overwrite bpf_func with native code; if it ever runs, that is a kernel bug. This is the mechanical proof that the interpreter is truly gone in an always-on build — there is no interpreter function to point at, only a poison stub.

The bpf_jit_enable Knob and Defaults

Whether the JIT runs (on a kernel that has both backends compiled in) is controlled at runtime by the sysctl net.core.bpf_jit_enable. Its documented values are (net.rst, 6.12):

  • 0 — disable the JIT (use the interpreter). The doc lists this as “(default value)”.
  • 1 — enable the JIT.
  • 2 — enable the JIT and dump the emitted code to the kernel log (debug).

There is an important subtlety in that “default value,” and it is exactly where the older bpf(2) man-page framing and the source diverge. The man page describes the historical picture: “Before Linux 4.15, the JIT compiler is disabled by default” (man7 bpf(2)) — matching the net.rst “0 (default value)” text. But the kernel variable is initialized as int bpf_jit_enable __read_mostly = IS_BUILTIN(CONFIG_BPF_JIT_DEFAULT_ON) (core.c, 6.12) — so the real boot-time default depends on CONFIG_BPF_JIT_DEFAULT_ON, which is def_bool ARCH_WANT_DEFAULT_BPF_JIT || BPF_JIT_ALWAYS_ON (kernel/bpf/Kconfig). On x86-64 and arm64 — which set ARCH_WANT_DEFAULT_BPF_JIT — the JIT is therefore on by default at runtime even without always-on. So the practical situation on a mainstream 64-bit 6.12 distro kernel is: JIT compiled in, JIT enabled by default, interpreter present-but-rarely-used. Under CONFIG_BPF_JIT_ALWAYS_ON the knob is frozen: the Kconfig help says “/proc/sys/net/core/bpf_jit_enable is permanently set to 1 and setting any other value than that will return failure,” and the man page confirms bpf_jit_enable “is initialized to 1 and is immutable.”

Two neighboring sysctls round out the JIT’s runtime controls (both documented in net.rst, 6.12). net.core.bpf_jit_harden (values 0/1/2, default 0) trades performance for JIT-spray mitigation by constant-blinding the emitted code — 1 hardens for unprivileged users only, 2 for all users; this is output-hardening, orthogonal to whether the interpreter exists, and is covered in JIT Hardening and Constant Blinding. net.core.bpf_jit_limit caps the total memory the JIT may allocate for compiled images across the system, “in order to reject unprivileged JIT requests once it has been surpassed” — a denial-of-service guard so unprivileged BPF cannot exhaust kernel memory by forcing endless JIT compilations. Neither knob changes the interpreter-vs-JIT selection; they harden and bound the JIT path once chosen.

Uncertain

Verify: which mainstream distributions ship CONFIG_BPF_JIT_ALWAYS_ON=y (interpreter fully removed) versus merely CONFIG_BPF_JIT=y + ARCH_WANT_DEFAULT_BPF_JIT (JIT-on-by-default but interpreter still present) on their 6.12-era kernels. The task brief’s phrasing “the interpreter is compiled out / disabled by default on most modern configs” conflates two distinct things: removed at compile time (ALWAYS_ON) vs not used at runtime because the JIT default is on (DEFAULT_ON). Reason: distro .configs were not inspected this pass; web search indicated several distros do not set ALWAYS_ON and keep the interpreter compiled in. To resolve: inspect /boot/config-* (grep BPF_JIT_ALWAYS_ON) on current Debian/Ubuntu/Fedora/RHEL 6.12-based kernels.

uncertain

Why the Interpreter Was Removed: Spectre v2

The security motivation for CONFIG_BPF_JIT_ALWAYS_ON is specific and historically important. In January 2018, Google Project Zero disclosed Spectre — speculative-execution side channels. Variant 2 (branch-target injection, CVE-2017-5715) lets an attacker steer speculative execution into chosen “gadget” instructions already present in the kernel’s text. The eBPF interpreter is an unusually convenient gadget source: it is a large body of kernel code that performs attacker-influenced loads and arithmetic in a tight dispatch loop. As Project Zero put it (quoted in the commit message), “the presence of the code in the host kernel’s text section is sufficient to make it usable for the attack, just like with ordinary ROP gadgets.”

The subtlety that makes this so dangerous is that the attack need not even load a BPF program. As LWN recounts of Jann Horn’s demonstration, “the exploit was not actually loading BPF code into the kernel; the speculative execution was using the interpreter on BPF instructions that lived in user space” (LWN, “BPF and security”). In other words, the mere existence of ___bpf_prog_run in the kernel’s text — a function that walks a buffer of “instructions” and performs the loads/arithmetic they encode — is the vulnerability; an attacker who can mis-speculate the CPU into that code can supply their own pseudo-”instructions” from userspace memory and use the interpreter as a universal gadget. This is precisely why no amount of verifier tightening fixes it: the verifier governs programs that are actually loaded, but the Spectre-v2 abuse bypasses program loading entirely.

The kernel’s response, commit 290af86629b25 “bpf: introduce BPF_JIT_ALWAYS_ON config” (first shipped in Linux 4.15, the Spectre/Meltdown response release, Jan 2018), was to make it possible to delete the interpreter from the kernel image. The Kconfig help is explicit: “Permanently enable BPF JIT and remove BPF interpreter … Enables BPF JIT and removes BPF interpreter to avoid speculative execution of BPF instructions by the interpreter.” With the interpreter gone, there is no interpreter dispatch loop in .text for an attacker to mis-speculate into, and every loaded BPF program is JIT-compiled native code (which is subject to its own hardening — constant blinding against JIT-spray — covered in JIT Hardening and Constant Blinding). This is the core security argument for “JIT-only”: fewer attacker-reachable gadgets, and a smaller, more controlled execution surface.

Two honest caveats round out the picture. First, removing the BPF interpreter does not make the kernel Spectre-proof: as BPF maintainer Alexei Starovoitov noted, “he believes that any interpreter in the kernel could be used in this way; there are at least three other interpreters, so the kernel is still not fully safe” (LWN). The BPF interpreter was simply the most attractive and best-understood gadget, not the only one. Second, JIT-only turned out to be a performance win as well as a security one: the same LWN account notes the interpreter-removal strategy “proved effective while also recovering performance lost to other Spectre v2 mitigations like retpolines” — retpolines penalize the interpreter’s hot indirect branch (goto *jumptable[insn->code]) heavily, whereas straight-line JIT-compiled native code has no such per-instruction indirect dispatch to slow down. So the security-motivated move to JIT-only also sidesteps the retpoline tax that the interpreter would otherwise pay.

Note that removing the interpreter is a different mitigation from bpf_jit_harden (covered above): the former shrinks the gadget surface by deleting interpreter code, while the latter hardens the JIT’s output against JIT-spray. The two are complementary, and a fully hardened kernel applies both. See BPF and Spectre Hardening for the broader Spectre story across the verifier and JIT.

Why the Interpreter Still Exists

Given all that, why keep the interpreter at all? Because not every architecture has a JIT backend. The eBPF JIT is per-architecture machine-code emission, and porting it is real work; architectures outside the supported list above simply have no bpf_int_jit_compile that produces native code. On those, CONFIG_BPF_JIT_ALWAYS_ON cannot even be selected — the Kconfig entry depends on BPF_SYSCALL && HAVE_EBPF_JIT && BPF_JIT, so always-on requires a JIT backend to exist in the first place. The interpreter is therefore the portability floor: it lets BPF run correctly (if slowly, and without the Spectre hardening) anywhere the kernel builds, while the JIT provides speed and security on the architectures that have one. The kernel doc captures the nuance for 32-bit machines: “32-bit architectures run 64-bit eBPF programs via interpreter. Their JITs may convert BPF programs that only use 32-bit subregisters into native instruction set and let the rest being interpreted” (classic_vs_extended.rst, 6.12) — i.e. partial JIT with interpreter fallback for the hard cases.

Failure Modes and Diagnosis

  • bpf() BPF_PROG_LOAD returns -ENOTSUPP. On an always-on kernel, this means the JIT could not compile the program (unsupported instruction/feature for this arch’s JIT, or a JIT bug) and there is no interpreter to fall back to. The same -ENOTSUPP appears for a kfunc-using program when the JIT fails, because kfuncs force jit_needed.
  • Program loads but runs slowly. The JIT may be disabled at runtime (net.core.bpf_jit_enable == 0) on a non-always-on kernel, so the program is interpreted. Check the sysctl; check fp->jited via bpftool prog show (a JITed program reports a jited_len).
  • bpf_jit_enable = 2 produces no log output. That mode dumps emitted JIT code to the kernel log; if nothing appears, the JIT did not run for that program (disabled, or the program wasn’t JITed).
  • Hitting __bpf_prog_ret0_warn (a WARN in dmesg). This should never happen in normal operation — it is the always-on poison stub firing, indicating the JIT failed to overwrite bpf_func, i.e. a kernel-level bug.

Alternatives and When Each Applies

The “alternatives” here are the two backends themselves, and the choice is mostly made for you by the kernel build:

  • JIT (native code). Use whenever available — it is faster and, in always-on form, removes the interpreter gadget. This is the default on x86-64/arm64 server and desktop kernels. The cost is per-architecture JIT code that must be maintained, and JIT-specific hardening (constant blinding) to defend the emitted code.
  • Interpreter (___bpf_prog_run). The fallback for architectures without a JIT, for partial-JIT cases on 32-bit, and historically for net.core.bpf_jit_enable = 0. Slower, and (being kernel .text exercising attacker-influenced operations) a Spectre-gadget liability — which is exactly why hardened kernels remove it.

For most production deployments on mainstream hardware, the decision reduces to: prefer a kernel with CONFIG_BPF_JIT_ALWAYS_ON=y for the security benefit; rely on the interpreter only where no JIT backend exists.

See Also

Graph View

Backlinks