bpftrace

bpftrace is a high-level tracing language and front-end for Linux that lets you express an entire dynamic-tracing program as a single awk-like one-liner. You attach a small action block to a kernel or userspace event (a function entry, a tracepoint, a timer tick, a hardware counter overflow), guard it with an optional /predicate/ filter, and inside the block you aggregate data in the kernel using maps — so the histogram or count is built where the event fires and only the summary is copied to userspace. Under the hood a bpftrace script is parsed into an Abstract Syntax Tree (AST), run through a fixed pipeline of 27 front-end passes plus type-checking, resource analysis and code generation, lowered through LLVM to extended Berkeley Packet Filter (eBPF) bytecode, and attached to the kernel via libbpf using BPF Type Format (BTF) for type information; the language deliberately borrows from awk, C, and the older DTrace and SystemTap tracers (bpftrace README). The project’s own mission statement is narrower and more honest than “a tracer”: “Provide a quick and easy way for people to write observability-based BPF programs, especially for people unfamiliar with the complexities of eBPF (e.g. the verifier, kernel/userspace interaction, attachment, program loading, memory access, and the various types of BPF maps)” (docs/design_principles.md, v0.26.1). This note is the language and front-end view; the eBPF virtual machine, verifier, JIT, and map implementation it compiles down to are owned by Linux eBPF MOC — cross-linked, not re-derived here.

Version pin

Everything below was read from the bpftrace v0.26.1 release tarball, fetched as https://codeload.github.com/bpftrace/bpftrace/tar.gz/refs/tags/v0.26.1 and read on disk — not from the master branch, which drifts. v0.26.1 was released 2026-06-02; v0.27-rc0 was tagged 2026-08-13 and is not covered here (both dates from the repository’s releases Atom feed, which needs no authentication and is not subject to the GitHub REST API’s rate limit). The kernel side is pinned to Linux v6.12, a maintained long-term-support (LTS) release. bpftrace is userspace software on its own release cadence: its own dependency policy states a minimum kernel version of 6.1 and support for “stable kernels and the 4 most recent LTS kernels” (docs/dependency_support.md), so bpftrace-version and kernel-version claims are dated separately throughout.

Mental Model — One Probe, One Predicate, One Action

The whole language reduces to a repeated triple: probe, optional predicate, action. A probe names an event the kernel can fire a BPF program on. A predicate — written between slashes, /expr/ — is a boolean guard: if it evaluates false, the action is skipped. The action is a brace-delimited block of statements that reads builtins, updates maps, and prints. You can list as many of these triples in one program as you like; bpftrace compiles a separate BPF program for each probe and they run concurrently as their events fire.

flowchart LR
  SRC["Event source<br/>kprobe / fentry / tracepoint /<br/>profile / uprobe / usdt"] -->|fires| PROG["BPF program<br/>(your action block)"]
  PROG -->|/predicate/| GUARD{"predicate<br/>true?"}
  GUARD -->|no| DROP["skip"]
  GUARD -->|yes| ACT["action:<br/>@map[key] = count()/hist()<br/>printf(...)"]
  ACT -->|aggregate| MAP["BPF map<br/>(kernel-side, often PERCPU)"]
  MAP -.->|"on end / exit / print()"| USER["userspace:<br/>drain and print summary"]
  ACT -.->|"per-event printf"| RB["ring buffer<br/>(can overflow -&gt; dropped events)"]
  RB -.-> USER

The bpftrace execution model. What it shows: every event source feeds a per-probe BPF program whose body is your action block, gated by a predicate; there are then two exits — aggregation into a kernel-resident map that is drained once at the end, and per-event output through a ring buffer that userspace must keep up with. The insight to take: those two exits have completely different cost curves. The map path is O(1) per event with no userspace involvement, so a one-liner that fires millions of times per second still ships only a few kilobytes of summary. The ring-buffer path is O(1) per event in the kernel but O(events) in userspace, and when userspace falls behind, events are dropped. “Aggregate, don’t stream” is not style advice — it is the difference between the two arrows.

The contrast worth internalizing: a tool like strace stops the traced process on every syscall and copies data across the kernel boundary twice per event. bpftrace instead runs a verified BPF program in-kernel that touches a map and returns — no context switch to a tracer, no per-event userspace copy. See In-Kernel Aggregation with BPF Maps for the map mechanics this depends on, and How strace Works for the mechanism being contrasted.

Anatomy of a Program

A bpftrace program is a sequence of probe definitions with an optional leading config block. There is no main, no declarations section, and no explicit attach step. Three delivery mechanisms exist, all equivalent to the compiler (man page): bpftrace -e 'program' for an inline one-liner, bpftrace file.bt for a script (conventionally with a #!/usr/bin/env bpftrace shebang), and bpftrace - reading from standard input.

config = { missing_probes = "warn"; max_map_keys = 65536 }   <-- optional, must precede all probes

kprobe:vfs_read , kprobe:vfs_write        <-- provider:option, comma-separated list, wildcards allowed
/ pid == $1 && comm != "bpftrace" /       <-- predicate: skip the action if false
{                                         <-- action block
  $delta = nsecs - @start[tid];           <-- $var = scratch variable, block-scoped, per-invocation
  @lat[comm] = hist($delta);              <-- @map = global BPF map, survives across invocations
  delete(@start, tid);                    <-- explicit map-key removal
}                                         <-- maps are printed automatically at exit

end { clear(@start); }                    <-- special probe: runs after all probes detach

The full grammar of a bpftrace program in one annotated example, assembled from docs/language.md (v0.26.1). Medium note: this is an annotated-source figure rather than a mermaid diagram, because what is being explained is the lexical structure of the text itself — mermaid has no notation for “this token means that”. What it shows: the four syntactic layers — optional config, probe list, predicate, action — and the two variable sigils. The insight to take: the sigil is the whole scoping model. $x lives on the BPF stack for one probe invocation and is block-scoped (since v0.22, per docs/migration_guide.md); @x lives in a BPF map shared across every CPU and every invocation, and is what you accumulate into. Getting these backwards — accumulating into a $ variable — is the first mistake everyone makes, and it fails silently by simply always reading zero.

Two syntactic conveniences do a lot of work. Probe lists: kprobe:tcp_reset,kprobe:tcp_v4_rcv { ... } associates one action with several probes, “as long as the action is valid for all specified probes.” Wildcards: kprobe:tcp_* expands at compile time against the kernel’s symbol list, and kprobe:tcp_reset,kprobe:*socket* combines both forms. Wildcard expansion is where a one-liner quietly becomes a thousand-probe program, which is why max_probes exists (see §Limits).

By default every probe must attach or the program aborts. This changed in v0.24: “Previously, if a probe with multiple attach points or a wildcard failed to attach, a warning would be printed and the program would continue to run. Now, if there are any attachment failures, the program will exit with an error” (docs/migration_guide.md). The missing_probes config variable takes it back — error (default), warn, or ignore.

Probe Types — The Event Catalog

A probe is written as a provider followed by colon-separated options: provider:option1:option2. Most providers have a short alias, and kprobe:f and k:f are identical. The v0.26.1 catalog, taken from the probe table in docs/language.md:

Provider	Short	What it attaches to	Argument access	Kernel mechanism
kprobe / kretprobe	k / kr	any kernel instruction — fn, fn+offset, module:fn, or a raw address	arg0…argN (untyped int64) / retval	kprobes — int3, ftrace, or optimised jump
fentry / fexit	f / fr	kernel function entry / exit; also running BPF programs (fentry:bpf:prog_name)	typed args.field, and retval together with args on fexit	BPF trampoline (register_ftrace_direct)
tracepoint	t	a TRACE_EVENT the kernel authors declared	typed args.field	Tracepoints — static key + trace-event
rawtracepoint	rt	the same tracepoint, without field formatting	args / argN, typed from BTF	BPF_TRACE_RAW_TP
uprobe / uretprobe	u / ur	a userspace symbol, binary@file:line, or an offset	arg0…argN / retval	uprobes
usdt	U	a static probe point the application author compiled in	argN	ELF notes + semaphores
profile	p	a timer, on every CPU — profile:hz:99, profile:ms:100	—	perf_event_open sampling
interval	i	a timer, on one CPU only — interval:s:1	—	perf_event_open
software	s	kernel software perf events, ~a dozen of them	—	perf_event_open software PMU
hardware	h	PMU counter overflow, ~ten named events	—	PMU counters
watchpoint	w	a memory address, watchpoint:0xADDR:8:rw	—	hardware debug registers
iter	it	a BPF iterator over kernel objects (iter:task, iter:task_file)	typed ctx	BPF_TRACE_ITER
begin / end	—	bpftrace’s own startup and shutdown	—	none — runtime-synthesised
self	—	a signal delivered to the bpftrace process itself (self:signal:SIGUSR1)	—	none — runtime-synthesised
test / bench	—	only under --test / --bench; unit tests and microbenchmarks of bpftrace code	—	none

The complete v0.26.1 provider catalog. The insight to take: the table splits three ways that the flat list hides. Rows 1–6 are instrumentation (something in the traced code fires); rows 7–11 are sampling (a timer or counter fires, and the traced code is merely whatever happened to be running); rows 12–15 are bpftrace’s own runtime, with no kernel event behind them at all. Mixing the first two categories in one script is normal and powerful — kprobe:vfs_read plus interval:s:1 { print(@); clear(@); } is the rolling-report idiom — but the argument builtins only mean anything in the first group.

Some details that only appear in the per-provider sections and routinely surprise people:

software and hardware take a sampling count. software:faults:100 fires on every hundredth page fault, not on every one; hardware:cache-misses:1e6 fires once per million cache misses. Omit the count and “a default value is used” — an unspecified default, which is a good reason to always state it. The event names are the perf ones: cpu-clock, task-clock, page-faults/faults, context-switches/cs, cpu-migrations, minor-faults, major-faults, alignment-faults, emulation-faults, dummy, bpf-output for software; cpu-cycles/cycles, instructions, cache-references, cache-misses, branch-instructions/branches, branch-misses, bus-cycles, frontend-stalls, backend-stalls, ref-cycles for hardware, “documented in the perf_event_open(2) man page.”

profile is per-CPU; interval is not. This is the single most consequential one-word difference in the catalog. profile:hz:99 fires 99 times a second on each CPU, which is what you want for sampling. interval:s:1 fires once a second somewhere, which is what you want for reporting. Using profile for a periodic report multiplies your output by the core count.

fentry:bpf:... traces BPF programs. A capability with no kprobe equivalent, added in v0.24.0 (2025-09-17) by PR #4354 — “Add ability to attach to running BPF programs and sub-programs via fentry:bpf:prog_name or fentry:bpf:prog_id:prog_name” (CHANGELOG.md, v0.26.1): fentry:bpf[:prog_id]:prog_name attaches to a running BPF program, including bpftrace’s own. The documentation’s example is tracing one bpftrace script from another. Only programs with a BTF id can be targeted, and args is not yet available for this variant.

begin/end are lowercase now. docs/language.md at v0.26.1 lists them as begin and end; the uppercase BEGIN/END spellings still appear throughout docs/tutorial_one_liners.md in the same release and still work. Note the trap documented alongside them: “specifying an end probe doesn’t override the printing of ‘non-empty’ maps at exit” — if you want a custom final report and nothing else, you must clear() every map in end.

flowchart TD
  Q["I want to observe X"] --> A{"userspace or kernel?"}
  A -->|userspace| B{"did the author compile in<br/>a static probe point?"}
  B -->|yes| USDT["usdt:<br/>stable, author-blessed,<br/>survives recompiles"]
  B -->|no| UP["uprobe: / uretprobe:<br/>needs the symbol;<br/>binary@file:line if DWARF"]
  A -->|kernel| C{"is it a periodic sample,<br/>not an event?"}
  C -->|"yes, per-CPU profile"| PROF["profile:hz:N"]
  C -->|"yes, one-off report"| INT["interval:s:N"]
  C -->|"yes, counter-driven"| HW["hardware:event:count<br/>software:event:count"]
  C -->|no, a real event| D{"does a tracepoint exist?<br/>bpftrace -l 'tracepoint:*X*'"}
  D -->|yes| TP["tracepoint:<br/>STABLE API, typed args,<br/>survives kernel upgrades"]
  D -->|"yes, but the field<br/>formatting costs too much"| RTP["rawtracepoint:"]
  D -->|no| E{"function entry or exit only?"}
  E -->|"no -- mid-function"| KP["kprobe:fn+offset<br/>the ONLY option;<br/>untyped argN, --unsafe for raw addrs"]
  E -->|yes| F{"kernel has BTF?<br/>bpftrace --info | grep BTF"}
  F -->|yes| FE["fentry: / fexit:<br/>typed args, near-zero overhead,<br/>fexit sees args AND retval"]
  F -->|no| KE["kprobe: / kretprobe:<br/>manual casts, and you must<br/>stash args in a map for the return"]

Choosing a probe provider. What it shows: the decision is made by four questions in order — kernel or userspace, event or sample, is there a stable hook, and entry-only or mid-function. The insight to take: two of these leaves are dead ends you should notice you have reached. kprobe:fn+offset is the only way to instrument a point inside a function, and it is also the only leaf with no typed arguments and no stability guarantee at all — reaching it means accepting that the script will break. Conversely, reaching tracepoint: means you have found the one option that survives a kernel upgrade, which is why the tree checks for it before it checks anything about function boundaries.

kprobe: versus fentry: — What Actually Differs

These two providers look interchangeable in a one-liner and are not. The difference is worth a section because choosing wrong costs you either portability or correctness, and the failure is silent in both directions.

They use different kernel machinery. A kprobe: in bpftrace becomes a BPF_PROG_TYPE_KPROBE program attached through the kernel’s kprobe subsystem, which on a modern kernel means an ftrace ops carrying FTRACE_OPS_FL_SAVE_REGS — ftrace materialises a full struct pt_regs on every hit so that the handler can decode registers out of it. A fentry: becomes a BPF_PROG_TYPE_TRACING program attached through a BPF trampoline: register_fentry() in kernel/bpf/trampoline.c (v6.12) resolves ftrace_location(ip) and calls register_ftrace_direct(), an ftrace direct call that jumps straight to a JIT-generated trampoline which saves only the argument registers the program declared it needs. Alexei Starovoitov’s original posting put the difference plainly: fentry/fexit are “roughly equivalent to kprobe/kretprobe. Unlike k[ret]probe there is practically zero overhead to call a set of BPF programs before or after kernel function” (Introduce BPF trampoline, Nov 2019). BPF trampolines merged in Linux v5.5 — verified by tag, kernel/bpf/trampoline.c returns HTTP 404 at v5.4 and HTTP 200 at v5.5. See kprobes for the kprobe side and BPF Links and Attachment Lifecycle for how the link is held.

sequenceDiagram
  autonumber
  participant F as traced kernel function
  participant FT as ftrace ops<br/>SAVE_REGS
  participant TCB as trace_call_bpf()
  participant TR as BPF trampoline<br/>(JIT generated)
  participant EN as __bpf_prog_enter_recur()
  participant P as your BPF program
  Note over F,P: kprobe: path
  F->>FT: call site patched to ftrace
  FT->>FT: build a full struct pt_regs
  FT->>TCB: kprobe handler calls trace_call_bpf(call, regs)
  TCB->>TCB: cant_sleep()
  alt bpf_prog_active on this CPU is already non-zero
    TCB-->>F: return 0 — event DROPPED,<br/>bpf_prog_inc_misses_counters()
  else this CPU is free
    TCB->>P: run with ctx = pt_regs; read args via arg0..argN
    P-->>F: return
  end
  Note over F,P: fentry: path
  F->>TR: call site patched by register_ftrace_direct()
  TR->>TR: save only the declared argument registers
  TR->>EN: __bpf_prog_enter_recur(prog, run_ctx)
  alt this prog is already active on this CPU
    EN-->>F: return 0 — event DROPPED,<br/>bpf_prog_inc_misses_counter(prog)
  else this program is not already running here
    EN->>P: run with ctx = typed args; read via args.field
    P-->>F: return
  end

The two call paths at a single hit, read from kernel/trace/bpf_trace.c and kernel/bpf/trampoline.c (v6.12). What it shows: the kprobe path materialises a full struct pt_regs before it can call anything, while the trampoline path saves only the registers the program’s BTF signature says it needs. The insight to take: look at the two alt blocks — both paths have a per-CPU recursion guard that silently drops events, but the scope of the guard differs. trace_call_bpf() tests bpf_prog_active, a global per-CPU counter, with the comment “since some bpf program is already running on this cpu, don’t call into another bpf program (same or different)”; so with kprobes, one probe firing on a CPU suppresses every other BPF tracing program on that CPU for the duration. The trampoline tests prog->active, which is per-program, so only genuine self-recursion is blocked. Neither drop is visible in bpftrace’s output — both are counted into the kernel’s recursion_misses statistic, readable with bpftool prog show, not into bpftrace’s own “Lost N events”. Note also cant_sleep() on line 4: this is the kernel asserting that a tracing program may not block, which is the root of every “cannot read a non-resident page” limitation further down.

They give you different arguments. With kprobe: you get arg0, arg1, … as untyped int64 values pulled out of registers, and you cast by hand:

kprobe:vfs_open {
  printf("open path: %s\n", str(((struct path *)arg0).dentry.d_name.name));
}

With fentry: the arguments are typed from BTF and you just name them. bpftrace -lv 'fentry:tcp_reset' prints the signature it derived:

fentry:tcp_reset
    struct sock * sk
    struct sk_buff * skb

and fexit uniquely sees the arguments and the return value in the same probe, which a kretprobe cannot:

fexit:fget {
  printf("fd %d name %s\n", args.fd, str(retval.f_path.dentry.d_name.name));
}

The kretprobe equivalent requires the stash-and-retrieve dance — save the argument into a map keyed by tid on entry, read it back on return, delete the key — which is three extra lines, a map allocation, and a correctness hazard if the function can return on a different thread or not return at all.

argN on a kprobe has a footgun that args does not. From docs/language.md (v0.26.1): “Whether arguments passed on stack or in a register depends on the architecture and the number or arguments used, e.g. on x86_64 the first 6 non-floating point arguments are passed in registers and all following arguments are passed on the stack. Note that floating point arguments are typically passed in special registers which don’t count as argN arguments which can cause confusion.” The worked example is worth memorising: for void func(int a, double d, int x), the third parameter x is arg1, not arg2, because the double went to an SSE register and consumed no argN slot. Arguments past the sixth are not reachable through argN at all — you need $stack_arg0 = *(int64*)(reg("sp") + 16) and to work out the offset yourself. bpftrace cannot help, because “bpftrace does not detect the function signature so it is not aware of the argument count or their type.”

	kprobe: / kretprobe:	fentry: / fexit:
Attach point	any instruction: fn, fn+offset, module:fn, raw address (--unsafe)	function entry / exit only
Kernel mechanism	kprobe subsystem — ftrace ops with SAVE_REGS, or int3 at a non-zero offset	BPF trampoline via register_ftrace_direct()
Requires BTF	no	yes (--info lists it under “Kernel features”)
Minimum kernel	long predates bpftrace’s 6.1 floor	v5.5 for trampolines
Argument typing	untyped argN; you cast; floating-point args shift the numbering	typed args.field from BTF
Return probe sees args	no — stash in a map keyed by tid	yes — args and retval together
Per-hit cost	ftrace trampoline + full pt_regs fill	trampoline saving only the declared arguments
Attaching to 1,000 functions	batched via kprobe_multi when no module is given	one trampoline registration per function
Survives a signature change	no (and no warning)	partially — BTF makes it “more resilient against small signature changes”
Can trace a BPF program	no	yes (fentry:bpf:prog_name)

kprobe versus fentry, from docs/language.md (v0.26.1) and kernel/bpf/trampoline.c / include/linux/bpf.h (v6.12). The insight to take: exactly two rows favour kprobe, and they are the first and third. If you need a point inside a function, or you are on a kernel without CONFIG_DEBUG_INFO_BTF=y, you must use kprobe. In every other case fentry/fexit is the better instrument, and the fexit-sees-both row alone eliminates the most common source of bugs in hand-written latency scripts.

The naming history is a small trap of its own. These probes were originally called kfunc/kretfunc and were “later renamed to fentry and fexit to match how these are referenced in the kernel and to prevent confusion with BPF Kernel Functions” — a genuinely different concept, covered in BPF Kernel Functions (kfuncs). The old names still work (bpftrace issue #2835), so old scripts and old blog posts both keep circulating with the confusing spelling.

BTF and CO-RE — The Change That Made bpftrace Portable

This is the most consequential thing about modern bpftrace and the least visible, because when it works you notice nothing at all.

The problem it solved. A BPF program that reads task->pid must be compiled with a byte offset for pid inside struct task_struct. That offset depends on the kernel’s config — which fields are compiled in, how they are packed, whether CONFIG_THREAD_INFO_IN_TASK moved things. The original BCC answer was to ship a full Clang/LLVM toolchain to every machine and compile the program there, against /lib/modules/$(uname -r)/build headers, at run time. That means every production host needs kernel headers and a compiler, every tool invocation pays hundreds of milliseconds of compilation, and the whole thing breaks the moment the headers are missing or mismatched.

BPF Type Format (BTF) is the fix: a compact debug-information format describing every type in the kernel, embedded in the running kernel image itself and exposed at /sys/kernel/btf/vmlinux. CO-RE (Compile Once — Run Everywhere) is the technique built on it: the compiler emits relocations rather than fixed offsets, and libbpf patches each one at load time using the target kernel’s own BTF. bpftrace uses both, and docs/language.md (v0.26.1) is unambiguous about which to prefer: “If the kernel version has BTF support, kernel types are automatically available and there is no need to include additional headers to use them. It is not recommended to mix definitions from multiple sources (ie. BTF and header files). Prefer to exclusively use BTF as it can never get out of sync on a running system. BTF is also less susceptible to parsing failures (C is constantly evolving). Almost all current linux deployments will support BTF.”

flowchart TB
  subgraph OLD["The BCC model (headers at run time)"]
    O1["tool.py with C source"] --> O2["Clang/LLVM<br/><b>on the production host</b>"]
    O2 --> O3["/lib/modules/$(uname -r)/build<br/>kernel headers"]
    O3 --> O4["BPF object with<br/><b>hard-coded offsets</b>"]
    O4 --> O5["load"]
  end
  subgraph NEW["The BTF/CO-RE model (bpftrace v0.26.1)"]
    N1["script.bt"] --> N2["bpftrace's embedded LLVM<br/>emits CO-RE relocations"]
    N3["/sys/kernel/btf/vmlinux<br/><i>the running kernel's own types</i>"] --> N2
    N3B["/sys/kernel/btf/&lt;module&gt;<br/><i>per-module BTF, Linux 5.11+</i>"] -.-> N2
    N3C["$BPFTRACE_BTF<br/><i>override for BTF-less hosts</i>"] -.-> N2
    N2 --> N4["BPF object with<br/><b>relocation records</b>"]
    N4 --> N5["libbpf resolves each<br/>relocation against target BTF"]
    N5 --> N6["load"]
  end
  O5 -.->|"needs headers + compiler<br/>on every host"| FAIL["breaks on a<br/>header-less host"]
  N6 -.->|"needs neither"| OK["one binary,<br/>heterogeneous fleet"]

How BTF replaced run-time header compilation. What it shows: the two pipelines side by side — BCC resolving struct offsets by compiling against headers on the target machine, bpftrace emitting relocations that libbpf fixes up against the kernel’s self-describing BTF. The insight to take: the win is not compile speed, it is that the type source moved into the kernel. /sys/kernel/btf/vmlinux is generated from the same build as the running code, so it “can never get out of sync”; a headers package can be, and routinely is, the wrong version. That is why one bpftrace binary can be pushed to a fleet running four different kernels, and why incident response with bpftrace does not begin with installing a compiler.

The requirements are specific and worth checking before you assume you have it (docs/language.md, “BTF Support”):

What you want	Requirement
BTF for vmlinux (all core kernel types)	Linux 4.18+ with CONFIG_DEBUG_INFO_BTF=y
	pahole v1.13+ used at kernel build time
	bpftrace v0.9.3+, built with libbpf v0.0.4+
BTF for kernel modules	additionally Linux 5.11+ with CONFIG_DEBUG_INFO_BTF_MODULES=y
Detecting it from inside a script	the preprocessor macro BPFTRACE_HAVE_BTF is defined when BTF is found
Detecting it from the shell	bpftrace --info lists BTF under “Kernel features”
Supplying it when the kernel lacks it	the BPFTRACE_BTF environment variable points at a BTF file

BTF prerequisites at bpftrace v0.26.1. The insight to take: the two rows that bite are the module row and the last one. Module BTF is a separate config option that some distributions still leave off, so kprobe:kvm:x86_emulate_insn can fail to type its arguments on a kernel where core types work perfectly. And BPFTRACE_BTF exists precisely because “almost all” is not “all” — on an embedded or ancient kernel you can generate BTF elsewhere and point bpftrace at the file.

One migration consequence is easy to miss and appears in docs/migration_guide.md under 0.24→0.25: “using the args builtin in tracepoint requires BTF for parsing the tracepoint argument types.” Tracepoint args used to be parsed from the tracefs format file and now needs BTF. On a BTF-less host, a tracepoint one-liner that worked on bpftrace 0.24 fails on 0.25 — the one case where the move to BTF took something away. See BTF (BPF Type Format) and CO-RE (Compile Once Run Everywhere) for the mechanisms themselves.

From Text to Running Program — The Compile Path

Nothing about bpftrace’s user experience suggests a compiler, which is exactly the point: bpftrace -e '...' looks like awk. Underneath, a full compiler pipeline runs on every invocation, and knowing its stages is what turns an opaque error message into a diagnosable one.

The pipeline is a pass manager — ast::PassManager in src/main.cpp (v0.26.1) — into which passes are pushed in a fixed order and then run once. Reading src/ast/passes/parse_passes.h, the front-end alone is 27 passes, and AllParsePasses() lists them in the order they execute:

Parse                     -> ConfigPass              -> ResolveRootImports
ImportExternalScripts     -> UnstableFeature         -> Deprecated
ParseAttachpoints         -> CheckAttachpoints       -> USDTImport
ImportInternalScripts     -> LoopReturn              -> ControlFlow
MacroExpansion            -> ParseBTF                -> PreExpansionBuiltins
ProbeAndApExpansion       -> ProbePrune              -> ArgsResolver
FieldAnalyser             -> ClangParse              -> FoldLiterals
Builtins                  -> CMacroExpansion         -> MapSugar
NamedParams               -> PidFilter               -> ResolveStatementImports

The 27 front-end passes, verbatim from AllParsePasses() in src/ast/passes/parse_passes.h (v0.26.1). Medium note: an ordered code listing rather than a diagram, because the content is a linear list whose only structure is its order — a mermaid flowchart of 27 boxes in a line would carry less information than the text. What it shows: the front-end is not “parse then check”; it is two dozen small, single-purpose rewrites of the AST. The insight to take: the comment in the source explains why one ordering constraint exists — external scripts are imported, then checked for unstable features, then internal scripts are imported, “This means that internal scripts are except [sic] from the unstable feature warning.” Pass order is semantics, not tidiness. Note also where ProbeAndApExpansion sits: wildcard expansion happens at pass 16, so everything after it operates on the fully expanded probe list, which is why a kprobe:* one-liner can make the compiler itself slow.

After the front end, CreateDynamicPasses() adds five more — ClangBuild, TypeSystem, PreTypeCheck, TypeResolver, Resource — and then the LLVM back end runs Compile, LinkBitcode, optionally Verify (only with --verify-llvm-ir), Optimize, Object, ExternObject, and Link. The Resource pass is the one whose name understates it: it walks the AST and computes what the program will need at run time — how many maps, of what types, with what key and value sizes, how many format strings, how much scratch space — because BPF requires all of that declared before load.

sequenceDiagram
  autonumber
  participant U as you
  participant BT as bpftrace<br/>(front end)
  participant BTF as /sys/kernel/btf/vmlinux
  participant LL as embedded LLVM
  participant LB as libbpf
  participant K as kernel
  U->>BT: bpftrace -e 'fentry:vfs_read { @[comm]=count(); }'
  BT->>BT: Parse -> AST (lexer/parser)
  BT->>BT: ConfigPass, imports, deprecation checks
  BT->>K: read /sys/kernel/debug/tracing/available_filter_functions,<br/>/proc/kallsyms, ELF symtabs
  K-->>BT: symbol universe for wildcard matching
  BT->>BT: ProbeAndApExpansion: 'vfs_*' -> N concrete probes
  BT->>BTF: ParseBTF / FieldAnalyser: resolve args.field types
  BTF-->>BT: struct layouts, function prototypes
  BT->>BT: TypeResolver, then Resource:<br/>how many maps, what sizes, what strings
  BT->>LL: codegen: one LLVM function per probe
  LL->>LL: Optimize, then emit BPF object (ELF, in memory)
  BT->>LB: bpf_object__open_mem(elf, size)
  LB->>BTF: resolve CO-RE relocations against target BTF
  LB->>K: bpf_object__load() -> BPF_MAP_CREATE, BPF_PROG_LOAD
  K->>K: verifier: <=1,000,000 insns analysed,<br/>512-byte stack, bounded loops
  alt verifier rejects
    K-->>U: error + verifier log (log_size, default 1,000,000 bytes)
  else accepted
    K-->>LB: prog fds, map fds
    LB->>K: bpf_program__attach_*() per probe type
    K-->>BT: bpf_link fds
    BT->>U: "Attaching N probes..."
  end
  loop while running
    K-->>BT: ring buffer records (printf output)
    BT->>K: poll event-loss counter
  end
  U->>BT: Ctrl-C
  BT->>K: close links, read maps back
  BT->>U: print maps, "Lost N events" if any

The compile-load-attach path for one one-liner. What it shows: the four distinct authorities the program passes through — bpftrace’s own passes, LLVM, libbpf, and the kernel verifier — and where each can reject you. The insight to take: error messages are only interpretable if you know which of the four produced them. A message about an unknown probe comes from CheckAttachpoints and means the symbol was not in available_filter_functions; a message about a struct field comes from FieldAnalyser and usually means missing BTF; “Looks like the BPF stack limit of 512 bytes is exceeded” comes from the kernel verifier, long after bpftrace was happy, which is why no amount of re-reading your script explains it. The -d flag exposes the boundaries: -d parse, -d ast, -d types, -d codegen, -d codegen-opt, -d dis, -d libbpf, -d verifier (src/bpftrace.h, v0.26.1) — and dis and parse are compiled in only for non-NDEBUG builds, so a distribution package may not have them.

The three flags worth knowing on a bad day are -d verifier, which dumps the kernel’s rejection log; -d codegen, which prints the LLVM Intermediate Representation (IR) before optimisation; and --dry-run, which compiles without attaching. The project’s own internals document is blunt about how much of the difficulty lives here: “Codegen. This is the most difficult part of bpftrace” (docs/internals_development.md), followed by a catalogue of verifier errors — “min value is negative”, “unbounded memory access”, “invalid stack”, “expected=PTR_TO_STACK; actual=PTR_TO_MAP_VALUE”, “stack limit exceeded”, “call to ‘memset’ is not supported” — each of which is a bpftrace bug surfacing as a kernel complaint. That list is a useful map of what the abstraction is protecting you from; see eBPF Verifier for what those messages mean on the kernel side.

Uncertain

Verify: that docs/internals_development.md still describes the current codegen. Reason: the document opens with its own disclaimer — “WARNING: The information below may be out of date. The codegen tips are still helpful but some of the generated code and verifier errors might have changed.” — and its worked examples show clang-6.0 and pre-opaque-pointer LLVM IR. To resolve: regenerate the examples with bpftrace -d codegen on a current build and compare. The pass list above is not affected; that was read from src/ast/passes/parse_passes.h and src/main.cpp in the same v0.26.1 tree, not from the prose document. uncertain

Variables, Maps, and the Aggregation Primitives

This is where bpftrace earns its overhead budget, and it is the part of the language most worth learning precisely.

Two sigils, two lifetimes

docs/language.md (v0.26.1) states the split in one sentence each. A scratch variable “is kept on the BPF stack”, its name starts with $, and it “cannot be accessed outside of [its] lexical block”. A map variable uses a BPF map, its name starts with @, and it exists “for the lifetime of bpftrace itself and can be accessed from all action blocks and user-space.” Types are inferred on first assignment and are then fixed: “The data type of a variable is automatically determined during first assignment and cannot be changed afterwards.” Declaration with let is optional and initialises to 0 when no value is given.

The per-thread idiom falls straight out of this: a map keyed on tid is the standard way to carry a value from an entry probe to a return probe.

kprobe:do_nanosleep {
  @start[tid] = nsecs;                                   // stash on entry
}

kretprobe:do_nanosleep /has_key(@start, tid)/ {          // guard: did we see the entry?
  printf("slept for %d ms\n", (nsecs - @start[tid]) / 1000000);
  delete(@start, tid);                                   // free the key, or the map fills
}

The stash-and-retrieve pattern, from docs/language.md (v0.26.1). What it shows: the three obligatory parts — store keyed on tid, guard the return probe with has_key(), and delete() the key. The insight to take: the delete() is not tidiness. max_map_keys defaults to 4,096; a script that stashes and never deletes silently stops recording new keys once the map is full, and the histogram you get back is of whatever happened in the first few milliseconds. The has_key() guard matters for the symmetric reason: bpftrace attaches the entry and return probes at slightly different instants, so the first few returns can have no matching entry, and without the guard they compute a delta against zero — producing a spurious bucket at “since the epoch”.

Map value functions

Assigning the result of a map value function to a map is what makes the aggregation in-kernel. The complete v0.26.1 set, from docs/stdlib.md:

Function	Signature	What it accumulates	Backing map
count()	count_t count()	number of calls	PERCPU
sum(n)	sum_t sum(int64 n)	running total of n	PERCPU
min(n) / max(n)	min_t min(int64 n)	smallest / largest n seen	PERCPU
avg(n)	avg_t avg(int64 n)	keeps count and total; divides in userspace at print	PERCPU
stats(n)	stats_t stats(int64 n)	count, avg and sum in one value	PERCPU
hist(n[, k])	hist_t hist(int64 n[, int k])	log2 histogram, 2^k buckets per power of two, 0 <= k <= 5, default k=0	histogram map
lhist(n, min, max, step)	lhist_t lhist(...)	linear histogram: M = (max-min)/step buckets over [min,max), plus two for the under- and over-flow tails, so M+2 total	histogram map
tseries(n, interval_ns, num_intervals[, agg])	tseries_t tseries(...)	rolling time series of up to num_intervals windows; agg is avg/max/min/sum, default “last value in the interval”	time-series map

The v0.26.1 map value functions. The insight to take: the third column is doing something you would not get from an ordinary counter — hist() and lhist() build a distribution in kernel memory at O(1) per event. That is the entire reason bpftrace can answer “what is the latency distribution” on a production host: the alternative, shipping every latency sample to userspace and histogramming it there, costs a ring-buffer record per event. Note the lhist arithmetic carefully: lhist(x, 0, 100, 10) gives you 12 buckets, not 10, because values below min and above max each get their own.

tseries is behind an unstable-feature flag (unstable_tseries, default warn — docs/language.md), meaning it works but the syntax may change; plan on pinning your bpftrace version if a script depends on it.

Why @ = count() is not @++

This distinction is stated three times in docs/stdlib.md and is the single most useful piece of mechanism in the language.

flowchart TB
  subgraph RAW["@x++ — a shared BPF_MAP_TYPE_HASH slot"]
    R1["CPU 0 reads @x = 41"] --> R3["CPU 0 writes 42"]
    R2["CPU 1 reads @x = 41"] --> R4["CPU 1 writes 42"]
    R3 --> RLOST["final value 42<br/><b>one increment lost</b>"]
    R4 --> RLOST
  end
  subgraph PC["@x = count() — BPF_MAP_TYPE_PERCPU_HASH"]
    P1["CPU 0 bumps its own slot<br/>no lock, no contention"] --> PSUM
    P2["CPU 1 bumps its own slot"] --> PSUM
    P3["CPU N bumps its own slot"] --> PSUM
    PSUM["userspace reads all N slots<br/>and sums them at print time"] --> PGOOD["final value 42<br/><b>correct</b>"]
  end
  PC -.->|"but: an in-kernel read<br/>such as /@x &gt; 10/ must<br/>iterate every CPU"| COST["expensive<br/>synchronous read"]

Raw increment versus count(). What it shows: the lost-update race in @x++ and how a per-CPU map removes it by giving each CPU a private slot that only userspace ever combines. The insight to take: docs/stdlib.md states the hazard outright — “This differs from ‘raw’ writes (e.g. @++) where multiple writers to a shared location might lose updates, as bpftrace does not generate any atomic instructions for ++.” So @++ is not merely slower-and-equivalent; on a busy multi-core box it is wrong, and quietly so. The cost you pay for correctness is on the read side: any expression that needs the value in-kernel — if (@ > 10), a cast like (int64)@, a predicate — forces bpftrace to “iterate over all the cpus to collect and sum these values”. Aggregate freely; read sparingly.

The same asymmetry applies to sum() versus @ += n, and to avg(), min(), max() and stats(), all of which docs/stdlib.md describes as “thread-safe, fast writes, slow reads”. avg() is a small object lesson in the design: rather than keeping a running mean (which cannot be merged across CPUs), it keeps a count and a total and lets userspace divide — “The average is computed in user-space when printing by dividing the total by the count.”

Declaring the map type

Since the map-declaration feature (marked experimental in v0.26.1, tracked in issue #4077), you can choose the underlying kernel map type in the global scope:

let @a = hash(100);          // BPF_MAP_TYPE_HASH,          100 entries
let @b = lruhash(100);       // BPF_MAP_TYPE_LRU_HASH
let @c = percpuhash(100);    // BPF_MAP_TYPE_PERCPU_HASH
let @d = percpulruhash(20);  // BPF_MAP_TYPE_LRU_PERCPU_HASH

The single mandatory argument is max entries. The documentation’s advice is worth quoting because it names both failure directions: “it’s best practice to declare maps up front as using the default can lead to lost map update events (if the map is full) or over allocation of memory if the map is intended to only store a few entries.” The LRU variants carry their own warning — they “evict the approximately least recently used elements… Adding a single new element may cause one or multiple elements to be deleted if the map is at capacity” — so an LRU map is the right choice for a bounded cache of in-flight requests and the wrong choice for a complete histogram. See BPF Maps for the kernel-side map types themselves.

Synchronous versus asynchronous actions

A detail the documentation marks but does not dwell on: fourteen builtins in v0.26.1 are tagged async — cat, clear, errorf, exit, join, ksym, print, printf, strftime, system, time, unwatch, usym, warnf, zero. An async action does not do its work in the probe; it enqueues a request that userspace performs later. That is why they are cheap in-kernel, and it is also the source of a genuinely confusing bug documented in docs/stdlib.md under print:

Note that maps are printed by reference while scalar values are copied. This means that updating and printing maps in a fast loop will likely result in bogus map values as the map will be updated before userspace gets the time to dump and print it.

So interval:ms:1 { print(@); clear(@); } does not print a snapshot — it asks userspace to go and read @ at some later moment, by which time the clear() (also async, also queued) and thousands of further updates may have landed. The print(@); clear(@); idiom is safe at second-granularity and unreliable at millisecond-granularity, for a reason that has nothing to do with your script’s logic.

Worked Examples — Reading Real Tools

The repository ships 39 .bt scripts under tools/ (v0.26.1), most of them ports of Brendan Gregg’s BCC tools. Three of them, read line by line, teach more than any synthetic example.

runqlat.bt — scheduler run-queue latency

#ifndef BPFTRACE_HAVE_BTF        // 1  fall back to headers only where BTF is absent
#include <linux/sched.h>
#else
#define TASK_RUNNING 0           // 2  BTF has the struct but not the enum-free #define
#endif

tracepoint:sched:sched_wakeup,
tracepoint:sched:sched_wakeup_new
{
  @qtime[args.pid] = nsecs;      // 3  a task became runnable: start the clock
}

tracepoint:sched:sched_switch
{
  if (args.prev_state == TASK_RUNNING) {
    @qtime[args.prev_pid] = nsecs;   // 4  involuntarily preempted -> still runnable, restart clock
  }
  if (args.next_pid == 0) {
    return;                          // 5  the idle task is not "waiting"; drop it
  }
  $ns = @qtime[args.next_pid];       // 6  copy into a scratch var: one map read, not two
  if $ns {
    @usecs = hist((nsecs - $ns) / 1000);
    $ignore = delete(@qtime, args.next_pid);  // 7  consume the return value on purpose
  }
}

tools/runqlat.bt, v0.26.1 (comments added). What it shows: six separate design decisions packed into fifteen lines. The insight to take: line 7 is the one nobody guesses. delete() returns a boolean, and “if the return value for delete is discarded, and deletion fails, you will get a warning” (docs/stdlib.md). Here deletion failing is expected — a task can be switched to without ever having been woken while we were tracing — so the script assigns the result to a throwaway $ignore purely to suppress the warning, with the comment “Swallowing deletion failures as they are expected”. Line 1 is the second lesson: the script is written to work with and without BTF, using the BPFTRACE_HAVE_BTF preprocessor macro to decide whether to pull in linux/sched.h. And line 4 is the domain knowledge: a task that is switched out while still TASK_RUNNING was preempted, not blocked, so its run-queue clock restarts rather than stopping.

biolatency.bt versus biolatency-kp.bt — the stability argument, with a date on it

The repository ships the same tool twice. biolatency-kp.bt is the 2018 original, built on kprobes:

kprobe:blk_account_io_start,
kprobe:__blk_account_io_start
{ @start[arg0] = nsecs; }                       // arg0 == struct request *, used as a unique id

kprobe:blk_account_io_done,
kprobe:__blk_account_io_done
/@start[arg0]/
{ @usecs = hist((nsecs - @start[arg0]) / 1000); delete(@start, arg0); }

biolatency.bt is the replacement, built on tracepoints:

tracepoint:block:block_bio_queue
{ @start[args.sector] = nsecs; }

tracepoint:block:block_rq_complete,
tracepoint:block:block_bio_complete
/@start[args.sector]/
{ @usecs = hist((nsecs - @start[args.sector]) / 1000); delete(@start, args.sector); }

The header comment on the kprobe version explains why: “Note that these do not exist or are inlined on newer kernels (since kernel version 6.4) and therefore this version will not work.”

That claim is verifiable, and I verified it. Fetching block/blk-mq.c at three tags:

Tag	__blk_account_io_start / __blk_account_io_done	blk_account_io_start / blk_account_io_done
v6.3	present as static void (out-of-line, has a symbol) at lines 1002 / 979	static inline wrappers that call them
v6.4	absent	static inline only, at lines 977 / 958
v6.12	absent	static inline only, at lines 999 / 976

Verified by curl against raw.githubusercontent.com/torvalds/linux/<tag>/block/blk-mq.c on 2026-09-04. The insight to take: this is what “kprobes are not a stable API” means concretely. Between v6.3 and v6.4 the out-of-line __blk_account_io_* helpers were folded into their static inline callers. static inline functions have no symbol to attach to, so both kprobe attach points simply ceased to exist — not deprecated, not warned about, just gone in a point release. The tracepoint version keeps working because block:block_bio_queue is part of the kernel’s declared trace-event ABI. The kprobe version’s config = { missing_probes = ignore; } block is the author admitting defeat gracefully: on a modern kernel the script attaches nothing and prints an empty histogram rather than erroring out.

There is a second, subtler difference. The kprobe version keys its map on arg0 — the raw address of the struct request — because on a kprobe you get untyped registers and a pointer value is the only unique identifier available. The tracepoint version keys on args.sector, a typed field with actual meaning. Keying on a kernel pointer works but is fragile: if the request is freed and the allocator hands the same address to a different request before your completion probe fires, you record a nonsense latency. This is the general hazard of using addresses as correlation keys, and it is invisible in the output.

execsnoop.bt — the whole tool in eight lines

BEGIN { printf("%-15s %-7s %-7s %s\n", "TIME", "PID", "PPID", "ARGS"); }

tracepoint:syscalls:sys_enter_exec*
{
  printf("%15s %-7d %-7d ", strftime("%H:%M:%S.%f", nsecs), pid, ppid);
  join(args.argv);
}

tools/execsnoop.bt, v0.26.1. What it shows: the per-event streaming mode, as opposed to the aggregation mode of the previous two tools. The insight to take: the wildcard sys_enter_exec* matches both sys_enter_execve and sys_enter_execveat, so one line covers both syscalls without the author needing to know which one a given process used. join() is the specialised builtin for the char **argv shape — a null-terminated array of string pointers is not something printf("%s") can express, so the language provides a dedicated (async) helper. And note what this tool does not do: it has no map, so every exec produces a ring-buffer record. That is fine for exec, which happens tens of times a second on a normal host; the same structure applied to sys_enter_read would drop events immediately.

Ahead-of-Time Compilation

bpftrace --aot output.btaot script.bt compiles a script into a self-contained executable that carries the BPF object with it and never runs LLVM again. It exists for the case bpftrace is otherwise bad at: an embedded or resource-constrained target where you cannot afford a ~100 MB LLVM, or a hot path where the several hundred milliseconds of compile time on every invocation is the dominant cost.

The mechanism, read from src/aot/aot.cpp (v0.26.1), is deliberately unclever. bpftrace runs the normal pipeline up to CreateObjectPass() to get a BPF ELF, serialises the RequiredResources structure — the map inventory, format strings and everything else userspace needs to interpret the output — using the cereal library, concatenates a 48-byte header + resources + ELF, and then shells out to objcopy --add-section .btaot=<tempfile> <shim> <output>, cloning a prebuilt runtime shim binary called bpftrace-aotrt and injecting the payload into it as a new ELF section. Running the result makes the shim find its own .btaot section with libelf, deserialise the resources, and load the embedded ELF — no parser, no LLVM, no BTF lookup.

packet-beta
0-15: "magic (0xA07)"
16-31: "unused"
32-63: "header_len (= 48)"
64-127: "version — Robert Sedgwick hash of BPFTRACE_VERSION"
128-191: "rr_off — offset of serialised RequiredResources"
192-255: "rr_len"
256-319: "elf_off — offset of the BPF ELF object"
320-383: "elf_len"

The 48-byte .btaot section header, transcribed field-by-field from struct Header in src/aot/aot.cpp (v0.26.1), which carries a static_assert(sizeof(Header) == 48). What it shows: an intentionally minimal container — a magic number, a version gate, and two (offset, length) pairs pointing at the two payloads that follow it in the same section. The insight to take: the version field is the whole compatibility story, and the source comment says so: “We don’t worry about versioning the header b/c we enforce that an AOT compiled script may only be run with the corresponding runtime shim. We enforce it through the version field, which is the ‘Robert Sedgwicks hash’ of BPFTRACE_VERSION.” There is no forward or backward compatibility at all — an AOT binary built by v0.26.1 refuses to run on any other version’s shim. The magic field’s comment is a nice touch of paranoia: 16 bits so it “can be useful to detect endianness”.

What AOT gives up

CreateAotPasses() in src/main.cpp differs from the dynamic pipeline by exactly one extra pass at the front: CreatePortabilityPass(). That pass, src/ast/passes/portability_analyser.cpp, is a short AST visitor that rejects five things outright, and reading its comments tells you precisely why each one is impossible:

Rejected	Error text	Why
Positional parameters ($1, $2)	“AOT does not yet support positional parameters”	codegen embeds them directly into the bytecode; they are only known at run time
curtask	“AOT does not yet support accessing curtask”	struct task_struct “is unstable across kernel versions and configurations… We must block it until we support field access relocations”
kaddr(), uaddr(), cgroupid()	“AOT does not yet support <name>()”	all three resolve a name to a number during codegen and bake the number in; a cgroup id or kernel address is not portable across machines
Any struct cast	“AOT does not yet support struct casts”	field offsets would have to be relocated at load time; only args (tracepoint and fentry) is left working, because those are typed by the kernel

What the portability analyser blocks, from src/ast/passes/portability_analyser.cpp (v0.26.1). The insight to take: the last row is the load-bearing one, and it is broader than it looks — ((struct path *)arg0).dentry is a struct cast, so the entire idiomatic kprobe style is unavailable under AOT. What remains portable is tracepoints and fentry with args.field, which is not a coincidence: those are exactly the two providers whose types the kernel itself guarantees. AOT is therefore not “bpftrace, but faster to start” — it is a strictly smaller language, and the shape of what survives is another argument for the stable providers.

The pass’s own header comment sets expectations honestly: “Over time, we expect to relax these restrictions as AOT supports more features.” The restrictions exist because bpftrace’s dynamic mode resolves things at compile time on the target machine, and AOT moves compilation off that machine — every one of the five rejections is a place where “compile time” and “the machine we will run on” had been the same thing and no longer are. That is the same problem CO-RE solves for struct offsets; AOT simply has not yet adopted the same solution for the rest.

Limits, Overhead, and What You Cannot See

bpftrace’s abstraction is good enough that its limits are surprising when you hit them. This section is the honest inventory: the hard ceilings, the measured costs, the ways events go missing, and the things that are simply not observable.

The anatomy of one probe hit

stateDiagram-v2
  [*] --> EventFires: kernel/user event reaches<br/>the attach point
  EventFires --> RecursionGuard: BPF program entered
  state "Recursion guard<br/>(only if the script attaches fentry/fexit<br/>to a spin-lock internal)" as RecursionGuard
  RecursionGuard --> EarlyExit: atomic xchg found flag<br/>already set on this CPU
  RecursionGuard --> Predicate: flag was clear, so set it
  EarlyExit --> LostEvent: ++__bt__event_loss_counter
  LostEvent --> [*]
  Predicate --> Skipped: /expr/ evaluated false
  Skipped --> [*]
  Predicate --> Action: /expr/ true (or absent)
  Action --> MapUpdate: aggregate into a map<br/>with count(), hist(), sum() ...
  Action --> RingBuffer: printf, print, cat, system<br/>async, enqueued rather than executed
  MapUpdate --> Return
  state "Ring buffer submit" as RingBuffer
  RingBuffer --> RBFull: no space left,<br/>userspace is behind
  RingBuffer --> Return: record queued
  RBFull --> LostEvent2: ++__bt__event_loss_counter
  LostEvent2 --> Return
  Return --> [*]: probe returns; guard flag cleared

The state machine of a single probe invocation. What it shows: the four exits — dropped by the recursion guard, skipped by the predicate, aggregated into a map, or queued to the ring buffer — and the two distinct places an event can be lost rather than merely skipped. The insight to take: a skipped event and a lost event look identical in the output (both are simply absent) but mean opposite things. A predicate skip is your filter working. A loss is data you needed and did not get, and it is only visible because bpftrace keeps a counter for it: the per-CPU global __bt__event_loss_counter, living in the ELF section .data.event_loss_counter (src/globalvars.h, v0.26.1), which userspace polls each loop and prints as “Lost N events”, plus “Total lost event count: N” at exit (src/bpftrace.cpp, src/output/text.cpp). If you see that line, your histogram is wrong and you must not report its numbers.

The recursion guard deserves its own paragraph because it is the least-known correctness hazard in the tool. src/ast/passes/recursion_check.cpp (v0.26.1) hard-codes four kernel functions as dangerous — vmlinux:_raw_spin_lock, _raw_spin_lock_irqsave, _raw_spin_unlock_irqrestore, and queued_spin_lock_slowpath — and only for fentry/fexit, “as kprobes have kernel protections against this type of deadlock”. The comment explains the failure it prevents:

This prevents an ABBA deadlock when attaching to spin lock internal functions e.g. “fentry:queued_spin_lock_slowpath”. Specifically, if there are two hash maps (non percpu) being accessed by two different CPUs by two bpf progs then we can get in a situation where, because there are progs attached to spin lock internals, a lock is taken for one map while a different lock is trying to be acquired for the other map.

When it triggers, bpftrace warns: “Attaching to dangerous function: … bpftrace has added mitigations to prevent a kernel deadlock but they may result in some lost events.” The mitigation is the atomic exchange in the diagram above — a per-CPU flag set on entry and cleared on return; a nested hit finds it set, increments the loss counter, and returns immediately. So tracing spin-lock internals with fentry gives you a sample, not a census, and the tool tells you so only if you read the warning.

The hard ceilings

Limit	Value	Where it lives	What happens when you hit it
BPF stack	512 bytes	MAX_BPF_STACK, include/linux/filter.h:96 (v6.12)	verifier rejects: “Looks like the BPF stack limit of 512 bytes is exceeded”
Verifier instruction budget	1,000,000 analysed instructions	BPF_COMPLEXITY_LIMIT_INSNS, include/linux/bpf.h:1928 (v6.12)	verifier rejects a too-complex program
Verifier branch/state limits	8,192 jump-sequence depth; 64 states per instruction	BPF_COMPLEXITY_LIMIT_JMP_SEQ / _STATES, kernel/bpf/verifier.c:189-190 (v6.12)	same
Maps per program	64	MAX_USED_MAPS, include/linux/bpf_verifier.h:604 (v6.12)	load fails
Subprograms per program	256	BPF_MAX_SUBPROGS, include/linux/bpf_verifier.h:642 (v6.12)	load fails
BPF programs per trampoline	38 (27 on s390x)	BPF_MAX_TRAMP_LINKS, include/linux/bpf.h:1111-1113 (v6.12)	fentry attach fails when too many tracers share one function
fentry traceable arguments	12 total, of which 5 in registers	MAX_BPF_FUNC_ARGS / MAX_BPF_FUNC_REG_ARGS, include/linux/bpf.h:1039,1044 (v6.12)	arguments beyond the limit are unavailable
Map keys per map	4,096	max_map_keys, bpftrace config default	map updates silently fail; data disappears
Attached probes	1,024	max_probes, bpftrace config default	bpftrace refuses to attach; “can incur high performance overhead or even freeze/crash the system”
Generated BPF programs	1,024	max_bpf_progs, bpftrace config default	compile aborts, to stop bpftrace hanging on a wide wildcard
String length	1,024 bytes	max_strlen, bpftrace config default	strings truncate, with .. appended (str_trunc_trailer)
Object size kept on the BPF stack	32 bytes	on_stack_limit, bpftrace config default	larger objects move to pre-allocated memory (less memory-efficient, but avoids the 512-byte wall)
Ring/perf buffer	default sized from system memory; min 64 pages (256 KB), max 4,096 pages (16 MB), floor of 1 page × CPU count	perf_rb_pages, bpftrace config	events dropped when userspace falls behind
Verifier log buffer	1,000,000 bytes	log_size, bpftrace config default	a very long verifier log is truncated

Every ceiling that can end a bpftrace session, split by owner. Kernel values were read directly from the v6.12 headers by curl; bpftrace values are the documented defaults in docs/language.md at v0.26.1. The insight to take: the top six rows are the kernel’s and cannot be raised from bpftrace — they are properties of the eBPF virtual machine (see eBPF Verifier). The bottom seven are bpftrace’s own, are all settable from a config = { ... } block or a BPFTRACE_* environment variable, and every one of them is a safety valve rather than a physical limit: raising max_map_keys costs memory and startup time, raising max_probes costs system stability. The row that catches people is max_map_keys = 4,096, because exceeding it produces no error — just a quietly incomplete result.

Probe overhead, with dates attached

Hard numbers for probe cost are scarce and age badly, so treat these as ratios, not as predictions for your hardware.

The kernel’s own Documentation/trace/kprobes.rst (still shipping at v6.12) states: “On a typical CPU in use in 2005, a kprobe hit takes 0.5 to 1.0 microseconds to process… A return-probe hit typically takes 50-75% longer than a kprobe hit.” Its optimised-probe table, measured on an Intel Xeon E5410 at 2.33 GHz, reports for x86-64: unoptimised kprobe 0.99 µs, boosted 0.43 µs, optimised (jump-patched) 0.06 µs; unoptimised kretprobe 1.24 µs, optimised 0.30 µs.

Uncertain

Verify: whether these figures resemble anything on current hardware. Reason: the text explicitly dates the first set to 2005 and the optimised table to Xeon E5410-era silicon (2007), and neither has been re-measured in the document since; Spectre/Meltdown mitigations have materially changed the cost of the kernel entry paths involved. To resolve: run the kernel’s own tools/testing/selftests/bpf/benchs/bench_trigger.c (bench trig-kprobe, trig-fentry) on the machine you care about. The ratios — optimised kprobe roughly an order of magnitude cheaper than the int3 path, return probes 50–75% dearer than entry probes — are the durable part and are consistent with the mechanism described in kprobes. uncertain

On the fentry side the claim is qualitative but comes from the author of the mechanism: BPF trampolines are “roughly equivalent to kprobe/kretprobe. Unlike k[ret]probe there is practically zero overhead to call a set of BPF programs before or after kernel function” (Starovoitov, Introduce BPF trampoline, Nov 2019).

For uprobes there is one recent measured number. At LSFMM+BPF 2024, Jiri Olsa reported that replacing the int3 breakpoint used by return probes with a dedicated uretprobe() system call is “about a 31% speedup on Intel CPUs, and a 10% speedup on AMD CPUs” — a gap an audience member attributed to Intel needing both Spectre and Meltdown mitigations where AMD needs only Spectre (LWN, Faster uprobes). The asymmetry is the real lesson: userspace probe cost is dominated by the kernel-entry path, not by your BPF program, which is why uprobe: on a hot function is in a different cost class from fentry: on one. See uprobes.

What you genuinely cannot see

• Inlined and static functions have no symbol, so no kprobe and no fentry. The biolatency-kp.bt breakage above is exactly this. Check with bpftrace -l 'kprobe:name*' before assuming.
• Blacklisted functions. “Kprobes can probe most of the kernel except itself… Probing (trapping) such functions can cause a recursive trap (e.g. double fault) or the nested probe handler may never be called” (Documentation/trace/kprobes.rst, v6.12). Functions marked NOKPROBE_SYMBOL() are rejected at registration.
• notrace functions, unless you pass --unsafe: “it makes kprobe/kretprobe checks less restrictive. It can be used to probe functions that bpftrace reports as not traceable, but are supported if the kernel is configured to allow probing notrace functions” (man page, v0.26.1).
• Arguments past the sixth on a kprobe, and any argument displaced by a floating-point parameter — see the argN footgun in the kprobe versus fentry section above.
• bpftrace tracing itself. “debuggability (no gdb or self-tracing)” is an explicit language non-goal in docs/design_principles.md. The one exception is the v0.26 fentry:bpf:prog_name provider, which traces a BPF program, including another bpftrace instance’s.
• Anything that requires blocking. BPF programs in these contexts cannot sleep, so a userspace page that is not resident cannot be faulted in to be read; str() on a swapped-out pointer returns an error rather than the string.
• Events dropped by the kernel’s own recursion guard. As traced in the kprobe versus fentry diagram above, trace_call_bpf() and __bpf_prog_enter_recur() both refuse to run a program that is already active on the CPU. Those misses are counted in the kernel (bpf_prog_inc_misses_counter, surfaced as recursion_misses by bpftool prog show) and are not included in bpftrace’s own “Lost N events” total, so the only way to see them is to look outside bpftrace.
• Events during a drop window. Once the ring buffer overflows, the events are gone; there is no replay.

Finally, the design document is explicit about the boundary of the project itself. Under Language Non-Goals it lists “BPF security, LSM, XDP, Scheduling” and “BPF concepts that don’t pertain to observability or can’t be abstracted cleanly” (docs/design_principles.md, v0.26.1). bpftrace will not grow into a general eBPF language; if your program needs to change behaviour rather than observe it, you are in libbpf and C territory. See XDP Express Data Path and BPF-LSM (Security Hooks) for those neighbours.

Version Skew — What Changed and When

bpftrace breaks compatibility deliberately and documents each break. docs/design_principles.md states the policy: “We value API stability… However, due to the speed of kernel development, especially in the BPF space, we need to keep up PUSH the community forward, which means sometimes we need to break things. We prefer the stability in the sense of ‘It is heavily used in production, and when something changes, there is a clear migration path’.” In practice that means a script copied off a blog post is version-dated whether or not it says so, and docs/migration_guide.md is the file that tells you which version it belongs to.

timeline
    title bpftrace Breaking Changes by release, from CHANGELOG.md at v0.26.1
    0.22.0 / 2025-01-07 : lexical block scoping added for scratch variables (PR 3367)
                        : multi-map delete removed — one map plus one key per call (PR 3506)
                        : pid and tid builtins return uint32 rather than uint64 (PR 3441)
                        : default SIGUSR1 map dump replaced by self signal probes (PR 3522)
    0.23.0 / 2025-03-25 : -kk removed; some BPF errors surfaced by default (PR 3784)
                        : listed under Fixed, not Breaking — pid, tid and ustack become PID-namespace aware (PR 3428)
    0.24.0 / 2025-09-17 : ANY probe attach failure now errors; missing_probes config added (PR 4097)
                        : majority of DWARF support dropped, only uprobe argument parsing remains (PR 3921, 3950)
                        : rawtracepoints now require kernel BTF (PR 3944)
                        : BPF_MAP_TYPE_RINGBUF becomes a hard requirement (PR 3974)
                        : text mode moves all non-script output to stderr (PR 4504)
                        : strcontains and has_key return booleans instead of 1 and 0 (PR 4280)
    0.25.0 / 2026-03-13 : the args builtin on a tracepoint now requires BTF (PR 4864)
                        : exit() no longer allowed inside a loop (PR 4587)
                        : script licenses restricted to GPL-compatible strings (PR 4677)
                        : deprecated sarg builtin removed (PR 4686)
                        : experimental watchpoint func+arg attach points dropped (PR 4890)
    0.26.0 / 2026-05-26 : pcomm becomes an alias of task.real_parent.comm, was task.group_leader.comm (PR 5132)
                        : subtraction and decrement now produce int64 rather than uint64 (PR 5138)
    0.26.1 / 2026-06-02 : three bug fixes only — the release this note documents
    0.27-rc0 / 2026-08-13 : tagged per the releases Atom feed, not covered here

Every entry from the Breaking Changes heading of each release in CHANGELOG.md (v0.26.1), plus one entry the CHANGELOG files under Fixed. What it shows: five of the last six minor releases broke something, and the breaks cluster into three kinds — language semantics (scoping, integer types, operator results), hard kernel requirements (BTF for rawtracepoints and then for tracepoint args, ring buffer mandatory), and output contracts (stderr versus stdout, booleans versus 1/0). The insight to take: the dangerous entries are the ones that change results rather than producing an error. The PID-namespace change is the sharpest example, and it is the one row the project itself files under Fixed: from 0.23.0, pid and tid report the value inside the process’s own PID namespace rather than the initial one, so the same script run in a container prints different numbers with the same formatting and no warning. pid(init)/tid(init) restore the old behaviour but only from 0.24; the migration guide states flatly that “there are no equivalent workarounds in 0.23.x”. The 0.24.0 move of attach notifications and errors to stderr is the second: a pipeline that captured only stdout silently stopped seeing “Lost N events”. Version-pin anything you rely on.

Two source-hygiene notes on those dates. They come from CHANGELOG.md inside the v0.26.1 tarball, which is internally consistent (0.24.0 → 0.24.1 → 0.24.2 dated 2025-09-17, 2025-10-03, 2025-12-12). GitHub’s tags.atom feed disagrees for one entry — it reports v0.24.0 as 2025-10-08, later than v0.24.1’s 2025-10-03, which cannot be right and is most likely a re-tag. Where they conflict, the CHANGELOG is the record of what the project says it released; the Atom feed records when a git tag object was last touched. The feed is still the right tool for the newest tags, which is how v0.27-rc0 is dated here — it has no CHANGELOG entry yet.

The 0.22 block-scoping change is the one most likely to bite a reader of older material, because the old behaviour was undefined rather than wrong:

begin {
    if (0) { $x = "hello"; }
    print(($x));       // <= 0.21: printed an empty line (undefined behaviour)
}                      // >= 0.22: "ERROR: Undefined or undeclared variable: $x"

From docs/migration_guide.md (v0.26.1). The insight to take: the fix is let $x; or an initialisation in the outer scope — and note the second, sharper rule stated in the same entry: declaring $x in both branches of an if/else still does not make it visible afterwards, because “$x still needs to exist in the outer scope”. Shadowing is also now rejected outright. These are ordinary lexical-scoping rules, but they arrived in a language whose earlier versions did not have them.

Alternatives and When to Choose Them

The detailed head-to-head lives in bpftrace vs BCC vs ftrace; what belongs here is the decision, framed by bpftrace’s own stated goals. docs/design_principles.md (v0.26.1) lists the language goals in explicit priority order — “1. conciseness / one-liners, 2. readability / easy to understand, 3. clean abstraction from eBPF, 4. ability to quickly iterate, 5. composability, 6. good performance in both kernel and userspace, 7. speed of program initialization/start-up”. Performance is sixth. Any comparison that treats bpftrace as a performance tool competing with hand-written libbpf C has misread the project.

Tool	Reach for it when	The cost
bpftrace	ad-hoc investigation; you want an answer in under a minute; the shape of the question is “distribution of X by Y”	limited output formatting; no arbitrary userspace post-processing; needs LLVM on the host unless AOT-compiled
BCC	a tool that will be run repeatedly, needs argument parsing, custom output, or JSON; you are shipping it to others	verbose; historically shipped a run-time compiler (the Python interface was marked deprecated in favour of libbpf C on 2020-11-04, per Gregg’s eBPF page)
libbpf + C + BPF skeletons	a daemon or agent; you need full control of program structure, pinning, and lifecycle; you need a stand-alone binary with no toolchain dependency	you now own the verifier, the map plumbing, and the loader by hand — everything the bpftrace mission statement lists as what it protects you from
perf	CPU profiling, PMU counters, stack walking, or you need perf’s recording and reporting pipeline	“Difficult, not yet well documented” for BPF use (Gregg)
ftrace	the box has no BPF, no LLVM, and no packages; you need function graphs or the latency tracers	text-file interface; no in-kernel aggregation beyond histogram triggers
ply	embedded targets: “Powerful one-liners, small binary, for embedded” (Gregg) — it “emits instructions directly, whereas bpftrace uses llvm’s IR API” (Gregg, 2018)	“Limited control of code and output”; far smaller feature set and community

The front-end choice. The insight to take: the axis is not power, it is iteration speed versus packaging. bpftrace wins overwhelmingly when the program will be written once, run once, and thrown away — which is what an incident is. It loses as soon as the program acquires users, because the things a tool needs (options, stable output, no compiler on the target) are the things bpftrace’s design deliberately does not prioritise. Notice that AOT compilation is exactly an attempt to buy back the last of those without leaving the language.

flowchart TD
  Q["I need to observe something<br/>on a Linux box"] --> A{"will this program be run<br/>more than a handful of times?"}
  A -->|"no — I am in an incident<br/>or exploring"| BT["<b>bpftrace</b><br/>one-liner or short .bt script"]
  A -->|yes| B{"does it need options, custom<br/>output formats, or JSON?"}
  B -->|no| C{"is a compiler acceptable<br/>on the target host?"}
  C -->|yes| BT
  C -->|"no — embedded / minimal image"| D{"how small must it be?"}
  D -->|"small-ish"| AOT["<b>bpftrace --aot</b><br/>no LLVM at run time,<br/>but no struct casts,<br/>no curtask, no $1"]
  D -->|"tiny"| PLY["<b>ply</b><br/>emits BPF directly,<br/>no LLVM at all"]
  B -->|yes| E{"is it a long-running daemon<br/>or a shipped product?"}
  E -->|no| BCC["<b>BCC</b><br/>Python or C tool,<br/>full argument parsing"]
  E -->|yes| LIBBPF["<b>libbpf + C + skeleton</b><br/>stand-alone binary,<br/>you own the verifier"]
  Q --> F{"is BPF unavailable —<br/>old kernel, locked-down,<br/>no packages?"}
  F -->|yes| FT["<b>ftrace</b> via tracefs,<br/>or <b>perf</b>"]

Choosing between the eBPF front-ends. What it shows: the branch that actually decides is the first one — how many times the program will be run — not how powerful the tool is. The insight to take: every arrow leaving bpftrace is leaving it for a packaging reason (options, output format, no toolchain on the target, a daemon lifecycle), never because bpftrace could not express the measurement. That is the design working as intended: bpftrace’s stated goals put “conciseness / one-liners” first and “good performance” sixth, so a tool that is graduating out of bpftrace is a tool that has stopped being a one-liner. The --aot branch exists precisely to keep one of those exits closed.

Two historical points worth carrying. bpftrace is directly descended from DTrace in intent — Brendan Gregg introduced it as “shaping up to be a DTrace version 2.0: more capable, and built from the ground up for the modern era of the eBPF virtual machine” (bpftrace (DTrace 2.0) for Linux 2018) — which is why the syntax feels familiar to anyone who used DTrace and why the provider:module:function shape survives. And bpftrace was created by Alastair Robertson, not by the Netflix/Facebook engineers most associated with it; that same post credits him and describes the early internals plainly: “Internally, bpftrace uses a lex/yacc parser to convert programs to AST, then llvm IR actions, then BPF” — a description that is still structurally accurate seven years and 27 passes later.

Production Notes

Aggregate by default; stream only when the event is rare. Every production one-liner that survives contact with a busy host has the shape @[key] = count() or @ = hist(delta). printf() per event is a debugging convenience, not a production instrument: Gregg’s framing of why eBPF tooling is deployable at all rests on this — of tcplife, he writes that it “does not trace every packet like older techniques, which can add too much performance overhead. Instead, it only traces TCP session events, which are much less frequent. This makes the overhead so low we can run this tool in production, 24x7” (Learn eBPF Tracing: Tutorial and Examples). The choice of which event to trace is the overhead decision; the choice of aggregate-versus-stream is the second one.

Prefer static instrumentation, and say so in the script. Gregg’s own advice, from the bpftrace write-up: “When you write bpftrace programs, try to use the static types first, before the dynamic ones, so that your programs are more stable” (A thorough introduction to bpftrace). The biolatency pair above is the worked cost of ignoring that advice: a tool that was correct for six years stopped working at Linux 6.4 with no warning, and the fix was to rewrite it against tracepoints.

Bound your maps explicitly. Declaring let @start = lruhash(10000); up front is better than inheriting the 4,096-key default, both because the default is often too small (silent data loss) and because for in-flight-request tracking an LRU map degrades gracefully — it evicts the oldest stragglers instead of refusing all new keys.

Always read the last lines of output. “Lost N events” and “Total lost event count: N” are the difference between a measurement and a guess. So is the “Attaching to dangerous function” warning. bpftrace prints these to stderr (src/output/text.cpp), so a pipeline that captures only stdout throws away the evidence that its numbers are wrong.

Check --info before you trust a script on a new host. It reports whether the kernel has BTF, which BPF features and helpers are available, and which probe types can be used. A script that works on your workstation and produces typed args there can fall back to nothing on a distribution kernel built without CONFIG_DEBUG_INFO_BTF_MODULES=y.

Run it under systemd for long sessions. bpftrace built with -DENABLE_SYSTEMD=1 supports notify-style service startup: systemd-run --unit=bpftrace --service-type=notify bpftrace -e '...' (docs/language.md, v0.26.1), which means systemd waits until the probes are actually attached before considering the unit started — the difference between “the service is up” and “we are actually recording”.

Know the memory-lock caveat. src/main.cpp (v0.26.1) calls enforce_infinite_rlimit_memlock() before loading, with a maintainer FIXME attached: “maybe we don’t want to always enforce an infinite rlimit?”. On kernels old enough to charge BPF maps against RLIMIT_MEMLOCK rather than the memory cgroup, a bpftrace invocation therefore removes that ceiling for itself; on a memory-constrained host a wide wildcard plus a large max_map_keys can allocate a great deal before anything complains.

Understand what privilege it needs. Loading tracing programs requires CAP_BPF plus CAP_PERFMON (or root), and reading arbitrary kernel memory means bpftrace is a full kernel-read primitive — see CAP_BPF and BPF Privilege Model and Unprivileged BPF and Its Restrictions. src/main.cpp also checks kernel lockdown: if the system is in confidentiality mode, bpftrace refuses to run at all rather than failing obscurely later.

See Also

The mechanisms bpftrace compiles down to

• eBPF Verifier · The eBPF Verifier as a Static Analyzer · Reading and Debugging Verifier Errors · Verifier Complexity Limits and State Pruning
• BPF Maps · Per-CPU Maps · In-Kernel Aggregation with BPF Maps · BPF Ring Buffer
• The bpf() Syscall · libbpf and the BPF Loader · BPF Links and Attachment Lifecycle · BPF Program Types · eBPF Tracing Program Types
• BTF (BPF Type Format) · BTF and vmlinux.h · CO-RE (Compile Once Run Everywhere) · CO-RE Relocations and Field Access

The event sources behind each provider

• kprobes · kretprobes · Optimized kprobes and the Breakpoint to Jump Path · kprobe and uprobe BPF Programs
• fentry fexit and BPF Trampolines · Dynamic ftrace and the mcount fentry Hook · fprobe and the Modern Function-Probe Path
• Tracepoints · The TRACE_EVENT Macro · Tracepoint BPF Programs · Syscall Tracepoints sys_enter and sys_exit · Static Keys and Tracepoint Patching
• uprobes · USDT Userspace Statically Defined Tracing
• The perf_event_open Syscall · The Performance Monitoring Unit · Hardware vs Software Performance Events

Neighbouring front-ends and tools

• bpftrace vs BCC vs ftrace · BCC BPF Compiler Collection · BPF Skeletons and bpftool
• ftrace Framework · The Function Tracer · The Function Graph Tracer · perf Profiling Tool · trace-cmd and KernelShark
• How strace Works · ptrace and Syscall Tracing · Capturing Packets with tcpdump

Doing it safely, and the cost of doing it

• Observability Overhead and Safety · The Observer Effect in Performance Measurement · Cardinality and the Cost of Observability
• CAP_BPF and BPF Privilege Model · Unprivileged BPF and Its Restrictions · BPF and Spectre Hardening

Where bpftrace deliberately stops

• XDP Express Data Path · BPF-LSM (Security Hooks) · sched_ext and BPF-Defined Schedulers · tc and cls_bpf (Traffic Control Hooks)

Maps of content

• Linux Tracing and Observability MOC — the parent map
• Linux eBPF MOC — the virtual machine, verifier, maps, and loader this note sits on top of