vault

eprobe Event-Based Probes

13 min read

eprobe Event-Based Probes

An eprobe (“event probe”) is the odd one out among Linux dynamic events: instead of attaching to a code location the way a kprobe attaches to a kernel instruction or a uprobe attaches to a userspace instruction, an eprobe attaches to an already-existing trace event — a tracepoint or another dynamic event — and produces a new, derived event that extracts and reformats fields out of the original. It is, almost literally, “a probe on an event.” The point is reshaping: a verbose kernel tracepoint may emit a dozen fields and dereference nothing useful, but with an eprobe you can pull out exactly the two fields you care about, dereference a pointer the original only logged as an address, and emit a lean custom event — all without writing or loading any eBPF. eprobes live behind the tracefs dynamic_events file alongside kprobe-events, uprobe-events, fprobe-events, and synthetic events, and are created with the e: syntax. This note describes the v6.12 LTS implementation in kernel/trace/trace_eprobe.c: the syntax, how an eprobe hooks the source event’s trigger list, and how it copies fields out of the source record.

Uncertain

Verify: where eprobe is documented at v6.12. Reason: the v6.12 Documentation/trace/ tree (listed via the GitHub contents API at tag v6.12) contains no eprobetrace.rst and no dynamic-events.rst — both are present only in later kernels. The model’s read of v6.12 events.rst also found no eprobe/dynamic_events section. So the prose below is grounded in the source (trace_eprobe.c, trace_dynevent.c) and the shared FETCHARGS grammar from kprobetrace.rst, not a dedicated v6.12 eprobe doc. To resolve: read the eprobe doc that shipped in the kernel that first added Documentation/trace/eprobetrace.rst and back-check the syntax against v6.12 trace_eprobe.c. uncertain

Mental Model — A Filter and Reshaper That Sits On Another Event

The cleanest way to think about an eprobe is as a small, declarative transform inserted onto an event’s output. A kernel tracepoint such as sched/sched_switch or syscalls/sys_enter_openat already fires and already writes a structured record into the trace ring buffer whenever it triggers. An eprobe says: “every time that event fires, also build my event by reading these particular fields out of its record (optionally dereferencing pointers, applying a type), and only when this filter matches.” Crucially the eprobe does not instrument any code — it adds itself to the trigger list of the source event, so it runs as a side effect of the source event’s normal firing.

flowchart LR
  SRC["Source event fires<br/>(e.g. sched/sched_switch)"]
  REC["record 'rec' written<br/>(fields at known offsets)"]
  TRIG["source event's trigger list<br/>file-&gt;triggers"]
  EF["eprobe_trigger_func()"]
  GET["get_event_field():<br/>read rec + field-&gt;offset<br/>(deref pointers, apply TYPE)"]
  NEW["new derived event committed<br/>(only the fields you asked for)"]
  SRC --> REC
  REC --> TRIG
  TRIG --> EF
  EF --> GET
  GET --> NEW

How an eprobe derives a new event from an existing one. What it shows: the source event fires and writes its record rec; because the eprobe registered itself on that event’s trigger list, eprobe_trigger_func runs, reads the requested fields straight out of rec by offset (dereferencing pointers and applying the requested type), and commits a fresh, slimmer event. The insight to take: an eprobe is pure post-processing of an event that was going to fire anyway — no new code instrumentation, no eBPF, just a field-extracting transform riding the source event’s trigger hook.

This is why eprobes are described as belonging to the event layer rather than the probe layer: a kprobe creates an event by hooking an instruction; an eprobe creates an event by hooking another event. The mechanism is the event-trigger infrastructure already used for things like traceon/traceoff, repurposed to mean “build a derived event.”

The dynamic_events Interface and the e: Syntax

All of Linux’s runtime-defined trace events share one tracefs file: /sys/kernel/tracing/dynamic_events. The dispatcher lives in kernel/trace/trace_dynevent.c, whose top comment names it the “Generic dynamic event control interface.” A line written to this file is routed to a handler by its first character, via a list of registered dyn_event_operations. From the v6.12 create_dyn_event:

static int create_dyn_event(const char *raw_command)
{
    struct dyn_event_operations *ops;
    int ret = -ENODEV;
 
    if (raw_command[0] == '-' || raw_command[0] == '!')
        return dyn_event_release(raw_command, NULL);
 
    mutex_lock(&dyn_event_ops_mutex);
    list_for_each_entry(ops, &dyn_event_ops_list, list) {
        ret = ops->create(raw_command);
        if (!ret || ret != -ECANCELED)
            break;
    }
    ...
}

Two facts fall straight out of this. First, - or ! means remove: a line starting with - deletes a single named event, and !/truncating the file clears events. Second, the first letter selects the type — p/r are kprobe entry/return events, u is a uprobe event, s is a synthetic event, f is an fprobe event, and e is an eprobe. So eprobes are not a special file or API; they are one row in this shared dispatch table.

The eprobe argument grammar is documented in trace_eprobe.c itself, in the comment above __trace_eprobe_create:

/*
 * Argument syntax:
 *      e[:[GRP/][ENAME]] SYSTEM.EVENT [FETCHARGS] [if FILTER]
 * Fetch args (no space):
 *  <name>=$<field>[:TYPE]
 */

Reading this symbol by symbol: the leading e selects the eprobe handler. [:[GRP/][ENAME]] is the optional name of the new event you are creating — a group GRP and event name ENAME (if omitted, the kernel derives them). SYSTEM.EVENT names the attached (source) event — note the dot separator, e.g. sched.sched_switch or syscalls.sys_enter_openat — this is the event whose firing your eprobe rides. [FETCHARGS] is the list of fields to extract; [if FILTER] is an optional filter expression evaluated against the source event before your derived event is emitted.

The fetch-arg form <name>=$<field>[:TYPE] is the eprobe-specific part: <name> is what the field will be called in your event; $<field> references a field of the source event by name; and [:TYPE] optionally casts it (u32, x64, string, etc.). The full type and dereference vocabulary is shared with kprobe-events and is documented in kprobetrace.rst — the relevant forms are the type suffixes (u8/u16/u32/u64, the hex x8..x64, string, ustring, symbol, bitfields b<width>@<offset>/<size>, arrays type[N]) and, critically for “dereference a pointer the source only logged as an address,” the memory-dereference operator:

+|-[u]OFFS(FETCHARG)  : Fetch memory at FETCHARG +|- OFFS address

So if a source event carries a pointer field ptr, an eprobe arg like name=+0(+8($ptr)):string reads the pointer, walks to offset 8, dereferences again, and renders the result as a string — turning an opaque address in the original event into human-readable data in the derived one. That dereferencing power, applied to a field of an event you did not write, is the headline capability of eprobes.

Mechanical Walk-through — Creation, Hooking, and Field Extraction

__trace_eprobe_create parses the line above. It validates the leading e, splits out the optional new-event name, parses SYSTEM.EVENT and looks up the real struct trace_event_call for the source event, collects the fetchargs and optional filter, and allocates a struct trace_eprobe. In v6.12 that structure is:

struct trace_eprobe {
    const char *event_system;       /* attached event's system  */
    const char *event_name;         /* attached event's name     */
    char       *filter_str;         /* optional source filter    */
    struct trace_event_call *event; /* reference to source event */
    struct dyn_event devent;
    struct trace_probe tp;          /* probe def + fetchargs     */
};

event is the handle to the source event; tp (a struct trace_probe, the same type kprobe/uprobe events use) holds the parsed fetchargs and the layout of the derived event’s record.

When the derived event is enabled, the eprobe inserts itself into the source event’s trigger list. From trace_eprobe.c:

trigger = new_eprobe_trigger(ep, eprobe_file);
if (IS_ERR(trigger))
    return PTR_ERR(trigger);
 
list_add_tail_rcu(&trigger->list, &file->triggers);

file here is the source event’s per-instance trace_event_file, and file->triggers is exactly the list ftrace walks every time that event fires. The eprobe is now one of the source event’s triggers. (Disabling the derived event removes the node from this list.) Because attachment is via the trigger list and not via patching code, an eprobe is essentially free to attach and detach and adds no instrumentation overhead beyond the per-event trigger walk.

When the source event fires, ftrace walks file->triggers and calls the eprobe’s trigger callback:

static void eprobe_trigger_func(struct event_trigger_data *data,
                                struct trace_buffer *buffer, void *rec,
                                struct ring_buffer_event *rbe)
{
    struct eprobe_data *edata = data->private_data;
 
    if (unlikely(!rec))
        return;
 
    __eprobe_trace_func(edata, rec);
}

The key argument is rec — a pointer to the source event’s record as just written into the buffer. __eprobe_trace_func reserves space in the ring buffer for the derived event, copies the requested fields out of rec, and commits:

static inline void __eprobe_trace_func(struct eprobe_data *edata, void *rec)
{
    dsize = get_eprobe_size(&edata->ep->tp, rec);
    entry = trace_event_buffer_reserve(&fbuffer, edata->file,
                       sizeof(*entry) + edata->ep->tp.size + dsize);
    store_trace_args(&entry[1], &edata->ep->tp, rec, NULL,
             sizeof(*entry), dsize);
    trace_event_buffer_commit(&fbuffer);
}

The actual field read is in get_event_field, and it is gratifyingly simple — a field is just an offset into the source record:

static nokprobe_inline unsigned long
get_event_field(struct fetch_insn *code, void *rec)
{
    struct ftrace_event_field *field = code->data;
    unsigned long val;
    void *addr;
 
    addr = rec + field->offset;
 
    if (is_string_field(field)) {
        switch (field->filter_type) {
        case FILTER_DYN_STRING:
            val = (unsigned long)(rec + (*(unsigned int *)addr & 0xffff));
        ...

field->offset was learned at creation time from the source event’s format file (see Trace Event Format Files); at runtime the eprobe just adds it to rec to find the value. For string fields it follows the dynamic-string indirection; for scalar fields it reads 1/2/4/8 bytes honoring signedness. Once a field value is in hand, any further +|-OFFS(...) dereference operators in the fetcharg are applied by the shared process_fetch_insn machinery, exactly as for kprobe-events. The result is stored into the derived event’s record and committed. The reader sees a brand-new event in trace/trace_pipe containing precisely the named fields.

Configuration — Worked Examples

A minimal eprobe that reshapes a syscall tracepoint down to one field:

cd /sys/kernel/tracing
# Create event 'eopen' in group 'myg' from syscalls/sys_enter_openat,
# pulling just the 'filename' field, rendered as a string.
echo 'e:myg/eopen syscalls.sys_enter_openat file=+0($filename):ustring' >> dynamic_events
echo 1 > events/myg/eopen/enable
cat trace_pipe
# ... eopen: (...) file="/etc/passwd"

Line-by-line: e:myg/eopen creates a new event eopen in group myg; syscalls.sys_enter_openat is the attached source event; file=+0($filename):ustring defines a derived field file by taking the source field filename (a userspace pointer), dereferencing it (+0(...)), and rendering it as a userspace string. Enabling events/myg/eopen/enable is what actually inserts the trigger into sys_enter_openat’s trigger list. Removing it:

echo 0 > events/myg/eopen/enable           # detach the trigger
echo '-:myg/eopen' >> dynamic_events        # delete the eprobe (leading '-')

A second example with a filter, to show the if clause and a scalar field:

echo 'e:myg/big_switch sched.sched_switch prev=$prev_pid next=$next_pid if prev_prio < 100' \
     >> dynamic_events

This derives big_switch from sched/sched_switch, keeping only prev_pid and next_pid, and only when the source event’s prev_prio is below 100 — the filter is evaluated against the source event’s fields via the same create_event_filter(... ep->event ...) path the code uses for trace_eprobe_parse_filter.

Failure Modes and Common Misunderstandings

The most common confusion is the separator and the field namespace. The source event is SYSTEM.EVENT with a dot, not a slash — sched.sched_switch, not sched/sched_switch (the slash form is the directory path under events/, not the eprobe argument). And $field names must match the source event’s exact field names as listed in its format file; a typo yields a creation error, not a silent empty field. Inspect events/<sys>/<event>/format first.

A second pitfall is dereferencing the wrong pointer flavor. A field that holds a kernel pointer needs a kernel-memory dereference; a field holding a userspace pointer (like a syscall path argument) needs the u-prefixed form (+u0(...) / :ustring). Using the wrong one reads garbage or faults the access (the kernel’s probe-read machinery catches the fault and reports (fault) rather than crashing, but the data is useless).

A third subtlety: an eprobe rides the source event’s trigger, so the source event must actually be enable-able and firing. If you derive from a tracepoint that is compiled out or never triggers in your workload, your eprobe produces nothing — there is no code path to hook, because there is no independent code path; the eprobe has none of its own. This is the structural difference from a kprobe, which creates its own attach site.

Finally, note the documentation gap flagged at the top: at v6.12 there is no dedicated eprobe doc file, so the authoritative reference for syntax is the trace_eprobe.c source comment and the shared kprobetrace.rst fetcharg grammar.

Alternatives and When to Choose Them

Against an eBPF program (bpftrace): an eprobe is the right tool when all you want is to reshape or filter an existing tracepoint into the exact fields you need, with no toolchain, no compiler, no BPF verifier, and no loaded program — just a line echoed into a file. It cannot compute, aggregate, or maintain state; the moment you need a histogram, a counting map, or arbitrary logic, reach for bpftrace/BCC and BPF map aggregation instead. Against a kprobe-event: choose a kprobe-event when no suitable tracepoint exists and you must hook a code location directly; choose an eprobe when a tracepoint already exposes the data and you only want a cleaner, narrower view of it (eprobes are also more stable, since they ride a maintained tracepoint rather than an internal symbol). Against triggers: if you only need to filter a tracepoint, the event’s own filter file suffices; the eprobe earns its keep when you additionally want to extract specific fields and dereference pointers into a slimmer derived event. Against synthetic events: a synthetic event combines fields from two events (e.g. start and end) keyed by a value; an eprobe transforms one event into a derived one — different jobs.

Production Notes

eprobes shine in two situations. The first is trimming a noisy tracepoint in a production trace: a high-frequency tracepoint with many fields floods the ring buffer, but an eprobe selecting two fields with a filter cuts the volume dramatically while keeping the records you need — all without recompiling or attaching BPF, which matters on locked-down hosts where loading BPF is restricted but tracefs is available. The second is surfacing data hidden behind a pointer: many tracepoints log a pointer (a struct address) rather than its contents; an eprobe’s dereference fetchargs (+OFFS(...), :string) expose the actual fields without anyone having to patch the original TRACE_EVENT macro.

Uncertain

Verify: whether v6.12 eprobes can attach to another dynamic event (e.g. a kprobe-event) and not only to static tracepoints, and whether attaching to an fprobe-event is supported at 6.12. Reason: trace_eprobe.c resolves any registered trace_event_call, which structurally permits chaining, but the fetched primary sources did not show a worked v6.12 example of an eprobe-on-a-kprobe-event. To resolve: test e:g/x kprobes.mykprobe ... on a 6.12 box, or find an upstream test/selftest exercising eprobe-on-dynamic-event at that tag. uncertain

See Also

Graph View

Backlinks