vault

ftrace Filtering and Triggers

15 min read

ftrace Filtering and Triggers

A bare function tracer or a flood of enabled trace events is a firehose — far more data than any ring buffer can hold and far more than a human can read. ftrace’s filtering and trigger machinery is how you narrow that firehose to exactly the events you care about, and then act on them, entirely inside tracefs and without writing a line of eBPF. There are three distinct layers, often confused. Function filters (set_ftrace_filter/set_ftrace_notrace) restrict which kernel functions the function tracer follows, using glob patterns and module scoping. Event filters (a per-event filter file) attach a predicate expression over an event’s fields so a sched_wakeup event is only recorded when, say, prio < 100. Event triggers (a per-event trigger file) fire an action when an event hits — take a stacktrace, grab a snapshot, toggle traceon/traceoff, enable another event, or aggregate into an in-kernel hist histogram — optionally gated by the same predicate syntax. Together they make tracefs a surprisingly capable conditional-tracing and lightweight-aggregation engine.

This note covers all three layers and how they compose, pinned to LTS 6.12. The underlying patch mechanism that makes function filtering cheap lives in Dynamic ftrace and the mcount fentry Hook; the event records and format files the predicates run over live in The Trace Event Subsystem; the filesystem surface is The tracefs Filesystem. When the in-kernel aggregation you need outgrows hist triggers, the next step up is eBPF — see the cross-links at the end.

Mental Model — Three Knobs, Two Domains

The single biggest source of confusion is that “filter” means two different things depending on which tracer you are driving. The function tracer operates on functions — its filter is a list of function names, narrowed by set_ftrace_filter. The trace-event subsystem (tracepoints, syscall events, kprobe events) operates on events with fields — its filter is a predicate expression on those fields, set in a per-event filter file. They are unrelated code paths that happen to share the word “filter.” Triggers belong to the event domain: they are actions bolted onto an event, fired when (after filtering) that event occurs.

flowchart TB
  subgraph FN["Function domain (function tracer)"]
    AFF["available_filter_functions<br/>(every traceable function)"]
    SFF["set_ftrace_filter<br/>(allow-list, globs, :mod:)"]
    SFN["set_ftrace_notrace<br/>(deny-list)"]
    PID1["set_ftrace_pid /<br/>set_ftrace_notrace_pid"]
    AFF --> SFF --> TRACED["functions actually patched<br/>to call ftrace_caller"]
    SFN -.->|"subtract"| TRACED
    PID1 -.->|"per-task gate"| TRACED
  end
  subgraph EV["Event domain (trace events)"]
    ENABLE["events/.../enable<br/>or set_event"]
    FILT["events/.../filter<br/>(predicate on fields)"]
    TRIG["events/.../trigger<br/>(action: stacktrace,<br/>snapshot, traceoff,<br/>enable_event, hist...)"]
    PID2["set_event_pid"]
    ENABLE --> FILT --> RECORD["record kept?"]
    RECORD -->|"yes"| TRIG
    PID2 -.->|"per-task gate"| FILT
  end

The two filtering domains and where triggers sit. What it shows: function filtering subtracts a notrace deny-list from a glob-matched allow-list to choose which functions get patched in; event filtering evaluates a field predicate to decide whether each event record is kept, and only surviving events fire their triggers. The insight to take: set_ftrace_filter chooses functions by name; an event filter chooses records by field value — never mix them up. Triggers are the event domain’s way to do something other than “write to the ring buffer.”

Function Filters — set_ftrace_filter and set_ftrace_notrace

The function tracer can, in principle, trace every kernel function — available_filter_functions “lists the functions that ftrace has processed and can trace” (ftrace.rst, v6.12). In practice you almost always want a handful. set_ftrace_filter is an allow-list: “Echoing names of functions into this file will limit the trace to only those functions.” Its counterpart set_ftrace_notrace is a deny-list, and the deny-list wins: “If a function exists in both set_ftrace_filter and set_ftrace_notrace, the function will not be traced.”

The crucial property is that writing these files changes which call sites get patched (see Dynamic ftrace and the mcount fentry Hook): narrowing the filter shrinks the set of functions whose nop is rewritten into a call, so a tight filter is not just less output — it is less overhead, because untraced functions stay nops.

Filtering supports glob patterns so you do not enumerate names by hand. Writing sched* matches every function whose name starts with sched; *lock* matches any function with lock in its name; *sh matches names ending in sh. Append with >> to add to the current set, or overwrite with >; write an empty string to clear:

# Trace only the scheduler's pick/switch family
echo 'pick_next_task*' > /sys/kernel/tracing/set_ftrace_filter
echo '__schedule'    >> /sys/kernel/tracing/set_ftrace_filter
 
# Trace everything except the spinlock fast paths (too noisy)
echo '*spin_lock*' > /sys/kernel/tracing/set_ftrace_notrace
 
# Clear the filter (back to all functions)
echo > /sys/kernel/tracing/set_ftrace_filter

To scope to a module, use the :mod: syntax, which restricts a glob to functions belonging to one module — echo ':mod:ext4' > set_ftrace_filter traces only ext4’s functions (ftrace.rst, v6.12). This matters because module function names are not unique across the kernel and a bare glob could match unrelated functions.

The same allow/deny model exists for the function-graph tracer via set_graph_function and set_graph_notrace: “Functions listed in this file will cause the function graph tracer to only trace these functions and the functions that they call” — i.e. it traces a whole subtree rooted at the named function, not just the function itself.

Per-Task Filtering — set_ftrace_pid

Function tracing can also be gated by task. Writing thread IDs into set_ftrace_pid limits the function tracer to those tasks; set_ftrace_notrace_pid excludes tasks, and the exclusion takes precedence. The function-fork option propagates the filter across fork(): “When set, tasks with PIDs listed in set_ftrace_pid will have the PIDs of their children added to set_ftrace_pid when those tasks fork. Also, when tasks with PIDs in set_ftrace_pid exit, their PIDs will be removed from the file” — and the same propagation applies to set_ftrace_notrace_pid (ftrace.rst, v6.12). This is how you trace a process and its descendants without manually chasing fork.

Function Filter Commands — Acting at a Function

set_ftrace_filter does more than select functions: appending a colon and a command turns a function into a trigger. The syntax is function:command:count, where the optional count limits how many times the command fires before deactivating (ftrace.rst “Filter commands”, v6.12). The commands include traceon/traceoff (start/stop tracing when the function is hit), snapshot, stacktrace, dump/cpudump, enable_event/disable_event, and mod. Two canonical examples from the docs:

# Start tracing the moment schedule() runs
echo 'schedule:traceon' > /sys/kernel/tracing/set_ftrace_filter
 
# Stop tracing the first 3 times do_fault() runs, then stop firing
echo 'do_fault:traceoff:3' > /sys/kernel/tracing/set_ftrace_filter

This is the function-tracer analogue of event triggers (below): it lets you catch a transient condition (“freeze the buffer when this rare function fires”) without polling from userspace.

Event Filters — Predicates on Event Fields

The trace-event subsystem is a different world. Each event (a tracepoint like sched/sched_wakeup, a syscall event, a dynamic kprobe event) emits a structured record whose fields are described by its format file. Every event directory also has a filter file holding a predicate expression; the event is only recorded when the predicate is true. The generic filter engine “implements a generic filtering system for Linux kernel trace events… converts filter expressions into an optimized program that evaluates predicates against event data” (trace_events_filter.c, v6.12).

The operator set depends on the field type (events.rst, v6.12):

  • Numeric fields: ==, !=, <, <=, >, >=, and & (bitwise AND, for testing flag bits).
  • String fields: ==, !=, and ~ — where ~ is a glob match, accepting * wildcards (e.g. prev_comm ~ "*sh" matches any command ending in sh, prev_comm ~ "sh*" matches any starting with sh).

Predicates combine with the logical operators && and ||, negate with !, and group with parentheses: “A filter expression consists of one or more ‘predicates’ that can be combined using the logical operators && and ||.” Concrete examples straight from the documentation:

# Only record wakeups where the preempt count was high
echo 'common_preempt_count > 4' \
  > /sys/kernel/tracing/events/sched/sched_wakeup/filter
 
# A compound predicate with grouping
echo '((sig >= 10 && sig < 15) || sig == 17) && comm != bash' \
  > /sys/kernel/tracing/events/signal/signal_generate/filter
 
# Clear the filter
echo 0 > /sys/kernel/tracing/events/sched/sched_wakeup/filter

Internally predicate_parse() compiles the infix expression into a small jump-threaded program: a first pass converts to an imperative form with branch targets handling precedence and parentheses via a stack, a second pass collapses redundant jumps, and a third applies inversion masks. At event time filter_match_preds() runs that program against the record, short-circuiting between predicates until it reaches a TRUE/FALSE verdict (trace_events_filter.c, v6.12). The compilation matters for overhead: a complex filter on a hot event is evaluated as a tight predicate program, not re-parsed each time, but it still runs per event, so a cheap filter on a screaming event still costs. String-field matching supports several layouts — static fixed-size arrays, dynamic variable-length strings, and char * pointers in kernel or user space — each with front/end/middle/full glob modes.

A useful subtlety: errors in a filter expression are reported with position information (the filter_parse_error machinery), and reading the filter file back shows the parsed filter — so a malformed predicate is rejected at write time rather than silently dropping events.

Event Selection and Per-Task Event Filtering

Before a filter matters, the event must be enabled — write 1 to the event’s enable file, or use the central set_event file: echo sched_wakeup >> set_event enables one event, !sched_wakeup disables it, 'irq:*' enables a whole subsystem, and *:* enables everything (events.rst, v6.12). Per-task event filtering uses set_event_pid: write PIDs into it and only those tasks’ events are recorded — echo $$ > set_event_pid then echo 1 > events/enable traces only the current shell’s events. Additional PIDs are appended with >>. This is the event-domain analogue of set_ftrace_pid.

Event Triggers — Acting When an Event Fires

A trigger turns an event from “record into the buffer” into “do something.” Each event directory has a trigger file, and the general syntax is command[:count] [if filter]: an optional firing count, and an optional if clause that is the same predicate language as event filters. Removing a trigger prepends ! (events.rst, v6.12). The supported commands, from trace_events_trigger.c (authored by Tom Zanussi, 2013) and the docs:

  • traceon / traceoff — start or stop global tracing when the event fires; traceoff is a post-trigger (runs after the event is written). Both accept :N to fire only N times.
  • snapshot — swap the main buffer into the snapshot buffer (requires CONFIG_TRACER_SNAPSHOT); :N limits firings.
  • stacktrace — dump a kernel stack trace into the buffer at the point the event fired (requires CONFIG_STACKTRACE); also a post-trigger.
  • enable_event / disable_event — turn another event on or off, syntax enable_event:system:event[:N]. This is how you build conditional cascades: event A, when it fires, enables event B.
  • hist — aggregate into an in-kernel histogram (next section).

Each accepts the conditional if <filter> suffix, so triggers fire only on matching events. Worked examples from the documentation:

# Dump a stack trace, but only for large allocations, max 5 times
echo 'stacktrace:5 if bytes_req >= 65536' \
  > /sys/kernel/tracing/events/kmem/kmalloc/trigger
 
# Freeze the buffer once, when a block device sees a long unplug queue
echo 'traceoff:1 if nr_rq > 1' \
  > /sys/kernel/tracing/events/block/block_unplug/trigger
 
# Take one snapshot under the same condition
echo 'snapshot:1 if nr_rq > 1' \
  > /sys/kernel/tracing/events/block/block_unplug/trigger
 
# When a read syscall is entered, enable the kmalloc event once
echo 'enable_event:kmem:kmalloc:1' \
  > /sys/kernel/tracing/events/syscalls/sys_enter_read/trigger
 
# Remove a trigger by prefixing '!'
echo '!stacktrace:5 if bytes_req >= 65536' \
  > /sys/kernel/tracing/events/kmem/kmalloc/trigger

Mechanically, each trigger registers a struct event_command (with name, trigger_type, parse(), reg()/unreg(), get_trigger_ops(), set_filter()), and the per-trigger event_trigger_ops supplies .trigger(), .print(), .init(), .free() (trace_events_trigger.c, v6.12). The traceoff/stacktrace “post-trigger” distinction exists because they must run after the record is committed — you want the stack trace to appear after the event line, and traceoff must not drop the very event that triggered it.

Histogram Triggers — In-Kernel Aggregation Without eBPF

The most powerful trigger is hist, which “can be used to aggregate trace event data into histograms” by “collecting kernel trace events into in-memory hash tables” (histogram.rst, v6.12). Instead of shipping every event to userspace and counting there, the kernel keeps a hash table keyed by chosen fields and sums chosen values in place — the classic “count by key” you would otherwise write a BPF map for. The syntax:

hist:keys=<field1[,field2,...]>[:values=<field1[,...]>]
  [:sort=<field1[,...]>][:size=#entries][:pause][:continue]
  [:clear][:name=histname1][:nohitcount]

The semantics: “When a matching event is hit, an entry is added to a hash table using the key(s) and value(s) named… Values must correspond to numeric fields — on an event hit, the value(s) will be added to a sum kept for that field.” If you give no values, the table just counts hits per key. Field names take modifiers appended with a dot: .hex (display as hex), .sym/.sym-offset (resolve an address to a symbol), .execname (show a PID as its program name), .log2 (bucket by power of two — the trick for latency histograms), and .buckets=N (group into fixed-size buckets). Worked examples verbatim from the docs:

# How many bytes each kmalloc call-site requests, bucketed by 32
echo 'hist:key=call_site.hex:val=bytes_req.buckets=32' \
  > /sys/kernel/tracing/events/kmem/kmalloc/trigger
 
# Same, but resolve call-site to a symbol and sort by total bytes desc
echo 'hist:key=call_site.sym:val=bytes_req:sort=bytes_req.descending' \
  > /sys/kernel/tracing/events/kmem/kmalloc/trigger
 
# Count read() syscalls per process name, busiest first
echo 'hist:key=common_pid.execname:val=count:sort=count.descending' \
  > /sys/kernel/tracing/events/syscalls/sys_enter_read/trigger

You read the result from the event’s hist file: “keys are printed first and are delineated by curly braces, and are followed by the set of value fields for the entry.” Multiple hist triggers on the same event coexist and can share data if given the same :name= — “If a hist trigger is given a name using the ‘name’ parameter, its histogram data will be shared with other triggers of the same name,” which is the basis for cross-event correlation (a hist on a start event and a hist on an end event sharing a key). Remove a histogram by prefixing ! to its exact command. The histogram engine (trace_events_hist.c) is also the foundation of synthetic events and inter-event latency tracking, where a value captured at one event is matched against another — a deeper topic that belongs in its own note.

Failure Modes and Common Misunderstandings

The most common mistake is confusing the two filter domains: trying echo 'pid == 100' > set_ftrace_filter (function tracer wants a function name, not a predicate) or echo 'schedule' > events/sched/sched_switch/filter (an event filter wants a field predicate, not a function name). Both fail, often confusingly, because the file accepts text but rejects it on parse.

A second trap is forgetting if filters reuse the event filter parser — a typo in the field name inside stacktrace:5 if byte_req >= 65536 (note the missing s) is rejected because byte_req is not a field of kmalloc. Read the file back to confirm the trigger took.

For histograms, the silent footgun is size=: the hash table is bounded, and once it fills, new keys are dropped, so a high-cardinality key (e.g. raw addresses without .sym) can overflow the default table and undercount. The hist file reports dropped entries; if your counts look low, raise size or reduce key cardinality.

Finally, overhead is per-event, not free. An event filter or hist trigger on a multi-million-events-per-second tracepoint runs its predicate/aggregation on every hit. ftrace’s in-kernel evaluation is cheap relative to shipping each event to userspace, but it is not zero — a histogram on sched_switch system-wide is measurable. This is exactly the threshold at which eBPF’s per-CPU maps and richer programs become the better tool.

Alternatives — When to Reach Past tracefs

These tracefs facilities are the right tool when you want conditional tracing and simple aggregation with no toolchain — they are always present, need no compiler, and survive in minimal/embedded environments. Their limits define when to move up:

  • bpftrace / BCC. When you need arbitrary in-kernel logic (per-CPU maps, multiple aggregations correlated across events, custom math, reading complex structures), an eBPF program is far more expressive than a hist trigger and aggregates per-CPU to cut contention. Choose tracefs hist for a one-off “count X by Y”; choose bpftrace when the aggregation has logic.
  • perf. For sampling profiles and PMU counters rather than per-event filtering, perf is the right front-end; it can also apply filters to the events it records.
  • trace-cmd / KernelShark. A friendlier driver over these same tracefs files — it sets filters and triggers for you and records to a .dat file. The semantics are identical; only the UI differs.

Production Notes

The decisive production advantage of tracefs filtering is dependency-freedom and safety: there is no JIT, no verifier, no compiler — just file writes — so it works on locked-down or ancient kernels where eBPF is unavailable or disabled, and it cannot crash the kernel the way an unsafe out-of-tree module could. The standard production pattern is a tight loop: enable one event, attach a narrow filter, add a traceoff:1 if <rare condition> trigger, run the workload, and let the buffer freeze itself the instant the rare condition occurs — capturing the lead-up without a userspace poller racing the event. For “how often / how big / by whom” questions, a single hist trigger answers in-kernel what would otherwise be a BCC script. The escalation path is well understood: start with set_ftrace_filter and an event filter to cut volume, add triggers to act on conditions, reach for a hist trigger to aggregate, and only graduate to bpftrace when the logic exceeds what a histogram modifier can express.

See Also

Graph View

Backlinks