vault

Housekeeping CPUs and Tickless Isolation

12 min read

Housekeeping CPUs and Tickless Isolation

When some CPUs are isolated for latency-critical work, the routine kernel chores those CPUs would normally have done — running unbound timers, servicing unbound workqueues, processing Read-Copy-Update (RCU) callbacks, hosting unpinned kernel threads, and keeping the system clock advancing — do not vanish; they are pushed onto the CPUs that remain un-isolated. Those remaining CPUs are the housekeeping CPUs. The kernel models “which CPUs may do which category of background work” with a set of per-category cpumasks indexed by an enum hk_type of housekeeping types (HK_TYPE_*), built once at boot from the isolcpus= and nohz_full= parameters and read all over the kernel via housekeeping_cpu() and friends (isolation.c, v6.12; isolation.h, v6.12). Layered on top is full-dynticks (adaptive-ticks) operation: an isolated CPU running exactly one task can stop its periodic scheduler tick entirely, but only because a housekeeping CPU keeps the global timekeeping tick alive on its behalf — which is why nohz_full mandates at least one housekeeping CPU and forbids you from making every CPU tickless. This note covers that machinery: the HK_TYPE_* flags (and the 6.14-era refactor that collapsed nine of them into three), the tick_nohz_full_mask, the “can the tick stop” decision, the timekeeping-CPU requirement, and the residual ~1 Hz remote tick.

This note pins behaviour to Linux 6.12 LTS (released 2024-11-17) and 6.18 LTS (released 2025-11-30), per kernel.org/releases. The HK_TYPE_* enum differs materially between these two releases; the difference is documented below. For the user-facing boot parameters that feed this machinery, see the sibling note CPU Isolation isolcpus and nohz_full.

Mental Model — A Pool of Absorber CPUs

The way to picture housekeeping is as a small pool of “absorber” CPUs that soak up everything the isolated CPUs are refusing to do. The kernel does not have one global “isolated” flag; instead, for each category of background work it keeps a cpumask of CPUs allowed to do it. An isolated CPU is simply one that has been cleared from one or more of those masks.

flowchart LR
  BOOT["Boot params:<br/>isolcpus= / nohz_full="] --> SETUP["housekeeping_setup()<br/>builds per-type cpumasks"]
  SETUP --> MASKS["housekeeping.cpumasks[HK_TYPE_*]<br/>one mask per work category"]
  MASKS --> Q1{"housekeeping_cpu(cpu, TYPE)?"}
  Q1 -->|yes| HK["Housekeeping CPU:<br/>runs unbound timers, WQ,<br/>RCU callbacks, kthreads,<br/>keeps the timekeeping tick"]
  Q1 -->|no| ISO["Isolated CPU:<br/>that category of work<br/>is steered elsewhere"]
  HK --> TK["tick_do_timer_cpu<br/>one HK CPU owns timekeeping"]
  ISO --> DYN["May enter full-dynticks<br/>if sched_can_stop_tick()"]
  DYN -.residual 1Hz.-> HK

The housekeeping dispatch. What it shows: boot parameters are parsed once into a set of per-category cpumasks; every kernel call site that wants to schedule background work asks housekeeping_cpu(cpu, HK_TYPE_X) “may this CPU do work of category X?” and routes the work to a housekeeping CPU if the answer is no. One housekeeping CPU (tick_do_timer_cpu) additionally owns global timekeeping. The insight to take: isolation is not a single boolean — it is per-category masking, which is exactly why nohz_full (tick/RCU/timer categories) and isolcpus=domain (the load-balancing-domain category) are independent knobs, and why the kernel can keep one CPU ticking for timekeeping while the rest go silent.

The HK_TYPE Flags — and the 6.14 Consolidation

The categories are an enumeration, enum hk_type, in include/linux/sched/isolation.h. In Linux 6.12 LTS there are nine distinct categories (isolation.h, v6.12):

  • HK_TYPE_TIMER — unbound (non-pinned) kernel timers.
  • HK_TYPE_RCU — RCU callback processing / grace-period kthreads.
  • HK_TYPE_MISC — miscellaneous offloadable work.
  • HK_TYPE_SCHED — scheduler statistics work.
  • HK_TYPE_TICK — the periodic scheduler tick (full-dynticks).
  • HK_TYPE_DOMAIN — membership in scheduler load-balancing domains.
  • HK_TYPE_WQ — unbound workqueue work.
  • HK_TYPE_MANAGED_IRQ — kernel-managed device interrupts.
  • HK_TYPE_KTHREAD — placement of unpinned kernel threads.

Each call site reads the relevant mask through the inline helpers in the same header: housekeeping_cpu(cpu, type) returns true if cpu may do work of type; housekeeping_cpumask(type) returns the whole allowed mask (defaulting to cpu_possible_mask when isolation is not active); housekeeping_any_cpu(type) returns some housekeeping CPU to offload to (NUMA-closest where possible); and housekeeping_affine(task, type) pins a kthread to the housekeeping set (isolation.c, v6.12). All of this is gated behind a static_branch (housekeeping_overridden) so that on a system with no isolation the helpers compile down to “true” / “all CPUs” with zero runtime cost.

Version change — flags consolidated in 6.14

In Linux 6.18 LTS the enum is different. A patch series by Waiman Long (Red Hat), Acked-by Frederic Weisbecker, merged in the 6.14 cycle (committed to the scheduler tree 2024-12-02; verified present in v6.14 and absent in v6.13 via the GitHub commit history of isolation.h), did two things (lore.kernel.org/…125248-4; lore.kernel.org/…125248-2):

  1. Removed HK_TYPE_SCHED — it “is defined but not set anywhere. So any code that tries to use HK_TYPE_SCHED are essentially dead code.”
  2. Collapsed the six “nohz_full-only” types into one HK_TYPE_KERNEL_NOISE. Because HK_TYPE_TICK, HK_TYPE_TIMER, HK_TYPE_RCU, HK_TYPE_MISC, HK_TYPE_WQ, and HK_TYPE_KTHREAD are only ever set together by nohz_full= (or isolcpus=nohz), their cpumasks were always identical, so there was no point storing nine. The number of cpumasks “drops from 9 to 3.”

The result in 6.18 is just three live categories: HK_TYPE_DOMAIN, HK_TYPE_MANAGED_IRQ, and HK_TYPE_KERNEL_NOISE — with the old names kept as compatibility aliases all equal to HK_TYPE_KERNEL_NOISE (isolation.h, v6.18). The commit message stresses “there should be no other functional change” — old call sites using HK_TYPE_TICK still work, they just resolve to the same mask. If you are reading scheduler code, the takeaway is: don’t assume the tick, RCU, and timer housekeeping CPUs can differ — since 6.14 they are by construction the same set.

HK_TYPE_DOMAIN and HK_TYPE_MANAGED_IRQ survived as distinct types in both releases precisely because they are settable independently by isolcpus= (the domain and managed_irq flags) without touching the tick/RCU/timer group.

How the Masks Are Built at Boot

All three masks come out of one function, housekeeping_setup() in kernel/sched/isolation.c, called once per relevant boot parameter (isolation.c, v6.12). The flow:

  1. The parameter’s CPU list (e.g. 2-7) is parsed into non_housekeeping_mask — the CPUs to isolate.
  2. The housekeeping set is the complement: cpumask_andnot(housekeeping_staging, cpu_possible_mask, non_housekeeping_mask) — every possible CPU minus the isolated ones.
  3. A guard ensures at least one housekeeping CPU exists. The code computes first_cpu = cpumask_first_and(cpu_present_mask, housekeeping_staging); if that comes back empty (you tried to isolate everything), it force-adds the boot CPU back into housekeeping and removes it from the isolated set, warning "Housekeeping: must include one present CPU, using boot CPU:%d".
  4. For each housekeeping type implied by this parameter’s flags, the staging mask is copied into housekeeping.cpumasks[type].
  5. If two parameters are given (nohz_full= and isolcpus=nohz), the second call checks the overlapping types match and rejects a mismatch with "Housekeeping: nohz_full= must match isolcpus=".

housekeeping_nohz_full_setup() (the nohz_full= parser) sets the tick/timer/RCU/misc/wq/kthread group; housekeeping_isolcpus_setup() defaults to HK_FLAG_DOMAIN and adds the tick group only if the nohz sub-flag is present (isolation.c, v6.12). The exact flag-to-parameter mapping (and the [Deprecated] status of isolcpus) lives in CPU Isolation isolcpus and nohz_full; here the point is simply that the masks are derived this way, and that the “at least one housekeeping CPU” invariant is enforced structurally in step 3 and re-checked in housekeeping_init(), which WARN_ON_ONCEs if any housekeeping mask is empty.

Full-Dynticks: tick_nohz_full_mask and the Timekeeping CPU

The set of CPUs eligible to run tickless is tick_nohz_full_mask, a cpumask exported from kernel/time/tick-sched.c, with tick_nohz_full_running as the global “is anyone tickless” boolean. tick_nohz_full_setup() simply copies the isolated mask into it at boot and sets the flag (tick-sched.c, v6.12). tick_nohz_full_cpu(cpu) then tests membership cheaply at runtime.

The timekeeping requirement is enforced by tick_nohz_cpu_hotpluggable(). One housekeeping CPU is designated tick_do_timer_cpu — it “handles housekeeping duty (unbound timers, workqueues, timekeeping, …) on behalf of full dynticks CPUs. It must remain online when nohz full is enabled,” so the function returns false (not hot-unpluggable) for that CPU (tick-sched.c, v6.12). This is the kernel-side reason for the rule stated in no_hz.rst: “At least one non-adaptive-tick CPU must remain online to handle timekeeping tasks in order to ensure that system calls like gettimeofday() returns accurate values on adaptive-tick CPUs … your system must have at least two CPUs in order for CONFIG_NO_HZ_FULL=y to do anything” (no_hz.rst, v6.12). tick_nohz_init() additionally bails out — clearing the mask and disabling full dynticks — if the architecture cannot deliver IRQ-work self-IPIs ("NO_HZ: Can't run full dynticks because arch doesn't support IRQ work self-IPIs"), because the tick-restart machinery depends on them (tick-sched.c, v6.12).

The “One Runnable Task” Constraint — sched_can_stop_tick

A nohz_full CPU is only eligible to be tickless; whether the tick actually stops at a given instant is decided by two functions. can_stop_full_tick() in tick-sched.c checks the tick dependencies: a global mask, a per-CPU mask, and per-task and per-signal masks (current->tick_dep_mask, current->signal->tick_dep_mask). If any dependency bit is set — for example TICK_DEP_MASK_POSIX_TIMER, TICK_DEP_MASK_PERF_EVENTS, TICK_DEP_MASK_SCHED, TICK_DEP_MASK_RCU — the tick cannot stop (tick-sched.c, v6.12). This is why arming a POSIX CPU timer or registering many perf events brings the tick back.

The TICK_DEP_MASK_SCHED bit is owned by the scheduler and set/cleared from sched_can_stop_tick() in kernel/sched/core.c, which encodes the actual “one runnable task” policy per scheduling class (core.c, v6.12):

  • Any SCHED_DEADLINE task on the run queue → tick stays. “Deadline tasks, even if single, need the tick” (the Constant Bandwidth Server enforcement runs on the tick).
  • SCHED_RR real-time: if exactly one round-robin task, the tick can stop (no peer to round-robin with); if more than one, the tick must stay to enforce the round-robin time-slice rotation.
  • SCHED_FIFO real-time: the tick can stop even with many FIFO tasks — “no forced preemption between FIFO tasks,” so there is nothing for the tick to do.
  • Fair class (EEVDF/CFS): if cfs.nr_running > 1 the tick must stay (it drives involuntary preemption between fair tasks); a single fair task lets the tick stop, unless that task is under CFS bandwidth (cpu.max) constraints, in which case the tick is kept to enforce the quota (__need_bw_check / cfs_task_bw_constrained).
  • sched_ext (BPF schedulers): the BPF scheduler is asked via scx_can_stop_tick().

So “one runnable task” is the headline rule, but the precise predicate is class-aware: deadline always needs it, multiple-RR needs it, multiple-fair needs it, FIFO never does. The no_hz.rst docs note a known limitation here — the current implementation still ticks a CPU that has one SCHED_FIFO task plus several low-priority SCHED_OTHER tasks, even though the FIFO task will run to the exclusion of the others, so the tick is technically unnecessary in that case (no_hz.rst, v6.12).

The Residual ~1 Hz Remote Tick

Even a fully tickless CPU needs some scheduler bookkeeping to advance, or quantities like load average, the EEVDF/CFS vruntime of the running entity, and avenrun would freeze. The kernel solves this with a remote tick: a housekeeping CPU ticks the tickless CPU roughly once per second on its behalf. The implementation is sched_tick_remote() in kernel/sched/core.c, scheduled as delayed work on the system unbound workqueue (core.c, v6.12):

/* Run the remote tick once per second (1Hz). This arbitrary
 * frequency is large enough to avoid overload but short enough
 * to keep scheduler internal stats reasonably up to date. */
os = atomic_fetch_add_unless(&twork->state, -1, TICK_SCHED_REMOTE_RUNNING);
WARN_ON_ONCE(os == TICK_SCHED_REMOTE_OFFLINE);
if (os == TICK_SCHED_REMOTE_RUNNING)
    queue_delayed_work(system_unbound_wq, dwork, HZ);

The work item, only started for non-HK_TYPE_TICK CPUs (sched_tick_start() returns early for housekeeping CPUs), checks whether the target CPU is actually tick-stopped (tick_nohz_tick_stopped_cpu()), takes the remote run-queue lock, calls curr->sched_class->task_tick(rq, curr, 0) to do the per-class accounting, updates calc_load_nohz_remote(), and re-queues itself HZ jiffies later — i.e. once per second (core.c, v6.12). There is even a WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3) guard that fires if the remote tick has somehow been starved for more than three seconds.

This is the “residual 1Hz tick” the isolcpus=nohz documentation warns about — and because it runs on the unbound workqueue, by default it can land on the isolated CPU itself. That is precisely why the boot docs tell you to affine the global workqueue mask off the isolated CPUs via /sys/devices/virtual/workqueue/cpumask after boot, or to use isolcpus=domain so the workqueue follows the domain isolation (kernel-parameters.txt, v6.12). The sched_tick_offload_init() allocator that backs this is invoked from housekeeping_init() only when the tick housekeeping flag is set (isolation.c, v6.12).

Failure Modes

  • All CPUs isolated → boot CPU silently reclaimed. If nohz_full=/isolcpus= names every present CPU, housekeeping_setup() force-adds the boot CPU back to housekeeping and warns. You did not get what you asked for; check dmesg for "using boot CPU".
  • nohz_full= without CONFIG_NO_HZ_FULL=y. Ignored with "Housekeeping: nohz unsupported. Build with CONFIG_NO_HZ_FULL". Verify the running kernel’s config.
  • Mismatched nohz_full= and isolcpus=nohz masks. Rejected with "Housekeeping: nohz_full= must match isolcpus="; the two must name the identical CPU set.
  • Trying to hot-unplug the timekeeping CPU. tick_nohz_cpu_hotpluggable() returns false for tick_do_timer_cpu; the offline attempt fails with -EBUSY.
  • Tick “won’t stop” on an isolated CPU. Almost always a lingering tick dependency: a POSIX CPU timer, too many perf events, a second runnable task (often a stray kernel thread or the unbound 1 Hz remote-tick worker itself), or a deadline/RR task. Trace with the tick_stop tracepoint or the dynticks-testing suite referenced in no_hz.rst.

Alternatives and Relationship to Other Mechanisms

Housekeeping/full-dynticks is the tick and background-work half of isolation; it pairs with, but is distinct from, domain isolation (Scheduling Domains and CPU Topology, driven by HK_TYPE_DOMAIN) and from cpuset isolated partitions (Cpusets and CPU Partitioning), which the kernel also folds into its cpu_is_isolated() test via cpuset_cpu_is_isolated() (isolation.h, v6.12). Note that cpusets can move CPUs in and out of domain isolation at runtime, but they do not control the tick/RCU housekeeping masks — those remain boot-time only (the no_hz.rst “Known Issues” explicitly say “A reboot is required to reconfigure both adaptive idle and RCU callback offloading”). So the practical division of labour is: cpusets for reversible load isolation, boot parameters for the tickless/RCU-offload machinery this note describes.

See Also

Graph View

Backlinks