vault

Capacity-Aware and Energy-Aware Scheduling

18 min read

Capacity-Aware and Energy-Aware Scheduling

On a symmetric multiprocessor every CPU is interchangeable, so the scheduler need only decide whether to move a task, never which kind of core to move it to. Modern mobile and increasingly desktop/server silicon breaks that assumption: asymmetric (heterogeneous) topologies — Arm’s big.LITTLE, DynamIQ, and Intel’s Performance/Efficient hybrid cores — pair a few high-throughput “big” cores with several low-power “little” ones. Two scheduler features cope with this. Capacity-aware scheduling teaches the fair class that CPUs differ in raw throughput (a per-CPU capacity value normalized to 1024) and that a task carries a utilization requirement that must “fit” the CPU it lands on; tasks that no longer fit are flagged misfit and actively migrated to a bigger core. Energy-Aware Scheduling (EAS) goes further: using an Energy Model (EM) of the CPUs’ power costs, it overrides the wake-up placement decision to pick the CPU predicted to get the work done at the lowest energy, rather than the fastest or least-loaded one (sched-energy.rst §1; sched-capacity.rst §1). Both rest on the same foundation — capacity values and Per-Entity Load Tracking (PELT) utilization signals, both expressed on the same 0–1024 scale so a task’s “size” can be compared directly against a CPU’s “size.” This note explains that foundation, the capacity-fitness criterion, misfit migration, and the find_energy_efficient_cpu() placement path, leaning on (and cross-linking, not re-deriving) Scheduling Domains and CPU Topology and Scheduler Load Balancing.

This note is pinned to Linux 6.12 LTS (2024-11-17) and re-verified against 6.18 LTS (2025-11-30): find_energy_efficient_cpu(), compute_energy(), update_misfit_status(), and cpu_overutilized() are all present and structurally unchanged in the 6.18 fair.c (v6.18 fair.c). EAS has been mainline since 5.0 (2019); the mechanism described here is the mature, current form.

Mental Model

Picture every CPU as a bucket of a certain size and every task as a volume of water. On a symmetric machine all buckets are the same size, so only the water level matters. On a heterogeneous machine, big cores are large buckets and little cores are small ones. Capacity is the bucket size (1024 for the most capable CPU in the system, proportionally less for slower ones); utilization is how much water a task pours. The capacity-fitness rule is simply: don’t pour more water than the bucket holds. A task whose utilization exceeds its current CPU’s capacity is overflowing — a misfit — and must be moved to a bigger bucket. EAS adds a price tag to each bucket: filling a big bucket to the brim costs more energy than filling a little one, but the relationship is non-linear and depends on the operating frequency, so the scheduler consults a per-CPU power cost table (the Energy Model) and simulates placing the task on each candidate before choosing the cheapest.

flowchart TB
  WAKE["task wakeup<br/>select_task_rq_fair()"] --> Q{"EAS enabled<br/>and rd not<br/>over-utilized?"}
  Q -- "yes" --> FEEC["find_energy_efficient_cpu()<br/>(EAS path)"]
  Q -- "no" --> CAS["capacity-aware<br/>load-based selection<br/>(util_fits_cpu)"]
  FEEC --> SPARE["per perf-domain:<br/>pick CPU with max<br/>spare capacity"]
  SPARE --> CE["compute_energy()<br/>simulate task on<br/>each candidate"]
  CE --> CHEAP["pick lowest-energy CPU<br/>vs. staying on prev_cpu"]
  CAS --> RUN["task runs"]
  CHEAP --> RUN
  RUN --> TICK["tick:<br/>update_misfit_status()"]
  TICK --> MIS{"task no longer<br/>fits this CPU?"}
  MIS -- "yes" --> ALB["misfit active<br/>load balance →<br/>migrate to bigger CPU"]
  TICK --> OU["cpu_overutilized()<br/>util > 80% cap?"]
  OU -- "yes" --> DIS["set rd->overutilized<br/>→ disable EAS,<br/>re-enable load balancer"]

Figure 1 — the capacity/energy decision flow. What it shows: at wakeup, if EAS is enabled and the root domain is below the over-utilization tipping point, placement goes through the energy-aware path (feec), which finds the highest-spare-capacity CPU per performance domain and uses the Energy Model to pick the cheapest; otherwise it falls back to capacity-aware, load-based selection. The periodic tick re-checks fitness (misfit → active migration) and over-utilization (cross the 80 % line → EAS switches off, the load balancer comes back on). The insight to take: EAS is a light-load optimization — it deliberately steps aside under heavy load, handing control back to the throughput-oriented load balancer.

Capacity: What It Means and Where It Comes From

CPU performance is roughly work_per_hz × max_freq. Capacity captures both axes: the kernel defines capacity(cpu) = work_per_hz(cpu) × max_freq(cpu), normalized so the most capable CPU in the system has capacity 1024 (SCHED_CAPACITY_SCALE) and slower CPUs get proportionally smaller values (sched-capacity.rst §1.1). Asymmetry has two sources: a different microarchitecture (deeper pipeline, bigger caches on the bigs) and a different maximum operating performance point (OPP) reachable via Dynamic Voltage and Frequency Scaling (DVFS). A big.LITTLE system typically differs on both.

The kernel distinguishes two capacity values (the documentation uses the terms loosely; the code is precise):

  • Original capacity — the CPU’s maximum attainable performance. It is class-agnostic and returned by the architecture callback arch_scale_cpu_capacity(cpu). The kernel cannot discover this on its own; the architecture must supply it. On arm/arm64/RISC-V it comes from the capacity-dmips-mhz device-tree binding fed through the arch_topology driver (sched-capacity.rst §1.2, §3.1).
  • Capacity — the original capacity minus performance stolen by non-fair work: time the CPU spends in IRQ handling and in higher scheduling classes (RT, deadline). This is what the fair class actually plans against. In code it is the cpu_capacity field of struct rq, read via capacity_of(cpu) which simply return cpu_rq(cpu)->cpu_capacity; (fair.c capacity_of, v6.12; sched.h struct rq, v6.12).

Capacity values share the same 0–1024 scale as the PELT utilization signals, which is the whole point: it lets the scheduler compare a task’s “size” against a CPU’s “size” with a single inequality.

Task Utilization and Why It Must Be Invariant

A task’s utilization is meant to express its throughput demand — at its simplest, its duty cycle (fraction of wall time spent executing). A 100 % utilization task is a busy loop; a 10 % task sleeps most of the time (sched-capacity.rst §2.1). But raw duty cycle is not a stable signal on a DVFS, heterogeneous machine, because the same work shows a different duty cycle depending on how fast the CPU was running and how capable it was:

  • Run a fixed workload at frequency F → 25 % duty cycle; run it at F/2 → 50 % duty cycle, for identical work. Frequency invariance corrects this: task_util_freq_inv = duty_cycle × (curr_freq / max_freq).
  • Run the same work on a CPU of capacity C vs. one of capacity C/3 → 25 % vs. 75 % duty cycle. CPU invariance corrects this: task_util_cpu_inv = duty_cycle × (capacity(cpu) / max_capacity).

Composing both yields the truly invariant signal the scheduler uses everywhere:

                                  curr_freq(cpu)    capacity(cpu)
task_util_inv(p) = duty_cycle(p) × -------------- × --------------
                                  max_freq(cpu)     max_capacity

Read symbol by symbol: duty_cycle(p) is the raw fraction-of-time-running; multiplying by curr_freq/max_freq rescales it as if the CPU were always at top frequency; multiplying by capacity(cpu)/max_capacity rescales it as if the task were on the most capable CPU. The result describes the task as though it always ran on the biggest core at full speed — a portable “size” that means the same thing regardless of where the task happens to be running (sched-capacity.rst §2.2–2.4). Achieving invariance requires the architecture to implement arch_scale_freq_capacity() (often via APERF/MPERF on x86, the AMU on arm64, or cpufreq transition hooks) — and EAS hard-requires these callbacks (sched-capacity.rst §3.2; sched-energy.rst §6.5).

The scheduler cannot see the future, so it cannot know a task’s true utilization the instant it becomes runnable. It uses PELT (Per-Entity Load Tracking, covered in its own leaf) to maintain a decaying average, util_avg. PELT alone has a flaw for placement: when a periodic task blocks, its util_avg decays toward zero, so on its next wakeup the scheduler would underestimate it and place it on too small a CPU. util_est (utilization estimation) fixes this. The kernel snapshots a task’s utilization at dequeue (when it goes to sleep) and remembers it as an exponentially-weighted moving average, so the estimated utilization survives sleep periods:

/* kernel/sched/fair.c, v6.12 */
static inline unsigned long _task_util_est(struct task_struct *p)
{
    return READ_ONCE(p->se.avg.util_est) & ~UTIL_AVG_UNCHANGED;
}
static inline unsigned long task_util_est(struct task_struct *p)
{
    return max(task_util(p), _task_util_est(p));   /* never below the estimate */
}

task_util_est() returns the larger of the instantaneous PELT average and the saved estimate — so a periodic task wakes up “remembering” how big it was, and capacity/energy decisions use that. Per-CPU, the same estimate is aggregated into cfs_rq->avg.util_est at enqueue/dequeue (util_est_enqueue/util_est_dequeue) so a runqueue’s expected load is known too (fair.c, v6.12). This estimated utilization is the input to both the fitness criterion and the energy calculation.

Capacity-Aware Placement: Fitness, uclamp, and Misfits

The core fair-class criterion is capacity fitness — a task should fit on its CPU:

task_util(p) < capacity(task_cpu(p))

If violated, the task needs more throughput than the CPU can supply and becomes CPU-bound (sched-capacity.rst §5.1.1). uclamp (utilization clamping) lets userspace bias this: via sched_setattr(2) or the cgroup interface a task gets a uclamp_min and uclamp_max, and the criterion becomes

clamp(task_util(p), uclamp_min(p), uclamp_max(p)) < capacity(cpu)

This gives userspace real leverage: a high uclamp_min forces a small but latency-critical task onto a big core (boosting), while a low uclamp_max lets a busy loop be confined to little cores (capping) — useful for background work you want kept off the energy-hungry bigs (sched-capacity.rst §5.1.2). In code, fitness is evaluated by util_fits_cpu() and wrapped by task_fits_cpu(), which feeds task_util_est(p) and the effective uclamp values.

Wakeup placement is the fast path to apply fitness, but it has a pathological gap: a task that rarely sleeps rarely wakes, so there may be no wakeup at which to fix a bad placement. Such a task — running on a CPU too small for it — is a misfit. The mechanism that rescues it shares the name. On every tick, update_misfit_status() evaluates the running task:

/* kernel/sched/fair.c — update_misfit_status(), v6.12 */
if (!sched_asym_cpucap_active())
    return;                                   /* only on asymmetric systems */
if (!p || (p->nr_cpus_allowed == 1) ||
    (arch_scale_cpu_capacity(cpu) == p->max_allowed_capacity) ||
    task_fits_cpu(p, cpu)) {
    rq->misfit_task_load = 0;                 /* fits, or nowhere bigger to go */
    return;
}
rq->misfit_task_load = max_t(unsigned long, task_h_load(p), 1);  /* flag it */

Reading it: skip entirely on symmetric systems (sched_asym_cpucap_active() is the static-key guard); a task fits if it’s pinned to one CPU, is already on the biggest CPU it’s allowed, or genuinely fits — otherwise the runqueue records a non-zero misfit_task_load. That flag is picked up by the active load balancer (the part of Scheduler Load Balancing that migrates a currently running task, not just queued ones): a misfit active balance fires to push the task onto a CPU with more capacity (sched-capacity.rst §5.1.3). The RT and deadline classes have their own, simpler capacity-aware wakeup rules — RT honours uclamp_min ≤ capacity “best effort” subject to priority, and deadline checks task_bandwidth(p) < capacity(cpu) (the bridge to The Constant Bandwidth Server and Admission Control) (sched-capacity.rst §5.2–5.3).

Detecting Asymmetry: SD_ASYM_CPUCAPACITY

When the scheduling domains are built, the scheduler discovers whether the system is heterogeneous and sets two flags plus a system-wide static key (sched-capacity.rst §4):

  • sched_asym_cpucapacity — a static key enabled if any capacity asymmetry exists in the system. It guards the asymmetry-specific code paths (like update_misfit_status above) so symmetric machines pay nothing.
  • SD_ASYM_CPUCAPACITY — set on any sched_domain spanning CPUs of differing capacity.
  • SD_ASYM_CPUCAPACITY_FULL — set on the lowest sched_domain level that spans all unique capacity values in the system. EAS keys off this flag specifically.

The static key is system-wide but the domain flags are local, and the difference matters: with exclusive cpusets you can carve out an SMP island of only-little CPUs. That island’s domain hierarchy will not have SD_ASYM_CPUCAPACITY set even though the system-wide key is on, so the canonical guard pattern is “check the static key, then also check the domain flag for the specific CPUs you care about” (sched-capacity.rst §4). The domain hierarchy itself is the subject of Scheduling Domains and CPU Topology.

Energy-Aware Scheduling: the Energy Model and feec

EAS answers a different question than fitness: of the CPUs where the task fits, which one costs the least energy? It does so only on heterogeneous topologies — on symmetric platforms there is little to gain, so EAS refuses to enable (sched-energy.rst §1, §6.1). The optimization target is, in the documentation’s own framing, to maximize performance / power (equivalently, minimize energy / instruction) while still “getting the job done” (sched-energy.rst §2).

The Energy Model (EM) supplies the cost data. It is not maintained by the scheduler — a separate framework (Documentation/power/energy-model.rst) holds a power cost table per performance domain, where a performance domain is a group of CPUs that scale frequency together (usually 1:1 with a cpufreq policy). Each table entry maps an OPP to a (capacity, power) pair. The data may be real micro-Watts or an “abstract scale” — EAS only ever compares differences, so the absolute units don’t matter (energy-model.rst §2.3; sched-energy.rst §6.2). The scheduler attaches a per-root-domain linked list of the performance domains intersecting that root domain, RCU-protected against hotplug (sched-energy.rst §3). It maintains the sched_energy_present static key, enabled only when at least one root domain meets all the EAS conditions (§6: asymmetric topology, EM present, schedutil governor, invariant signals, no SMT).

The placement path. When EAS is on and the root domain is below the over-utilization line, select_task_rq_fair() calls find_energy_efficient_cpu() (often abbreviated feec). Its strategy, stated in the source comment, is: in each performance domain find the CPU with the maximum spare capacity (capacity − utilization) — that is the candidate that will let the domain run at the lowest frequency — then use the Energy Model to estimate total system energy with the task placed there, and compare against leaving it on prev_cpu (fair.c find_energy_efficient_cpu comment, v6.12):

/* kernel/sched/fair.c — find_energy_efficient_cpu(), v6.12 (abridged) */
for (; pd; pd = pd->next) {                       /* each performance domain */
    for_each_cpu(cpu, cpus) {
        ...
        util = cpu_util(cpu, p, cpu, 0);
        fits = util_fits_cpu(util, util_min, util_max, cpu);
        if (!fits)
            continue;                              /* skip CPUs the task won't fit */
        lsub_positive(&cpu_cap, util);             /* spare capacity */
        if (cpu == prev_cpu) { prev_spare_cap = cpu_cap; ... }
        else if (/* better fit or more spare */)   /* track best per-PD candidate */
            { max_spare_cap = cpu_cap; max_spare_cap_cpu = cpu; ... }
    }
    base_energy = compute_energy(&eenv, pd, cpus, p, -1);          /* energy without p */
    if (prev_spare_cap > -1)
        prev_delta = compute_energy(..., prev_cpu) - base_energy;  /* cost of staying */
    if (max_spare_cap_cpu >= 0 && ...)
        cur_delta  = compute_energy(..., max_spare_cap_cpu) - base_energy; /* cost of moving */
    /* keep the candidate with the smallest energy delta */
}
if ((best_fits > prev_fits) ||
    ((best_fits > 0) && (best_delta < prev_delta)) || ...)
    target = best_energy_cpu;                       /* move only if it actually helps */

The structure is a per-performance-domain scan that, for each domain, computes a baseline energy (dst_cpu = -1, task excluded) and then the marginal energy delta of placing the task on the best candidate vs. on prev_cpu. compute_energy() is the kernel’s energy estimator:

/* kernel/sched/fair.c — compute_energy(), v6.12 */
max_util  = eenv_pd_max_util(eenv, pd_cpus, p, dst_cpu);   /* peak util drives the OPP */
busy_time = eenv->pd_busy_time;
if (dst_cpu >= 0)
    busy_time = min(eenv->pd_cap, busy_time + eenv->task_busy_time);
energy = em_cpu_energy(pd->em_pd, max_util, busy_time, eenv->cpu_cap);

It computes the domain’s peak utilization (which sets the OPP the whole domain will run at — CPUs in a performance domain share a frequency) and its total busy time, then asks the Energy Model (em_cpu_energy()) for the predicted energy at that operating point (fair.c, v6.12). The final decision moves the task only if the energy genuinely improves (or fit improves) — otherwise it stays on prev_cpu, preserving cache warmth. Forkees are deliberately excluded from feec: a brand-new task has no utilization history, so EAS cannot forecast its impact and lets the normal slow path place it (fair.c comment, v6.12).

The Example 2 in the kernel doc walks a concrete decision (a util_avg = 200 task on a 2-little/2-big machine) and shows the energy of three placements computed as util/cap × power per CPU, summed across all four — landing on the little CPU1 as cheapest (total 1364 vs. 1485 for a big core vs. 1437 for staying put). The non-obvious lesson the doc draws out: a little CPU is not always cheapest — packing a small task onto an already-busy little cluster can raise the whole cluster’s OPP and cost more than running it alone on a big core, which is exactly why a model-based comparison beats any fixed heuristic (sched-energy.rst §4).

The over-utilization tipping point

EAS is a light-to-medium load optimization. Under heavy load every core needs all its capacity and there is no slack to trade for energy; worse, the PELT signals stop accurately representing task “size” once CPUs saturate. So the scheduler watches for over-utilization: a CPU is flagged once it exceeds 80 % of its capacity. cpu_overutilized() is the test:

/* kernel/sched/fair.c — cpu_overutilized(), v6.12 */
static inline bool cpu_overutilized(int cpu)
{
    if (!sched_energy_enabled())
        return false;
    rq_util_min = uclamp_rq_get(cpu_rq(cpu), UCLAMP_MIN);
    rq_util_max = uclamp_rq_get(cpu_rq(cpu), UCLAMP_MAX);
    return !util_fits_cpu(cpu_util_cfs(cpu), rq_util_min, rq_util_max, cpu);
}

The over-utilization detection accounts for capacity stolen by higher classes and IRQ, not just CFS, so the “80 %” is against available capacity (sched-energy.rst §5). As long as no CPU in a root domain is over-utilized, EAS overrides wake-up balancing and the periodic load balancer is disabled (so it won’t undo EAS’s energy-optimal — but deliberately unbalanced — placement). The moment one CPU crosses the line, rd->overutilized is set, EAS switches off, and the load balancer is re-enabled to chase throughput and respect nice values. This hand-off is the single most important operational fact about EAS: it is not always in control (sched-energy.rst §5).

Why schedutil Is Mandatory for EAS

EAS predicts energy by assuming each CPU’s OPP follows its utilization — that the frequency will be set proportional to load. Only one cpufreq governor makes that true: schedutil, which derives the requested frequency directly from the scheduler’s utilization signal (sched-energy.rst §6.4). Its formula is, with frequency-invariant utilization:

next_freq = C × max_freq × util / max     where C = 1.25

The C = 1.25 headroom puts the frequency tipping point at util/max = 0.8 — i.e. a CPU at 80 % utilization requests its maximum frequency, leaving 25 % headroom so the chosen OPP isn’t perpetually saturated. (Note the same 0.8 appears as the over-utilization line above; the two are deliberately aligned.) In code, get_next_freq() calls map_util_freq(util, freq, max) to apply this and then rounds up to the lowest driver-supported frequency ≥ the raw request (cpufreq_schedutil.c get_next_freq, v6.12). Because schedutil’s frequency requests and EAS’s energy predictions use the same utilization signal, they stay consistent — which is precisely why EAS with any other governor is unsupported (sched-energy.rst §6.4).

Failure Modes and Common Misunderstandings

  • “I enabled EAS but it’s not placing tasks for energy.” Check every §6 condition: asymmetric topology (SD_ASYM_CPUCAPACITY_FULL set), an Energy Model registered for the platform, the schedutil governor active, frequency/CPU-invariant signals (arch_scale_freq_capacity + arch_scale_cpu_capacity implemented), and SMT off (EAS is SMT-unaware and disabled on SMT). Miss any one and sched_energy_present stays off (sched-energy.rst §6). The scheduling domains must also be rebuilt after the EM registers.
  • “EAS works at idle but stops under load.” Working as designed — once a CPU passes the 80 % over-utilization point, rd->overutilized is set, EAS yields to the load balancer for the duration (sched-energy.rst §5).
  • Assuming “little core = always lower energy.” False. A small task can be cheaper on a big core if placing it on a little cluster would raise that cluster’s shared OPP for everyone else. This is exactly what the model-based compute_energy() exists to catch (sched-energy.rst §4).
  • New tasks land “wrong.” Forkees skip feec (no utilization history) and use the normal slow path; their placement only becomes energy-aware after they accumulate PELT/util_est history (fair.c, v6.12).
  • A CPU-bound task stuck on a little core. If it never sleeps, wakeup placement can’t fix it; the misfit active-balance path on the tick is the safety net — but only on asymmetric systems and only if a bigger allowed CPU exists (sched-capacity.rst §5.1.3).
  • EAS on a symmetric SMP box. Not supported — no savings have been demonstrated, so the topology code never enables it (sched-energy.rst §6.1).

EAS’s restriction to asymmetric topologies (and its exclusion of SMP and SMT) is confirmed unchanged in the 6.18 sched-energy.rst: “EAS does not support platforms with symmetric CPU topologies,” “EAS is only supported on platforms with asymmetric CPU topologies for now,” and “EAS on SMT is not supported” all survive verbatim, with the 80 % tipping point intact (v6.18 sched-energy.rst).

See Also

Graph View

Backlinks