vault

vCPU Threads and Host Scheduling

12 min read

vCPU Threads and Host Scheduling

A virtual CPU (vCPU) is not a special kernel object the scheduler treats differently — it is an ordinary host thread belonging to the userspace Virtual-Machine Monitor (VMM) process (QEMU, Firecracker, Cloud Hypervisor). That thread spends its life inside an ioctl(KVM_RUN) call: it asks KVM to enter guest mode, the CPU runs the guest natively until a VM exit, and the thread returns. Because the vCPU is just a thread, everything the host scheduler does applies unchanged — it is time-sliced by EEVDF (formerly CFS), it competes against every other runnable task on the box, and its CPU time is constrained by [[The cgroup v2 CPU Controller|cpu.weight and cpu.max]] exactly like any other process. This single fact — “a VM is just a process, a vCPU is just a thread” — is KVM’s entire value proposition and the source of its hardest performance pathologies: overcommit (more vCPUs than physical CPUs), steal time (guest time the host scheduled away), and the need for pinning to get latency-sensitive guests off the shared scheduler. When the guest goes idle it executes HLT, which traps; KVM halt-polls briefly and then blocks the thread (schedule()), freeing the physical CPU for someone else (kvm_main.c, v6.12).

Mental Model

Picture the VMM process as a normal multithreaded program. It has one thread per vCPU plus a few I/O threads. Each vCPU thread runs a tight loop: prepare guest state, call KVM_RUN, drop into the guest, come back on a VM exit, handle it, loop. While inside the guest the thread is “running” from the host’s point of view — it is consuming a physical CPU and accruing utime/stime. While blocked on an idle guest, it is sleeping like any thread waiting on an event.

flowchart TB
  subgraph PROC["VMM process (one in 'top' / ps)"]
    T0["vCPU thread 0<br/>(host task)"]
    T1["vCPU thread 1<br/>(host task)"]
    IO["I/O thread(s)"]
  end
  SCHED["Host scheduler (EEVDF)<br/>runqueues per physical CPU"]
  CG["cgroup cpu controller<br/>cpu.weight / cpu.max"]
  PCPU["Physical CPUs"]
  T0 --> SCHED
  T1 --> SCHED
  IO --> SCHED
  CG -.->|"constrains"| SCHED
  SCHED -->|"picks a runnable task"| PCPU
  T0 -.->|"ioctl(KVM_RUN): enter guest"| PCPU
  PCPU -.->|"VM exit / HLT"| T0

The vCPU as a host thread. What it shows: vCPU threads sit in the same runqueues as every other task; the EEVDF scheduler picks among them and the cgroup cpu controller constrains how much CPU the VM’s threads collectively get. A vCPU “runs the guest” only while it holds a physical CPU inside KVM_RUN. The insight to take: there is no separate “VM scheduler” — guest scheduling decisions (which guest thread runs) happen inside the guest on whatever physical time the host scheduler grants the vCPU thread. Two levels of scheduling stack: host-schedules-vCPU, then guest-schedules-its-own-tasks on the slice it got.

The Idle Guest Path — HLT, Halt-Poll, Block

The most important interaction between guest and host scheduler is the idle path, because an idle guest must give the physical CPU back or overcommit would be impossible. When a guest OS has nothing to run, its idle loop executes the HLT instruction (halt). On bare metal HLT parks the core until an interrupt; under virtualization it causes a VM exit, which KVM turns into a thread sleep.

The handler kvm_emulate_halt() → __kvm_emulate_halt() records the halt and, if the local APIC is in-kernel, sets the vCPU’s machine-state to halted so the run loop will park it; otherwise it exits to userspace (x86.c, v6.12):

static int __kvm_emulate_halt(struct kvm_vcpu *vcpu, int state, int reason)
{
	/*
	 * The vCPU has halted, e.g. executed HLT.  Update the run state if the
	 * local APIC is in-kernel, the run loop will detect the non-runnable
	 * state and halt the vCPU.  Exit to userspace if the local APIC is
	 * managed by userspace, in which case userspace is responsible for
	 * handling wake events.
	 */
	++vcpu->stat.halt_exits;
	if (lapic_in_kernel(vcpu)) {
		if (kvm_vcpu_has_events(vcpu))
			vcpu->arch.pv.pv_unhalted = false;
		else
			vcpu->arch.mp_state = state;
		return 1;
	} else {
		vcpu->run->exit_reason = reason;
		return 0;
	}
}

++vcpu->stat.halt_exits counts the halt (this is the halt_exits you see in kvm_stat); lapic_in_kernel(vcpu) checks whether KVM emulates the local interrupt controller itself (the fast common case); if so it sets mp_state to halted and the run loop calls kvm_vcpu_halt(). That function implements the clever halt-polling optimization: rather than immediately blocking (an expensive sleep/wake round-trip), it busy-waits for a short window in case a wake event arrives almost immediately:

/*
 * Emulate a vCPU halt condition, e.g. HLT on x86, WFI on arm, etc...  If halt
 * polling is enabled, busy wait for a short time before blocking to avoid the
 * expensive block+unblock sequence if a wake event arrives soon after the vCPU
 * is halted.
 */
void kvm_vcpu_halt(struct kvm_vcpu *vcpu)
{
	...
	do_halt_poll = halt_poll_allowed && vcpu->halt_poll_ns;
	...
	if (do_halt_poll) {
		ktime_t stop = ktime_add_ns(start, vcpu->halt_poll_ns);
		do {
			if (kvm_vcpu_check_block(vcpu) < 0)
				goto out;
			cpu_relax();
			poll_end = cur = ktime_get();
		} while (kvm_vcpu_can_poll(cur, stop));
	}
 
	waited = kvm_vcpu_block(vcpu);
	...
}

The polling loop spins (cpu_relax(), a PAUSE instruction) checking kvm_vcpu_check_block() until either an event arrives — in which case it goto out without ever sleeping, the win — or the poll window vcpu->halt_poll_ns elapses, at which point it falls through to kvm_vcpu_block() and genuinely sleeps. The block itself is the textbook Linux idiom:

bool kvm_vcpu_block(struct kvm_vcpu *vcpu)
{
	...
	for (;;) {
		set_current_state(TASK_INTERRUPTIBLE);
		if (kvm_vcpu_check_block(vcpu) < 0)
			break;
		waited = true;
		schedule();
	}
	...
}

set_current_state(TASK_INTERRUPTIBLE) marks the thread sleeping, and schedule() hands the physical CPU to another runnable task — this is exactly how an idle guest releases its core for overcommit. The thread is woken by an injected interrupt, timer, or KVM_REQ_UNBLOCK.

The poll window adapts. KVM starts at a default (halt_poll_ns, initialized from the arch constant KVM_HALT_POLL_NS_DEFAULT, which is 200000 ns = 200 µs on x86, per kvm_host.h, v6.12) and grows or shrinks per-vCPU based on success: if the guest woke during the poll, growing the window pays off (grow_halt_poll_ns, default halt_poll_ns_grow = 2 doubles it); if the vCPU ended up blocking anyway, shrinking saves the wasted spin (shrink_halt_poll_ns). Halt-polling is a latency-vs-utilization knob: it burns host CPU spinning to shave wake-up latency for bursty, frequently-idle guests, but on an overcommitted host that spinning steals time from other guests — which is why it is tunable via the halt_poll_ns module parameter and KVM_CAP_HALT_POLL.

Uncertain

Verify: that KVM_HALT_POLL_NS_DEFAULT is 200000 ns (200 µs) on x86 in 6.12 and 6.18. Reason: confirmed in v6.12 arch/x86/include/asm/kvm_host.h (#define KVM_HALT_POLL_NS_DEFAULT 200000), but this is an arch-specific constant that has changed historically and could differ on 6.18 or other architectures. To resolve: grep the same constant in the v6.18 tag. uncertain

Guest CPU Time Is the VMM’s CPU Time — cgroup Limits Apply

Because the vCPU thread runs the guest natively while inside KVM_RUN, the host kernel accounts all guest execution as that thread’s CPU time. Run top on the host and the QEMU/Firecracker process shows the guest’s busy CPU; the kernel even has a dedicated %guest accounting column. Critically, this means the [[The cgroup v2 CPU Controller|cgroup v2 cpu controller]] governs guest CPU exactly as it governs any process: a VM placed in a cgroup with cpu.max 50000 100000 is throttled to 50% of one CPU across all its vCPU threads, and cpu.weight sets its proportional share under contention. This is how cloud and container-VM platforms (Kata, KubeVirt) cap and fair-share VMs — they simply put the VMM process in a cgroup. There is no virtualization-specific limiter; the ordinary scheduler does the work.

A subtle consequence: CFS/EEVDF bandwidth throttling (cpu.max) can pause a vCPU thread mid-guest-instruction-stream at a quota boundary. From inside the guest this looks like the CPU mysteriously freezing — the basis of the “steal time” the guest observes (next section). The interaction between guest timekeeping and host throttling is a classic source of guest clock skew; see Timer Virtualization (kvmclock and TSC).

Overcommit and Steal Time

Overcommit means provisioning more total vCPUs than physical CPUs (pCPUs) — e.g. ten 2-vCPU guests on an 8-pСPU host. It works precisely because idle guests release their pCPUs via the HLT/block path above, so the sum of busy vCPUs rarely exceeds the pСPU count. When it does — when more vCPUs are runnable than there are pCPUs — the host scheduler time-multiplexes them, and a guest’s vCPU spends time runnable but not running. From the guest’s perspective its CPU has stalled for no reason it can see.

KVM exposes this lost time honestly through paravirtualized steal time so the guest can account for it. The guest enables it by writing the steal-time MSR (MSR_KVM_STEAL_TIME, “steal time can be enabled by writing to msr 0x4b564d03” per cpuid.rst, v6.12), pointing it at a shared struct kvm_steal_time page. KVM updates that page from record_steal_time() (x86.c, v6.12), and the key line is:

	unsafe_get_user(steal, &st->steal, out);
	steal += current->sched_info.run_delay -
		vcpu->arch.st.last_steal;
	vcpu->arch.st.last_steal = current->sched_info.run_delay;
	unsafe_put_user(steal, &st->steal, out);

Walking it: current->sched_info.run_delay is the host scheduler’s own accounting of how long this thread sat on a runqueue waiting to run — pure scheduling delay. KVM takes the increment since the last update (run_delay - last_steal) and adds it to the guest-visible steal counter, then remembers the new baseline. So steal time is literally the host scheduler’s run-queue wait time, exported into the guest. The guest kernel reads this and reports it as the %st (“steal”) column in top/mpstat and in /proc/stat. High %st inside a guest is the definitive signal of host CPU overcommit or cgroup throttling.

KVM also marks a vCPU preempted in the same shared structure when the host deschedules it, via kvm_steal_time_set_preempted() writing KVM_VCPU_PREEMPTED (x86.c, v6.12; the bit is defined in kvm_para.h, v6.12 as KVM_VCPU_PREEMPTED (1 << 0)). A guest that knows a lock-holder vCPU has been preempted can avoid spinning on its spinlock — the paravirtualized spinlock optimization that prevents the notorious “Lock-Holder Preemption” problem, where a guest vCPU spins for an entire host time-slice waiting for a lock held by another vCPU the host descheduled. The same structure even carries a KVM_VCPU_FLUSH_TLB request so the host can do a TLB flush on the guest’s behalf and skip an expensive inter-processor interrupt (see record_steal_time’s guest_pv_has(vcpu, KVM_FEATURE_PV_TLB_FLUSH) branch).

Pinning and isolcpus — Taking vCPUs Off the Shared Scheduler

For latency-sensitive guests (real-time, network functions, telco) the shared scheduler is the enemy: any host task can preempt a vCPU, and steal time becomes unbounded jitter. The fix is CPU pinning — binding each vCPU thread to a dedicated physical CPU (taskset/sched_setaffinity on the thread, or <vcpupin> in libvirt) so it is the only vCPU on that core. This gives one-to-one vCPU↔pCPU mapping with no overcommit, eliminating steal time at the cost of utilization (the core is reserved even when the guest is idle).

Pinning alone is insufficient if other host tasks can still land on those cores. The kernel boot parameter isolcpus= removes CPUs from the general scheduler’s load-balancing domains so no ordinary task is placed there automatically; only explicitly-affined threads (the pinned vCPUs) run on them.

Uncertain

Verify: the precise current semantics and deprecation status of the isolcpus= boot parameter in 6.12/6.18. Reason: isolcpus (the bare-CPU-list form) has long been documented as deprecated in favor of the cpuset cgroup / cpuset.cpus.partition “isolated” mechanism, but the exact wording and whether the legacy form still works in 6.12/6.18 was not re-read from Documentation/admin-guide/kernel-parameters.txt for this note. To resolve: read the isolcpus entry in the v6.12 kernel-parameters.txt. uncertain

Production-grade low-latency setups layer several mechanisms: isolcpus (or an isolated cpuset partition) to clear the cores, vCPU pinning to occupy them, nohz_full to stop the periodic timer tick on those cores, and IRQ affinity to steer device interrupts away. The interaction with interrupt delivery is why ioeventfd and hardware posted interrupts matter so much for pinned guests — they let interrupts reach a running vCPU without a VM exit, so a pinned vCPU never has to be descheduled to take an interrupt.

Failure Modes and Diagnosis

High guest %st (steal). The guest’s top/mpstat shows large steal percentages. Root cause: host CPU overcommit, or cgroup cpu.max throttling. Diagnose on the host with cat <cgroup>/cpu.stat (look for throttled_usec) and by counting runnable threads vs pCPUs. Fix: reduce overcommit, raise the cap, or pin.

Lock-holder preemption / guest soft-lockups. A guest reports RCU stalls or soft-lockups under host contention because a vCPU holding a lock was descheduled for a full host slice while others spun. Mitigation: ensure paravirtualized spinlocks and KVM_VCPU_PREEMPTED are active, and reduce overcommit.

Halt-polling burning host CPU. On an overcommitted host, a frequently-idle guest’s halt-polling spins consume pCPU that other guests need, raising everyone’s steal. Diagnose via kvm_stat (halt_poll_success/halt_poll_fail) and tune or disable halt_poll_ns.

Jitter on a “pinned” guest. If isolcpus was not set (or the cpuset partition is wrong), kernel threads and IRQs still land on the pinned cores. Diagnose with cat /proc/interrupts and per-CPU runqueue inspection; fix the isolation, not the pinning.

See Also

Graph View

Backlinks