vault

Dirty Page Tracking and Live Migration

18 min read

Dirty Page Tracking and Live Migration

Live migration moves a running virtual machine (VM) from one physical host to another without the guest noticing — no reboot, sub-second pause, established TCP connections survive. The hard part is the guest’s RAM, which is gigabytes that keep changing while you copy them. The dominant technique is pre-copy: while the guest keeps running on the source, the virtual-machine monitor (VMM) copies all of guest RAM to the destination, then iteratively re-copies only the pages the guest dirtied since the last pass, until the remaining dirty set is small enough to transfer during a brief stop-and-copy pause. This demands that the hypervisor tell the VMM which pages were written. KVM (the Kernel-based Virtual Machine) does this by dirty-tracking guest memory — historically by write-protecting pages so a write traps, more recently by hardware Page-Modification Logging (PML) — and exposes the dirty set to userspace either as a per-slot bitmap via the KVM_GET_DIRTY_LOG ioctl or, since Linux 5.11, as a per-vCPU dirty ring (KVM_CAP_DIRTY_LOG_RING). When pre-copy fails to converge because the guest writes faster than the network drains, the fallback is post-copy, which switches the guest to the destination immediately and demand-pages its RAM back from the source. This note is pinned to the Linux 6.12 / 6.18 long-term-support (LTS) kernels and the modern QEMU migration code.

The recurring tension throughout is convergence: the migration races the guest’s own write activity. Every mechanism here — write-protection, PML, the dirty ring, auto-converge throttling, the dirty-limit, and ultimately post-copy — exists to win or sidestep that race. The other recurring theme is cost shape: a bitmap costs O(guest size) to scan each round regardless of how many pages actually changed, while a ring costs O(pages actually dirtied) — which is exactly why the ring was added.

Mental Model — A Race Between Copier and Scribbler

Think of pre-copy migration as photocopying a book that someone is still editing. You copy every page once (the full pass). By the time you finish, the editor has changed a few pages, so you go back and re-copy just those. Each pass is shorter than the last — if the editor edits slower than you copy. When only a handful of pages still differ, you tell the editor to stop for a moment (the stop-and-copy pause), copy those last few, hand over the book, and let the editor resume on the new copy. The whole trick rests on one capability: knowing which pages changed since you last looked. That is dirty tracking.

flowchart TD
  START["Migration begins<br/>enable dirty logging on all memslots"]
  FULL["Full pass: copy ALL guest RAM<br/>to destination (guest still running)"]
  SYNC["Fetch dirty set<br/>(KVM_GET_DIRTY_LOG bitmap OR dirty ring)"]
  RESEND["Re-send only dirtied pages"]
  CHECK{"remaining dirty set<br/>small enough to copy<br/>within downtime budget?"}
  STOP["STOP-AND-COPY:<br/>pause guest, copy last dirty pages<br/>+ device/CPU state, switch over"]
  CONV{"converging?<br/>(dirty rate &lt; transfer rate)"}
  THROTTLE["throttle vCPUs<br/>(auto-converge / dirty-limit)<br/>OR switch to POST-COPY"]
  DONE["Guest runs on destination"]

  START --> FULL --> SYNC --> RESEND --> CHECK
  CHECK -->|"yes"| STOP --> DONE
  CHECK -->|"no, still too many"| CONV
  CONV -->|"yes"| SYNC
  CONV -->|"no, diverging"| THROTTLE --> SYNC

Figure: the pre-copy iteration loop and its escape hatches. What it shows: after one full pass, the loop repeatedly syncs the dirty set and re-sends only what changed; it converges when the residual dirty set fits in the downtime budget. The insight to take: the loop only terminates if the guest dirties memory slower than the link transfers it — when that fails, the VMM must either slow the guest down (throttle) or change strategy entirely (post-copy). Dirty tracking is the sensor that feeds every decision in this loop.

Mechanical Walk-through

Why you cannot just copy RAM once

Guest RAM under KVM is ordinary host anonymous memory that the VMM mmap’d and registered as a memory slot (see Guest Physical Memory and Memory Slots). If you copied it in a single pass while the guest ran, pages you copied early would already be stale by the time you finished — the destination would have a torn, inconsistent memory image. You could instead pause the guest, copy everything, and resume on the destination (this is stop-and-copy migration in its pure form), but for a multi-gigabyte guest over a 10 Gbit/s link that pause is many seconds — far past the ~hundreds-of-milliseconds downtime users tolerate. Pre-copy threads the needle: copy iteratively while running, so the final pause only has to transfer the small residual working set.

Enabling dirty tracking: the memslot flag

The VMM turns on dirty tracking per memory slot by setting KVM_MEM_LOG_DIRTY_PAGES (= (1UL << 0)) in the slot’s flags when calling KVM_SET_USER_MEMORY_REGION (kvm.h, v6.12). In QEMU this happens when memory_global_dirty_log_start() fires, which causes KVM to allocate a per-slot kernel bitmap via kvm_create_dirty_bitmap() and to begin tracking writes (terenceli, 2018).

Tracking writes, method one: write-protection

The classic mechanism reuses the memory-management unit. When dirty logging is enabled, KVM calls kvm_mmu_slot_remove_write_access() to strip write permission from every shadow page-table entry (SPTE) in the slot — concretely, it clears the writable bit in the second-dimension page tables that back NPT). The guest can still read freely, but the first write to such a page causes an EPT/NPT violation — a VM exit into KVM. KVM’s fault handler (fast_pf_fix_direct_spte()) does two things: it calls mark_page_dirty() to set the page’s bit in memslot->dirty_bitmap, and it restores write permission so subsequent writes to that same page do not trap again (terenceli, 2018). The cost: one VM exit per newly dirtied page per iteration. For a write-heavy guest this is brutal — every working-set page costs a trap on the first touch of each round.

Tracking writes, method two: hardware PML

Intel’s Page-Modification Logging (PML) removes the per-page write fault. Kai Huang’s original KVM PML patch series describes it: “PML logs dirty GPA automatically to a 4K PML memory buffer when CPU changes EPT table’s D-bit from 0 to 1” (Huang, LKML 2015). Three new VMCS (Virtual Machine Control Structure) fields appear: a PML Address pointing at a 4 KiB buffer, and a PML Index. The buffer holds up to 512 guest-physical addresses (8 bytes each); the PML Index starts at 511 and the CPU decrements it as it logs each dirtied guest-physical address. Crucially, with PML the page is not write-protected — KVM only clears the EPT entry’s dirty (D) bit, so the guest writes at full speed and the hardware records the address. When the buffer fills, the CPU raises a PML-buffer-full VM exit; KVM drains the 512 addresses into its dirty bitmap and resets the index. KVM also manually flushes every vCPU’s PML buffer when userspace queries the dirty set, so no in-flight dirty addresses are missed. Huang’s benchmarks showed roughly 4.18%–5% better guest performance under active dirty logging versus write-protection (Huang, LKML 2015). PML still incurs a VM exit, but one exit per 512 dirty pages instead of one per page.

Uncertain

Verify: that on a v6.12/6.18 host KVM uses PML by default on PML-capable Intel CPUs and falls back to write-protection only where PML is unavailable, and that AMD’s equivalent dirty-bit-based tracking (without a logging buffer) behaves analogously. Reason: the PML mechanism is documented from the 2015 introduction patch, not re-verified against the v6.12 arch/x86/kvm/vmx/ code in this task; the AMD path was not consulted. To resolve: read vmx_flush_pml_buffer() and the dirty-logging enable path in arch/x86/kvm/vmx/vmx.c and arch/x86/kvm/mmu/ at tag v6.12. uncertain

Harvesting the dirty set, interface one: the bitmap

The original userspace interface is KVM_GET_DIRTY_LOG (a VM ioctl, _IOW(KVMIO, 0x42, struct kvm_dirty_log)). Userspace passes a slot id and a userspace buffer; KVM fills in a bitmap, “one bit per page,” where “Bit 0 is the first page in the memory slot” (api.rst, v6.12):

struct kvm_dirty_log {
	__u32 slot;
	__u32 padding1;
	union {
		void __user *dirty_bitmap; /* one bit per page */
		__u64 padding2;
	};
};

By default “the bits in the dirty bitmap are cleared before the ioctl returns” — and clearing means re-arming tracking, i.e. re-write-protecting the pages. The defect of this interface is its shape: the bitmap is sized to the whole slot, so each iteration the VMM copies and the kernel walks a bitmap proportional to total guest RAM, even if the guest dirtied only a thousand pages. For a 256 GiB guest the bitmap alone is 8 MiB (256 GiB / 4 KiB / 8 bits), scanned every round.

The get-then-clear refinement: KVM_CAP_MANUAL_DIRTY_LOG_PROTECT2

A subtle inefficiency in the auto-clearing bitmap is that KVM re-protects a page the instant KVM_GET_DIRTY_LOG reports it, but the VMM may not actually send that page for many milliseconds. In that window the guest re-dirties the page, taking another write fault and producing a false-positive in the next round. KVM_CAP_MANUAL_DIRTY_LOG_PROTECT2 (capability 168) splits the operation: with the flag KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE (1 << 0) set, “KVM_GET_DIRTY_LOG will not automatically clear and write-protect all pages that are returned as dirty. Rather, userspace will have to do this operation separately using KVM_CLEAR_DIRTY_LOG” (api.rst, v6.12). The doc gives two reasons: KVM_CLEAR_DIRTY_LOG “can operate on a 64-page granularity rather than requiring to sync a full memslot” (so KVM does not hold MMU spinlocks for long), and manual reprotection “helps reducing this time, improving guest performance and reducing the number of dirty log false positives.” The companion flag KVM_DIRTY_LOG_INITIALLY_SET (1 << 1) initializes the whole bitmap to 1 at creation so dirty logging “can be enabled gradually in small chunks on the first call to KVM_CLEAR_DIRTY_LOG.” The clear ioctl takes a sub-range:

struct kvm_clear_dirty_log {
	__u32 slot;
	__u32 num_pages;
	__u64 first_page;
	union {
		void __user *dirty_bitmap; /* one bit per page */
		__u64 padding2;
	};
};

first_page must be a multiple of 64; num_pages must also be a multiple of 64 unless it reaches the end of the slot. (Note: KVM_CAP_MANUAL_DIRTY_LOG_PROTECT2 superseded the buggy original KVM_CAP_MANUAL_DIRTY_LOG_PROTECT (capability 166, marked obsolete); userspace “should not try to use” the old one.)

Harvesting the dirty set, interface two: the dirty ring

The bitmap’s O(guest size) scan cost is fixed by the dirty ring (KVM_CAP_DIRTY_LOG_RING, capability 192; available since Linux 5.11). “KVM is capable of tracking dirty memory using ring buffers that are mmapped into userspace; there is one dirty ring per vcpu” (api.rst, v6.12). Each entry is:

struct kvm_dirty_gfn {
	__u32 flags;
	__u32 slot; /* as_id | slot_id */
	__u64 offset;
};

The VMM calls KVM_ENABLE_CAP “right after KVM_CREATE_VM” and before creating any vCPU, passing the ring size, which “must be a power of two”; the doc recommends “at least 64 KiB (4096 entries).” Each kvm_dirty_gfn records one dirtied guest-frame as it happens, so the ring’s content is proportional to the number of pages actually written, not to total RAM. Entries flow through a three-state machine via the flags field — KVM_DIRTY_GFN_F_DIRTY = BIT(0), KVM_DIRTY_GFN_F_RESET = BIT(1), KVM_DIRTY_GFN_F_MASK = 0x3:

        dirtied         harvested        reset
   00 -----------> 01 -------------> 1X -------+
    ^                                          |
    +------------------------------------------+

KVM publishes a dirtied page by writing the entry and setting KVM_DIRTY_GFN_F_DIRTY (kvm_dirty_ring_push() in dirty_ring.c, v6.12 does smp_wmb() then kvm_dirty_gfn_set_dirtied()). Userspace walks the mmap’d ring, and for each entry whose DIRTY bit is set it copies the page, then sets bit 1 to mark it “harvested” (1X). After harvesting some entries the VMM calls the VM ioctl KVM_RESET_DIRTY_RINGS (_IO(KVMIO, 0xc7)); KVM walks the harvested entries (kvm_dirty_ring_reset()), re-protects each collected guest-frame via kvm_arch_mmu_enable_log_dirty_pt_masked(), and resets the entry to 00. The doc is emphatic about ordering: this reset “must be called before reading the content of the dirty pages.”

The ring as a throttle — the convergence win hiding in plain sight

The most important behavioral property of the ring is what happens when it fills. In kvm_dirty_ring_push(), if the ring crosses its soft_limit (its size minus a small reserve), KVM raises KVM_REQ_DIRTY_RING_SOFT_FULL on the vCPU. The next time that vCPU tries to enter the guest, kvm_dirty_ring_check_request() notices the request, sets vcpu->run->exit_reason = KVM_EXIT_DIRTY_RING_FULL, and returns to userspace instead of running the guest — and the comment is explicit: “The VCPU isn’t runnable when the dirty ring becomes soft full … to prevent the VCPU from running until the dirty pages are harvested and the dirty ring is reset by userspace” (dirty_ring.c, v6.12). This means a vCPU that dirties memory faster than the VMM can harvest is automatically stalled — a built-in back-pressure that directly helps migration converge, because a paused vCPU dirties nothing. QEMU’s dirty-limit feature (QEMU 8.1) builds on exactly this: it measures each vCPU’s dirty rate from “how many dirty pages a virtual CPU has had since the last KVM_EXIT_DIRTY_RING_FULL exception” and selectively throttles only the high-writing vCPUs, so “virtual CPUs tied with writing processes … get penalized, whereas virtual CPUs involved with read processes will not” (QEMU dirty-limit doc).

Uncertain

Verify: the exact exit-reason name. The kernel UAPI header defines KVM_EXIT_DIRTY_RING_FULL = 31 (kvm.h, v6.12) and virt/kvm/dirty_ring.c sets that constant, but the v6.12 Documentation/virt/kvm/api.rst text says the vCPU “will return with exit reason KVM_EXIT_DIRTY_LOG_FULL” — a name that does not exist in the header. Reason: this is a documentation typo in the kernel tree (code says ..._RING_FULL, prose says ..._LOG_FULL). To resolve: trust the header/source — the real constant is KVM_EXIT_DIRTY_RING_FULL (value 31). uncertain

A flush caveat unique to the ring

One correctness subtlety: with the bitmap, KVM_GET_DIRTY_LOG itself flushes the processor’s dirty buffers. With the ring, “it’s still possible that the kernel has not yet flushed the processor’s dirty page buffers into the kernel buffer” — so to be sure all dirty frames are visible before the final stop-and-copy, “one needs to kick the vcpu out of KVM_RUN using a signal. The resulting vmexit ensures that all dirty GFNs are flushed to the dirty rings” (api.rst, v6.12).

Pages dirtied without a vCPU: the ring-with-bitmap hybrid

Not every write comes from a vCPU. Some guest memory is dirtied by KVM itself outside vCPU context — e.g. on arm64 when saving the virtual GIC/ITS interrupt-controller tables. Because such a write has no vCPU and therefore no ring to push into, the ring alone would miss it. KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP (capability 225) addresses this: it keeps a per-slot bitmap as a backup for “dirty guest pages without vcpu/ring context.” The doc warns this is “only beneficial if there is only a very small amount of memory that is dirtied out of vcpu/ring context” — otherwise just use the standalone bitmap. It also cannot be enabled unless KVM_CAP_DIRTY_LOG_RING_ACQ_REL is enabled first and no memslot yet exists.

Memory ordering: the ACQ_REL variant

On weakly-ordered architectures (e.g. arm64), userspace reads and writes of a ring entry’s flags field must use load-acquire/store-release semantics so the kernel and VMM agree on entry state. KVM_CAP_DIRTY_LOG_RING_ACQ_REL (capability 223) signals that requirement: it “is the only capability that should be exposed by weakly ordered architecture.” A total-store-ordering (TSO) architecture like x86 may expose both KVM_CAP_DIRTY_LOG_RING and KVM_CAP_DIRTY_LOG_RING_ACQ_REL (api.rst, v6.12).

When pre-copy cannot converge: post-copy

If the guest’s dirty rate stubbornly exceeds the link’s transfer rate, pre-copy never reaches a small enough residual set — the iteration runs forever, burning bandwidth. Post-copy inverts the order: pause the guest, send only the minimal CPU and device state, and immediately restart the guest on the destination while its RAM still lives on the source. “In postcopy the destination CPUs are started before all the memory has been transferred, and accesses to pages that are yet to be transferred cause a fault that’s translated by QEMU into a request to the source QEMU” (QEMU postcopy doc). The destination registers guest RAM with the kernel’s userfaultfd in missing-page mode; a guest access to an un-arrived page raises a userfault, QEMU sends a page request to the source, receives the page, and “wakes” the faulting thread. The source also background-streams pages it has not yet sent, “in much the same way as precopy.” Post-copy guarantees a bounded total transfer (each page moves exactly once) and therefore always converges — but at a steep cost in risk, covered next.

Failure Modes and Common Misunderstandings

Non-convergence (the central failure). A guest with a large, hot, randomly-written working set dirties pages faster than the link drains them; the residual dirty set never shrinks below the downtime budget and pre-copy loops indefinitely. QEMU’s two countermeasures both slow the guest: auto-converge throttles all vCPUs by stealing CPU time (injecting sleeps), and the newer dirty-limit uses the dirty ring to throttle only the heavy-writing vCPUs. Both trade guest performance during migration for the ability to finish.

Uncertain

Verify: auto-converge’s exact throttle schedule (commonly described as starting around 20% CPU throttling and escalating in steps until convergence). Reason: the primary QEMU Features/AutoconvergeLiveMigration wiki page was access-denied during this task; the mechanism (CPU-time throttling of vCPUs) is corroborated by the QEMU live-migration overview but the specific percentages were not verified against a primary source. To resolve: read the QEMU migration/throttle code or the auto-converge feature page directly. uncertain

Post-copy’s catastrophic failure mode. During post-copy “a failure of either side causes the guest to be lost,” because “VM data resides in both source and destination QEMU instances” (QEMU postcopy doc) — the authoritative RAM image is split across two hosts and the network between them. By contrast, pre-copy can always abort cleanly: the source still holds a complete, authoritative copy until the instant of switch-over, so a failed pre-copy just resumes on the source. This is why production systems typically start in pre-copy and only transition to post-copy if convergence fails — getting the convergence guarantee while minimizing the window of unrecoverable risk.

Confusing “dirty bit” with “dirty log.” The CPU’s per-page-table-entry dirty bit (set by hardware on write) is a building block; the dirty log is KVM’s accumulated set of which guest frames were written since the VMM last asked. PML bridges them: hardware sets the EPT D-bit, the CPU logs the address, KVM aggregates into the log. Write-protection skips the D-bit entirely and uses page faults as the signal.

Forgetting the final flush. With the ring, failing to signal-kick vCPUs before the last harvest can leave dirty frames stuck in a processor buffer, producing silent memory corruption on the destination. The bitmap path hides this because the get ioctl flushes for you.

Alternatives and When to Choose Them

  • Stop-and-copy (pure offline migration). Pause, copy everything, resume on destination. Simplest, always converges, but downtime ≈ (guest RAM / link bandwidth) — unacceptable for large live guests. Fine for small VMs or when downtime does not matter.
  • Pre-copy. The default. Lowest risk (source stays authoritative until switch-over), low downtime for typical workloads, but can fail to converge under heavy write pressure — mitigated by throttling.
  • Post-copy. Bounded, guaranteed-converging transfer with constant low downtime, but a single host/network failure during the post-copy phase loses the VM. Best as a fallback from pre-copy.
  • Hybrid pre-copy → post-copy. Run pre-copy passes; if not converging within a deadline, flip to post-copy. QEMU supports this; it captures pre-copy’s safety for the common case and post-copy’s guarantee for the pathological one.

The dirty-ring vs bitmap choice is orthogonal: prefer the ring for large guests (cost scales with dirtying, plus free back-pressure throttling); the bitmap remains necessary for memory dirtied outside vCPU context (hence the with-bitmap hybrid).

Production Notes

QEMU’s pre-copy loop is concretely ram_save_iterate() → ram_find_and_save_block(), syncing the dirty set each round via migration_bitmap_sync() → kvm_physical_sync_dirty_bitmap() → the KVM_GET_DIRTY_LOG ioctl, whose kernel handler kvm_vm_ioctl_get_dirty_log() “copies the dirty bitmap to userspace and also set the spte to write protection” (terenceli, 2018). The completion criterion is a downtime budget: QEMU estimates the time to transfer the residual dirty set at current bandwidth and triggers stop-and-copy once that estimate drops below max_downtime. The dirty-ring-driven dirty-limit (QEMU 8.1) is the current best-practice convergence tool because it throttles surgically rather than blindly (QEMU dirty-limit doc).

A migration-specific interaction with confidential computing: when guest memory is hardware-encrypted (AMD SEV-SNP, Intel TDX), the host cannot read the plaintext pages it is trying to copy, which breaks naive pre-copy and demands hardware-assisted migration paths — see Confidential Computing and the Trust Boundary. Dirty tracking itself also sits directly on the host memory manager: tracked guest pages are still subject to swap, Kernel Samepage Merging, transparent huge pages, and NUMA balancing (cross-link Linux Memory Management MOC), and the generic page-write-back machinery is a useful contrast — see Dirty Pages and Writeback for the unrelated filesystem sense of “dirty page,” which this note is careful not to conflate with KVM’s guest-memory dirty tracking.

See Also

Graph View

Backlinks