vault

The Time Stamp Counter and TSC Reliability

18 min read

The Time Stamp Counter and TSC Reliability

The Time Stamp Counter (TSC) is a 64-bit model-specific register (MSR) inside every x86 core that counts upward and is read with a single unprivileged instruction — rdtsc (Read Time-Stamp Counter), which loads the count into the EDX:EAX register pair (felixcloutier RDTSC). Because reading it costs a handful of cycles and — crucially — can be done from userspace without a system call, the TSC is the Linux kernel’s preferred clocksource on x86 and the engine behind the vDSO fast path for clock_gettime. But the TSC carries decades of baggage: on older CPUs it counted actual core cycles, so it sped up and slowed down with frequency scaling, drifted between cores, and froze in deep idle. Modern processors fix this with the invariant TSC — a counter that ticks at a fixed rate in every power state — exposed in Linux as the pair of feature flags constant_tsc and nonstop_tsc (arch/x86/kernel/tsc.c, v6.12). This note explains the hardware, the historical failure modes, and the layered machinery Linux uses to decide whether to trust the TSC at all: the boot-time synchronization check and the runtime clocksource watchdog.

This note pins to the Linux 6.12 LTS kernel (released 2024-11-17) as tracked by Linux Time and Timers MOC.

Mental Model

Think of the TSC as a stopwatch baked into the silicon of each core. The earliest stopwatches were geared directly to the engine: when the engine (the core clock) sped up under load, the stopwatch ran fast; when it idled, the stopwatch stopped. That made it useless as a clock even though it was a perfectly good cycle counter. The fix was to drive the stopwatch from a separate, fixed-rate crystal that never changes speed and never stops — the invariant TSC. Once every core’s stopwatch is driven by that same fixed reference and they all start at the same instant, you have something close to a free, monotonic, system-wide clock.

The kernel does not take that “close to” for granted. It runs the TSC through two trust gates. The first is a one-time synchronization check at boot, when each secondary CPU comes online: do the cores actually read the same value at the same moment? The second is a continuously running watchdog that, for the whole life of the system, cross-checks the TSC against an independent, slower-but-trusted timer (the High Precision Event Timer or the ACPI power-management timer) and yanks the TSC offline the instant the two disagree by more than a tolerance.

flowchart TB
  RD["rdtsc / rdtscp<br/>(userspace, ~20-40 cycles)"] --> TSCMSR["TSC MSR<br/>(per-core 64-bit counter)"]
  TSCMSR -->|"fixed rate?"| INV{"invariant TSC?<br/>constant_tsc AND nonstop_tsc"}
  INV -->|"no (old CPU)"| BAD["counts actual cycles<br/>halts in C-states<br/>-> untrustworthy"]
  INV -->|"yes"| SYNC{"boot sync check<br/>check_tsc_sync_*"}
  SYNC -->|"warp detected"| UNSTABLE["mark_tsc_unstable()<br/>fall back to HPET"]
  SYNC -->|"in sync"| WD{"clocksource watchdog<br/>vs HPET / PM-timer"}
  WD -->|"skew > threshold"| UNSTABLE
  WD -->|"agrees"| TRUST["clocksource_tsc<br/>rating 300, VALID_FOR_HRES<br/>used by vDSO"]

The TSC trust pipeline in Linux x86. What it shows: the raw per-core counter must clear two independent gates — a boot-time pairwise synchronization test and a perpetual runtime watchdog comparison against a slower reference clock — before it becomes clocksource_tsc (rating 300, the highest on x86) and is offered to the vDSO. Any failure at any stage routes to mark_tsc_unstable() and a fallback to the HPET. The insight to take: the TSC is fast but historically untrustworthy, so Linux never trusts it blindly — it earns its place as the primary clocksource by passing verification, and can lose that place at runtime.

The Hardware: rdtsc, rdtscp, and the Invariant TSC

The TSC is a 64-bit MSR that the processor “monotonically increments… every clock cycle and resets… to 0 whenever the processor is reset” in the original definition (Intel SDM Vol 3, p. 615 mirror). Two instructions read it. rdtsc reads the 64-bit count into EDX:EAX (high 32 bits in EDX, low 32 in EAX). It is not a serializing instruction: “It does not necessarily wait until all previous instructions have been executed before reading the counter” (felixcloutier RDTSC). On an out-of-order core that is a real hazard for micro-benchmarking — the processor may execute the rdtsc before, or after, the code you meant to time. The fix is a fence: software inserts LFENCE (or MFENCE) before the read to pin it to a program point.

rdtscp (Read Time-Stamp Counter and Processor ID) is the slightly heavier sibling. It reads the same 64-bit TSC into EDX:EAX and loads the IA32_TSC_AUX MSR (address 0xC0000103) into ECX (felixcloutier RDTSCP). Critically, rdtscp “waits until all previous instructions have executed and all previous loads are globally visible” before reading — a partial ordering guarantee rdtsc lacks, though it is still not a full serializing instruction. Linux populates IA32_TSC_AUX so the low 12 bits hold the logical CPU number and the next 12 bits the NUMA node/socket number; this is exactly what lets a vDSO reader know which core’s TSC it just read and detect if it migrated mid-read.

The reliability story is told by three capability tiers, which Linux maps onto CPUID feature bits in arch/x86/include/asm/cpufeatures.h (v6.12):

  • Variant TSC (oldest): the counter is geared to the actual core clock. There is no flag for this — it is simply the absence of constant_tsc.
  • Constant TSC: X86_FEATURE_CONSTANT_TSC at bit 3*32+8, named "constant_tsc", comment “TSC ticks at a constant rate” (cpufeatures.h v6.12). The counter no longer changes rate with P-states (frequency scaling), but it may still halt in deep C-states (idle).
  • Nonstop / invariant TSC: add X86_FEATURE_NONSTOP_TSC at bit 3*32+24, named "nonstop_tsc", comment “TSC does not stop in C states”. The combination of constant_tsc and nonstop_tsc is what the Intel architecture calls the invariant TSC, advertised by CPUID.80000007H:EDX[8], which “will run at a constant rate in all ACPI P-, C-, and T-states” (Intel Community: Invariant TSC support). On Linux you see both constant_tsc and nonstop_tsc in the flags line of /proc/cpuinfo on any reasonably modern machine.

Two more flags matter for trust. X86_FEATURE_TSC_ADJUST (bit 9*32+1, "tsc_adjust", “TSC adjustment MSR 0x3B”) gives the kernel the IA32_TSC_ADJUST MSR — a per-logical-CPU offset that is added to the hardware counter, letting software correct skew without disturbing the underlying crystal. And X86_FEATURE_TSC_RELIABLE (bit 3*32+23, "tsc_reliable", “TSC is known to be reliable”) is a synthetic flag the kernel or hypervisor sets to assert the TSC needs no verification at all.

The Historical Reliability Problems

Three distinct hardware behaviors made the TSC notoriously dangerous to use as a clock, and understanding them explains every line of Linux’s defensive code.

1. Frequency changes (the SpeedStep / P-state problem). On variant-TSC CPUs the counter incremented once per actual core cycle. When Intel SpeedStep or AMD PowerNow dropped the core to a lower P-state to save power, the TSC slowed down with it; under turbo boost it sped up. A program timing a fixed wall-clock interval would read wildly different cycle counts depending on what the frequency governor happened to be doing (oliveryang, Pitfalls of TSC usage). The constant-TSC fix decouples the counter from the core clock: it now ticks at the processor’s nominal/base frequency regardless of the current P-state. The counter no longer measures “cycles of work done” — it measures elapsed time, which is what a clock wants.

2. Halting in deep C-states. Even with a constant rate, early implementations stopped the counter when the core entered a deep idle state (C3 and below). A CPU that idled for a second would, on wakeup, show a TSC that had not advanced by a second’s worth of ticks. Linux warns about this case — without nonstop_tsc it cannot trust the counter across idle. The nonstop/invariant fix keeps the counter running in every C-state, driven from an always-on power domain.

3. Per-core and per-socket skew. Each core has its own TSC MSR. Nothing in the architecture guarantees they all read the same value at the same instant — cores can be released from reset at slightly different times, sockets can be reset independently, and BIOS code can write the MSR per-core. The symptom is time going backwards: a thread reads the TSC on core 0, migrates to core 1, reads again, and gets a smaller number. On invariant-TSC single-package systems the cores share a reset and a clock domain, so they start together and stay together; across sockets it depends on the platform wiring synchronizing the reset signal. Multi-board NUMA machines (the classic example is SGI UV) give no cross-blade sync guarantee at all (oliveryang).

Uncertain

Verify: that all invariant-TSC multi-socket systems keep the TSC synchronized across sockets via a shared reset/clock distribution. Reason: the Intel SDM guarantees a constant rate per package but the cross-socket start-value synchronization is a platform/firmware property, not an architectural one — secondary sources assert it is “usually” synchronized on modern boards. To resolve: consult the specific platform’s firmware documentation, or rely on the kernel’s own boot sync check, which is exactly why Linux verifies rather than assumes.

Mechanical Walk-through: How Linux Decides to Trust the TSC

The decision lives in tsc_init() in arch/x86/kernel/tsc.c (v6.12), and it proceeds in stages.

Stage 1 — static reliability assessment. tsc_init() calls check_system_tsc_reliable(). That function sets tsc_clocksource_reliable = 1 if the CPU advertises X86_FEATURE_TSC_RELIABLE (for example a hypervisor told us so, or a known-good SoC like the OLPC Geode). More importantly, it decides whether to skip the runtime watchdog entirely. The condition, quoted from the v6.12 comment, disables the watchdog when the system has: “TSC running at constant frequency”, “TSC which does not stop in C-States”, “the TSC_ADJUST register which allows to detect even minimal modifications”, and “not more than four packages” — in code:

if (boot_cpu_has(X86_FEATURE_CONSTANT_TSC) &&
    boot_cpu_has(X86_FEATURE_NONSTOP_TSC) &&
    boot_cpu_has(X86_FEATURE_TSC_ADJUST) &&
    topology_max_packages() <= 4)
        tsc_disable_clocksource_watchdog();

The reasoning is precise: a constant + nonstop TSC with the adjust MSR available, on a system of at most four sockets, is trustworthy enough that the perpetual cross-check would only burn cycles. The four-package limit is a heuristic ceiling on cross-socket skew risk.

Stage 2 — synchronization verdict. tsc_init() then calls unsynchronized_tsc(), an “educated guess” (the comment’s word) about whether the TSC is the same across all CPUs. It returns 1 (unsynchronized) if the TSC feature is absent or tsc_unstable is already set, or if the box is an APIC-clustered machine. It returns 0 (synchronized) immediately if constant_tsc is present, or if tsc_clocksource_reliable was set. The interesting fallback: for non-Intel vendors it assumes multi-socket systems are not synchronized (topology_max_packages() > 1 → return 1), because — per the comment — “Intel systems are normally all synchronized. Exceptions must mark TSC as unstable.” If unsynchronized_tsc() returns true, tsc_init() calls mark_tsc_unstable("TSCs unsynchronized") and gives up on the TSC.

Stage 3 — clocksource registration. If the TSC survives, tsc_init() registers clocksource_tsc_early (rating 299) right away for early boot. The full-fidelity clocksource_tsc is registered later, by tsc_refine_calibration_work(), once a longer calibration against a hardware reference (HPET or the PM timer, over roughly one second) has refined the frequency. The two clocksources differ:

  • clocksource_tsc_early: rating 299, flags CLOCK_SOURCE_IS_CONTINUOUS | CLOCK_SOURCE_MUST_VERIFY.
  • clocksource_tsc: rating 300 (the highest on x86, beating the HPET’s 250 and the PM timer’s lower rating), flags CLOCK_SOURCE_IS_CONTINUOUS | CLOCK_SOURCE_VALID_FOR_HRES | CLOCK_SOURCE_MUST_VERIFY | CLOCK_SOURCE_VERIFY_PERCPU.

The VALID_FOR_HRES flag is what lets the TSC drive high-resolution timer mode; MUST_VERIFY enrolls it in the watchdog; VERIFY_PERCPU tells the watchdog to additionally cross-check the per-CPU readings against each other (catching skew the single-reference watchdog would miss). Its .read callback is read_tsc(), which simply returns rdtsc_ordered() (an rdtsc with the appropriate fence). When this clocksource is enabled, tsc_cs_enable() calls vclocks_set_used(VDSO_CLOCKMODE_TSC), wiring the TSC into the vDSO so userspace clock_gettime reads it directly. See Clocksource Rating and Selection for how the rating numbers drive the choice.

The Boot-Time Synchronization Check

When a secondary CPU comes online, arch/x86/kernel/tsc_sync.c (v6.12) runs a pairwise duel: the already-running boot CPU runs check_tsc_sync_source() while the newly-arrived CPU runs check_tsc_sync_target(). The target first calls tsc_store_and_check_tsc_adjust(), which reads the IA32_TSC_ADJUST MSR and compares it to the reference value for its package; if a sibling already established a per-package adjust value, the new CPU writes the same value to its own MSR so the two cores agree. This is the mechanism that lets the kernel correct small offsets rather than condemn the whole TSC.

The core test is check_tsc_warp(), run by both CPUs simultaneously under a shared raw spinlock (sync_lock). Each CPU repeatedly reads the TSC with rdtsc_ordered() and records the maximum backward jump it sees relative to the last value any CPU wrote into a shared last_tsc variable:

if (unlikely(prev > now)) {
    arch_spin_lock(&sync_lock);
    max_warp = max(max_warp, prev - now);
    ...
}

A “warp” is the smoking gun of skew: if CPU B reads a value smaller than one CPU A already published, the two clocks are not the same clock. The test runs for a duration set by loop_timeout(cpu) — roughly 20 ms when the CPU is the only one in its socket, dropping to about 2 ms when several cores are already present. The source CPU runs the test up to test_runs times: 1 if there is no TSC_ADJUST MSR, 3 if there is — between runs it can nudge the adjust MSR and re-test, so a correctable offset is fixed rather than fatal.

If, after the retries, a warp persists, the boot CPU schedules tsc_sync_mark_tsc_unstable() via a workqueue, which calls mark_tsc_unstable("check_tsc_sync_source failed"). There is one important escape hatch: some firmware resets sockets asynchronously, deliberately leaving a non-zero TSC_ADJUST on the boot CPU; tsc_sanitize_first_cpu() forces that MSR back to 0 unless tsc_async_resets is set, in which case the firmware’s value is preserved on purpose (a FW_BUG workaround).

The Runtime Watchdog

Passing the boot check is not a lifetime guarantee — a TSC can drift later (thermal effects, exotic power management, a buggy BIOS SMM handler). So the generic clocksource watchdog in kernel/time/clocksource.c (v6.12) keeps watching any clocksource flagged CLOCK_SOURCE_MUST_VERIFY, which includes the TSC. A timer_list fires every WATCHDOG_INTERVAL, defined (HZ >> 1) — half a second. On each tick clocksource_watchdog() reads both the watchdog reference (a trusted clocksource that is not flagged MUST_VERIFY — typically the High Precision Event Timer or the ACPI PM timer, selected by clocksource_select_watchdog() as the highest-rated trusted source) and the clocksource under test, in a tight window with interrupts disabled, and converts both to nanoseconds.

The tolerance is WATCHDOG_THRESHOLD, defined (NSEC_PER_SEC >> 5) — about 0.0312 s, and “when doubled: 0.0625s” per the source comment. If the TSC and the reference disagree by more than this margin (combined with each clocksource’s uncertainty_margin, whose floor is WATCHDOG_MAX_SKEW, the 500 ppm lower bound borrowed from NTP’s limits), the TSC is condemned. The read itself is retried by cs_watchdog_read(): an SMI, NMI, or VCPU preemption between the two reads would inflate the apparent delay, so the function retries up to clocksource_get_max_watchdog_retry() times, returning WD_READ_SKIP (suspend the watchdog briefly) if the system is simply overloaded rather than the clock being broken.

When the verdict is “unstable,” __clocksource_unstable() strips the offending flags — “cs→flags &= ~(CLOCK_SOURCE_VALID_FOR_HRES | CLOCK_SOURCE_WATCHDOG); cs→flags |= CLOCK_SOURCE_UNSTABLE” — and schedules clocksource_watchdog_work() to re-select. Losing VALID_FOR_HRES drops the TSC out of high-resolution timer service; the watchdog then promotes the next-best clocksource (the HPET) to primary. For the TSC specifically, the callback tsc_cs_mark_unstable() additionally sets tsc_unstable = 1, calls clear_sched_clock_stable() so the scheduler stops trusting TSC timestamps (see sched_clock and Scheduler Timestamps), and prints “Marking TSC unstable due to clocksource watchdog.” Because the TSC also carries VERIFY_PERCPU, the watchdog periodically runs clocksource_verify_percpu(), sampling the TSC on multiple CPUs and reporting any that are cpus_ahead or cpus_behind — catching per-core skew the single-reference comparison cannot. The number of CPUs sampled is bounded by clocksource.verify_n_cpus= (default 8).

Command-Line Knobs

The tsc= parameter overrides the kernel’s judgment. The v6.12 Documentation/admin-guide/kernel-parameters.txt documents it verbatim:

  • tsc=reliable — “mark tsc clocksource as reliable, this disables clocksource verification at runtime, as well as the stability checks done at bootup. Used to enable high-resolution timer mode on older hardware, and in virtualized environment.” This sets tsc_clocksource_reliable = 1. Dangerous if you are wrong: you turn off both gates.
  • tsc=unstable — “mark the TSC clocksource as unstable, this marks the TSC unconditionally unstable at bootup and avoids any further wobbles once the TSC watchdog notices.” Calls mark_tsc_unstable("boot parameter"). Use when you know the TSC is bad and want to skip the discovery latency.
  • tsc=nowatchdog — “disable clocksource watchdog. Used in situations with strict latency requirements (where interruptions from clocksource watchdog are not acceptable).” The periodic watchdog read is itself a source of jitter, undesirable for nohz_full real-time isolation.
  • tsc=recalibrate — “force recalibration against a HW timer (HPET or PM timer) on systems whose TSC frequency was obtained from HW or FW using either an MSR or CPUID(0x15). Warn if the difference is more than 500 ppm.”
  • tsc=watchdog — “Use TSC as the watchdog clocksource with which to check other HW timers (HPET or PM timer), but only on systems where TSC has been deemed trustworthy.” The inverse role: trust the TSC enough to police the HPET.

You can also force a different clocksource entirely with clocksource=hpet (or clocksource=acpi_pm) per the clocksource= parameter, useful for diagnosing whether a bug is TSC-specific.

Common Misunderstandings

  • “rdtsc gives you cycle counts.” On an invariant TSC it does not — it counts at the nominal frequency, not the current one. A 3.0 GHz nominal core running at 1.2 GHz still advances the TSC at 3.0 GHz-equivalent. The TSC measures time, not work; to count actual cycles you need the performance counters (CPU_CLK_UNHALTED), not the TSC.
  • “rdtsc is ordered like any other read.” It is not serializing; without an LFENCE the CPU may hoist or sink it. Naive rdtsc-around-a-loop micro-benchmarks are routinely wrong for this reason.
  • “If the kernel picked TSC, it’s perfectly synchronized forever.” The watchdog exists precisely because that is false. A TSC that was synchronized at boot can be marked unstable hours later; check dmesg for “Marking TSC unstable” and /sys/devices/system/clocksource/clocksource0/current_clocksource to see if it fell back to hpet.
  • “More sockets is fine.” check_system_tsc_reliable() keeps the watchdog on above four packages by design — large multi-socket boxes are exactly where cross-socket skew bites, so the kernel refuses to disable verification there.

Alternatives and When to Choose Them

The TSC’s competitors are the High Precision Event Timer (rating 250) and the ACPI power-management timer (a ~3.58 MHz I/O-port counter, slow but rock-solid). The honest comparison: the TSC wins on read cost by an order of magnitude (a register read versus an MMIO or I/O-port access that can take hundreds of nanoseconds), and it is the only x86 clocksource reachable from the vDSO without a syscall. The HPET and PM timer win on trustworthiness without verification — they are platform devices with no per-core skew and no frequency-scaling history, which is exactly why they serve as the watchdog reference. Choose the HPET as primary only when the TSC has been condemned (or you force it for debugging); choose the PM timer when even the HPET is buggy. In a virtual machine, neither bare-metal answer is ideal — see Timer Virtualization (kvmclock and TSC) for how kvmclock and a virtualized, host-synchronized TSC change the calculus.

Production Notes

The single most common real-world symptom is a workload that was using the TSC silently falling back to the HPET after the watchdog fires, with a visible performance cliff: clock_gettime latency jumps from a few nanoseconds (vDSO + TSC) to hundreds of nanoseconds (HPET MMIO), and anything calling it in a hot loop — databases, tracing, high-frequency trading stacks — degrades sharply. The diagnostic ritual is to check current_clocksource, grep dmesg for TSC, and confirm constant_tsc nonstop_tsc are present in /proc/cpuinfo. For latency-isolated cores under nohz_full, operators commonly pass tsc=nowatchdog to eliminate the half-second watchdog interrupt as a jitter source — but only after independently verifying the TSC is genuinely invariant, since they are switching off the safety net. Cloud and virtualized environments frequently rely on tsc=reliable (or the hypervisor setting X86_FEATURE_TSC_RELIABLE) because the host guarantees a synchronized, non-stop virtual TSC and the guest’s own watchdog would only add noise.

See Also

Graph View

Backlinks