vault

Suspend Resume and Timekeeping

14 min read

Suspend Resume and Timekeeping

When a system enters a deep sleep state such as ACPI S3 (“suspend-to-RAM”), the hardware counter the kernel uses to tell time — the clocksource — frequently stops counting: the x86 Time Stamp Counter (TSC) halts when the CPU loses power, and an architected timer may even reset to zero on the way back up. The kernel therefore cannot learn how long it slept by reading the same counter it normally reads. Instead, on the way down it snapshots a persistent clock (a battery-backed Real-Time Clock, RTC) and stops the timekeeper; on the way back up it reads that persistent clock again, computes the elapsed wall-clock interval, and injects that interval into its time bookkeeping so the clocks resume showing sane values. This injection is the entire reason CLOCK_BOOTTIME advances across suspend while plain CLOCK_MONOTONIC does not — they are deliberately fed differently. The machinery lives in timekeeping_suspend() and timekeeping_resume() in kernel/time/timekeeping.c (v6.12 source).

This note pins to the Linux 6.12 LTS tree (released 2024-11-17). Mechanisms are stable across the 6.x series, but line-level details (function names, the three-source preference order) were read directly from v6.12.

Mental Model

The kernel maintains two complementary notions of time. A monotonically rising counter (the clocksource) answers “how many nanoseconds have ticked since some reference”; this is what CLOCK_MONOTONIC and CLOCK_REALTIME are built on. But that counter is only trustworthy while the CPU is powered and the counter keeps incrementing. Across a power-down it is dead. The persistent clock — an RTC chip on a coin-cell battery, the MC146818 CMOS clock on legacy x86, or its ACPI/EFI equivalent — keeps ticking regardless. The kernel uses the persistent clock as an external witness to the passage of wall-clock time over the gap.

sequenceDiagram
    participant App as Userspace
    participant TK as timekeeper (tk_core)
    participant CS as clocksource (TSC/arch timer)
    participant RTC as persistent clock (RTC)
    Note over CS: counting normally
    App->>TK: clock_gettime() works
    Note over TK,CS: --- suspend begins ---
    TK->>RTC: read_persistent_clock64() → T_suspend
    TK->>TK: timekeeping_forward_now() then suspended=1
    TK->>CS: halt_fast_timekeeper(), clocksource_suspend()
    Note over CS: STOPPED (no power)
    Note over TK,CS: --- machine asleep N seconds ---
    Note over TK,CS: --- resume begins ---
    TK->>RTC: read_persistent_clock64() → T_resume
    TK->>TK: delta = T_resume − T_suspend
    TK->>TK: __timekeeping_inject_sleeptime(delta)
    TK->>CS: re-base cycle_last, suspended=0
    App->>TK: clock_gettime() works again

The suspend/resume timekeeping handshake. What it shows: the timekeeper takes a persistent-clock reading on the way down (T_suspend), freezes itself, and on the way back up takes a second reading (T_resume); the difference is the slept-away wall-clock interval that gets injected back into the clocks. The insight to take: the kernel cannot trust its own clocksource across the gap, so it borrows a second, always-on time source as a witness and reconciles the two on resume.

Mechanical Walk-through

The persistent clock abstraction

The persistent clock is exposed through the weak symbol read_persistent_clock64() (timekeeping.c:1713):

void __weak read_persistent_clock64(struct timespec64 *ts)
{
	ts->tv_sec = 0;
	ts->tv_nsec = 0;
}

The default is a no-op returning zero — meaning “this architecture has no persistent clock the timekeeper can read with interrupts off.” Architectures that do have one override it. On x86 (arch/x86/kernel/rtc.c:108):

void read_persistent_clock64(struct timespec64 *ts)
{
	x86_platform.get_wallclock(ts);
}

which on a PC typically routes to mach_get_cmos_time(), reading the MC146818 RTC over I/O ports 0x70/0x71. Crucially the RTC has one-second resolution (now->tv_nsec = 0 is hard-coded in mach_get_cmos_time), so the sleep interval the kernel learns this way is only accurate to a second — a fact that drives the drift-compensation logic discussed below.

There is a second, related weak function used only at boot, read_persistent_wall_and_boot_offset(), which returns both the wall time and an estimate of how long the machine has already been up (using local_clock() as a fallback). That establishes the initial relationship wall time + wall_to_monotonic = boot time in timekeeping_init(). It matters here because the same wall_to_monotonic offset gets adjusted during sleep-time injection.

Suspend: timekeeping_suspend()

timekeeping_suspend() and timekeeping_resume() are registered as syscore_ops (timekeeping.c:2038), meaning they run very late on suspend and very early on resume — after device drivers are quiesced and before they come back, with interrupts off on the boot CPU. The suspend path does the following (timekeeping.c:1968):

  1. Snapshot the persistent clock: read_persistent_clock64(&timekeeping_suspend_time). This records the wall-clock instant at which we are about to go to sleep. If the value is non-zero, the kernel learns at runtime that a persistent clock exists and sets persistent_clock_exists = true.
  2. Flag that sleep timing will be needed: suspend_timing_needed = true. This flag coordinates with the RTC subsystem (see “Three sources” below).
  3. Forward the timekeeper to now: timekeeping_forward_now(tk) reads the clocksource one last time and advances xtime to the current instant, so cycle_last holds the final pre-suspend counter value. Then it sets timekeeping_suspended = 1.
  4. Start suspend timing on the clocksource: clocksource_start_suspend_timing(curr_clock, cycle_now). Some clocksources (notably those backed by a non-stop counter that survives suspend) can report the slept interval themselves; this primes that path.
  5. Drift compensation: if a persistent clock exists, the kernel computes delta = xtime − timekeeping_suspend_time and compares it to the old_delta from the previous suspend. Because the RTC has one-second granularity, every suspend/resume cycle can introduce up to ~1 second of rounding error; left uncorrected this would accumulate. By tracking the change in the offset (delta_delta) and nudging timekeeping_suspend_time to keep the system-vs-persistent offset roughly constant, repeated suspends do not drift the clock by a second each time. If delta_delta jumps by ≥2 seconds the kernel assumes a real time correction happened and resets its baseline.
  6. Halt the fast timekeeper: halt_fast_timekeeper(tk) freezes the NMI-safe fast readout path so that lockless readers (tracing, ktime_get_mono_fast_ns()) return a frozen-but-consistent value rather than touching a dead clocksource. Finally tick_suspend(), clocksource_suspend(), and clockevents_suspend() shut down the tick and timer hardware.

Resume: timekeeping_resume()

On the way back up (timekeeping.c:1906):

  1. Read the persistent clock again: read_persistent_clock64(&ts_new). Then clockevents_resume() and clocksource_resume() revive the timer hardware.
  2. Compute the slept interval from the best available source. The code tries three sources in a strict preference order, and this is the heart of the note:
cycle_now = tk_clock_read(&tk->tkr_mono);
nsec = clocksource_stop_suspend_timing(clock, cycle_now);
if (nsec > 0) {
	ts_delta = ns_to_timespec64(nsec);
	inject_sleeptime = true;
} else if (timespec64_compare(&ts_new, &timekeeping_suspend_time) > 0) {
	ts_delta = timespec64_sub(ts_new, timekeeping_suspend_time);
	inject_sleeptime = true;
}
  • First choice — a non-stop clocksource. If the clocksource kept counting through suspend (some platforms have an always-on counter), clocksource_stop_suspend_timing() returns the elapsed nanoseconds directly. This is the highest-resolution, most accurate source.
  • Second choice — the persistent clock. Otherwise, if ts_new (the resume reading) is later than timekeeping_suspend_time (the suspend reading), the difference is the slept interval, at one-second RTC resolution.
  • Third choice — the RTC, handled elsewhere. If neither works, inject_sleeptime stays false here and the RTC subsystem injects the interval later via timekeeping_inject_sleeptime64() (see below).
  1. Inject: if a delta was found, __timekeeping_inject_sleeptime(tk, &ts_delta) does the actual bookkeeping (next section).
  2. Re-base the counter: tk->tkr_mono.cycle_last = cycle_now (and the raw counterpart) sets the timekeeper’s reference cycle to the current counter value, so that subsequent reads measure forward from now rather than counting the dead gap as elapsed cycles. timekeeping_suspended = 0 re-enables normal operation.
  3. Notify the rest of the kernel: tick_resume() brings the tick back, and timerfd_resume() wakes timerfd consumers because, from their perspective, resume is equivalent to a clock_was_set() event — the wall clock jumped forward.

The injection itself: __timekeeping_inject_sleeptime()

This is the function that makes the monotonic/boottime distinction concrete (timekeeping.c:1817):

static void __timekeeping_inject_sleeptime(struct timekeeper *tk,
					   const struct timespec64 *delta)
{
	if (!timespec64_valid_strict(delta)) {
		printk_deferred(...); /* "Invalid sleep delta value!" */
		return;
	}
	tk_xtime_add(tk, delta);
	tk_set_wall_to_mono(tk, timespec64_sub(tk->wall_to_monotonic, *delta));
	tk_update_sleep_time(tk, timespec64_to_ktime(*delta));
	tk_debug_account_sleep_time(delta);
}

Walk it line by line, because each line targets a different clock:

  • tk_xtime_add(tk, delta) — xtime is the realtime base. Adding the slept interval advances CLOCK_REALTIME by the full slept amount. This is correct: wall-clock time really did pass.
  • tk_set_wall_to_mono(tk, wall_to_monotonic − delta) — wall_to_monotonic is the offset such that monotonic = realtime + wall_to_monotonic. By subtracting delta from this offset at the same instant xtime gains delta, the two changes cancel out for the monotonic clock. The net effect: CLOCK_MONOTONIC does not jump — it appears frozen across the sleep gap, which is exactly the documented behaviour (“This clock does not count time that the system is suspended,” per clock_gettime(2)).
  • tk_update_sleep_time(tk, delta) — this adds delta to tk->offs_boot (timekeeping.c:168):
static inline void tk_update_sleep_time(struct timekeeper *tk, ktime_t delta)
{
	tk->offs_boot = ktime_add(tk->offs_boot, delta);
	tk->monotonic_to_boot = ktime_to_timespec64(tk->offs_boot);
}

offs_boot is the offset added to the monotonic reading to produce CLOCK_BOOTTIME: boottime = monotonic + offs_boot. Because CLOCK_MONOTONIC was held still across the gap but offs_boot just grew by the slept interval, CLOCK_BOOTTIME advances by exactly the suspended duration. This is the single mechanism that distinguishes the two clocks. You can see it directly in the fast accessor (timekeeping.c:529): ktime_get_boot_fast_ns() returns ktime_get_mono_fast_ns() + offs_boot.

So the same injection touches three clocks differently: REALTIME gains the interval via xtime; MONOTONIC is held constant by the compensating wall_to_monotonic change; BOOTTIME gains the interval via offs_boot. That is by design.

The three time sources and the RTC fallback path

When the timekeeper has neither a non-stop clocksource nor a persistent clock readable with interrupts off (some embedded RTCs are only reachable over a sleeping bus like I2C), the RTC subsystem performs the injection itself, after devices have resumed. The code documents the preference order explicitly (timekeeping.c:1834):

1) non-stop clocksource
2) persistent clock (RTC accessible when irqs are off)
3) RTC (accessible only with irqs on)

Two coordination predicates glue this together:

  • timekeeping_rtc_skipresume() returns !suspend_timing_needed — i.e., “the timekeeper already injected the sleep time, so the RTC core does not need to.” It prevents double-counting the interval.
  • timekeeping_rtc_skipsuspend() returns persistent_clock_exists — “the persistent clock will definitely be used, so the RTC core can skip its own suspend snapshot.”

When the RTC core must do the work, it calls timekeeping_inject_sleeptime64() (timekeeping.c:1879):

void timekeeping_inject_sleeptime64(const struct timespec64 *delta)
{
	raw_spin_lock_irqsave(&timekeeper_lock, flags);
	write_seqcount_begin(&tk_core.seq);
	suspend_timing_needed = false;
	timekeeping_forward_now(tk);
	__timekeeping_inject_sleeptime(tk, delta);
	timekeeping_update(tk, TK_CLEAR_NTP | TK_MIRROR | TK_CLOCK_WAS_SET);
	write_seqcount_end(&tk_core.seq);
	raw_spin_unlock_irqrestore(&timekeeper_lock, flags);
	clock_was_set(CLOCK_SET_WALL | CLOCK_SET_BOOT);
}

It clears suspend_timing_needed (so nobody injects twice), forwards the timekeeper to now, runs the same __timekeeping_inject_sleeptime() reused from the in-band path, and — importantly — calls clock_was_set(CLOCK_SET_WALL | CLOCK_SET_BOOT) to tell the rest of the kernel that both the realtime and boottime clocks discontinuously moved. clock_was_set() re-evaluates absolute-time hrtimers (a CLOCK_REALTIME timer set for “3pm” must be re-checked because 3pm may now be in the past) and notifies timerfd consumers. The seqcount write barrier (write_seqcount_begin/end) ensures lockless readers using the seqlock retry pattern never observe a half-updated xtime/offs_boot.

Why MONOTONIC historically did not advance — and the nuance

The behaviour above — MONOTONIC frozen across suspend, BOOTTIME advancing — is the current and historical contract for CLOCK_MONOTONIC. CLOCK_BOOTTIME exists precisely because applications sometimes want suspend time included; the man page defines it as “identical to CLOCK_MONOTONIC, except that it also includes any time that the system is suspended” (clock_gettime(2)). A timeout computed against CLOCK_MONOTONIC will therefore appear to “not count down” while the laptop lid is closed; the same timeout against CLOCK_BOOTTIME will have fully elapsed on resume. This is why timerfd and epoll timeouts that should expire across suspend must use CLOCK_BOOTTIME (or TFD_TIMER_CANCEL_ON_SET semantics for realtime).

On resume the kernel fires clock_was_set() for the wall and boot clocks but not for monotonic — monotonic did not jump, so there is nothing to re-arm against it. The realtime hrtimers and timerfds are re-evaluated because the wall clock leapt forward.

Uncertain

Verify: the precise statement that CLOCK_MONOTONIC “historically did not advance during suspend” and whether there was ever a kernel version where it did (the brief frames this as historical). The v6.12 source and the current clock_gettime(2) man page both state MONOTONIC excludes suspend time, which I verified. Reason: I did not trace the full pre-3.x git history to confirm there was no era where monotonic included suspend. To resolve: review the changelog around the introduction of CLOCK_BOOTTIME (Linux 2.6.39) and ktime_get_boottime. uncertain

Uncertain

Verify: that on x86 the TSC specifically halts in ACPI S3 and the architected timer “may reset” on some ARM platforms. Reason: this is hardware/platform-specific and I cited it from general clocksource/suspend behaviour, not a single authoritative datasheet read during this task. The kernel’s three-source fallback design strongly implies clocksources can stop across suspend, which is the load-bearing claim. To resolve: cross-check the Intel SDM TSC behaviour in C-states/S-states and the ARM Generic Timer (CNTPCT) reset behaviour per SoC. uncertain

Failure Modes

  • Double-counting the sleep interval. If both the timekeeper and the RTC core injected the same gap, the clock would leap forward by twice the slept time. The suspend_timing_needed flag and the timekeeping_rtc_skipresume()/timekeeping_rtc_skipsuspend() predicates exist exactly to prevent this. A bug that mis-sets these flags manifests as wall-clock time roughly doubling the actual sleep duration.
  • One-second-per-suspend drift. Because the RTC has one-second granularity, naive injection rounds and accumulates error across many suspends. The delta_delta compensation in timekeeping_suspend() mitigates this; without it, a laptop suspended/resumed dozens of times a day would drift visibly until the next NTP correction (see NTP and Kernel Clock Steering).
  • Negative or absurd deltas. A glitchy RTC that reads backwards across suspend yields a negative delta; __timekeeping_inject_sleeptime() guards with timespec64_valid_strict() and logs “Invalid sleep delta value!” rather than corrupting the clock. The resume path also only injects when ts_new > timekeeping_suspend_time.
  • Readers during the frozen window. Between timekeeping_suspended = 1 and resume, the clocksource is dead. halt_fast_timekeeper() snapshots a consistent base so NMI-safe fast readers return a frozen value instead of reading garbage from an unpowered counter. The slow path is gated by WARN_ON(timekeeping_suspended) checks scattered through the accessors.
  • CLOCK_MONOTONIC timeouts that “never fire” across suspend. Not a kernel bug but an application bug: code that expects a CLOCK_MONOTONIC-based watchdog to elapse while the machine sleeps will be surprised. The fix is CLOCK_BOOTTIME.

Alternatives and When to Choose Them

  • Non-stop clocksource vs persistent clock. A platform with a counter that survives suspend (reported via CLOCK_SOURCE_SUSPEND_NONSTOP) gives nanosecond-accurate sleep timing and is always preferred; the RTC is the coarse fallback. There is no user choice here — the kernel picks the best available source automatically.
  • CLOCK_BOOTTIME vs CLOCK_MONOTONIC for application timing. Choose CLOCK_BOOTTIME when an interval must include time the machine spent asleep (e.g., “expire this lease 30 minutes from now even if the laptop sleeps”). Choose CLOCK_MONOTONIC when you want to measure active elapsed time and treat suspend as a pause. See Monotonic Realtime and Boottime Clocks.
  • CLOCK_REALTIME vs CLOCK_TAI across suspend. Both jump forward by the slept interval on resume; neither is suitable for measuring durations, because they are also subject to NTP steering and leap seconds. Use a monotonic family clock for durations.

Production Notes

The reason systemd timers, container runtimes, and mobile power managers care about this is concrete: a RealtimeTimer and a MonotonicTimer behave differently across laptop sleep, and getting the choice wrong means either missed deadlines or premature firings after a long suspend. The timerfd_resume() call wired into timekeeping_resume() is what lets epoll-driven event loops notice that their TFD_TIMER_CANCEL_ON_SET realtime timers were invalidated by the resume-time wall-clock jump. On servers that never suspend, all of this is dormant; on laptops, phones, and any S3-capable hardware it runs on every lid close.

See Also

Graph View

Backlinks