vault

Thin Provisioning and Snapshots dm-thin

16 min read

Thin Provisioning and Snapshots dm-thin

dm-thin is the Linux device-mapper target that implements thin provisioning and space-efficient snapshots, implemented in drivers/md/dm-thin.c with its on-disk metadata in drivers/md/dm-thin-metadata.c (v6.12 source). Thin provisioning means a volume can advertise a large virtual size while only consuming physical blocks as they are actually written — allocate-on-write. A thin-pool is built from two backing devices, a data device (where blocks live) and a metadata device (which records the virtual-to-physical block mapping in B-trees); any number of thin volumes are then carved out of the pool, each addressing the shared data device through its own mapping tree. Snapshots are nearly free: a snapshot simply shares the origin’s data blocks by pointing its mapping tree at the same physical blocks, and a write to either side triggers a copy-on-write break-sharing of just the affected block. The single most important idea is that unwritten blocks cost nothing and shared blocks cost nothing until written — so you can over-provision storage and take dozens of snapshots cheaply, at the price of having to monitor the pool so it never runs out of real space.

This is the kernel mechanism underneath LVM thin pools (lvcreate --thin); the LVM tooling is the usual way to drive it, but the target stands alone and can be configured raw with dmsetup. It is the block-layer sibling of filesystem-level CoW snapshots like Btrfs Copy-on-Write Subvolumes and Snapshots — same copy-on-write idea, one layer lower.

Mental Model

Think of a thin-pool as a bank of fixed-size data blocks plus a ledger. The data device is the bank — a flat array of blocks of size data_block_size (a power-of-two between 64 KiB and 1 GiB). The metadata device is the ledger — a set of B-trees recording, for each thin volume, which of its virtual blocks map to which physical data block. A thin volume hands out a virtual address space; when a read hits a virtual block that has no ledger entry, the pool returns zeroes; when a write hits an unmapped virtual block, the pool allocates a fresh data block from the bank, records the mapping in the ledger, and only then performs the write. Nothing is pre-allocated — a 1 TiB thin volume on a 100 GiB pool consumes zero data blocks until you write to it.

Snapshots reuse the ledger trick. Taking a snapshot clones the root of the origin’s mapping tree, so both trees now point at exactly the same data blocks — no data is copied. The pool tracks a reference count per data block; a block referenced by two trees has refcount 2 and is “shared.” The first write to a shared block triggers break-sharing: allocate a new block, copy the old contents into it, repoint one tree at the copy, and decrement the shared block’s count. The cost of a snapshot is therefore one cloned tree-root now, plus one block-copy per first write to each shared block later — not a full duplication.

flowchart TB
  subgraph POOL["thin-pool"]
    MD["metadata device<br/>(B-trees: dev id -> virtual blk -> phys blk, time)"]
    DATA["data device<br/>(fixed-size blocks)"]
  end
  T0["thin vol 0 (origin)"] -->|"mapping tree 0"| MD
  T1["thin vol 1 (snapshot)"] -->|"mapping tree 1<br/>(cloned root)"| MD
  MD -->|"refcount=2 -> SHARED"| B["data block 42"]
  T0 -.->|"write -> break-sharing:<br/>copy 42 -> new block 99,<br/>repoint tree 0"| B
  DATA --- B

The pool, two volumes, and a shared block. What it shows: the origin and its snapshot both map their virtual blocks through the metadata B-trees onto the same physical data block 42 (refcount 2, shared); a write to the origin copies block 42 into a fresh block 99 and repoints only the origin’s tree. The insight to take: snapshots are cheap because sharing is the default and copying is deferred to the exact moment of first divergent write — and that copy is one data block, not the whole volume.

Mechanical Walk-through

Allocate-on-write

The heart of thin provisioning is what happens on a write to an unmapped virtual block. dm-thin handles bios in a worker thread (do_worker() → process_thin_deferred_bios()); when a write maps to a virtual block with no entry in the mapping B-tree, it calls provision_block(), which calls alloc_data_block() to pull a free block from the data space map (dm-thin.c, v6.12). On success it schedules the actual data placement — either schedule_zero() (zero the new block first, unless skip_block_zeroing/zero_new_blocks says otherwise) or schedule_external_copy() (for external-origin snapshots, copy from the read-only origin) — and queues the new mapping for insertion into the B-tree via dm_thin_insert_block(). The provisioning state lives in a struct dm_thin_new_mapping, whose prepare_actions counter tracks the outstanding quiesce/copy/zero steps; when it reaches zero the mapping moves to the prepared_mappings list and the worker commits it.

Two special cases short-circuit allocation, per the code’s own comments: “Remap empty bios (flushes) immediately, without provisioning” — a flush carries no data, so it needs no block — and “Fill read bios with zeroes and complete them immediately” — a read of an unprovisioned block is definitionally zeroes, so no allocation happens on the read path. Only writes provision. This asymmetry is what makes a freshly created multi-terabyte thin volume occupy zero data blocks: until something writes, every block reads as zero from the mapping’s absence alone.

Snapshots and break-sharing

A snapshot is created by the pool message create_snap <new_id> <origin_id>, which, in the metadata layer, “clone[s] the root node of the origin btree. After this there is no concept of an origin or a snapshot. They are just two device trees that happen to point to the same data blocks” (dm-thin.c comment). That symmetry is important: dm-thin does not privilege the origin — both volumes are now first-class, and either can be written, deleted, or snapshotted again.

The shared blocks are detected by reference count (maintained in the metadata space map, below). When a write lands on a shared data block, dm-thin runs the break-sharing sequence the code documents step by step:

  1. plug io further to this physical block — hold off other I/O to that block;
  2. quiesce any read io to that shared data block — let in-flight reads drain;
  3. copy the data block to a newly allocate[d] block — break_sharing() calls alloc_data_block() for a fresh block and schedule_internal_copy(), which uses kcopyd to duplicate the contents;
  4. insert the new mapping into the origin’s btree — repoint the writer’s tree at the copy;
  5. unplug io to this physical block — release the held I/O.

Only after the copy completes is the write applied to the new, private block. The other volume still sees the original block, now with a decremented refcount. This is copy-on-write at the pool (block) level, as opposed to the filesystem-level CoW of btrfs.

A subtlety the code calls out: sharing detection uses a timestamp (“block time”), and “the timestamp magic isn’t perfect, and will continue to think that [a] data block in the snapshot device is shared even after the write to the origin has broken sharing.” The consequence is conservative — dm-thin may break-share a block that is no longer actually shared, wasting one copy, but it never fails to break-share a truly shared block, so correctness holds; only efficiency suffers slightly.

The metadata B-trees and space maps

The metadata device, managed by dm-thin-metadata.c and the kernel persistent-data library, holds several structures rooted in a superblock (dm-thin-metadata.c, v6.12). The data mapping B-tree is “a hierarchical btree, with 2 levels which effectively maps (thin dev id, virtual block) → block_time” — the first level keys on the 24-bit thin device id, the second on the virtual block, and the leaf value packs the physical block with a creation timestamp (“Block time is a 64-bit field holding the time in the low 24 bits, and block in the top 40 bits”). A separate device details tree maps each thin dev id to a struct disk_device_details (recording mapped_blocks, transaction_id, creation_time, snapshotted_time). The superblock itself is at THIN_SUPERBLOCK_LOCATION 0 with magic THIN_SUPERBLOCK_MAGIC 27022010 (the format’s “birthday,” 27 Feb 2010) and THIN_VERSION 2; it carries the roots data_mapping_root and device_details_root plus the serialized roots of two space maps.

The space maps (dm_space_map, one for data data_sm, one for metadata metadata_sm) are what track free space and the reference counts that make sharing work. They store refcounts compactly: a bitmap encodes “pairs of bits. With the meaning being: 0 - ref count is 0, 1 - ref count is 1, 2 - ref count is 2, 3 - ref count is higher than 2,” and for any block with refcount > 2 the exact count lives “in a second btree that directly maps the block_address to a uint32_t ref count.” A data block is shared precisely when its refcount exceeds 1; data_block_inc()/data_block_dec() adjust counts as snapshots are taken and broken. This is the single source of truth that lets a write decide whether it must break-share.

Crucially, the metadata itself is updated copy-on-write through a dm_transaction_manager: dm_tm_shadow_block() shadows (copies) a metadata block before modifying it, and changes are made durable atomically by __commit_transaction(), which pre-commits the space maps, serializes their roots with save_sm_roots(), then writes and commits the superblock last via dm_tm_commit(). Because the superblock flip is the final atomic step, an interrupted commit (power loss) leaves the previous consistent transaction intact — the pool recovers to the last committed state rather than to a torn one.

Out-of-space behavior

Over-provisioning means the pool can advertise more virtual space than the data device holds, so it can genuinely run out of real blocks. dm-thin models this with a pool_mode state machine (dm-thin.c):

enum pool_mode {
    PM_WRITE,                 /* normal: provisioning allowed */
    PM_OUT_OF_DATA_SPACE,     /* data device full */
    PM_OUT_OF_METADATA_SPACE, /* metadata device full */
    PM_READ_ONLY,             /* metadata frozen, no provisioning */
    PM_FAIL,                  /* unrecoverable: all I/O errors */
};

The intended safety valve is the low water mark: the pool target carries a low_water_blocks threshold and, when free data blocks drop below it, raises a dm event to userspace so a volume manager can extend the pool before it fills — “Ideally we’d never hit this state; the low water mark would trigger userland to extend the pool before we completely run out of data space.” If that fails and data blocks are exhausted, alloc_data_block() returns -ENOSPC and set_pool_mode() transitions to PM_OUT_OF_DATA_SPACE. What happens next depends on the error_if_no_space feature flag:

  • Default (queue_if_no_space): writes that need a block are held (retry_bios_on_resume() requeues them) for up to no_space_timeout_secs (default 60 s), giving an administrator a window to extend the pool, after which they fail. The bet is that out-of-space is transient and recoverable.
  • error_if_no_space: out-of-space writes fail immediately with BLK_STS_NOSPC. This is safer for systems that prefer a hard error over a hang.

If the metadata device fills, the pool goes PM_OUT_OF_METADATA_SPACE/PM_READ_ONLY — no new mappings can be recorded. PM_FAIL is the unrecoverable terminal state where every I/O errors; the pool status then shows needs_check, and the device must be examined offline with thin_check/thin_repair before it will mount read-write again. Running a thin-pool out of space (especially with the filesystem above unaware) is the canonical dm-thin failure and the reason monitoring is mandatory.

Discard passdown

When a filesystem above frees blocks (TRIM/discard), dm-thin can return them to the pool’s free list and optionally pass the discard down to the data device (so an SSD can reclaim flash). With discard_passdown enabled this is a careful two-phase dance: process_prepared_discard_passdown_pt1() removes the virtual mapping and bumps refcounts on the affected data blocks to prevent reallocation while the passdown is in flight, then pt2() decrements them after the device discard completes; passdown_double_checking_shared_status() ensures a block is genuinely unshared before issuing the underlying discard (you must never discard a block another snapshot still references). With no_discard_passdown the mapping is removed but no discard is sent down.

Configuration and Worked Example

The kernel docs show the raw two-step dmsetup setup (thin-provisioning admin guide); LVM hides all of this behind lvcreate, but seeing it raw makes the mechanism concrete.

# 0. The metadata device must start zeroed so the pool knows it's fresh.
dd if=/dev/zero of=$metadata_dev bs=4096 count=1
 
# 1. Create the pool. Table: "<start> <len> thin-pool <meta> <data>
#    <data_block_size> <low_water_mark> [<#feature_args> <feature_args>...]"
dmsetup create pool --table \
  "0 20971520 thin-pool $metadata_dev $data_dev 1024 32768 1 skip_block_zeroing"
  • 0 20971520 — the pool device exposes 20971520 sectors (10 GiB) of virtual address space mapping is not the point here; the pool target itself is the management device.
  • thin-pool $metadata_dev $data_dev — the two backing devices: ledger and bank.
  • 1024 — data_block_size in 512-byte sectors = 512 KiB. Allowed range is 128–2097152 sectors (64 KiB–1 GiB), a multiple of 128 (DATA_DEV_BLOCK_SIZE_MIN_SECTORS.._MAX_SECTORS in the source). Bigger blocks (e.g. 1024/512 KiB) suit pure thin provisioning (less metadata, less overhead); smaller blocks (128/64 KiB) suit heavy snapshotting (finer CoW granularity, so a small write breaks-shares less data).
  • 32768 — low_water_mark in data blocks: when free data blocks fall below this, the pool raises a dm event.
  • 1 skip_block_zeroing — one feature arg: don’t pre-zero new blocks (faster, but a new block may briefly read old data unless the layer above overwrites it). Other feature args: ignore_discard, no_discard_passdown, read_only, error_if_no_space.
# 2. Create a thin volume inside the pool (24-bit dev id chosen by caller).
dmsetup message /dev/mapper/pool 0 "create_thin 0"
# Activate it as a usable block device. Table: "<start> <len> thin <pool> <dev_id>"
dmsetup create thin --table "0 2097152 thin /dev/mapper/pool 0"
 
# 3. Snapshot it. Suspend the origin FIRST so its in-flight I/O can't corrupt
#    the metadata mid-clone, take the snap, then resume.
dmsetup suspend /dev/mapper/thin
dmsetup message /dev/mapper/pool 0 "create_snap 1 0"   # new id 1, snapshot of id 0
dmsetup resume /dev/mapper/thin
dmsetup create snap --table "0 2097152 thin /dev/mapper/pool 1"

The thin target’s table is just thin <pool_dev> <dev_id> [<external_origin_dev>] — a thin volume is nothing but a (pool, id) pair, optionally backed by a read-only external origin (a snapshot of a plain device that lives outside the pool). The pool status line is what you monitor: <transaction_id> <used_meta>/<total_meta> <used_data>/<total_data> <held_metadata_root> rw|ro|out_of_data_space [no_]discard_passdown [error|queue]_if_no_space needs_check|- <metadata_low_watermark>. The used_data/total_data pair is the over-provisioning gauge; needs_check means metadata repair is required.

LVM equivalents, for orientation: lvcreate -L 100G --thinpool tp vg builds the pool (sizing the metadata LV automatically — roughly 48 * data_size / data_block_size, rounded up to 2 MiB, capped at 16 GiB); lvcreate -V 1T --thin -n vol vg/tp makes a 1 TiB thin volume on it; lvcreate -s vg/vol -n snap snapshots it. Metadata integrity tooling — thin_check, thin_repair, thin_dump (the device-mapper-persistent-data package) — operates on the metadata device offline.

Failure Modes and Gotchas

  • Pool runs out of data space. The defining dm-thin failure. With default queue_if_no_space, writes hang for no_space_timeout_secs then error; the filesystem above (which thinks it had space) sees write failures and may go read-only or corrupt in-flight data. You must monitor used_data/total_data and extend before the low-water event escalates. This is why production dm-thin always pairs with monitoring (dmeventd, Prometheus on LVM thin metrics).
  • Metadata exhaustion is worse than data exhaustion. A full metadata device can’t even record the mapping that frees space, pushing the pool to PM_READ_ONLY. Size metadata generously; the 16 GiB cap is rarely the binding constraint.
  • needs_check / PM_FAIL. A metadata operation failed or corruption was detected; the pool refuses normal operation until thin_check/thin_repair runs offline. Never force-activate a needs_check pool read-write blindly.
  • Snapshots are not backups. A thin snapshot shares blocks with its origin on the same pool/device; if that data device dies, snapshot and origin die together. Snapshots protect against logical mistakes (bad deploy, fat-fingered rm), not media failure.
  • skip_block_zeroing can leak stale data. New blocks may read whatever was there before. Safe only when the layer above always writes a full block before reading it; otherwise a thin volume could expose another tenant’s freed data.
  • Block-size choice is permanent and consequential. Too large wastes space on small writes and bloats CoW copies; too small bloats metadata. It cannot be changed after pool creation.
  • Forgetting to suspend the origin before create_snap risks taking the snapshot mid-write, capturing inconsistent state — hence the suspend/resume dance above (LVM does it for you).

Alternatives and When to Choose Them

  • The older dm-snapshot target. The original device-mapper snapshot used a separate copy-on-write exception store per snapshot, with O(snapshots) write amplification and a fixed-size store that fills and invalidates the snapshot when exceeded — fine for one short-lived snapshot (e.g. a consistent backup window) but poor for many long-lived ones. dm-thin’s pool-shared CoW scales to dozens of snapshots and lets snapshots-of-snapshots be cheap. Choose dm-snapshot only for a single transient snapshot of a non-thin volume.
  • Btrfs / ZFS snapshots. Filesystem-native CoW: the filesystem itself does block sharing and snapshotting, so it can be file-aware (send/receive, per-subvolume policy, checksummed) where dm-thin is block-blind. Choose these when you want an integrated CoW filesystem; choose dm-thin when you need block-level thin provisioning under an ordinary filesystem (ext4/xfs) or under VMs.
  • Thick (fully-allocated) LVM volumes. No over-provisioning risk, simplest to reason about, slightly faster (no allocate-on-write indirection). Choose when you don’t need snapshots or over-provisioning and predictability matters most.
  • Qcow2 / VM-image thin provisioning. Hypervisors thin-provision at the image-file level. dm-thin moves that below the filesystem so every tenant benefits, and is what LVM-backed VM storage (e.g. some Kubernetes/oVirt setups) uses.

Production Notes

dm-thin is the engine behind LVM thin pools (lvmthin(7)), which is how most production systems consume it — container storage drivers (the now-deprecated Docker devicemapper driver used dm-thin directly), VM disk backends, and any setup wanting cheap, frequent snapshots over ext4/xfs. The hard-won operational lesson is uniform across deployments: a thin pool is a shared resource that can be exhausted, and exhaustion is a correctness event, not just a performance one. Real incidents trace to unmonitored pools filling silently — a runaway log, an un-reaped snapshot — after which the filesystem above takes write errors. The mitigations are: always run dmeventd/monitoring on used_data and used_metadata; prefer error_if_no_space where a hard failure is safer than a hang; size the metadata device with headroom; and remember the data-block-size trade-off is set in stone at creation. Snapshots being block-shared also means a “deleted” file in a snapshot still pins its blocks until the snapshot is removed, so reclaiming space requires reaping snapshots, not just deleting files.

See Also

Graph View