vault

Time-Series Databases

12 min read

Time-Series Databases

A time-series database (TSDB) is a store specialized for data that arrives as a stream of (timestamp, value) points attached to an identity (a metric name plus labels), where the workload is overwhelmingly append-heavy, timestamp-ordered, and range-queried: monitor a fleet of servers, ingest millions of samples per second, keep each point immutable once written, read back windows of time to graph or aggregate, and expire old data by age. Because the timestamps march forward at near-fixed intervals and neighbouring values change little, TSDBs achieve compression ratios ordinary databases cannot — Facebook’s Gorilla (Pelkonen et al., VLDB 2015) squeezed a 16-byte point down to an average of 1.37 bytes, a 12× reduction, using delta-of-delta timestamps and XOR-encoded floats, the two techniques that now underpin Prometheus, InfluxDB, TimescaleDB, VictoriaMetrics, and the rest of the field. This note covers what makes time-series data special, the specialized compression it enables, and how the leading engines are built.

Mental Model — An Append-Only Log Per Series, Compressed to the Bone

Think of a TSDB as millions of independent, append-only logs — one per time series — each an ever-growing list of (timestamp, value) points in timestamp order. A “series” is identified not by an opaque key but by a set of dimensions: Prometheus identifies it by a metric name plus a set of key/value labels (http_requests_total{method="GET",status="200"}); InfluxDB by a measurement plus tags; Gorilla by a single string key. Writes almost always append to the head of the newest series; reads almost always scan a contiguous time window across one or many series and aggregate.

flowchart TB
  subgraph INGEST["Write path — append-mostly, timestamp-ordered"]
    S1["series A: (t0,v) (t1,v) (t2,v) ...→ head"]
    S2["series B: (t0,v) (t1,v) ...→ head"]
    S3["series C ... (millions of series)"]
  end
  HEAD["In-memory head block<br/>+ write-ahead log (crash safety)"]
  BLOCK["Immutable time-range blocks on disk<br/>(2h → compacted → larger)"]
  COMP["Compression: delta-of-delta timestamps<br/>+ XOR floats → ~1.3 bytes/point"]
  RET["Retention/downsampling:<br/>drop or roll up old blocks by age"]
  INGEST --> HEAD -->|"flush closed range"| BLOCK
  BLOCK --> COMP
  BLOCK --> RET

What it shows: samples land in an in-memory head block (fronted by a WAL for durability), and once a time range closes it is flushed as an immutable, heavily-compressed block; old blocks are compacted into bigger ones and eventually dropped by retention. Insight to take: the two properties that define time-series data — monotonic timestamps and immutable, time-bounded blocks — are exactly what make its compression and its retention cheap. Deleting a month of data is dropping a directory, not a range-delete; compressing values is exploiting that consecutive samples barely differ. Everything else in a TSDB follows from these two facts.

Why Time-Series Data Needs a Specialized Store

A general-purpose row-store can hold (series, timestamp, value) rows, but it fights the workload at every step. The write rate is enormous and unrelenting (Gorilla’s target was “700 million data points added per minute”), which punishes B-tree in-place updates; timestamps are near-uniform, so generic compression misses the structure; the cardinality of distinct series can reach billions (Gorilla: “2 billion unique time series identified by a string key”); and retention is by age, which a generic store implements as an expensive range-delete rather than dropping a whole partition. Purpose-built TSDBs answer each of these: LSM-style append-and-merge for ingest, delta/XOR encodings for compression, per-series indexing over labels for cardinality, and time-partitioned blocks so expiry is a file drop.

Gorilla — The Compression Algorithm Everyone Copied

Gorilla was Facebook’s in-memory TSDB, “a write-through cache for monitoring data written to an HBase data store” holding 26 hours of data in RAM for fast queries (Pelkonen et al. 2015). Its lasting contribution is the compression scheme. Each monitoring point is a 3-tuple: a string key, a 64-bit integer timestamp, and a double-precision float value. Timestamps and values are compressed separately, into blocks partitioned by time (a two-hour window).

Timestamp compression — delta-of-delta. Because samples arrive at a fixed interval (say every 60 s), the delta between consecutive timestamps is nearly constant, so the delta of the delta is almost always zero. Formally, for timestamp t_n the stored quantity is

D = (t_n − t_{n−1}) − (t_{n−1} − t_{n−2})

— the current inter-arrival gap minus the previous inter-arrival gap. The block header stores the starting timestamp aligned to a two-hour window, and the first timestamp t0 is stored as a 14-bit delta from it (14 bits spans a bit more than 4 hours, so no larger block would fit). Each subsequent D is variable-length encoded:

  • D = 0 → store a single '0' bit.
  • D ∈ [−63, 64] → store '10' followed by the value in 7 bits.
  • D ∈ [−255, 256] → store '110' followed by 9 bits.
  • D ∈ [−2047, 2048] → store '1110' followed by 12 bits.
  • otherwise → store '1111' followed by D in 32 bits.

The payoff: on real Facebook data, “96% of all time stamps can be compressed to a single bit” — because a perfectly-periodic scrape has D = 0 every time.

Value compression — XOR. Consecutive metric values are usually close, so their IEEE-754 bit patterns share leading bits. Gorilla XORs each value with the previous one:

  • If the XOR is 0 (value unchanged) → store a single '0' bit.
  • If non-zero → store '1', then a control bit: '0' if the block of “meaningful” (non-zero) bits falls within the previous value’s window of leading/trailing zeros (reuse that window, store just the meaningful bits); '1' otherwise, followed by 5 bits for the number of leading zeros, 6 bits for the length of the meaningful field, then the meaningful XORed bits.

On real data ~51% of values compress to a single bit, ~30% to the control-10 case (avg 26.6 bits), ~19% to control-11 (avg 36.9 bits). Combined over a two-hour block, the scheme reaches 1.37 bytes per point, versus 16 bytes uncompressed — the 12× figure. (Gorilla noted blocks longer than two hours give diminishing returns.)

Prometheus — 2-Hour Blocks, a WAL, and Gorilla-Style Chunks

Prometheus’s local storage is a purpose-built TSDB. Ingested samples are grouped into blocks covering two-hour time ranges; the block currently receiving writes is the in-memory head block, made crash-safe by a write-ahead log in the wal/ directory (128 MB segments) (Prometheus storage). A completed block is an immutable directory containing a chunks/ subdirectory of series samples, an index mapping metric names and labels to those chunks, and metadata. A background process compacts two-hour blocks into larger ones “containing data spanning up to 10% of the retention time, or 31 days, whichever is smaller.” Deletes are tombstoned rather than applied in place. Prometheus stores “an average of only 1–2 bytes per sample,” which drives its capacity-planning formula disk = retention_seconds × samples_per_second × bytes_per_sample.

That 1–2 bytes is Gorilla’s scheme, generalized. Prometheus’s chunk encoding uses delta-of-delta for timestamps and XOR for float values, packed with a variable-bitwidth (“varbit”) encoding — varbit_ts for the signed timestamp delta-of-deltas (1–68 bits) and varbit_xor for the XORed values (1–77 bits) (chunks format spec). Prometheus’s own analysis reports varbit reaching “as far down as 1.28 bytes per sample… even slightly better than the 1.37 bytes per sample reported for Gorilla” (varbit chunks blog). Prometheus is also pull-based — it scrapes targets on an interval — which is why its inter-sample gaps are so regular that delta-of-delta collapses to one bit.

TimescaleDB — A Time-Series Layer Inside PostgreSQL

TimescaleDB (now developed as Tiger Data) takes the opposite architectural bet: rather than a bespoke engine, it is a PostgreSQL extension that adds time-series superpowers while staying fully SQL- and PostgreSQL-compatible. Its core abstraction is the hypertable — a virtual table that is automatically partitioned by time into many child tables called “chunks,” each holding a specific time range, with optional secondary partitioning on a space dimension (Tiger Data hypertables). Chunks are created automatically as data arrives; queries transparently prune to the relevant chunks (chunk exclusion), so a query for last hour’s data never touches last year’s chunks. “You interact with hypertables in the same way as you would with regular PostgreSQL tables. All the optimization magic happens behind the scenes.” On top of chunking, TimescaleDB adds continuous aggregates (incrementally-maintained materialized views that pre-compute rollups) and columnar compression of older chunks (converting row chunks into compressed columnar segments). The appeal is keeping the full PostgreSQL ecosystem — joins, SQL, indexes, transactions — while getting time-partitioned ingest and retention.

Uncertain

Verify: the specifics of TimescaleDB continuous aggregates and columnar compression (incremental refresh policy, per-chunk compression mechanism, current version behaviour). Reason: the hypertables doc page I fetched described chunking well but only mentioned continuous aggregates/compression in passing; I did not fetch their dedicated pages this pass. To resolve: consult the Tiger Data docs for continuous aggregates and hypercore/columnar compression directly. uncertain

InfluxDB — From the TSM Engine to the IOx/Arrow Rewrite

InfluxDB is the most dramatic recent example of a TSDB re-architecting itself. Versions 1.x/2.x used the TSM (Time-Structured Merge tree) engine — an LSM-tree variant with a WAL, in-memory cache, and compressed columnar TSM files, purpose-built for metrics. InfluxDB 3.0 (previously “InfluxDB IOx”) is a ground-up rewrite in Rust onto the FDAP stack: Apache Flight (RPC), DataFusion (query engine), Apache Arrow (in-memory columnar format), and Apache Parquet (on-disk columnar format) (InfluxDB 3.0 architecture). Data is persisted as compressed Parquet files (10–100× smaller than raw) on object storage (S3/Azure/GCS), with a PostgreSQL-compatible catalog for metadata and a local write-ahead log for crash recovery. The system splits into independently-scalable components — ingesters (validate, partition by day, deduplicate via sort-merge, persist Parquet), queriers (build DataFusion plans with predicate/projection pushdown, combine persisted Parquet with un-persisted ingester data), compactors (merge small Parquet files into larger non-overlapping ones), and a garbage collector (retention). This is the same open columnar lakehouse stack described in Columnar Analytical Databases — a specialized TSDB converging on general columnar infrastructure, separating compute from storage so each scales alone.

Retention, Downsampling, and Rollups

The defining lifecycle of time-series data is that its value density falls with age: you want per-second resolution for the last hour but only per-hour averages for last year. TSDBs handle this with two mechanisms. Retention drops data past a cutoff — cheap because it means deleting whole time-partitioned blocks/chunks/Parquet files, never a row-by-row range-delete. Downsampling / rollups / continuous aggregates pre-compute coarser summaries of old data (e.g. min/max/avg per 1-minute bucket) and often discard the raw points afterward: Prometheus does this with recording rules feeding longer-retention storage, TimescaleDB with continuous aggregates, InfluxDB with scheduled tasks. The combination lets a TSDB serve fast dashboards over recent high-resolution data while storing years of history compactly.

Failure Modes and Misunderstandings

  • Cardinality explosion — the #1 way to kill a TSDB. Each unique combination of label/tag values is a separate series with its own index entry and chunk. Putting a high-cardinality value (a user ID, a request ID, a full URL, an email) into a label multiplies series into the millions and blows up memory and index size. The rule is: labels must be low-cardinality dimensions, never unbounded identifiers. Most Prometheus/InfluxDB outages trace to a bad label.
  • Out-of-order and backfilled writes. The compression and block model assume monotonically-increasing timestamps. Late-arriving or historical points either get rejected, force a special out-of-order path, or trigger expensive rewrites of already-compacted blocks. (Newer versions of several engines added out-of-order ingestion, but it is a bolted-on capability, not the happy path.)
  • Treating a TSDB as a general database. TSDBs are weak at high-cardinality joins, arbitrary updates to old points, and per-point deletes. If you need those, a general store (or TimescaleDB, which keeps SQL) fits better.
  • Querying enormous time ranges at full resolution. Scanning a year at per-second granularity reads billions of points; this is what downsampling exists to avoid — query the rolled-up series for long ranges.
  • Under-provisioning the head/WAL. The in-memory head block plus WAL must hold the active time window; a sustained ingest spike can OOM the process before compaction catches up.

Alternatives and When to Choose Them

  • Prometheus (+ long-term stores like Thanos/Cortex/Mimir): pull-based metrics monitoring of cloud-native infrastructure; PromQL; ecosystem standard. Choose for Kubernetes/service monitoring.
  • TimescaleDB: choose when you want time-series ingest/retention and full SQL, joins, and the PostgreSQL ecosystem in one system.
  • InfluxDB 3.0: choose for high-cardinality metrics/events with SQL and an object-storage-backed, compute/storage-separated lakehouse architecture.
  • VictoriaMetrics: a Prometheus-compatible TSDB advertising even better-than-Gorilla compression and lower resource use; choose for very large Prometheus-style deployments. (Its compression claims are the vendor’s; treat comparative numbers as as-of and workload-dependent.)
  • A generic columnar warehouse (ClickHouse): frequently used as a time-series store because its columnar layout, sparse index, and codecs (delta, delta-of-delta, Gorilla, T64) suit metrics well — the line between “TSDB” and “columnar OLAP store” is genuinely blurry (see Columnar Analytical Databases).

Production Notes

The field has visibly converged. The compression core — delta-of-delta timestamps + XOR floats — first published in Gorilla now appears, essentially unchanged, in Prometheus’s varbit chunks and as explicit codecs in ClickHouse. Simultaneously the storage architecture is converging on the same Arrow/Parquet/object-storage lakehouse that OLAP engines adopted, most visibly in InfluxDB’s IOx rewrite. The practical lesson for operators is that the make-or-break decision is almost never the engine — it is schema/label design: keep cardinality bounded, batch writes, and lean on downsampling for long-range queries. A TSDB that is well-fed with low-cardinality, in-order, batched samples is astonishingly efficient; one fed unbounded labels and out-of-order backfill will fall over regardless of vendor.

See Also

Graph View