vault

Centralized Configuration System Design

34 min read

Centralized Configuration System Design

A centralized configuration system stores configuration data (key-value pairs, often hierarchical), serves it to client processes via either pull or push, validates it against a schema on write, versions it for rollback, audits all changes, and supports staged rollout (canary, percentage-based, gradual ramp). The canonical industrial deployments are Apache ZooKeeper (Hunt et al. USENIX ATC 2010, https://www.usenix.org/legacy/event/usenix10/tech/full_papers/Hunt.pdf — the original open-source distributed coordination service inspired by Google Chubby), Google Chubby (Burrows OSDI 2006, https://research.google/pubs/the-chubby-lock-service-for-loosely-coupled-distributed-systems/ — the foundational paper), etcd (CoreOS 2013, now CNCF graduated, used by Kubernetes for cluster state, https://etcd.io/), HashiCorp Consul (2014, https://www.consul.io/), AWS AppConfig (2019, https://docs.aws.amazon.com/appconfig/), Facebook Configerator (Tang et al. SOSP 2015, https://research.facebook.com/publications/holistic-configuration-management-at-facebook/), LaunchDarkly (feature flags as a service, founded 2014), and Spring Cloud Config (https://spring.io/projects/spring-cloud-config). The interview problem is rich because configuration sits at the dangerous intersection of distributed consensus (the storage backend must be consistent), real-time push delivery (clients want updates within seconds), and operational risk (a single bad change can take down a fleet in seconds — the Knight Capital 2012 incident is the cautionary tale, where a botched code deployment combined with a repurposed feature flag cost $440 million in 45 minutes; per the SEC order it was fundamentally a deployment-and-flag failure, not a config-service mistake, but it remains the canonical lesson that flag/config changes must follow deploy discipline). Almost every advanced design decision is a balance between the data-integrity goals of consensus protocols (slow, careful) and the rollout-velocity goals of feature flags (fast, gradual).

1. Why This System Exists

The naïve approach is a config.yaml file deployed with the application binary, read at startup. This breaks at the first need to change configuration without redeploying: emergency feature kill-switches, A/B-test cohort assignments, traffic routing during incidents, dynamic service discovery, gradual rollouts, multi-tenant settings.

Once you need runtime config changes, you need three things the file approach cannot give you: (a) a single source of truth that all processes read from consistently; (b) a notification mechanism so processes learn about changes within seconds, not minutes (re-reading on every request is too expensive, polling is wasteful); (c) safety against bad changes — atomic multi-key updates, schema validation, audit, rollback.

The interview lens is that this system spans every reliability concern in distributed systems. The storage layer requires consensus (you cannot have two configs both claiming to be current). The delivery layer requires efficient watch primitives. The change-management layer requires staged rollout to limit blast radius. The audit layer requires immutable history. And the operational reliability requires the system itself to be deployed on independent infrastructure — because “the config service is down so we can’t deploy a fix” is a deadlock that has bricked production at multiple companies.

2. Requirements

2.1 Functional Requirements

  1. Store key-value config. Keys are hierarchical strings (/services/checkout/db/connection_pool_size) with values that are typed (string, integer, boolean, JSON, binary blob).
  2. Get by key. Single-key read, multi-key prefix range read (getRange('/services/checkout/') returns the subtree).
  3. Set with optimistic concurrency. Compare-and-swap (CAS): “set this key from value V₁ to V₂ only if its current value is V₁.” Without CAS, two concurrent updaters race and one wins silently.
  4. Watch. Client subscribes to a key (or prefix) and receives an event stream of changes. The watch must be reliable in the face of brief disconnections — clients receive missed events on reconnect or are notified to do a full re-read.
  5. Atomic multi-key updates (transactions). “Update keys K₁, K₂, K₃ together or none.” Critical for keeping related configs consistent.
  6. Schema validation. Each key has a schema (JSON Schema, protobuf descriptor); writes that violate the schema are rejected at the write boundary.
  7. Versioning. Every write produces a monotonically increasing revision number. Clients can read at a specific revision; operators can roll back to a prior revision.
  8. Audit log. Every change records who, what, when, why (mandatory comment on write).
  9. Multi-tenant namespacing. Keys are namespaced by tenant / team / environment so a change in one team’s namespace cannot accidentally affect another’s.
  10. Feature flag pattern. A specific flavor of config: flag_name → { enabled: bool, targeting: rules } where rules are predicates evaluated against context (user_id, org_id, region, percent bucket).
  11. Gradual rollout. Push a config change to 1% of clients, then 10%, then 100%, with the ability to halt and roll back at any stage.

2.2 Non-Functional Requirements

  1. Strict consistency for writes. A write either succeeds (and is visible to all subsequent reads) or fails. No “the write looked successful but actually wasn’t applied to all replicas.” This is what motivates Raft / Paxos as the storage backbone.
  2. Read availability. 99.99% — clients get some answer, even if it’s slightly stale. Reads should not require write-quorum availability.
  3. Push latency. Config write → all clients see the new value < 1 second.
  4. Read latency. Single-key read < 50 ms p99 from cached client. Cold read (not in client cache) < 100 ms.
  5. Scale. 100K keys per namespace; 10K namespaces per cluster; 1M+ clients watching simultaneously; 1000 writes/sec sustained, 10K writes/sec burst.
  6. Durability. Once a write is acknowledged, the value survives any single-node failure. Three-way replication standard.
  7. Reliable rollback. Last 1000 versions retained; arbitrary rollback to any of them in < 5 seconds.

3. Capacity Estimation

3.1 Storage

Average config value is ~1 KB (small JSON or string). 100K keys × 1 KB × ~10K namespaces × 1000 versions retained:

keys × bytes × namespaces × versions
= 100,000 × 1024 × 10,000 × 1000
= 10^15 bytes ≈ 1 PB if every key has 1000 distinct historical versions

In practice, only a fraction of keys change frequently; most have 1–10 historical versions. Realistic storage is 10–100 GB per cluster — small enough that the storage tier can be a Raft-replicated KV that fits entirely in memory on commodity hardware. This is what makes etcd’s “all data in RAM, snapshot to disk” architecture viable.

3.2 Read QPS

If a service instance reads its config on startup and on every change notification (perhaps every few minutes during normal operation), 100K instances × 1 read/min ≈ 1.6K reads/sec. Watches multiply: 1M open watches each tracking a few keys → millions of change events per second of cluster activity, but the steady-state watch traffic is just heartbeats (~1/min).

Burst case: a config change to a “global” key watched by all 1M clients triggers 1M notifications in a few seconds. The push-out fanout is the load constraint.

3.3 Write QPS

Configuration changes are rare compared to reads — typically 10–1000 writes/sec across an entire cluster, with occasional bursts during operational incidents (kill-switch flips, traffic-shift events). The system is write-rare, read-and-watch-frequent.

3.4 Watch Channel Bandwidth

If 1M clients each maintain a watch (long-poll or stream), the server must keep 1M open connections. At ~10 KB of memory per open connection (TCP buffers, watch state), that’s 10 GB across the cluster — manageable on a few mid-tier servers acting as connection terminators.

A change to a globally-watched key sends the new value (~1 KB) to all 1M watchers — 1 GB of fanout bandwidth in the burst window. This is large but spreads over the few-second window the change rolls out.

4. API Design

4.1 etcd v3 API (gRPC)

The de facto standard, since etcd backs Kubernetes:

service KV {
  rpc Range(RangeRequest) returns (RangeResponse);
  rpc Put(PutRequest) returns (PutResponse);
  rpc DeleteRange(DeleteRangeRequest) returns (DeleteRangeResponse);
  rpc Txn(TxnRequest) returns (TxnResponse);          // multi-key atomic
  rpc Compact(CompactionRequest) returns (CompactionResponse);
}
 
service Watch {
  rpc Watch(stream WatchRequest) returns (stream WatchResponse);   // bidirectional stream
}
 
service Lease {
  rpc LeaseGrant(LeaseGrantRequest) returns (LeaseGrantResponse);  // for ephemeral keys
  rpc LeaseKeepAlive(stream LeaseKeepAliveRequest) returns (stream LeaseKeepAliveResponse);
}

The Txn operation is etcd’s atomic multi-op primitive:

Txn:
  if  (key=K, mod_revision = expected_rev)        // CAS condition
  then put K = V', put K2 = V2'                    // applied if condition true
  else get K                                       // applied if condition false

This is enough to build distributed locks, consensus checkpoints, and atomic config updates. Kubernetes uses exactly this for controller leader election and resource version management.

4.2 The Watch Pattern

import etcd3
client = etcd3.client(host='etcd.example.com')
 
# initial read
value, metadata = client.get('/services/checkout/config')
 
# subscribe to changes
events_iterator, cancel = client.watch_prefix('/services/checkout/')
for event in events_iterator:
    if isinstance(event, etcd3.events.PutEvent):
        print(f"key {event.key} updated to {event.value}")
        # re-read the relevant subtree to apply
        apply_config_change(event.key, event.value)
    elif isinstance(event, etcd3.events.DeleteEvent):
        print(f"key {event.key} deleted")

The watch is a long-lived streaming connection. Under normal operation it sends nothing until a change occurs. When a change occurs, the server emits an event with the new key-value and the revision number. On reconnect after a brief network partition, the client passes its last-known revision; the server replays missed events (within a configurable history window).

4.3 Feature Flag API (LaunchDarkly-style)

GET /flags/checkout-new-flow/evaluate
Body: {
  "context": {
    "kind": "user",
    "key":  "user-12345",
    "country": "US",
    "tier": "enterprise"
  }
}
Response: { "value": true, "variation": "treatment", "reason": { "kind": "RULE_MATCH" } }

The server evaluates targeting rules — “enabled if country in ['US', 'CA'] and tier == 'enterprise' and user_id falls in a 50% bucket” — and returns the resolved value. SDKs cache evaluations and stream rule updates.

The targeting rules themselves are config that the operator pushes:

{
  "key": "checkout-new-flow",
  "variations": [false, true],
  "default_variation": 0,
  "rules": [
    { "if": "tier == 'enterprise'", "rollout": { "percent": 100, "variation": 1 } },
    { "if": "country in ['US', 'CA']", "rollout": { "percent": 25, "variation": 1, "bucketBy": "user_id" } }
  ]
}

5. Data Model

5.1 Key Hierarchy

Keys are typically string-encoded paths separated by /:

/services/checkout/db/connection_pool_size       42
/services/checkout/feature_flags/new_flow         true
/services/checkout/feature_flags/new_flow/rules   { ... }
/tenants/acme/regions/us-east-1/quotas/api_qps    1000

The hierarchy is a convention — etcd treats keys as opaque bytes — but range queries (getRange('/services/checkout/', '/services/checkout/~')) make hierarchical reads efficient because keys are stored sorted.

5.2 Value with Versioning

Each key stores not just the current value but a chain of versions:

struct KeyValue {
    key:                bytes
    value:              bytes
    create_revision:    int64        // revision when created
    mod_revision:       int64        // revision of last modification
    version:            int64        // count of modifications (0 = key never set)
    lease:              int64        // 0 = no lease (permanent); else lease_id
}

struct HistoricalEntry {
    revision:    int64
    value:       bytes
    written_at:  time
    written_by:  user_id
    reason:      string              // mandatory audit comment
}

mod_revision is the basis of CAS: Put(key, value, if_mod_revision=N) succeeds only if the key’s current mod_revision = N.

5.3 The Raft Replication Layer

The data sits behind a Raft state machine. Every write is proposed by the leader, replicated to followers, committed once a majority acknowledges, and then applied to the state machine on each replica. Reads are served from any replica with one of three consistency levels:

  • Linearizable read — read goes through Raft (the leader sends a read-index heartbeat, waits for quorum acknowledgement, then serves from the leader’s state). Latency is one round-trip; consistency is strict.
  • Serializable read — read from any replica without Raft. Latency is local; may return stale data (lag = replication lag).
  • Stale-but-bounded read — read from any replica but check that the local revision is within N of the leader. A pragmatic middle ground.

For most config use cases, serializable is fine — the staleness is bounded by replication lag (ms) and the cost of strictly up-to-date reads is rarely worth the latency.

5.4 Watch State

Each watch is a server-side state object: {client_id, prefix, last_revision_sent}. When a change at revision R affects a key in the prefix, the server iterates active watchers, finds matches, and pushes the event. To bound memory, server limits the number of watches per client and the total watch count per server.

5.5 Lease (Ephemeral Keys)

A lease is a server-side timer; keys associated with a lease auto-delete when the lease expires. The client must KeepAlive the lease periodically. This is the primitive for service registration: a service writes its address under /services/checkout/instances/<host> with a 10-second lease; if the service crashes, the keepalive stops, the lease expires, the key is deleted, and watchers are notified — effectively a heartbeat-based service discovery.

6. High-Level Architecture

flowchart TB
    subgraph Cluster[etcd / Consul / ZooKeeper Cluster]
        L[Leader]
        F1[Follower 1]
        F2[Follower 2]
        L <-->|Raft heartbeats + log replication| F1
        L <-->|Raft heartbeats + log replication| F2
        F1 <-.gossip.-> F2
    end
    Operator[Operator / CD Pipeline] --> Admin[Admin API<br/>gRPC + Auth]
    Admin --> L
    Admin --> Audit[(Audit Log<br/>append-only)]
    Admin --> Validator[Schema Validator]
    Validator --> L
    L --> Storage[(BoltDB / RocksDB<br/>per replica)]
    F1 --> Storage1[(per-replica storage)]
    F2 --> Storage2[(per-replica storage)]
    subgraph Clients
        SDK1[App SDK<br/>local cache + watch]
        SDK2[App SDK<br/>local cache + watch]
        Sidecar[Consul Agent /<br/>Sidecar Pull]
    end
    SDK1 -->|gRPC Watch stream| F1
    SDK2 -->|gRPC Watch stream| F2
    Sidecar -->|HTTP poll / long-poll| L
    Sidecar -->|distribute| App1[App Process]
    Sidecar -->|distribute| App2[App Process]
    CDN[Edge CDN<br/>LaunchDarkly model] -->|cached eval results| Mobile[Mobile / Web Clients]
    L -.replicate.-> CDN

What this diagram shows. The center is a 3-node cluster running Raft consensus — one leader, two followers. Writes go to the leader, replicate via Raft log, commit on majority. The operator path (top): admin API receives writes from operators or CI/CD pipelines, validates against schema, writes audit entry, and forwards to the leader. The client path (bottom): clients connect over gRPC streaming connections to any replica (typically a follower) for both reads and watch streams. Followers serve serializable reads locally. The sidecar pattern (Consul-style, middle): instead of every app process maintaining a direct etcd connection, a node-local sidecar agent pulls config and distributes it to colocated app processes via a local Unix socket — reducing connection count to the cluster from “number of app processes” to “number of nodes.” The edge cache pattern (right, LaunchDarkly-style): for feature flags evaluated by mobile/web clients in the field, push the evaluation rules to a CDN edge so the round-trip is to an edge POP, not the cluster.

The most important architectural property is that the storage tier is small and consensus-replicated, while the delivery tier is fanout-optimized. A config service that put millions of clients on direct connections to the Raft cluster would melt under watch-fanout load; the sidecar / edge tiers buffer and amplify the leader’s small write throughput into millions of effective recipients.

7. Request Flow

7.1 Write with CAS and Schema Validation

sequenceDiagram
    participant Op as Operator
    participant API as Admin API
    participant V as Schema Validator
    participant L as Raft Leader
    participant F1 as Follower 1
    participant F2 as Follower 2
    participant Audit as Audit Log
    participant W as Watcher (any client)

    Op->>API: PUT /v3/kv/put<br/>{key, value, prev_kv_revision=N, reason}
    API->>V: validate(key, value)
    V-->>API: ok
    API->>L: Put(key, value, if_mod_revision=N)
    L->>L: assemble Raft proposal
    par parallel replication
        L->>F1: AppendEntries(proposal)
        L->>F2: AppendEntries(proposal)
    end
    F1-->>L: ack
    F2-->>L: ack
    Note over L: majority committed → apply
    L->>L: apply to state machine; update watch index
    L-->>API: success at revision R
    API->>Audit: append {actor, key, old_value, new_value, R, reason, ts}
    Audit-->>API: ack
    API-->>Op: 200 OK, revision=R
    L-->>W: Watch event {key, value, R} (broadcast to matching watchers)

The CAS protects against concurrent writes: if two operators both load mod_revision = N and both try to write, only the first succeeds at N+1; the second fails because the current mod_revision is now N+1, not N. The second operator must re-read and reconcile.

7.2 Watch with Reconnection

sequenceDiagram
    participant C as Client
    participant S as Server (any replica)
    participant L as Leader

    C->>S: Watch stream open<br/>{prefix=/services/checkout/, start_revision=R0}
    S-->>C: ack stream open at R0
    Note over C,S: idle stream, server may send heartbeat
    L->>L: write at R1 affecting /services/checkout/...
    L->>S: replicate; S applies
    S-->>C: WatchResponse {events: [(key, value, R1)]}
    Note over C,S: client processes; updates local cache
    Note over C: network partition (5 min)
    C->>S: reconnect; resume from R1
    S->>S: any events R2..Rcurrent in window?
    alt window has events
        S-->>C: events R2..Rcurrent
    else outside window
        S-->>C: WatchCancel(reason="compacted")
        C->>S: full re-read with getRange
        S-->>C: current state at Rcurrent
    end

The history window (compaction_window) is the lookback for replay. etcd compacts old revisions to bound storage; if a watcher disconnects long enough that its last_revision is past the compaction horizon, the server cannot replay — it cancels the watch with a compacted error, and the client must do a full re-read.

7.3 Gradual Rollout

sequenceDiagram
    participant Op as Operator
    participant API as Admin API
    participant Cfg as Config Cluster
    participant Canary as 1% of Clients
    participant Rest as 99% of Clients
    participant Mon as Metrics System

    Op->>API: PUT new value, rollout_percent=1
    API->>Cfg: write new value with targeting:percent=1
    Cfg-->>Canary: watchers in the 1% bucket see new value
    Cfg-->>Rest: watchers in the 99% bucket see old value
    Canary->>Mon: emit metrics with new behavior
    Op->>Mon: check error rate, latency
    alt healthy
        Op->>API: bump rollout_percent=10
        API->>Cfg: update targeting:percent=10
        Cfg-->>Rest: 9% additional watchers see new value
        Note over Op: continue ramping...
    else regression detected
        Op->>API: rollback (set rollout_percent=0)
        API->>Cfg: write previous value
        Cfg-->>Canary: rollback to old value
    end

The percent-bucket scheme is implemented client-side: the client computes bucket = hash(client_id) % 100 once at startup; the config sets a threshold t; the client uses the new value if bucket < t. This is deterministic — the same client_id always lands in the same bucket — which is what gives “1% of users see new flow” stable semantics across SDK reads.

8. Deep Dive

8.1 Why Raft (or Paxos) Is Mandatory

The temptation to “just use a database” for config storage misses the consistency requirement. Two properties matter:

Property 1: Single source of truth on writes. If two operators write conflicting config simultaneously, exactly one wins; the winner is recorded and visible. With eventually-consistent storage (DynamoDB without conditional writes, leader-less Cassandra), both can succeed and the read order varies by replica — a recipe for split-brain config.

Property 2: Linearizability under failure. When the leader crashes mid-write, the next leader must reconstruct the most recent committed state. Raft guarantees this through the “election commits” mechanism (only candidates with up-to-date logs can win elections — Ongaro & Ousterhout 2014, https://raft.github.io/raft.pdf §5.4).

Storage backends like Postgres can achieve this with synchronous replication, but they don’t support the watch primitive natively, and bolting a watch layer onto Postgres’ WAL is non-trivial. etcd, ZooKeeper, and Consul are purpose-built for the config-and-coordination workload: small data volumes, mandatory consistency, watch-heavy reads.

8.2 The Watch Primitive — Why It’s Hard

A watch is a long-lived subscription. Three sub-problems:

Sub-problem 1: Connection multiplexing. A million clients each holding a separate TCP connection to the cluster overwhelms ephemeral port limits and connection-table memory. Solutions: connection multiplexing on the server (one TCP socket carries many logical watches via a stream protocol like gRPC), client-side sidecar (one connection per node, fanning out to local processes via Unix socket).

Sub-problem 2: Bounded reliability under disconnect. When a network partition severs the watch, can the client trust that no events were missed once it reconnects? Etcd’s answer: events are tagged with revision numbers; on reconnect the client passes its last revision; the server replays from history if it still has the events. The history window is bounded (you cannot keep all events forever), so very long disconnects fall back to full re-read. The client must handle both cases.

Sub-problem 3: Fanout under high-rate keys. A key watched by 1M clients getting written 100/sec produces 100M event deliveries/sec. The server must avoid serializing this through one socket. Solutions: per-watcher event queues with bounded depth; drop-and-cancel on overflow (“you’re slow; reconnect and re-read”); rate-limit per-client.

8.3 Schema Validation and Typed Config

A bare KV store treats values as opaque bytes. Production systems layer schemas on top:

JSON Schema (https://json-schema.org/). Operators register a schema per key (or per prefix); writes are validated client-side and re-validated server-side at the admin tier. JSON Schema’s properties, required, pattern, enum cover most config validation needs.

Thrift descriptors (configuration-as-code). Facebook Configerator embraces a configuration-as-code model (Tang et al. 2015): engineers do not hand-edit JSON. Instead they write a Apache Thrift-language schema file (job.thrift) defining the typed config object, then write Python files (create_job.cinc, cache_job.cconf) that programmatically manipulate that Thrift object, plus a Python validator (job.thrift-cvalidator) expressing invariants; the Configerator compiler runs the Python, type-checks against the Thrift schema, invokes the validator, and emits a JSON config (Tang et al. 2015, §2 and Figure 2, per the paper PDF read this pass). Because the schema is Thrift and the authoring is code, misnamed fields and invariant violations fail at compile time rather than in production. The build system generates language-specific accessor code so applications consume config as typed objects.

Linting beyond schema. “This config string must be a valid IP address.” “This integer must be between 1 and 1000.” Custom linters at the admin API layer reject obviously-bad values early.

The schema layer’s most underrated benefit is backward-compatibility enforcement: a schema change that removes a field breaks consumers using the old field; the validator can detect this against the prior schema version and refuse the change.

8.4 The Knight Capital Lesson — Why Gradual Rollout Matters

Knight Capital lost 440 million pre-tax loss.

It is worth being precise about the root cause, because the incident is routinely mis-told as a “config-system” failure: it was fundamentally a code-deployment failure (one server missed the new binary) compounded by flag-reuse semantics (the same flag meant different things in old vs new code). The configuration-system implication is nonetheless real and direct: a flag flip is a deployment. Any change — code, config, or flag — must follow the same discipline: canary, observe, gradually expand, automatic rollback on regression. A “global flag flip” that takes 1 second to propagate to a 10,000-server fleet is exactly the no-time-to-react failure mode Knight experienced.

The recommended pattern (canonical in Facebook Configerator, Tang et al. 2015 §6.1):

  1. Canary tier. Push new config to a small percentage (1–5%).
  2. Observation window. Monitor error rates, latency, business metrics for at least N minutes.
  3. Auto-rollback. If observation triggers (error rate > threshold), automatically revert.
  4. Tier expansion. Successive expansion: 5% → 25% → 50% → 100%.
  5. Snapshot before each tier. The previous version remains in storage for instant rollback.

LaunchDarkly’s “gradual rollout” feature implements exactly this for feature flags (https://docs.launchdarkly.com/home/getting-started/feature-flags/percentage-rollouts).

8.5 Audit and Rollback

Every change is recorded:

struct AuditEntry {
    revision:   int64
    timestamp:  time
    actor:      user_id
    action:     enum { put, delete, txn }
    keys:       list[bytes]
    old_values: list[bytes]
    new_values: list[bytes]
    reason:     string                  // mandatory comment
    correlation_id: string              // links to the change ticket
}

Rollback is, conceptually, a write that takes the old_values of the rollback target and writes them as new values. The trick is transactional rollback — a write that touched 5 keys must roll back all 5 atomically, lest you end up with a half-rolled-back configuration that’s inconsistent with both the old and new state. etcd’s Txn primitive handles this.

8.6 Why Configuration Often Lives on the Same Plane as Distributed Locks

The same Raft cluster powering config service typically also powers Distributed Lock Service System Design. The reason: both need (a) consistent serialization of operations, (b) ephemeral keys with leases, (c) watch primitives. ZooKeeper, etcd, and Chubby are coordination services; configuration is just one application of the underlying KV+lease+watch primitive.

The Kubernetes architecture is the textbook example: etcd holds the cluster state (which is one giant configuration), and the same etcd is used for controller leader election (a lock pattern: “who is the active replica controller?”). Multi-purposing the cluster is a design economy and a single point of dependency.

8.7 Edge-Cached Feature Flag Delivery (LaunchDarkly Architecture)

LaunchDarkly’s published architecture (https://launchdarkly.com/blog/launchdarkly-flag-delivery-network/) is instructive:

  • Origin: a centralized cluster holds the truth.
  • Distribution layer: a global CDN of edge POPs holds the rule sets — not the resolved values per user, but the rules themselves.
  • SDK: the SDK fetches rules from its nearest edge POP, caches them locally, and evaluates locally — so a flag check is a local function call, not an RPC. Latency is sub-millisecond.
  • Streaming updates: the SDK opens a streaming connection to the edge POP that pushes rule changes. New rules propagate in seconds globally.

The architecture is the inverse of “centralized evaluation, cached results.” It pushes the policy, not the decisions, to the edge — which is what makes it scale to billions of flag evaluations per minute.

9. Scaling Strategy

9.1 Stage 1 — Single etcd Cluster

3-node etcd cluster, ~10K writes/sec, ~100K watchers. Sufficient for a single Kubernetes cluster of ~5K nodes.

9.2 Stage 2 — Sidecar Pattern

Deploy a node-local agent (Consul Agent style) that maintains the watch on behalf of all colocated app processes. Reduces direct watcher count from “number of processes” to “number of nodes” — a 10–100× reduction.

9.3 Stage 3 — Read Replicas / Followers as Read Path

All followers serve serializable reads. The leader handles writes only. Read-heavy workloads scale horizontally by adding followers; the leader’s write throughput is the bottleneck.

9.4 Stage 4 — Federation Across Namespaces

Multi-tenant deployments shard by namespace: namespace A’s data on cluster 1, namespace B on cluster 2. Cross-namespace reads federate. Each cluster scales independently.

9.5 Stage 5 — Edge Push Layer

For feature-flag-style global flags watched by millions of clients (mobile, web, IoT), push the rule sets to a CDN. The cluster handles operator writes; the CDN handles SDK reads. Like LaunchDarkly’s flag-delivery-network architecture.

9.6 What Breaks First

In production: (1) etcd disk write latency on the leader during peak writes — solved by SSD and proper fsync settings; (2) connection-table exhaustion at the cluster when client count exceeds tens of thousands without a sidecar layer; (3) compaction lag — etcd retains history; old history must be compacted or storage grows unboundedly; (4) leader instability under high CPU — leader elections cause brief unavailability; (5) network partition leaving a minority partition unable to write (correct, but operationally painful).

10. Real-World Examples

Google Chubby (Burrows OSDI 2006, https://research.google/pubs/the-chubby-lock-service-for-loosely-coupled-distributed-systems/). The foundational paper. Originally built as a coarse-grained lock service; quickly became the de-facto Google primary-election and configuration system. Each cell is a 5-replica Paxos group serving thousands of clients with millions of files (small KV pairs). The paper’s most influential observation: clients want a file-system-like hierarchical interface to a lock-and-config service; the API need not be a fancy database.

Apache ZooKeeper (Hunt et al. USENIX ATC 2010, https://www.usenix.org/legacy/event/usenix10/tech/full_papers/Hunt.pdf). Yahoo’s open-source successor to Chubby concepts. Used by Apache Kafka (broker coordination), HBase (master election), Solr (collection state), and countless other Apache-stack systems. Implements ZAB (ZooKeeper Atomic Broadcast), a Paxos-like atomic-broadcast protocol. The “watcher” primitive is a one-time trigger: per the ZooKeeper Programmer’s Guide, “Standard watches are one time triggers; if you get a watch event and you want to get notified of future changes, you must set another watch.” This contrasts sharply with etcd’s stream-based watches, which deliver an ongoing event stream over a single bidirectional gRPC stream without re-registration — a major usability and correctness difference (one-shot watches have an inherent race: between a watch firing and the client re-registering, a second change can be missed unless the client re-reads).

etcd (https://etcd.io/, CoreOS 2013, CNCF graduated). Implements Raft consensus (chosen because of Raft’s pedagogical clarity vs Paxos). Backs Kubernetes for cluster state — every k8s API operation is an etcd transaction. The v3 gRPC API is the de facto modern interface.

HashiCorp Consul (https://www.consul.io/, 2014). Multi-datacenter aware: each datacenter has its own Raft-replicated KV; gossip-based service discovery across datacenters. Adds health-checking and DNS-based service discovery on top of the KV. Often deployed with sidecar agents on every node.

AWS AppConfig (https://docs.aws.amazon.com/appconfig/, 2019). AWS-managed, integrated with the IAM control plane. Strong story for staged rollout (configurable per-deployment validation periods, automatic rollback triggered by CloudWatch alarms). No watch primitive — clients poll on a configurable interval.

Facebook Configerator (Tang et al. SOSP 2015, https://research.facebook.com/publications/holistic-configuration-management-at-facebook/ — full PDF read this pass). Facebook’s internal config system, managing hundreds of thousands of configs distributed to hundreds of thousands of servers across multiple continents, with thousands of configuration changes committed every day by thousands of engineers (Tang et al. 2015, §1 and §6). The storage and distribution backbone is not vanilla ZooKeeper: the ground truth is a git repository (configs are version-controlled source), and a “Git Tailer” continuously extracts committed changes and writes them to Zeus — a forked version of ZooKeeper with scalability and performance enhancements for Facebook scale — which fans them out through a three-level high-fanout distribution tree using a push model (Tang et al. 2015, §3, per the paper PDF). Measured propagation: the proxy on a server takes about 5 seconds to fetch a config change, and a change reaches hundreds of thousands of servers in roughly 4.5 seconds via the distribution tree (Tang et al. 2015, §6.2). Key innovation: configuration-as-code — configs are authored as Apache Thrift schemas plus Python generator/validator files, compiled by the Configerator compiler into JSON, and pushed through a CI-like pipeline (code review via Phabricator, integration tests via Sandcastle, automated canary tests via the Canary Service, commit via the “Landing Strip”). The note’s earlier drafts attributed the authoring language to a “Mustang” DSL and cited “4M config changes per month”; neither claim appears in the paper and both have been corrected to the paper’s actual figures.

LaunchDarkly (https://launchdarkly.com/, founded 2014). Feature-flags-as-a-service. Architecture (https://launchdarkly.com/blog/launchdarkly-flag-delivery-network/): central rule store, global CDN-cached rule distribution, SDK-side evaluation. Optimized for the “flag check” being a local function call.

Spring Cloud Config (https://spring.io/projects/spring-cloud-config). Backed by Git: configs live in a Git repository; the config server reads from Git and serves to Spring applications. Versioning is Git’s. No real watch primitive; clients poll via Spring Cloud Bus webhooks for change notifications.

Knight Capital 2012 Incident (SEC Release No. 34-70694, 2013, https://www.sec.gov/files/litigation/admin/2013/34-70694.pdf). The cautionary tale every config-system designer should read — though, precisely, a deployment failure rather than a config-service failure (see §8.4). New SMARS code was deployed to seven of eight servers; the eighth still ran old code in which a repurposed flag re-enabled the disabled “Power Peg” routing logic. Over ~45 minutes the runaway server produced ~4 million erroneous executions; direct loss: $440M. Lesson: code, config, and flag changes must follow deploy discipline (atomic, all-servers-or-none); reusing a flag’s meaning across a code change is a latent landmine; gradual rollout with observability is non-negotiable.

11. Tradeoffs

Design ChoiceOption AOption BWhen A winsWhen B wins
Storage backendRaft KV (etcd)RDBMS (Spring Cloud Config on Git)Need consistent linearizable writes + native watchesHave existing RDBMS / Git workflow
Delivery modelPush (server streams to client)Pull (client polls)Low push latency requiredSimpler ops, tolerable staleness
Watch granularityPer-keyPer-prefixKeys change often; key set boundedHierarchical configs; new keys appear
Schema enforcementStrict typed (Configerator/Protobuf)Loose (JSON Schema)Polyglot fleet, codegen possibleEvolving schemas, agility valued
Rollout policyAtomic (all clients at once)Gradual (canary → percent ramp)Trivial config, low riskAnything with operational risk
Audit commentOptionalMandatoryInternal team, low oversightPublic/regulated environments
TenancySingle cluster, namespace-isolatedPer-tenant clustersCost optimization; trust between tenantsHard isolation needed; large per-tenant scale
Edge distributionDirect from clusterCDN-cached rulesFew clients, low fanoutMobile/web fleet; global low-latency reads
Consistency levelLinearizable readSerializable readStale read unacceptableMs-level staleness fine

12. Pitfalls

  1. Bad config push as the leading cause of outages. A study of major outages at Facebook (Tang et al. 2015 §6) identifies config changes as the leading cause of incidents — not buggy code, not hardware. Treat every config push as a deploy.

  2. Watch leak. A client opens a watch, errors on the response, and never closes it. Watches accumulate on the server until memory is exhausted. Implement per-client watch limits and idle-watch reaping.

  3. Polling instead of watching. Clients that “didn’t bother with watches” poll every second. With 100K clients × 1Hz polling = 100K reads/sec just to detect changes. This is the dominant load pattern in many production deployments and the easiest fix (use the watch API).

  4. Forgetting rollout windows. Operator pushes the canary, doesn’t wait for the observation window, expands to 100% — and the bug only manifests at higher traffic, taking down the whole fleet. Enforce minimum observation windows; require an explicit “expand rollout” action with the previous tier’s metrics displayed.

  5. Cluster split-brain under network partition. A 3-node cluster split 2:1 has the majority side keep operating; the minority side goes read-only. Both sides serving stale reads can confuse clients. With 5 nodes and a 3:2 split, same dynamics. Always run an odd number of nodes; never 2.

  6. Compaction destroying watch replay. etcd keeps the full multi-version (MVCC) history of every key until it is compacted; a watcher whose last_revision is older than the compaction horizon gets a compacted error on resume and must do a full re-read. Crucially, etcd does not auto-compact by default — --auto-compaction-retention defaults to 0, which disables automatic compaction, so etcd retains all history until the backend hits its 2 GiB default quota and the cluster goes read-only (per etcd v3.6 configuration options and the v3.5 maintenance guide). Operators must explicitly enable periodic compaction (e.g. --auto-compaction-retention=1h, a common but operator-chosen value, not a default); Kubernetes’ kube-apiserver configures its own compaction loop (default every 5 minutes) on top of etcd. The trade-off is the same either way: short retention bounds storage but shortens the watch-replay window, so a client offline longer than the retention period must full re-read. If the full re-read is large (10K keys), this is expensive. Tune the retention window to the worst-case disconnect.

  7. Schema drift between writers and readers. A writer running new code writes a new field; readers running old code don’t know what to do with it. Always evolve schemas backward-compatibly: add optional fields, never rename or remove without a migration period.

  8. PII and secrets in config. A key like /services/api/secrets/db_password in plain text in the config store visible to anyone with read access. Use a separate secret-management system (HashiCorp Vault, AWS Secrets Manager) with stricter access control; the config service references secret IDs, not raw values.

  9. The “config service is down so we can’t deploy a fix” deadlock. When the config service is the only mechanism to reach apps, and the config service itself is down, you cannot push the fix. Always have an out-of-band push mechanism (deploy with hardcoded fallback config) or a multi-tier config (read from local file if cluster unreachable, with explicit warning).

  10. Time-to-propagate underestimated. Operators expect “config change visible everywhere within 1 second.” With 1M clients, watch fanout takes 5–30 seconds depending on cluster topology. Provide a “propagation observed” feedback to the operator — show how many clients have applied the new value vs how many are still on the old.

  11. Unbounded value sizes. A developer stores a 10 MB JSON blob as a single config value. The Raft replication has to ship 10 MB across the network on every write; clients have to download 10 MB on every watch event. etcd caps the maximum request size at 1.5 MiB (1,572,864 bytes) by default (--max-request-bytes), and the backend quota is 2 GiB by default (--quota-backend-bytes), per the etcd v3.5 system-limits doc; respect the cap, and store large blobs in object storage with the config holding only a reference.

  12. Unintentional global flag flips. A “rolling-rollout” feature flag gets accidentally set to 100% in one click. Cushion this with confirmation dialogs, two-person review for global flips, and rate-limited “expand rollout” actions.

13. Common Interview Variants

  • “Implement a feature flag system.” Storage layer (etcd or DynamoDB with conditional writes); evaluation rules with percent buckets and targeting; SDK with local cache + watch; gradual rollout; A/B testing integration.
  • “How would you handle a config rollback that needs to be instant?” Versioning + audit log + transactional rollback (the Txn primitive of etcd). Pre-stage the rollback as a no-op write that flips a “version pointer” key — every client watches the pointer.
  • “Design distributed config for an air-gapped fleet (e.g., mobile/IoT).” Edge-distribution model: rules pushed to CDN; clients poll edges every N minutes; signed configs to prevent tampering.
  • “Build the watch primitive on top of a database that doesn’t support it.” Polling with delta detection: client passes last-known revision; server returns current revision and changed keys since. Drawback: O(polling-interval) latency.
  • “What if the config service must support millions of writes/sec for an experiment platform?” Reconsider the architecture — Raft caps at thousands of writes/sec. Use a sharded backend (per-experiment-namespace shards, each Raft-replicated independently) and accept the loss of cross-shard atomicity.
  • “How does Kubernetes use etcd?” Every k8s resource (Pod, Deployment, ConfigMap) is an etcd key. Controllers watch key prefixes (/registry/pods/) and react to events. The API server is a thin proxy with admission control + validation in front of etcd.
  • “Implement audit-trail rewind.” Append-only log of (revision, key, old_value, new_value, actor, ts). UI lets operator pick a revision and “rewind” — a synthetic write that takes old_values and writes them as new_values, atomically across all keys touched in that revision range.

14. Open Questions

The Knight Capital incident’s framing has been corrected against the SEC order (§8.4, §10): it was a code-deployment-plus-flag-reuse failure, not a config-service failure, and the prose now says so. The earlier uncertainty callout is resolved.

One genuine design-judgment question (not a factual uncertainty) is worth surfacing: should a Kubernetes installation treat etcd as its only configuration plane, or layer a config-as-Git system on top? In practice most teams do layer one on: Helm Charts templates and ArgoCD / Flux reconcile a Git repository of desired manifests down into Kubernetes objects, which the API server then persists in etcd. etcd thus holds cluster state (the reconciled truth) while Git holds the authored, reviewed source — directly analogous to Facebook Configerator’s “git is ground truth, Zeus distributes” split. There is reasonable disagreement on whether application-level dynamic config (feature flags, runtime tunables) belongs in etcd/ConfigMaps at all, or in a purpose-built feature-flag service (AB Testing Platform System Design, LaunchDarkly-style) that is decoupled from the cluster-state plane. This is an architecture trade-off, not an unresolved fact.

Uncertain uncertain

Verify: Facebook Configerator’s current internal architecture. Reason: the verified facts in this note (configuration-as-code via Thrift+Python, git ground-truth, Zeus distribution tree, thousands of changes/day, ~4.5s fan-out) come from the Tang et al. SOSP 2015 paper, which is a snapshot of 2015 production; Meta’s internal systems have certainly evolved in the decade since. To resolve: locate a more recent Meta engineering write-up (the system may have been renamed or re-architected). Treat the 2015 figures as historically accurate, not as current production.

15. See Also

Graph View

Backlinks