Google File System Design

The Google File System (GFS) is a distributed file system designed for very large (multi-gigabyte) files, append-mostly write workloads, and high-throughput sequential reads from many concurrent clients running on commodity hardware where component failures are routine rather than exceptional. Introduced by Ghemawat, Gobioff and Leung at the 2003 ACM Symposium on Operating Systems Principles (SOSP) (paper), GFS deliberately weakens the Portable Operating System Interface (POSIX) consistency model — most importantly, it exposes a special atomic record append primitive with at-least-once semantics — in exchange for radically simpler architecture and operational throughput at petabyte scale. Its open-source clone is the Hadoop Distributed File System (HDFS) which underpins Apache Hadoop, Apache HBase, and the broader 2010s “big data” stack. GFS itself was superseded internally at Google by Colossus, whose architecture Google has since described in a 2021 Cloud blog post (Google Cloud 2021): Colossus replaces GFS’s single in-memory master with a distributed metadata service whose Curators store file-system metadata in BigTable, which “allowed Colossus to scale up by over 100x over the largest GFS clusters” — directly addressing GFS’s most-discussed weakness, the single metadata master.

1. Functional and Non-Functional Requirements

GFS makes its design assumptions unusually explicit — the paper §2 “Design Overview” is essentially a list of requirements that drive every later choice. Restating them in interview-ready form:

Functional requirements.

• Store very large files — the unit of optimization is a multi-GB to multi-TB file, not a tiny config blob. The design openly says “small files are not optimized.”
• Provide a flat hierarchical namespace with create, delete, open, close, read, write, snapshot, and (the GFS-specific) record append operations. The interface is file-like (looks like a Unix filesystem from a programmer’s perspective) but does not implement POSIX. There are no symbolic links, no permissions in the Unix sense, no atomic move-across-directories.
• Support multi-reader concurrent access — many MapReduce mappers reading the same file in parallel is the prototypical workload.
• Support multi-writer concurrent appends — many distributed crawlers writing to one log file is the second prototypical workload. This is the motivation for atomic record append.
• Tolerate routine component failures — disk, machine, network, rack — without operator intervention. Failures are the norm, not exceptional events.
• Support batch-analytics workloads — high sustained throughput is more valuable than low individual-request latency. Trading latency for throughput is acceptable everywhere.

Non-functional requirements.

• Petabyte scale. The 2003 paper already discussed clusters with hundreds of nodes and hundreds of terabytes; production GFS clusters grew well beyond that.
• Commodity hardware. No expensive Storage Area Networks (SANs), no premium-priced fault-tolerant servers. Just thousands of cheap Linux boxes, each of which fails routinely.
• Throughput-oriented. The target is aggregate bandwidth — many gigabytes per second across the cluster — not single-request latency. Some operations (especially metadata operations through the master) are explicitly slow in absolute terms but are infrequent enough to be acceptable.
• Weak consistency for append. Successful concurrent appends are atomic (each append lands intact and contiguously, with no interleaving), but at-least-once rather than exactly-once: a record may appear multiple times if a primary chunkserver retries. Applications are expected to deduplicate by record-ID. This is a deliberate trade — see §8 below for the full discussion.
• Single-writer or coordinated-append semantics. Random in-place mutation of an existing file region is supported but discouraged; the workload assumption is that files are written once and then read many times, or appended to by multiple writers. Random in-place writes break the atomicity guarantees.

2. Capacity Estimation

A back-of-envelope sizing exercise mirroring the 2003 paper’s example cluster (roughly the published scale at SOSP). Suppose:

• Total storage: 10 PB (10 × 10^15 bytes) of user data.
• Replication factor: 3 (the GFS default).
• Total raw storage required: 30 PB.
• Per-chunkserver capacity: 2 TB raw. Then chunkserver count ≈ 30 PB / 2 TB ≈ 15,000 chunkservers. (At 2003 disk sizes the chunkserver count was lower; the math is the same.)
• Chunk size: 64 MB. Chunk count: 10 PB / 64 MB ≈ 156 million chunks (after deduplication of replication, since chunk count is per logical chunk, not per replica).
• Per-chunk metadata in the master: the paper states the master “maintains less than 64 bytes of metadata for each 64 MB chunk” (§2.6.1) — chunk handle, version number, and (rebuilt-not-persisted) replica locations. Total master memory for chunk metadata: 156M × 64 B ≈ 10 GB. This is the headline reason GFS requires a fat master with lots of RAM, and it is also the bottleneck Colossus addresses.
• Per-file metadata: the namespace tree, also held in master RAM, prefix-compressed. The paper states “the file namespace data typically requires less than 64 bytes per file” (§2.6.1). With ~10⁸ files this is another few GB.
• Total master RAM budget: tens of GB, sized to hold all metadata in-memory. The master never reads metadata from disk on a hot path.

Throughput.

• A 2003 commodity disk delivers ~50 MB/s sequential read. With 15,000 chunkservers (3 disks each, say) the cluster raw read bandwidth is 15,000 × 3 × 50 MB/s ≈ 2.25 TB/s, of which only a fraction is realized after network constraints.
• Each chunkserver has a 1 Gbps NIC (2003 era): 15,000 × 1 Gbps ≈ 15 Tbps aggregate, or roughly 1.9 TB/s, again capped at switch bandwidth.
• Per-client read: streaming 64 MB chunks from one chunkserver at NIC line rate ≈ 100 MB/s sustained per client.
• Master operations (create, delete, lookup): the master serves ~10⁴ operations per second from RAM. Above that you become master-bound — see Pitfalls.

The capacity story is “the master is the small bottleneck; chunkservers are the throughput plane.” Master sizing is a per-cluster constant; chunkserver count scales with data volume.

3. Application Programming Interface

GFS exposes a custom client library, not a Linux mountable filesystem. Calls (paraphrased from the paper):

fd        = create(path)
fd        = open(path, mode)
              close(fd)
n_bytes   = read(fd, offset, length, buf)
              write(fd, offset, data)        # serial, primary-coordinated
offset    = record_append(fd, data)          # atomic, at-least-once, GFS-special
              snapshot(src, dst)             # copy-on-write
              delete(path)                   # lazy / two-phase
fileinfo  = stat(path)
listing   = list(directory)

The two unusual entries are record_append and snapshot, both discussed below. There is no rename across directories, no link, no chmod in the Unix sense; access control is simpler and more coarse-grained than POSIX.

HDFS exposes a similar API in Java (FileSystem, FSDataInputStream, FSDataOutputStream) and over the WebHDFS REST surface. HDFS does not expose record-append in exactly the GFS form — it has append() semantics that have evolved over Hadoop versions and are weaker than GFS’s record-append in the original paper.

4. Data Model

The on-disk and in-memory data model has three layers.

File. A user-facing logical entity identified by a path. Has a namespace entry (in the master’s tree) and an ordered list of chunks.

Chunk. A fixed-size (64 MB) unit of storage, identified by a globally unique chunk handle — a 64-bit number assigned by the master at chunk creation. Each chunk is replicated to (typically) three chunkservers. Within a chunk, data is byte-addressable.

Replica. A physical copy of a chunk on a particular chunkserver, stored as a Linux file. Each replica carries a chunk version number that the master uses to identify stale replicas after a chunkserver outage. When a primary mutates a chunk, the master increments the version number; replicas that miss the update remain on the older version and are detected as stale on next contact.

The master holds three persistent metadata categories:

1. Namespace. The hierarchical tree of file and directory names, with full pathnames as keys. Implemented as a prefix-compressed table held in RAM and persisted via an operation log. Locking is per-path with a read-lock-on-prefix discipline that allows concurrent operations in disjoint subtrees.
2. File-to-chunk mapping. For each file, the ordered list of chunk handles. Persisted via the operation log.
3. Chunk-to-replica mapping. For each chunk handle, which chunkservers hold a replica and the chunk’s current version. Not persisted — the master rebuilds this entirely from chunkserver heartbeats at startup. This is a famous design choice: it removes a class of consistency bug (master and chunkserver disagree about replica location after a master crash) by making chunkservers the authoritative source for “what do you actually have on disk?”. The cost is a longer startup time for the master while it polls all chunkservers.

The operation log (master’s write-ahead log) records every namespace mutation. The master flushes it to local disk and a remote replica before applying the mutation in memory and acknowledging the client. To avoid replaying gigabytes of log on every restart, the master takes periodic checkpoints of in-memory state in a B-tree-friendly format that can be loaded directly into memory.

5. High-Level Architecture

flowchart TB
    subgraph Clients["Application Clients (linked against GFS client library)"]
        C1[Client A]
        C2[Client B]
    end

    subgraph Master["Single GFS Master"]
        NS[Namespace + file-to-chunk map<br/>persisted via operation log]
        CL[Chunk locations<br/>NOT persisted; rebuilt from heartbeats]
        OPLOG[(Operation Log<br/>local + remote replica)]
    end

    subgraph Shadow["Shadow Masters (read-only)"]
        SM1[Shadow Master 1]
    end

    subgraph CS["Chunkservers (thousands of commodity Linux boxes)"]
        CS1[Chunkserver 1]
        CS2[Chunkserver 2]
        CS3[Chunkserver 3]
    end

    C1 -->|metadata: 'where is /foo?'| Master
    C2 -->|metadata: 'where is /bar?'| Master
    Master -.cached chunk locations.-> C1
    Master -.cached chunk locations.-> C2
    C1 -->|bulk data read/write| CS1
    C1 -->|bulk data read/write| CS2
    C2 -->|bulk data read/write| CS3

    CS1 -->|heartbeat + chunk inventory| Master
    CS2 -->|heartbeat + chunk inventory| Master
    CS3 -->|heartbeat + chunk inventory| Master

    Master -.async log shipping.-> Shadow

What this diagram shows. GFS is a single-master, many-chunkserver architecture deliberately separated into a control plane (master) and a data plane (chunkservers). Clients first ask the master “where do I find chunk X of file F?”, cache that location for a long-ish lifetime (chunk leases on the order of 60 seconds), and then talk directly to chunkservers for bulk data transfer. The master never sees the user payload — it only sees metadata. This is the central architectural choice of GFS: by keeping the master out of the data path, the master’s RAM-bound bandwidth limit (a few GB/s of metadata) is not the cluster’s bandwidth limit (many TB/s of bulk data through the chunkservers). The shadow masters tail the operation log asynchronously and serve stale read-only metadata during a master outage; they are not promotion targets in the original paper. Heartbeats go upward — every chunkserver tells the master what chunks it holds, and the master uses these reports to maintain the chunk-to-replica index without persisting it.

6. Request Flow

6.1 Read

sequenceDiagram
    actor Client
    participant Master
    participant CS_Primary as Chunkserver (any replica)

    Client->>Master: lookup(filename, byte_offset)
    Note over Master: compute chunk_index = offset / 64MB
    Master-->>Client: chunk_handle, replica_locations[3], version
    Note over Client: cache for ~60s
    Client->>CS_Primary: read(chunk_handle, version, range)
    CS_Primary-->>Client: bytes

The read path’s elegant property is that the master is consulted once per chunk, not once per byte. A client streaming a 1 GB file makes ~16 master calls (1024 MB / 64 MB) and one chunkserver call per chunk. After the cache warms, the master’s load is minimal.

If a chunkserver returns an error or the version number is stale (a lagging replica), the client retries with a different replica from the cached set. If all three are stale or down, the client re-queries the master and gets a fresh location set.

6.2 Write — Primary-Coordinated Pipeline

sequenceDiagram
    actor Client
    participant Master
    participant Primary
    participant Secondary1
    participant Secondary2

    Client->>Master: who is primary for chunk C?
    Master-->>Client: primary=Primary, secondaries=[Secondary1, Secondary2], lease until T+60s
    Client->>Primary: data D (pushed in pipeline)
    Primary->>Secondary1: forward D
    Secondary1->>Secondary2: forward D
    Note over Client,Secondary2: data is in all 3 replica buffers but not yet committed
    Client->>Primary: COMMIT(C, D, write_id)
    Primary->>Primary: assign serial# in lease order; apply
    Primary->>Secondary1: APPLY at serial#
    Primary->>Secondary2: APPLY at serial#
    Secondary1-->>Primary: ack
    Secondary2-->>Primary: ack
    Primary-->>Client: success (or partial-failure error)

Two phases: data push (data flows in a chained pipeline along the network topology to minimize redundant inter-rack hops — chunkservers forward to the closest unfilled replica) and control flow (the client tells the primary to commit, and the primary chooses a serial number that all replicas obey). The primary’s role is only to serialize concurrent writes — it does not generate the data or buffer it for long. Secondaries hold the data in a per-write buffer until the primary tells them to apply at a particular serial number.

If a secondary fails to acknowledge the apply, the write is reported to the client as a partial failure. The replica state is now inconsistent across replicas — some have the data, some do not. The application is expected to retry; the consistency anomaly is documented and one of the deliberate weaknesses of GFS.

6.3 Record Append — The GFS Special

sequenceDiagram
    actor Client
    participant Primary
    participant Secondary1
    participant Secondary2

    Client->>Primary: record_append(file F, data D <= 16MB)
    Note over Primary: pad current chunk if D would cross 64MB boundary;<br/>create new chunk if needed
    Primary->>Secondary1: pipeline data
    Primary->>Secondary2: pipeline data
    Primary->>Primary: pick offset O at end of chunk; apply at O
    Primary->>Secondary1: APPLY at offset O
    Primary->>Secondary2: APPLY at offset O
    Secondary1-->>Primary: ack
    Secondary2-->>Primary: ack
    Primary-->>Client: success at offset O

    Note over Client,Secondary2: If any replica fails to ack, primary tells client to retry.<br/>Client retry may produce a duplicate record at a new offset.

The semantics: each replica gets the record written atomically at the same offset (or, on retry, at a new offset all three eventually agree on). Records may appear multiple times on success-after-retry. Records may also be interleaved with padding bytes when a chunk is closed mid-record. Applications consume the file with a deduplicating reader keyed on a per-record unique ID — a discipline the paper explicitly recommends.

The reason record-append exists: it lets many writers (often thousands of crawlers or log producers) hammer one file simultaneously without serializing through a single coordination point. Standard write requires the client to know the offset, which forces external coordination; record-append delegates offset assignment to the primary and pipelines the work.

7. Deep Dive — Selected Topics

7.1 Why a Single Master

The original GFS chose one master, not a Paxos-replicated quorum, because:

1. Simplicity. Distributed metadata with a consensus protocol is hard to get right; a single master moves the consistency problem to a single machine.
2. The metadata fits in RAM. With sub-100-byte per-file overhead, even tens of millions of files fit comfortably in ~64 GB. There is no need to shard metadata.
3. The master is not on the data path. Its IOPS budget is a few thousand metadata ops per second, well within reach of a single fat machine.

The master is made highly available by:

• Operation log replication. Every metadata mutation is logged locally and on a remote replica before it is applied. After a crash, the log replays from the last checkpoint to recreate the in-memory state.
• Checkpointing. Periodic compact-format snapshots of the in-memory state. Recovery time becomes (last checkpoint + log tail), not a full log replay.
• External monitoring. A health monitor outside the master process restarts the master on hard failure. Failover time is on the order of tens of seconds.
• Shadow masters. Read-only replicas of the master that lag the operation log; clients can fall back to them for stale-but-okay metadata reads while the master is restarting.

The single-master limit is real: by 2010, McKusick & Quinlan’s ACM Queue interview acknowledged that GFS clusters had grown to where “the master is now the bottleneck” — in particular, the per-chunk metadata footprint grew to where master RAM became scarce, and the operation rate grew to where the master’s CPU became scarce. Colossus, the GFS successor, attacks exactly this. Per Google’s 2021 Cloud blog post — the first reasonably detailed public description, though still not a peer-reviewed paper — Colossus’s metadata control plane consists of many Curators, to which clients talk directly for control operations such as file creation, and which “store file system metadata in Google’s high-performance NoSQL database, BigTable.” Pushing the namespace and chunk index into BigTable turns the single fat in-RAM master into a horizontally-scalable, sharded store — Google states this “allowed Colossus to scale up by over 100x over the largest GFS clusters,” with “a single cluster scalable to exabytes of storage and tens of thousands of machines.” Background Custodians handle durability and efficiency tasks (disk-space balancing, RAID reconstruction), and data flows directly between clients and “D” file servers (Google’s network-attached disks) — the same control-plane/data-plane split GFS pioneered, now with the control plane itself distributed. (BigTable runs on top of Colossus today, so the dependency was inverted over time; the original Colossus motivation, per the blog, was to scale metadata for Search.)

7.2 The Consistency Model

The paper’s table (§2.7) is the canonical reference. After a successful mutation, a region of a file is in one of these states:

• Defined. All replicas hold the same data; clients reading any replica see the bytes the writer intended. This is the case for serial successful writes and for successful record-appends (modulo possible duplicates at later offsets).
• Consistent but undefined. All replicas agree on what the bytes are, but the bytes are a mix of data from concurrent writers. Concurrent successful writes to overlapping regions produce this state — every replica has the same final bytes, but those bytes are some interleaving of the concurrent writes. The reader sees a consistent view but it is not what any single writer intended.
• Inconsistent. Replicas disagree. Writes that suffered a partial failure (some replicas committed, some did not) leave the region inconsistent. Clients reading different replicas may see different bytes.

The crucial weakening compared to POSIX: GFS does not offer linearizable writes to overlapping regions. Applications avoid this by either (a) writing to non-overlapping regions, (b) using record-append exclusively, or (c) writing each chunk via a single writer with external coordination. This is why GFS works for append-mostly workloads — they avoid the dangerous case by construction.

7.3 Garbage Collection — Lazy Delete

When a file is deleted, the master:

1. Renames the file to a hidden tombstone path (<original>.deleted_at_<timestamp>).
2. The chunk-to-file mapping points to a “soon-to-be-orphaned” entry.
3. After a grace period (default 3 days), the master removes the namespace entry.
4. When chunkservers heartbeat, the master replies with a list of “chunks you hold that I no longer reference”; the chunkserver deletes those replicas locally.

The benefits: simpler reasoning (no synchronous file-deletion across thousands of chunkservers), safety against accidental deletion (3-day undo window), and batched I/O for the deletes. The cost: storage is reclaimed lazily; aggressive deletion patterns can leave orphan chunks consuming space for days.

7.4 Snapshot via Copy-on-Write

snapshot(src, dst) makes a near-instant copy of a file or directory:

1. Master revokes outstanding chunk leases for chunks under src (so no in-flight writes can race with the snapshot).
2. Master logs and applies the namespace operation: dst is created pointing at the same chunk handles as src. Reference counts on chunks become 2.
3. Subsequent writes to src or dst trigger the master to instruct the chunkserver to copy-on-write: a new chunk handle is allocated, the old chunk’s bytes are copied into a new replica set, and the writer’s path is updated to point at the new handle.

This is conceptually identical to filesystem-level COW (Copy-on-Write — making lazy copies that materialize only when one side mutates) such as ZFS or btrfs snapshots, but applied at the chunk level. The snapshot itself is O(metadata) — instant to take regardless of file size — and the copy cost is paid lazily on first write to either side.

7.5 Replication Placement

Default replication factor: 3. Placement policy (§4.2 of the paper):

• Spread across machines. No two replicas of the same chunk on the same machine, obviously.
• Spread across racks. At least two of the three on different racks. This survives a top-of-rack switch failure (the most common rack-level outage).
• Below average disk utilization. New replicas prefer chunkservers below cluster mean disk usage to balance long-term load.
• Recently-failed avoidance. Chunkservers that recently restarted are deprioritized as new-chunk targets.

When the master detects under-replication (e.g., a chunkserver disappears for >10 minutes), it schedules re-replication with priority based on (a) how far below the target factor the chunk is — a chunk down to 1 replica is critical — and (b) how recently the file was accessed. Re-replication runs in the background using throttled bandwidth so it does not displace foreground reads.

8. Scaling Considerations

GFS scales in three dimensions:

• Storage scale (chunkservers). Linear: add more chunkservers, the master rebalances chunks via re-replication. The 2003 paper talked about hundreds of nodes; production GFS clusters reportedly grew to thousands.
• Read throughput. Linear in chunkserver count, since the master is bypassed once locations are cached. The bottleneck becomes network fabric (cross-rack bisection bandwidth) before it becomes any single component.
• Metadata scale (master). Not linear. The master’s RAM holds the entire namespace and chunk-to-file map; once that exceeds tens of GB, the master is at its single-machine limit. This is the GFS limit. Colossus addresses it with a sharded metadata layer (multiple master “shards” each owning a key range).

Beyond GFS: the HDFS evolution path. HDFS (Hadoop’s GFS clone) faced the same wall and responded with HDFS Federation (multiple independent NameNodes, each owning a namespace prefix) and HDFS High Availability (active/standby NameNode pair coordinated via a journal; ZooKeeper-based failover). HDFS HA was added in Hadoop 2.x; HDFS Federation was added around the same time. Together they give HDFS what Colossus gives GFS: namespace partitioning plus master fault tolerance.

9. Real-World Examples and Citations

• Google File System (Ghemawat et al. SOSP 2003, paper) — the original. Underpinned MapReduce, Bigtable, the early Google search index, and most of Google’s batch infrastructure for ~5 years.
• Hadoop Distributed File System (Shvachko et al. MSST 2010, paper) — Yahoo’s open-source clone of GFS, originally written by Doug Cutting (also of Lucene fame). The de facto big-data storage layer of the 2010s. Underpins Apache HBase, Apache Hive, Apache Spark (when run on HDFS), and innumerable enterprise data lakes.
• Colossus (Google internal, ~2010 onward) — the GFS successor. Discussed publicly in McKusick & Quinlan’s ACM Queue interview (2009), Andrew Fikes’ 2010 Faculty Summit talk, and most concretely in Google’s 2021 Cloud blog post. Confirmed changes: distributed metadata via Curators backed by BigTable (vs GFS’s single in-RAM master), Custodians for background durability/balancing, and direct client-to-”D”-file-server data flow. The blog notes that “applications built on top of Colossus use a variety of encodings to fine-tune performance and cost trade-offs,” including client-side “software RAID” — i.e. erasure coding is supported, but the exact codes (e.g. Reed-Solomon parameters) and any Spanner coupling are not spelled out publicly.
• Microsoft Azure Storage / Tectonic at Meta — distinct lineage but solving the same problem space; both reference the GFS paper as foundational.

Uncertain

Verify: Colossus’s exact erasure-coding parameters (Reed-Solomon k/m), per-data-class replication factors, and any coupling to Spanner. Reason: Google’s 2021 Cloud blog confirms the high-level architecture (Curators on BigTable, Custodians, D file servers, >100× metadata scale, exabyte single clusters) but deliberately abstracts the encodings (“a variety of encodings,” “software RAID”) and gives no numeric durability scheme. To resolve: a peer-reviewed Colossus paper (none exists as of 2026-05) or a more detailed Google engineering disclosure. The architectural shape above is now well-sourced; the numbers are not. uncertain

10. Tradeoffs

Decision	Choice	Alternative	Why GFS chose this
Master count	Single	Replicated quorum (Paxos)	Simpler; metadata fits in one machine’s RAM
Replica count	3	2 (cheaper) or 5 (safer)	Empirical balance of MTBF (mean time between failures) vs cost
Chunk size	64 MB	4 KB (POSIX-like)	Reduces master metadata 16,000×; large sequential I/O is the workload
Chunk locations	Not persisted	Persisted with master state	Master + chunkservers can disagree after master crash; rebuild from heartbeats removes that bug class
Consistency model	Weak (record-append at-least-once)	Linearizable	Concurrency without coordination; apps deduplicate
Delete	Lazy, 3-day window	Synchronous	Safety + batched I/O
Snapshot	Copy-on-write at chunk level	Eager copy	O(metadata) snapshot time
Network model	Direct client ↔ chunkserver	Through master	Master would be the bandwidth bottleneck otherwise
Read parallelism	Many readers per chunk	Single-reader-per-chunk lock	Workload is read-heavy; locking would serialize

11. Pitfalls

1. Single-master bottleneck. As cluster size grew, master CPU and master RAM both became scarce. Symptoms: slow open()/stat()/list() calls under load; long master restart times; metadata size exceeding fat-server RAM. Colossus and HDFS Federation are the two industry responses.

2. Small-file problem. Each file consumes ~100 bytes of master metadata regardless of size. A million 1-KB files consume the same metadata budget as a million 1-GB files, but only 1 GB of chunkserver storage. Workloads with many tiny files exhaust master RAM with negligible disk usage. The remediation is to bundle small files into a single large file (HBase’s HFile, Hadoop SequenceFile, Apache Parquet, Hadoop Archives) — this is one reason the 2010s “data-lake” stack gravitated to columnar formats.

3. Master memory is the cluster ceiling. Per-chunk metadata has a hard floor (~64 bytes/chunk in GFS). Doubling the chunk count doubles master RAM. This makes growth planning nonlinear — you cannot scale storage indefinitely on a fixed master.

4. Record-append duplicates surprise applications. Developers porting code from POSIX often write append-then-read code that assumes each record_append lands exactly once. Under failures, GFS will produce duplicates. The application must deduplicate by record ID; forgetting to causes silent data corruption.

5. Concurrent overlapping writes leave consistent-but-undefined regions. Two writers writing to overlapping byte ranges of the same file produce an interleaving that is consistent across replicas but is not what either writer intended. Applications avoid this by either (a) using record-append, (b) per-file single-writer discipline, or (c) external coordination (e.g., Chubby locks). Many production bugs are traced to ignoring this constraint.

6. Stale replicas after primary churn. When a primary’s lease expires and a new primary is elected, the chunk version is bumped. Clients with cached metadata pointing at old replicas may briefly read stale data. Mitigation: bound the lease lifetime (60 s default) so cached locations expire quickly.

7. Partial-failure writes leave inconsistent regions. Write semantics are not all-or-nothing across the three replicas; a write that succeeds at the primary but times out at one secondary leaves the file inconsistent (replicas disagree). The client retries, but if it gives up, the inconsistency persists until that chunk is rewritten. Re-replication of failed-write chunks is one corner case the master must handle explicitly.

8. HDFS NameNode failover is slow without HA configuration. In stock single-NameNode HDFS, NameNode failure means the cluster is read-only or down until the NameNode is restarted (minutes). HA mode pairs an active and standby NameNode coordinated via a JournalNode quorum and ZooKeeper; failover is then sub-minute. Many deployments still skip HA.

12. Common Interview Variants

• “Design HDFS” — same answer as design GFS, with HA NameNode + Federation if pressed on scaling.
• “Why didn’t Google replicate the master with Paxos?” — interviewer is fishing for the simplicity argument and the “metadata-fits-in-RAM” point. Bonus: explain why operation-log replication + checkpointing achieves practical HA without consensus.
• “What if the chunk size were 4 KB instead of 64 MB?” — master metadata explodes 16,000×; sequential I/O becomes per-chunk-overhead-bound. Use the calculation in §2.
• “How would you support small files efficiently?” — bundle into a sequence file or columnar format; do not store one-per-chunk.
• “How does GFS differ from Amazon S3?” — see Amazon S3 Object Storage System Design. Headline: GFS is a file system with sequential semantics and append; S3 is an object store with key-blob semantics and PUT/GET. GFS clients link a library; S3 is HTTP. GFS is in-cluster; S3 is internet-scale.
• “What replaced GFS at Google?” — Colossus. Distributed metadata; supports erasure coding; tight Spanner integration. No formal paper.
• “Walk me through a write.” — primary lease + pipeline data push + serial-numbered apply + ack. See §6.2.
• “Why is record-append at-least-once and not exactly-once?” — exactly-once requires either two-phase commit (slow) or persistent client identifiers with deduplication on the server (storage overhead). At-least-once + app-level dedup pushes the cost to the few apps that need it.
• “How does GFS handle a chunkserver coming back from a long outage?” — the chunkserver re-registers and reports its chunk inventory; the master sees the version numbers, marks any older-than-current-version replicas as stale, and schedules them for garbage collection. The chunkserver’s storage is effectively reclaimed.

13. Open Questions / Uncertain

• Colossus’s metadata partitioning is now known to be BigTable-backed Curators (Google Cloud 2021) — but the sharding key and rebalancing mechanics are still not public.
• How much does Colossus’s erasure coding reduce raw storage versus 3× replication, in production? (Exact codes unpublished; the blog only says “a variety of encodings.”)
• At what cluster size did Google internally migrate from GFS to Colossus? (Believed to be late 2000s; details internal.)
• HDFS Federation has known operational complexity (administering many namespaces); is it adopted at scale or do most sites stick with single-NameNode HA?

Uncertain

Verify: numeric and protocol-level Colossus claims (sharding key, exact erasure codes, Spanner coupling, migration timeline). Reason: the public record (talks + the 2021 Google Cloud blog) gives architecture but not internals. To resolve: an authoritative Google publication. The named components — Curators, Custodians, D file servers, BigTable metadata — are now officially sourced and safe to state; treat anything more specific as “industry inference.” uncertain

14. See Also

• Amazon S3 Object Storage System Design — the object-store cousin; different consistency story
• Distributed Key Value Store System Design — Dynamo-style; GFS is one of the original “commodity-hardware-tolerates-failure” arguments
• Distributed SQL Database System Design — Spanner runs on Colossus, the GFS successor
• LSM Tree — Bigtable (which runs on GFS/Colossus) uses LSM-style storage
• Bloom Filter — used in Bigtable’s SSTable lookups
• Consistent Hashing — not used by GFS itself; contrast to Dynamo’s choice
• Raft / Paxos High-Level — what GFS did not use for metadata
• Two-Phase Commit — a possible (and rejected) alternative for record-append exactly-once
• Major System Designs MOC
• SWE Interview Preparation MOC