vault

io Interface Design

14 min read

io Interface Design

The io package is the most-copied piece of API design in the Go standard library, and io.Reader and io.Writer are its core. Each is a single-method interface — Read(p []byte) (n int, err error) and Write(p []byte) (n int, err error) — and almost every byte that moves in a Go program moves through one of them. The package’s own documentation describes its job as “wrap existing implementations of such primitives … into shared public interfaces that abstract the functionality” (pkg.go.dev/io). The lasting lesson is the small-interface philosophy: an interface with one method is trivial for any type to satisfy, trivial to wrap, and trivial to compose — which is why a *os.File, a bytes.Buffer, a net.Conn, an gzip.Reader, and an HTTP response body are all interchangeable. This note dissects the contracts, the io.Copy fast paths, EOF semantics, and the wrapping types. The nil-interface trap and the two-word interface representation are covered elsewhere — see Interface Internals and Nil Interface vs Nil Pointer.

Mental Model

Think of io.Reader and io.Writer as the two ends of a byte pipe, and of every other type in the package as a pipe fitting: an adapter, a tee, a length limiter, or a concatenator. Because both interfaces have exactly one method, any new type — a decompressor, an encryptor, a rate limiter, a hash — becomes a fitting simply by implementing one method, and it then plugs into every function that already speaks Reader or Writer. The data does not know or care what is on the other end.

flowchart LR
    SRC["source<br/>os.File / net.Conn / bytes.Reader"]
    subgraph fittings["pipe fittings (each is itself a Reader)"]
        GZ["gzip.Reader<br/>(decompress)"]
        TEE["io.TeeReader<br/>(copy to a Writer)"]
        LIM["io.LimitReader<br/>(stop after N bytes)"]
    end
    COPY["io.Copy(dst, src)"]
    DST["sink<br/>os.File / bytes.Buffer / http.ResponseWriter"]
    SRC --> GZ --> TEE --> LIM --> COPY --> DST

    style fittings fill:#fff0e8

Diagram: bytes flow left to right through a chain of fittings, each of which is itself an io.Reader wrapping the previous one. The insight: composition is free because every fitting has the same one-method shape — io.Copy at the end neither knows nor needs to know how many layers it is draining.

Why Small Interfaces

Go’s interface satisfaction is structural and implicit: a type satisfies an interface merely by having the right methods — there is no implements keyword (see Interfaces in Go, Method Sets). This makes interface size a design lever. A one-method interface is the cheapest possible contract: any type with a Read method of the right signature satisfies io.Reader for free, even types in packages that have never heard of io.

Effective Go states the principle directly — “the interfaces and abstractions” should be small, and a type can satisfy several at once. The payoff compounds:

  • Anyone can implement it. strings.NewReader, bytes.Buffer, os.File, net.Conn, gzip.Reader, bufio.Reader, http.Response.Body — dozens of types across many packages are io.Readers without coordination.
  • Anyone can wrap it. A decorator that takes an io.Reader and returns an io.Reader (a decompressor, a counter, a hasher via io.TeeReader) composes infinitely.
  • Anyone can consume it. A function that needs to read bytes asks for an io.Reader and instantly accepts every one of those types.

The contrast is a “fat” interface — say a hypothetical File interface with twenty methods. A test double would need to stub all twenty; a wrapper would need to forward all twenty. Go’s answer is to split fat interfaces: io.ReadCloser is just Reader + Closer, and you ask for exactly the sub-interface you use. The Go proverb captures it: “The bigger the interface, the weaker the abstraction.”

The Reader Contract

type Reader interface {
    Read(p []byte) (n int, err error)
}

The interface is one line; the contract is a dense paragraph, and getting it wrong is a classic Go bug. The documentation says, verbatim (pkg.go.dev/io):

“Read reads up to len(p) bytes into p. It returns the number of bytes read (0 ⇐ n ⇐ len(p)) and any error encountered. … If some data is available but not len(p) bytes, Read conventionally returns what is available instead of waiting for more.

When Read encounters an error or end-of-file condition after successfully reading n > 0 bytes, it returns the number of bytes read. It may return the (non-nil) error from the same call or return the error (and n == 0) from a subsequent call. … The next Read should return 0, EOF.

Callers should always process the n > 0 bytes returned before considering the error err. …

Implementations of Read are discouraged from returning a zero byte count with a nil error, except when len(p) == 0. …

Implementations must not retain p.”

Five rules worth restating in your own words:

  1. A partial read is normal. Read may return fewer bytes than the buffer holds even when more data exists. Code that assumes one Read fills the buffer is wrong; that is what io.ReadFull and io.ReadAll are for.
  2. n > 0 and a non-nil err can come together. A reader may hand you the last 40 bytes and io.EOF in the same call. Therefore: always process n bytes first, then check err. A caller that does if err != nil { return } before consuming p[:n] silently drops the final chunk.
  3. EOF may arrive in the same call or the next one. Both are legal. Robust code handles both — which, again, is exactly what rule 2 buys you.
  4. (0, nil) means “nothing happened,” not EOF. Implementations are discouraged from returning it; callers must not treat it as end-of-stream — looping on it forever is a hang.
  5. Read must not retain p. Once Read returns, the caller may reuse or overwrite the buffer. An implementation that stashes p for later use has a data race waiting to happen.

The Writer Contract

type Writer interface {
    Write(p []byte) (n int, err error)
}

The Writer contract is stricter than Reader and shorter (pkg.go.dev/io):

“Write writes len(p) bytes from p to the underlying data stream. It returns the number of bytes written from p (0 ⇐ n ⇐ len(p)) and any error encountered that caused the write to stop early. Write must return a non-nil error if it returns n < len(p). Write must not modify the slice data, even temporarily.

Implementations must not retain p.”

The key asymmetry: a Reader returning fewer bytes than asked is normal, but a Writer returning n < len(p) is an error condition — Write is obliged to also return a non-nil error explaining the short write. This is why io.Copy’s inner loop (below) treats nr != nw as io.ErrShortWrite. And unlike Read (which “may use all of p as scratch space”), Write must not even temporarily mutate p.

EOF Semantics

io.EOF is a sentinel value, not an error condition in the usual sense:

“EOF is the error returned by Read when no more input is available. (Read must return EOF itself, not an error wrapping EOF, because callers will test for EOF using ==.) Functions should return EOF only to signal a graceful end of input.”

Two consequences. First, io.EOF must be returned bare, never fmt.Errorf("...: %w", io.EOF) — callers compare with err == io.EOF, and a wrapped EOF would fail that test. (errors.Is(err, io.EOF) would still match a wrapped one, but much standard-library code predates wrapping and uses ==.) Second, io.EOF is success, not failure: it means “the stream ended cleanly.” A truncation in the middle of a structured record is a different error — io.ErrUnexpectedEOF, defined as “EOF was encountered in the middle of reading a fixed-size block or data structure.” io.ReadFull formalizes this: it returns plain EOF only if zero bytes were read, and ErrUnexpectedEOF if it read some but not all of the requested bytes.

Functions that drain a stream deliberately swallow EOF. io.Copy “is defined to read from src until EOF, [so] it does not treat an EOF from Read as an error to be reported” — a successful Copy returns err == nil, never err == EOF. io.ReadAll does the same.

io.Copy and the ReaderFrom / WriterTo Fast Paths

io.Copy(dst, src) is the package’s workhorse. Its public face delegates to an unexported copyBuffer, whose source (from src/io/io.go) is worth reading line-by-line:

func copyBuffer(dst Writer, src Reader, buf []byte) (written int64, err error) {
    if wt, ok := src.(WriterTo); ok {        // 1  fast path A
        return wt.WriteTo(dst)
    }
    if rf, ok := dst.(ReaderFrom); ok {      // 2  fast path B
        return rf.ReadFrom(src)
    }
    if buf == nil {                          // 3  slow path: allocate a buffer
        size := 32 * 1024
        if l, ok := src.(*LimitedReader); ok && int64(size) > l.N { // 4
            if l.N < 1 {
                size = 1
            } else {
                size = int(l.N)
            }
        }
        buf = make([]byte, size)
    }
    for {                                    // 5  the generic copy loop
        nr, er := src.Read(buf)
        if nr > 0 {                          // 6  process bytes BEFORE error
            nw, ew := dst.Write(buf[0:nr])
            if nw < 0 || nr < nw {           // 7  guard against a broken Writer
                nw = 0
                if ew == nil {
                    ew = errInvalidWrite
                }
            }
            written += int64(nw)
            if ew != nil {                   // 8  write error: stop
                err = ew
                break
            }
            if nr != nw {                    // 9  short write: stop
                err = ErrShortWrite
                break
            }
        }
        if er != nil {                       // 10 read error or EOF
            if er != EOF {                   // 11 EOF is NOT reported
                err = er
            }
            break
        }
    }
    return written, err
}
  • Line 1, fast path A. If the source also implements io.WriterTo (WriteTo(w Writer) (int64, error)), Copy calls src.WriteTo(dst) and returns. The source knows best how to dump itself — a bytes.Buffer writes its backing slice in one Write; a *os.File may invoke sendfile(2).
  • Line 2, fast path B. Otherwise, if the destination implements io.ReaderFrom (ReadFrom(r Reader) (int64, error)), Copy calls dst.ReadFrom(src). A *os.File destination can call sendfile/copy_file_range; a *bytes.Buffer can Grow once and slurp.
  • Line 3–4. Only if neither fast path applies does Copy allocate a 32 KiB staging buffer — shrunk if the source is a *LimitedReader that will yield fewer bytes, to avoid over-allocating.
  • Line 5–11, the generic loop. Read into buf, write what was read. Line 6 encodes Reader-contract rule 2: bytes are written before the error is examined. Line 7 defends against a Writer that violates its contract. Line 9 turns a short write into io.ErrShortWrite exactly because the Writer contract forbids n < len(p) without an error. Line 11 is the EOF-is-success rule: a plain EOF ends the loop with err == nil.

The fast paths matter enormously in practice. When you io.Copy(httpResponseWriter, file) the *os.File is a WriterTo and the socket is a ReaderFrom, so the kernel can sendfile the bytes without ever copying them into a Go buffer — zero-copy, no 32 KiB allocation, no per-chunk loop. Use io.CopyBuffer to supply your own reusable staging buffer (it “panics” on a zero-length non-nil buf), and io.CopyN to bound the transfer.

The Wrapping Interfaces and Adapters

Beyond Reader/Writer, the package defines:

  • Closer — Close() error. The contract notes “the behavior of Close after the first call is undefined.” This is why you do not blindly defer x.Close() twice.
  • Seeker — Seek(offset int64, whence int) (int64, error), with whence one of SeekStart (0), SeekCurrent (1), SeekEnd (2).
  • Composite interfaces — ReadCloser, ReadWriter, ReadSeeker, WriteCloser, ReadWriteCloser, ReadSeekCloser, and more. Each is just an embedding of the singles: type ReadCloser interface { Reader; Closer }. Ask for the smallest one you actually use — an HTTP handler that only reads the body should take an io.Reader, not io.ReadCloser.

And the adapters, each itself a fitting:

  • io.LimitReader(r, n) returns a Reader (a *LimitedReader) that reads from r but returns EOF after n bytes — the standard defense against an unbounded upload.
  • io.MultiReader(r1, r2, ...) logically concatenates readers, draining each in turn and returning EOF only after the last one does.
  • io.TeeReader(r, w) returns a Reader that writes to w everything it reads from r — the idiomatic way to hash or log a stream as it passes (e.g. compute a SHA-256 of an upload while copying it to disk). It has “no internal buffering — the write must complete before the read completes.”
  • io.Pipe() returns a connected *PipeReader/*PipeWriter: a synchronous in-memory pipe with no buffering, used to glue code that produces via a Writer to code that consumes via a Reader — for instance, streaming a json.Encoder straight into an http.Request body.

Failure Modes and Common Misunderstandings

Checking err before consuming n. The single most common io.Reader bug: n, err := r.Read(buf); if err != nil { return } drops the final n bytes when a reader returns data and io.EOF together. Always handle buf[:n] first.

Treating (0, nil) as EOF. A reader that returns zero bytes and nil has done nothing — it is not end-of-stream. Looping until (0, EOF), not until (0, nil), is correct.

Wrapping io.EOF. Returning fmt.Errorf("read failed: %w", io.EOF) breaks every caller that does err == io.EOF. Return the sentinel bare.

Assuming Read fills the buffer. A single Read may return one byte even when the file is gigabytes. To get exactly len(buf) bytes use io.ReadFull; to get everything use io.ReadAll.

Retaining the slice. A Reader or Writer implementation that keeps a reference to p after returning violates the contract — the caller is free to overwrite that memory, producing a data race or corrupted data.

Forgetting to close — or closing twice. http.Response.Body must be closed to free the connection; closing it twice is undefined. Bound the lifetime with defer and close exactly once.

Alternatives and When to Choose Them

For buffered I/O — coalescing many small reads/writes into few syscalls — wrap with bufio.Reader/bufio.Writer; they are themselves io.Reader/io.Writer, so they slot into any chain. For in-memory accumulation, bytes.Buffer is both a Reader and a Writer and implements the WriterTo/ReaderFrom fast paths. For typed records rather than raw bytes, layer encoding/json, encoding/gob, or bufio.Scanner on top of an io.Reader. When you need backpressure between a producer and a consumer in separate goroutines, io.Pipe beats a bytes.Buffer because it is synchronous and bounded. The general rule: design your own APIs to accept io.Reader/io.Writer rather than concrete types, so callers can hand you a file, a socket, a buffer, or a test double interchangeably.

Production Notes

The io.Copy fast paths are why high-throughput Go servers can serve files at line rate: net/http’s file server ends up in (*os.File).WriteTo → sendfile, moving bytes kernel-to-kernel. The same fast path is a trap for the unwary — if you wrap a *os.File in a struct that does not forward WriteTo/ReadFrom, you silently lose zero-copy and fall back to the 32 KiB loop. The io.TeeReader pattern is ubiquitous in upload pipelines (hash-while-store, content-length-while-stream). And the contract’s “process n before err” rule is so easy to get wrong that the standard library, the race detector’s example suite, and countless code-review checklists call it out explicitly — it is the canonical interview question for “do you actually understand io.Reader?”

See Also

Graph View

Backlinks