vault

JSON Encoding and Decoding Internals

15 min read

JSON Encoding and Decoding Internals

The encoding/json package converts between Go values and JavaScript Object Notation text. It is reflection-driven: with no code generation and no schema, it inspects each value’s reflect.Type at run time, dispatches to a per-type encoder or decoder, and — critically — caches that work so the reflection cost is paid roughly once per type rather than once per value. This note traces the v1 mechanism (the encoderFunc cache, struct-field caching, the streaming Encoder/Decoder, the Marshaler/Unmarshaler escape hatches) and then the encoding/json/v2 redesign, which landed as an experiment behind GOEXPERIMENT=jsonv2 in Go 1.25 (go.dev/doc/go1.25) and remains experimental — not yet stable — as of Go 1.26 (released February 2026; the package docs still say it “only exists when building with the GOEXPERIMENT=jsonv2 environment variable set”, pkg.go.dev/encoding/json/v2). Its v1-quirk fixes and streaming architecture are the intended future of JSON in Go.

Mental Model

The right way to think about encoding/json v1 is a two-phase machine: a type-analysis phase that uses reflection and is expensive, and a value-walking phase that runs the analyzed plan over actual bytes and is comparatively cheap. The package’s whole performance story is moving the expensive phase out of the hot path with a sync.Map cache keyed by reflect.Type.

encoding/json/v2 reframes the whole thing along a different axis: it separates syntax from semantics. Syntax — is this well-formed JSON, where does this token end — is handled by a new, reflection-free package encoding/json/jsontext. Semantics — how does a JSON object map to a Go struct — is handled by encoding/json/v2 on top of jsontext. v1 fused the two; v2 splits them, which is what makes true streaming and pluggable customization possible.

graph TD
    subgraph v1["encoding/json (v1)"]
        M1["Marshal(v)"] --> TE["typeEncoder(reflect.Type)"]
        TE -->|cache hit| EC[(encoderCache sync.Map)]
        TE -->|cache miss| NTE["newTypeEncoder builds encoderFunc"]
        NTE --> EC
        EC --> RUN["encoderFunc walks value -> bytes"]
    end
    subgraph v2["encoding/json/v2 (experimental)"]
        SEM["json/v2: semantics<br/>Go value <-> JSON value"] --> SYN["json/jsontext: syntax<br/>tokens, validation, no reflection"]
    end

Diagram: v1 caches a per-type encoderFunc in a sync.Map so reflection is amortized; v2 instead splits the problem into a reflection-free syntax layer (jsontext) and a semantics layer (json/v2). The insight: v1’s performance trick is caching; v2’s architectural trick is layering — and the layering is what unlocks streaming.

Mechanical Walk-through — v1 Encoding

json.Marshal(v) allocates an encodeState — a wrapper around bytes.Buffer that accumulates output and tracks a ptrLevel counter and a ptrSeen map to detect pointer cycles (“Keep track of what pointers we’ve seen in the current recursive call path, to avoid cycles that could lead to a stack overflow”, src/encoding/json/encode.go). The cycle check engages only after the nesting depth passes a startDetectingCyclesAfter threshold of 1000 levels, so shallow data pays nothing for the bookkeeping; a genuine cycle (a struct pointing back at itself through pointers, maps, or slices) is caught past that depth and returns an UnsupportedValueError("encountered a cycle via …") rather than overflowing the stack. It then calls typeEncoder for the value’s reflect.Type.

typeEncoder consults encoderCache, a package-level sync.Map declared as var encoderCache sync.Map // map[reflect.Type]encoderFunc, mapping reflect.Type to encoderFunc (whose exact signature is func(e *encodeState, v reflect.Value, opts encOpts), per src/encoding/json/encode.go). On a hit, the cached function is used directly — no further reflection. On a miss, newTypeEncoder builds one. There is a subtle bootstrapping wrinkle for recursive types (a struct that contains itself, e.g. a tree node): the package stores an indirect := sync.OnceValue(func() encoderFunc { return newTypeEncoder(t, true) }) placeholder in the cache before the real encoder finishes building, so a recursive typeEncoder call during construction finds the placeholder — which lazily, exactly-once builds and delegates to the real function — instead of looping forever. The sync.OnceValue is what guarantees the recursive type’s encoder is constructed only once even under concurrent first use.

newTypeEncoder is a dispatch over the type’s characteristics, checked in order: does the type (or its pointer) implement json.Marshaler? encoding.TextMarshaler? Otherwise, switch on reflect.Kind — Bool, the integer kinds, the float kinds, String, Struct, Map, Slice, Array, Ptr, Interface — each producing a specialized encoder (boolEncoder, structEncoder, mapEncoder, …).

The struct path is where the real caching lives. newStructEncoder calls cachedTypeFields, which (via typeFields) does a one-time, breadth-first walk of the struct’s fields — resolving embedded/anonymous fields, parsing every json:"..." struct tag, applying the visibility and name-conflict rules, and computing each field’s index path. The result, a structFields value, holds the ordered field list plus lookup maps byExactName and byFoldedName (the latter for v1’s case-insensitive decoding). This whole analysis is cached per type; encoding a million structs of the same type parses the tags exactly once. At encode time structEncoder iterates the cached fields, applies omitempty/omitzero logic, writes the field name, and recurses into each field’s own (also cached) encoder.

Two struct-tag features worth pinning down precisely. omitempty omits a field whose value is an empty value — false, 0, a nil pointer/interface, or an empty array/slice/map/string (length zero). It does not omit a zero-valued struct, and it does not omit a non-nil pointer to a zero value — a long-standing source of confusion. Go 1.24 added omitzero, which omits a field equal to its type’s zero value (and consults an IsZero() bool method if present), fixing the “I want to omit a zero time.Time” case that omitempty never handled.

Mechanical Walk-through — v1 Decoding

json.Unmarshal(data, &v) runs a scanner over data to validate JSON syntax and then a recursive-descent decodeState that walks the JSON in lockstep with the destination’s reflect.Value. For a JSON object decoded into a Go struct, for each member name it looks up the matching struct field — first an exact, case-sensitive match in byExactName, then a case-insensitive fallback via byFoldedName. That case-insensitive fallback is a deliberate v1 feature and, as the v2 design post argues, “a surprising default, a potential security vulnerability, and a performance limitation” (go.dev/blog/jsonv2-exp).

Decoding into an interface{} (i.e., any) uses fixed default mappings: a JSON object becomes map[string]any, an array becomes []any, a string becomes string, true/false become bool, null becomes nil, and — the classic trap — every JSON number becomes a float64, because v1 has no other information to go on. Unmarshaling the integer 123456789012345678 into an any therefore loses precision; unmarshaling it into an int64 field does not. (Decoder.UseNumber switches numbers-into-any to the json.Number string type to avoid the loss.)

If the destination type implements json.Unmarshaler, the package hands it the raw bytes of the JSON value and lets it decode itself. v1’s Unmarshaler contract requires a complete JSON value with no trailing data — which forces the package to find the end of the value first and is part of why custom UnmarshalJSON methods can be slow (the value is effectively parsed twice — once to delimit it, once by the method).

Streaming — Encoder and Decoder

json.NewEncoder(w) and json.NewDecoder(r) (src/encoding/json/stream.go) wrap an io.Writer/io.Reader. They are how you process JSON without holding the whole document in memory:

dec := json.NewDecoder(resp.Body)   // 1
for dec.More() {                    // 2
    var rec Record
    if err := dec.Decode(&rec); err != nil { return err }  // 3
    process(rec)
}
  1. The Decoder reads from resp.Body incrementally.
  2. More() reports whether another element remains in the current array/object — the idiom for streaming a JSON array of records.
  3. Each Decode consumes exactly one JSON value from the stream.

The honest caveat: v1’s Decoder is not fully streaming at the value level. Decode buffers an entire JSON value into memory before unmarshaling it (it internally finds the value’s end, then runs the same decodeState over that buffer). Streaming a 10-GB array of small records works fine; decoding a single 10-GB object still allocates 10 GB. True token-level streaming is exactly what v2’s jsontext provides. Decoder.Token() does expose a token stream in v1, but it is a separate, lower-level API from Decode.

The v2 Redesign — Status and What It Fixes

Go 1.25 shipped an experiment: “Go 1.25 includes a new, experimental JSON implementation, which can be enabled by setting the environment variable GOEXPERIMENT=jsonv2 at build time” (go.dev/doc/go1.25). When enabled, two packages appear — encoding/json/v2 (the semantic revision, “a major revision of the encoding/json package”) and encoding/json/jsontext (which “provides lower-level processing of JSON syntax”, the reflection-free syntax layer) — and the existing encoding/json is re-implemented in terms of v2. Concretely, that re-implementation is itself conditional on the build tag: the standard encode.go carries //go:build !goexperiment.jsonv2 and is the standalone v1 encoder, while a parallel file built only under goexperiment.jsonv2 delegates to v2 — so the v1 API runs on v1 code by default and on v2 code only when the experiment is set. The new packages are not subject to the Go 1 compatibility promise and “only exist when building with the GOEXPERIMENT=jsonv2 environment variable set” (or the equivalent goexperiment.jsonv2 build tag).

Status as of Go 1.26 (released February 2026): still experimental, not graduated. The Go 1.25 blog post had held open that the experiment’s outcome “may result in anything from abandonment of the effort, to adoption as stable packages of Go 1.26” (go.dev/blog/jsonv2-exp), and that was the most optimistic timeline. It did not happen: the Go 1.26 release notes make no mention of encoding/json/v2, jsontext, or GOEXPERIMENT=jsonv2 (go.dev/doc/go1.26), and — the positive confirmation — the live encoding/json/v2 package documentation still states it “is experimental, and not subject to the Go 1 compatibility promise. It only exists when building with the GOEXPERIMENT=jsonv2 environment variable set. Most users should use [encoding/json]” (pkg.go.dev/encoding/json/v2). The governing proposal, #71497 (opened 31 January 2025), is labeled Proposal-Accepted but sits in the Backlog milestone with no release target, and a json/v2 working group continues to iterate on the API. So the design is accepted in principle, but the stable encoding/json/v2/jsontext packages had not shipped in any Go release as of this writing (May 2026). One concrete reason the graduation slipped: a real performance regression, #75026 (“encoding/json: skyrocketing memory allocation in encoding when jsonv2 experiment enabled”), found that with the experiment enabled, encoding a large map of long string values allocated dramatically more memory (≈260 MB vs ≈27 MB without it, with bytes.growSlice/bytes.(*Buffer).grow dominating) — a serious blocker precisely because v2 will eventually back the v1 API. That issue has since been closed/Done, but it illustrates that v2 is still being shaken out under real workloads.

v2 exists because v1’s defects “are inherent consequences of the existing API, making them practically impossible to fix within the Go 1 compatibility promise” (go.dev/blog/jsonv2-exp). The behavioral fixes:

  • Invalid UTF-8 — v1 accepts it; RFC 8259 forbids it; v2 “reports an error in the presence of invalid UTF-8.”
  • Duplicate object member names — v1 silently accepts them (last wins); v2 “reports an error if a JSON object contains a duplicate name.”
  • nil slice/map — v1 marshals a nil slice/map as JSON null; v2 “marshals a nil Go slice or Go map as an empty JSON array or JSON object, respectively” ([]/{}).
  • Case-insensitive matching — v1’s surprising default becomes a case-sensitive match in v2 (“unmarshals a JSON object into a Go struct using a case-sensitive match”).
  • omitempty redefined — v2 “redefines the omitempty tag option to omit a field if it would have encoded as an ‘empty’ JSON value (which are null, "", [], and {})” — a JSON-shaped definition rather than v1’s Go-value-shaped one, and closer to what most users expect.
  • time.Duration has no default representation — v2 “reports an error when trying to serialize a time.Duration, which currently has no default representation, but provides options to let the caller decide” (#71631).
  • Inconsistent MarshalJSON on pointer receivers — v1 “inconsistently called” pointer-receiver MarshalJSON methods due to an implementation detail (#22967); v2 fixes this.

All quotations above are from the v2 design blog post (go.dev/blog/jsonv2-exp).

And the architectural fixes. json/v2 adds functions that operate directly on io.Reader/io.Writer — alongside Marshal(in any, opts ...Options) ([]byte, error) there are MarshalWrite(out io.Writer, in any, opts ...Options) error and MarshalEncode(out *jsontext.Encoder, in any, opts ...Options) error, and symmetrically Unmarshal, UnmarshalRead(in io.Reader, …), and UnmarshalDecode(in *jsontext.Decoder, …) (pkg.go.dev/encoding/json/v2) — so you no longer construct an Encoder/Decoder just to read from a stream. Options become first-class variadic arguments threaded through every entry point, and they plumb through custom marshalers because v2 adds streaming-friendly interfaces — MarshalerTo with MarshalJSONTo(*jsontext.Encoder) error and UnmarshalerFrom with UnmarshalJSONFrom(*jsontext.Decoder) error. A MarshalerTo writes tokens directly to the output stream instead of allocating and returning a []byte, which is the source of v2’s biggest wins; the docs recommend it over the legacy Marshaler as “more performant and flexible,” and when a type implements both Unmarshaler and UnmarshalerFrom the latter takes precedence. There is also caller-side customization without any interface on the target type: you build a *Marshalers value with the generic constructors MarshalFunc[T](func(T) ([]byte, error)) or MarshalToFunc[T](func(*jsontext.Encoder, T) error), combine them with JoinMarshalers, and pass them via the WithMarshalers(*Marshalers) Options option (and the symmetric UnmarshalFunc/UnmarshalFromFunc/WithUnmarshalers). This lets a caller register, say, a custom encoding for *strconv.NumError purely from the call site.

On performance, the v2 post is careful and quotes precisely: “The Marshal performance of v2 is roughly at parity with v1. Sometimes it is slightly faster, but other times it is slightly slower. The Unmarshal performance of v2 is significantly faster than v1, with benchmarks demonstrating improvements of up to 10x” — with the largest gains reserved for types that implement the streaming MarshalerTo/UnmarshalerFrom interfaces (go.dev/blog/jsonv2-exp; benchmark detail at jsonbench). The Go 1.25 release notes phrase the same conclusion as “encoding performance is at parity between the implementations and decoding is substantially faster in the new one” (tip.golang.org/doc/go1.25).

Uncertain

Verify: that the “up to 10x” decode speedup and marshal “parity” generalize to your workload. Reason: the figures themselves are confirmed verbatim against two primary sources (the Go blog and the Go 1.25 release notes), but they are best-case, self-reported by the package authors, and highly workload-dependent — and the experiment is still being tuned (the encoder memory-allocation regression #75026 shows performance is not yet settled). To resolve: run jsonbench (or go test -bench) on representative payloads with and without GOEXPERIMENT=jsonv2. uncertain

Failure Modes and Common Misunderstandings

Numbers into any become float64. Decoding into interface{} turns every JSON number into a float64; large integers silently lose precision past 2^53. Decode into a concrete int64 field, or use Decoder.UseNumber().

Unexported fields vanish. encoding/json uses reflection and only touches exported (capitalized) struct fields — a lowercase field is neither marshaled nor unmarshaled, with no error.

omitempty does not omit zero structs or non-nil pointers. omitempty fires on the empty values listed earlier; a zero-valued nested struct, or a non-nil *T pointing at a zero T, is not empty and is emitted. Use omitzero (Go 1.24+) for true zero-value omission.

Unknown JSON fields are ignored by default. v1’s Unmarshal silently drops JSON members with no matching struct field. Decoder.DisallowUnknownFields() makes them an error — important for strict config parsing.

HTML escaping by default. Marshal escapes <, >, and & to < etc. so output is safe to embed in HTML. It surprises people comparing JSON byte-for-byte; disable it with Encoder.SetEscapeHTML(false).

Custom UnmarshalJSON can be slow. v1’s contract forces the value to be delimited before your method runs, effectively double-parsing. v2’s UnmarshalerFrom avoids this by handing you a *jsontext.Decoder.

json.RawMessage defers, it does not skip. A json.RawMessage field captures the raw bytes of a sub-value for later decoding — still validated as well-formed JSON, just not decoded into a typed value yet.

Alternatives and When to Choose Them

For most code, encoding/json is the right default — it is in the standard library, schema-free, and fast enough. When JSON serialization is a measured bottleneck, the options are: code-generation libraries (easyjson, ffjson) that emit per-type marshal/unmarshal code and skip reflection entirely — fastest, but add a build step and stale generated code; reflection-but-faster libraries (github.com/json-iterator/go, goccy/go-json) that are mostly drop-in and noticeably faster than v1; and encoding/json/v2 itself once it stabilizes, which closes much of that gap inside the standard library, especially for decoding and for types using the streaming interfaces. For non-JSON needs — schema evolution, smaller wire size, cross-language IDLs — Protocol Buffers, FlatBuffers, or CBOR are different tools entirely. The honest guidance: do not reach for a third-party JSON library on reputation; profile first, and if you are on a recent toolchain, benchmark GOEXPERIMENT=jsonv2 before adding a dependency.

Production Notes

The biggest real-world encoding/json lesson is that the reflection-and-cache model means the first marshal of each type is slower (it builds and caches the encoderFunc); steady-state throughput is what matters and what benchmarks should measure. High-throughput services that marshal the same handful of types per request benefit fully from the cache; services that marshal a vast variety of one-off types do not. The v2 experiment’s stated proving ground is exactly this kind of production use — the blog post cites a particular Kubernetes service whose recursive OpenAPI-spec parsing in UnmarshalJSON “significantly hurt performance,” and which improved “by orders of magnitude” after switching to the streaming UnmarshalJSONFrom method (go.dev/blog/jsonv2-exp; kubernetes/kube-openapi#315). Teams on Go 1.25+ are explicitly encouraged to run their test suites with GOEXPERIMENT=jsonv2 to surface the behavioral differences — duplicate-name rejection, nil-as-[] instead of null, case-sensitive matching — before v2 becomes the default, because those are observable wire-format changes that can break downstream consumers.

See Also

Graph View

Backlinks