vault

Rune and Byte Semantics

9 min read

Rune and Byte Semantics

Go has two predeclared type aliases for working with text at different granularities: byte is an alias for uint8, an 8-bit storage unit, and rune is an alias for int32, holding a single Unicode code point (Go spec, Numeric types). A Go string is a sequence of bytes; the characters it represents are runes, encoded as UTF-8. The two are not interchangeable: indexing a string (s[i]) gives a byte, while for range over a string decodes UTF-8 and gives runes. Understanding which granularity you are at — bytes on the wire, code points in the text — is the source of most “why is len wrong” and “why is my substring garbled” bugs in Go (Go blog, Strings, bytes, runes and characters in Go).

This note covers the rune/byte type semantics and UTF-8 iteration. The runtime layout of a string — the two-word header, immutability, the cost of conversions — is the sibling note String Internals; read it for the storage story.

Why Two Aliases Exist

byte and rune are aliases, not distinct types: type byte = uint8 and type rune = int32. The = matters — a byte is a uint8 for every purpose (assignability, method sets, reflect), it is purely a name that documents intent. byte says “this uint8 is a unit of raw data / UTF-8 encoding”; rune says “this int32 is a Unicode code point” (the term rune is Go’s coinage, borrowed from the Plan 9 lineage, to avoid the overloaded word character). Using the aliases makes code self-documenting: func(b []byte) reads as “bytes”, func(r rune) as “a code point”.

A code point is an integer in the Unicode codespace, 0x0 through 0x10FFFF. That range needs 21 bits, so int32 is the smallest standard integer that fits — hence rune = int32. The value is signed so that utf8.DecodeRune can return RuneError and code can compare against -1-style sentinels without surprise, and so rune arithmetic behaves like ordinary signed integer arithmetic.

Mental Model

flowchart TD
  T["text: 'Gö'"] --> CP["code points (runes):\nU+0047 'G', U+00F6 'ö'"]
  CP -->|"UTF-8 encode"| BY["bytes: 47  c3 b6"]
  BY --> STR["string value\n(len = 3 bytes, 2 runes)"]
  STR -->|"s[i]  → byte"| IDX["index: 0x47, 0xc3, 0xb6"]
  STR -->|"for range → rune"| RNG["range: (0,'G') then (1,'ö')"]

Diagram: the two views of the same text. The insight: a string stores bytes (here 3 of them), but represents runes (here 2). s[i] reads the byte layer; for range reads the rune layer by decoding UTF-8. The byte index in for range jumps by the UTF-8 width — 0 then 1 — never by 1-per-character.

The headline: a string’s len is bytes; its character count is runes, and for any non-ASCII text the two differ.

Mechanical Walk-through

UTF-8: the encoding Go assumes

UTF-8 (RFC 3629) is a variable-width encoding mapping each code point to 1–4 bytes:

  • U+0000..U+007F (ASCII) → 1 byte, 0xxxxxxx. ASCII text is its own UTF-8.
  • U+0080..U+07FF → 2 bytes, 110xxxxx 10xxxxxx.
  • U+0800..U+FFFF → 3 bytes, 1110xxxx 10xxxxxx 10xxxxxx.
  • U+10000..U+10FFFF → 4 bytes, 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx.

The leading byte’s high bits encode the length; every continuation byte starts 10. This is self-synchronizing: from any byte you can tell whether it is a start byte or a continuation byte, so a decoder can resynchronize after corruption. Go source files are themselves UTF-8, and string literals are stored as their UTF-8 bytes (Go spec, String literals).

Go does not enforce that a string is valid UTF-8 — a string is just bytes. UTF-8 is a convention the standard library and for range assume; you can put arbitrary bytes in a string.

for range over a string decodes runes

The spec is precise (Go spec, For statements): “For strings, the ‘range’ clause iterates over the Unicode code points in the string starting at byte index 0. On successive iterations, the index value will be the index of the first byte of successive UTF-8-encoded code points in the string, and the second value, of type rune, will be the value of the corresponding code point.”

So in for i, r := range s: i is the byte offset of where the current code point starts (it jumps by 1, 2, 3, or 4), and r is the decoded rune. The compiler lowers this to a loop that calls the runtime’s UTF-8 decoder (runtime/utf8.go’s decoderune).

Invalid UTF-8 has defined behavior: “If the iteration encounters an invalid UTF-8 sequence, the second value will be 0xFFFD, the Unicode replacement character, and only one byte will be consumed” (Go spec). U+FFFD (�) is the standard “something went wrong” code point; consuming exactly one byte lets the loop resynchronize at the next byte.

Indexed access reads bytes

By contrast s[i] is a plain byte load — O(1), no decoding, type byte. A C-style for i := 0; i < len(s); i++ { c := s[i] } walks bytes, so c may be a UTF-8 continuation byte with no standalone meaning. This is correct for byte-oriented work (hashing, I/O, ASCII parsing) and wrong for character-oriented work.

Conversions

  • []byte(s) — copies the string’s UTF-8 bytes into a mutable slice (see String Internals for the copy cost). One element per byte.
  • []rune(s) — decodes the UTF-8 and produces one int32 per code point. For ASCII this quadruples the memory (4 bytes/rune); for a 4-byte emoji it shrinks 4 bytes to one rune. Indexing a []rune then gives O(1) random access by code point — the only way to do that.
  • string(b) for b []byte — wraps the bytes (copying) as a string; no validation.
  • string(r) for r rune or []rune — encodes each rune to UTF-8. string(rune(0x4e2d)) → the 3-byte UTF-8 of 中. An out-of-range or surrogate rune encodes to U+FFFD’s bytes.

string(intValue) is the same rune-encoding conversion and a classic bug: string(65) is "A", not "65" — go vet flags it; use strconv.

Code: Bytes vs Runes

package main
 
import (
    "fmt"
    "unicode/utf8"
)
 
func main() {
    s := "Gö本"                                  // G=1B, ö=2B, 本=3B  → 6 bytes, 3 runes
 
    fmt.Println(len(s))                          // 1: 6   — bytes
    fmt.Println(utf8.RuneCountInString(s))       // 2: 3   — code points
 
    for i := 0; i < len(s); i++ {                // 3: byte loop
        fmt.Printf("%d:%x ", i, s[i])            //    0:47 1:c3 2:b6 3:e6 4:9c 5:ac
    }
    fmt.Println()
 
    for i, r := range s {                        // 4: rune loop — decodes UTF-8
        fmt.Printf("%d:%c(%d) ", i, r, r)        //    0:G(71) 1:ö(246) 3:本(26412)
    }
    fmt.Println()
 
    bad := "\xff"                                // 5: a lone 0xff — invalid UTF-8
    for i, r := range bad {
        fmt.Printf("%d:%U ", i, r)               // 6: 0:U+FFFD — replacement char, 1 byte consumed
    }
    fmt.Println()
 
    r, size := utf8.DecodeRuneInString(s[3:])    // 7: decode one rune explicitly
    fmt.Printf("%c is %d bytes\n", r, size)      //    本 is 3 bytes
}

Line 1 vs 2 is the whole note in two prints: len is bytes, RuneCountInString is characters. Line 3’s byte loop visits all 6 bytes and prints continuation bytes (c3 b6 etc.) that mean nothing alone. Line 4’s range loop visits 3 runes and the index skips — 0, 1, 3 — by UTF-8 width. Line 6 shows the invalid-UTF-8 contract: U+FFFD, one byte consumed. Line 7 uses unicode/utf8 to decode a single rune and learn its byte width.

Failure Modes and Common Misunderstandings

“One rune == one user-perceived character.” False. A grapheme cluster — what a human calls a character — can be several code points: é may be U+00E9 (precomposed, 1 rune) or e + U+0301 combining acute accent (2 runes). Emoji with skin-tone modifiers or ZWJ sequences span many runes. len([]rune(s)) counts code points, not graphemes; correct grapheme segmentation needs golang.org/x/text or rivo/uniseg. See the Go blog on text normalization (Go blog, Text normalization in Go).

Reversing a string by bytes corrupts it. Reversing s byte-by-byte scrambles multi-byte code points into invalid UTF-8. Reverse []rune(s) instead — and even that breaks combining sequences and emoji (you need grapheme-aware reversal for full correctness).

Slicing a string at an arbitrary byte index can split a code point. s[:3] may end in the middle of a 3-byte rune; the resulting substring is invalid UTF-8. Slice at boundaries known from a range index or utf8.DecodeRune.

'a' is a rune constant. A rune literal in single quotes has type rune (default) — 'a' is int32(97). A double-quoted "a" is a string. Mixing them is a type error.

Comparing strings for “equality of text” ignores normalization. Two strings can render identically yet differ byte-wise (precomposed vs decomposed). == compares bytes; visual equality needs Unicode normalization (golang.org/x/text/unicode/norm).

Alternatives and When to Choose Them

Work in bytes ([]byte, s[i], range []byte(s)) for I/O, hashing, and ASCII-only parsing — it is allocation-light and fast. Work in runes (for i, r := range s, []rune(s)) when you need code-point semantics: counting characters, classifying with the unicode package (unicode.IsLetter, unicode.ToUpper), random indexed access, or correct reversal. Use unicode/utf8 (DecodeRune, RuneCountInString, Valid, RuneLen) for explicit, streaming, allocation-free UTF-8 work without materializing a []rune. For grapheme-cluster correctness, drop to golang.org/x/text — the standard library deliberately stops at code points.

Production Notes

The recurring real-world bug is treating len(s) as a character count for input validation (“max 20 characters”) — a 20-byte limit rejects perfectly valid short non-ASCII input and is exploitable for truncation mischief; validate with utf8.RuneCountInString (or grapheme count if the spec really means user-perceived characters). The second is byte-index slicing of user text producing invalid UTF-8 in logs and databases. The Go blog’s Strings, bytes, runes and characters in Go is the canonical reference and is explicit that strings carry no UTF-8 guarantee — defensive code calls utf8.ValidString on untrusted input (Go blog).

See Also

Graph View

Backlinks