A waveform is the record of every signal’s value at every instant of a simulation, written to a file while the simulation runs and read afterwards in a viewer. It is the closest thing RTL has to a debugger: you cannot single-step a circuit, you cannot set a breakpoint on a flip-flop, and you cannot print a value from inside a module Verilator optimised away — but you can dump the whole state of the design, every cycle, and then go looking. Waveform debugging is the skill of doing that efficiently, and it has one central technique that is rarely taught explicitly: find the first cycle at which reality diverges from expectation, then walk backwards through the signals that feed the wrong value until you reach a signal that is right. Everything else — formats, viewers, dump scoping, disassembly correlation — is support for that one move.
The technique has a hard boundary, and it is worth stating up front: a waveform is a terrible way to find a bug you cannot localise. A million-cycle trace of a CPU that “gives the wrong answer” contains the bug and is nonetheless useless, because you have no cycle number to start from. That is why Testbenches and RTL Verification comes first in the build ladder — a self-checking test converts “it’s broken” into “cycle 1,483,209, x12 should be 0x2a”, and that is a question a waveform can answer in minutes. Every command line, log line, file size and timing number below was measured on Verilator 5.046 2026-02-28 rev fedora-5.046, Icarus Verilog 13.0, GTKWave 3.3.127, Surfer 0.7.0, clang 22.1.8 targeting riscv32 and llvm-objdump, on an AMD Ryzen AI MAX+ 395 (32 threads), Fedora 44, with trace files written to a tmpfs.
Mental Model
Think of a waveform file as a write-ahead log of a circuit’s state. It is not a picture; the picture is drawn by the viewer. What is on disk is a header declaring every signal in the design’s hierarchy, followed by a sequence of (timestamp, signal, new value) records — and crucially, records exist only where a value changed. A signal that holds still for a million cycles costs nothing after its first entry. This single property explains almost everything about waveform performance: trace cost is proportional to the number of value changes, not to the number of cycles simulated, a claim measured below.
flowchart LR
subgraph SIM["simulation — writes"]
MODEL["compiled RTL model"]
HOOK["trace instrumentation<br/>compiled in by --trace-vcd<br/>or --trace-fst"]
DUMP["tfp.dump(ctx.time())<br/>called by the testbench"]
MODEL -->|"every traced signal"| HOOK
DUMP -->|"'take a snapshot now'"| HOOK
end
HOOK -->|"only signals that CHANGED"| FILE[("trace.vcd<br/>or trace.fst")]
subgraph VIEW["analysis — reads"]
GTK["GTKWave 3.3.127"]
SURF["Surfer 0.7.0"]
SCRIPT["a script<br/>(the file is parseable)"]
end
FILE --> GTK
FILE --> SURF
FILE --> SCRIPT
GTK -->|"cursor at t, read signals"| ANSWER["'at t=36 a_q was<br/>0x6b8b4567 and b was 33'"]
How a trace is produced and consumed. What it shows: three separable stages — instrumentation chosen at Verilate time, snapshots requested at testbench time, and analysis done afterwards by any tool that can parse the file. The insight to take: the arrow labelled “only signals that CHANGED” is the whole economics of the format, and the arrow labelled “a script” is the escape hatch people forget — the file is a documented, machine-readable artifact, so when a question is “at which timestamp did pc first equal 0x8000_0100”, the answer is grep, not scrolling.
The second half of the mental model is about time direction. A circuit is a directed graph: values flow from inputs and registers, through combinational logic, into other registers. A bug is a place where the graph produces the wrong value. Debugging runs against that flow — you start at the wrong output, look at its inputs, and ask which one is wrong. When you find an input that is correct, the fault is in the logic between it and the output; when all inputs are correct and the output is wrong, you have localised the bug to one expression. This is the same procedure as bisecting a call stack, except that the “stack” is spatial (a cone of logic) rather than temporal, and the waveform is what lets you inspect the whole cone at one instant.
What a VCD File Actually Contains
VCD stands for Value Change Dump, and it is a plain-text format defined in the Verilog language standard (IEEE 1364; it has been part of the language since the original Verilog-XL era and is carried forward into IEEE 1800 SystemVerilog). Because it is text, you can read it, grep it, and parse it in twenty lines of Python — which is exactly what produced the ASCII waveforms in this note. Here is the real beginning of counter.vcd, from the counter testbench in Testbenches and RTL Verification:
$timescale 1ps — the unit of every timestamp in the file. Verilator’s default when the source carries no `timescale directive is 1ps/1ps (Verilator argument reference), so a testbench calling timeInc(5) is stepping 5 ps.
$scope module TOP / $upscope — the hierarchy, as a nested tree. Verilator wraps the user’s top module in a synthetic TOP scope, so counter appears one level down. (--no-trace-top suppresses the wrapper’s port copies.)
$var wire 8 % count [7:0] — a declaration with five fields: type, bit width, identifier code, name, and bit range. The identifier code is the compression trick: % is a short ASCII token standing in for the full hierarchical path everywhere else in the file.
$enddefinitions — the header ends; everything after is data.
flowchart TB
subgraph HDR["header — read once, builds the signal tree"]
V["$version<br/>Generated by VerilatedVcd"]
TS["$timescale 1ps<br/>the unit of every timestamp"]
SC["$scope module TOP<br/>… nested $scope / $upscope …"]
VAR["$var wire 8 (id) count (7:0)<br/>type · width · ID CODE · name · range"]
END["$enddefinitions"]
V --> TS --> SC --> VAR --> END
end
subgraph BODY["body — value changes only, in time order"]
T0["timestamp 0<br/>clk=0 rst=1 en=0<br/>count=00000000"]
T5["timestamp 5<br/>clk=1<br/>only the clock moved"]
T150["timestamp 150<br/>clk=1<br/>count=00001101"]
T0 --> T5 --> DOTS["…"] --> T150
end
END --> T0
VAR -.->|"the one-character ID code stands in<br/>for the full path everywhere below"| T0
Anatomy of a VCD file. What it shows: a header that declares the hierarchy once and assigns each signal a short ASCII identifier code, followed by a body that is nothing but timestamps and (code, new value) pairs. The insight to take: the dashed arrow is the format’s one idea — a signal’s full hierarchical path is written once, and every subsequent mention costs one or two characters. It is also why a VCD has no random access: to know a value at time t you must replay every change up to t.
The body is #<timestamp> lines followed by value changes. Scalars are <value><code> with no space; vectors are b<binary> <code> with a space:
At #0 everything is initialised: clk=0, rst=1, en=0, count=0, and the parameter WIDTH dumps as the 32-bit constant 8. At #5 only 1" appears — the clock rose and nothing else changed, so nothing else is written. At #150 the clock rose and the count became 13 (b00001101). The count value is emitted at #150 and then not again until #160; a viewer draws a flat line between them.
Two consequences follow directly from this structure and both matter in practice.
A signal’s value at time t is the last value written at or before t. There is no random access. Any tool answering “what was pc at t=1,483,209?” must scan forward from the header, which is why viewers build an index on load and why loading a multi-gigabyte VCD is slow.
Identifier codes are shared between aliased signals, and that is a debugging clue in its own right. In the differential ALU trace, Verilator assigned:
b_s2 and b share the code (. Verilator recognised that with the bug parameter set they are the same net and did not emit the value twice. So the file itself is telling you the bug: the second pipeline stage’s b operand is the input port, not the register. Two signals that should be different and share an identifier code are, by construction, wired together.
The whole design generated only 35 $var records for 14 distinct names, precisely because every port appears once per scope it is visible in and aliases collapse onto one code.
VCD versus FST — Format, Size, and Why It Matters
FST — Fast Signal Trace — is a binary, block-structured, compressed format written by Tony Bybell for GTKWave. It is not an IEEE standard; its specification is its reference implementation, fstapi.c/fstapi.h, which is MIT-licensed and vendored directly into Verilator — on this machine it sits at /usr/share/verilator/include/gtkwave/fstapi.h, headed Copyright (c) 2009-2018 Tony Bybell with SPDX-License-Identifier: MIT. Reading that header tells you the format’s design in one enum:
enum fstBlockType { FST_BL_HDR = 0, FST_BL_VCDATA = 1, FST_BL_BLACKOUT = 2, FST_BL_GEOM = 3, FST_BL_HIER = 4, FST_BL_VCDATA_DYN_ALIAS = 5, FST_BL_HIER_LZ4 = 6, FST_BL_HIER_LZ4DUO = 7, FST_BL_VCDATA_DYN_ALIAS2 = 8, FST_BL_ZWRAPPER = 254, /* indicates that whole trace is gz wrapped */ FST_BL_SKIP = 255 /* used while block is being written */};enum fstWriterPackType { FST_WR_PT_ZLIB = 0, FST_WR_PT_FASTLZ = 1, FST_WR_PT_LZ4 = 2 };
A file is a sequence of typed blocks: a header, a separately compressed hierarchy block (LZ4 or LZ4-twice), geometry, and one or more value-change blocks that may use dynamic aliasing to collapse signals that turn out to carry identical value streams. Verilator’s own writer, in verilated_fst_c.cpp, requests a compressed hierarchy and LZ4 value packing:
Three structural consequences distinguish FST from VCD for a working engineer. First, because the hierarchy is its own block, a viewer can build its signal tree without decompressing the value data — which is why opening a large FST is fast even though the file is dense. Second, blocks are independently compressed and indexed, so a viewer can seek to a time region rather than scanning from #0. Third, the format is not human-readable, so grep is off the table; the trade you are making is human accessibility for machine efficiency.
A third format exists and is easy to confuse with these two: SAIF, enabled by --trace-saif, “Specification of this format can be found in IEEE 1801-2018 (see Annex I)” (Verilator argument reference). SAIF records switching activity — how often each signal toggled — not the value history. It is a power-estimation input, not a debugging artifact, and it will not open in a waveform viewer as a trace.
VCD
FST
SAIF
Standard
IEEE 1364 / 1800
de facto; fstapi.c is the spec
IEEE 1801-2018 Annex I
Encoding
ASCII text
Binary, block-structured
ASCII text
Compression
None
LZ4 value data, LZ4 hierarchy, optional gzip wrapper
N/A
Readable with grep?
Yes
No
Yes
Random seek
No — scan from start
Yes — indexed blocks
N/A
Verilator flag
--trace-vcd (also plain --trace)
--trace-fst
--trace-saif
Multithreaded write
Follows --threads
--trace-threads (inert from 5.048)
—
Purpose
Debugging; interchange
Debugging at scale
Power/switching estimation
The three trace formats Verilator can emit. What it shows: VCD and FST are the same information in two encodings with opposite trade-offs; SAIF is a different question entirely and belongs in a power-analysis flow, not a debug flow. The insight to take: the row that decides most real choices is “readable with grep” versus “random seek”. Small traces you want to script against stay VCD; large traces you want to open interactively become FST.
Note the deprecation state, which is easy to get wrong from older tutorials: plain --trace is now documented as deprecated in favour of --trace-vcd, --trace-fst or --trace-saif, with --trace alone still meaning VCD for compatibility. And --trace-threads is “deprecated and has no effect” as of Verilator 5.048; before that it applied only to FST and could use at most two threads (Verilator argument reference).
The Cost of Tracing — Measured
Everyone repeats that tracing is expensive and that FST is smaller and faster than VCD. The first half is true and larger than most people expect; the second half is half wrong, and the difference matters when you are choosing what your regression suite builds. Here is the measurement.
The design is a deliberately signal-rich one: 32 independent 32-bit maximal-length linear-feedback shift registers plus a 32-way XOR reduction and an accumulator — 200 $var records in the VCD header, on the order of 1,100 traceable bits. The testbench times only the simulation loop (excluding model construction and file close), runs 1,000,000 clock cycles with two eval()s and two dump()s per cycle, and is built three ways from identical sources:
Run 1 — high activity. Every LFSR shifts every cycle, so roughly a thousand bits change per clock:
no-trace 1000000 cycles 0.045 s 22.021 MHz simulated acc=0x39849a88no-trace 1000000 cycles 0.045 s 21.993 MHz simulated acc=0x39849a88no-trace 1000000 cycles 0.047 s 21.063 MHz simulated acc=0x39849a88VCD 1000000 cycles 0.734 s 1.363 MHz simulated acc=0x39849a88VCD 1000000 cycles 0.715 s 1.399 MHz simulated acc=0x39849a88VCD 1000000 cycles 0.716 s 1.398 MHz simulated acc=0x39849a88FST 1000000 cycles 5.661 s 0.177 MHz simulated acc=0x39849a88FST 1000000 cycles 5.830 s 0.172 MHz simulated acc=0x39849a88FST 1000000 cycles 6.174 s 0.162 MHz simulated acc=0x39849a88
Run 2 — low activity. Identical design and testbench, except the LFSRs are enabled one cycle in 64, which is far closer to what a real CPU’s signals do:
no-trace 1000000 cycles 0.045 s 22.300 MHz simulated acc=0xda6d4126no-trace 1000000 cycles 0.046 s 21.722 MHz simulated acc=0xda6d4126VCD 1000000 cycles 0.137 s 7.279 MHz simulated acc=0xda6d4126VCD 1000000 cycles 0.136 s 7.364 MHz simulated acc=0xda6d4126FST 1000000 cycles 0.158 s 6.319 MHz simulated acc=0xda6d4126FST 1000000 cycles 0.154 s 6.497 MHz simulated acc=0xda6d4126
Configuration
Activity
Sim rate
Slowdown vs untraced
File size (1 M cycles)
Bytes/cycle
no trace
high
22.0 MHz
1.0×
—
—
--trace (VCD)
high
1.39 MHz
15.8×
2,453,793,413 B (2.45 GB)
2,454
--trace-fst
high
0.175 MHz
126×
180,964,331 B (181 MB)
181
--trace --trace-depth 1
high
2.99 MHz
7.4×
1.2 GB (40 $var)
~1,200
no trace
low (1 in 64)
22.0 MHz
1.0×
—
—
--trace (VCD)
low
7.32 MHz
3.0×
73,849,115 B (73.8 MB)
74
--trace-fst
low
6.41 MHz
3.4×
3,507,788 B (3.5 MB)
3.5
Measured trace cost, Verilator 5.046, 1,000,000 cycles, files on tmpfs, median of three (high activity) or two (low activity) runs. What it shows: three things at once — tracing costs between 3× and 126× depending on format and activity; FST is 13.6× smaller than VCD at high activity and 21× smaller at low activity; and FST is not faster than VCD — at high activity it is 8× slower, and at low activity the two are within 15% of each other. The insight to take: trace cost tracks toggle count, not cycle count. Compare the two “no trace” rows: identical, 22 MHz, because the design’s own work is the same. Then compare the VCD rows: 15.8× versus 3.0× slowdown, and 2,454 versus 74 bytes per cycle — a 33× difference driven entirely by how many signals moved.
The same numbers as bars, because the shape is the argument. What it shows: the two “no trace” bars are identical — the design’s own work does not change — while the traced bars collapse or do not depending entirely on toggle activity. The insight to take: at high activity FST is visibly the shortest bar in the chart; at low activity it is nearly as tall as VCD. The format’s cost is not a constant you can memorise, it is a function of how much your design moves.
So the correct statement of the FST trade-off is: FST buys you an order of magnitude of disk at the price of CPU, and the price is only steep when your design toggles a lot. For a CPU core — where most of the register file, most of the CSRs and most of the bus are idle on any given cycle — you are in the low-activity regime, FST costs essentially nothing extra over VCD, and it is 21× smaller. That is the real reason to prefer it, and it is a different reason from the one usually given.
Followed up — the ranking survives a disk-backed filesystem, with a caveat
All the numbers above were written to tmpfs (/tmp, RAM-backed), so the obvious objection is that VCD’s bulk cost nothing. Re-running 200,000 cycles with the output on a btrfs volume on an encrypted NVMe (/home) instead:
=== 200k cycles, DISK-backed ===VCD 200000 cycles 0.185 s 1.082 MHz simulatedVCD 200000 cycles 0.188 s 1.066 MHz simulatedFST 200000 cycles 0.987 s 0.203 MHz simulatedFST 200000 cycles 1.022 s 0.196 MHz simulated=== same 200k on tmpfs ===VCD 200000 cycles 0.136 s 1.474 MHz simulatedFST 200000 cycles 0.971 s 0.206 MHz simulated
VCD lost 27% (1.47 → 1.07 MHz); FST was unchanged. So VCD remains roughly 5× faster to write than FST even on real storage, and the ranking does not invert. The caveat is real, though: 468 MB written in 0.185 s is 2.5 GB/s, which is page-cache throughput, not device throughput — the timed loop returns before the data reaches the disk. A run long enough to exceed available page cache (this machine has 125 GB of RAM, so several tens of gigabytes of VCD) would eventually be bound by device bandwidth, and there FST’s 13–21× size advantage would dominate. Conclusion: for runs that fit in page cache, VCD is faster to write; for runs that do not, FST wins, and FST is the format that makes such runs feasible at all.
Uncertain
Verify: the claim in Verilator that FST files are “50 to 100 times smaller and faster to write” than VCD. Reason: measured here at 13.6× smaller (high activity) and 21× smaller (low activity), and slower to write in both regimes on Verilator 5.046. The size ratio is design-dependent and could reach 50× on a design with more redundancy between signals, but “faster to write” was not reproduced. To resolve: either re-measure on a design closer to whatever produced the original figure, or amend that note’s claim to separate size from speed. #uncertain
One more measured knob, because it is the cheapest large win available. --trace-depth 1 limits tracing to the top level — “Using a small number will decrease visibility, but significantly improve simulation performance and trace file size” (Verilator argument reference). On the high-activity design it cut the $var count from 200 to 40, doubled the simulation rate (1.39 → 2.99 MHz) and halved the file (2.45 GB → 1.2 GB). Not a tenfold win here, because the surviving top-level signals include the wide arrays that dominate the toggle count — which is itself the lesson: depth-limiting helps in proportion to how much of your toggle activity lives below the cut, not in proportion to how many names it removes.
Scoping the Dump — $dumpvars and —trace-depth
The reflex on a first bring-up is to dump everything, and on a design of twenty signals that is right. On a design of twenty thousand it is a mistake for three separate reasons: the file becomes unmanageable (2.45 GB per million cycles, measured above), the simulation slows by two orders of magnitude, and — least appreciated — the viewer becomes unusable, because finding the eight signals that matter inside a hierarchy of thousands is itself the work.
There are two mechanisms for scoping, and in Verilator one of them does not work the way every Verilog tutorial says it does.
$dumpvars — and Verilator ignores its arguments. In standard Verilog, $dumpvars(levels, scope) dumps levels deep starting at scope, with $dumpvars(0) meaning “everything”. Verilator’s documented behaviour is different:
$dumpvars and $dumpports module identifier is ignored; the traced instances will always start at the top of the design. The levels argument is also ignored; use tracing_on/tracing_off pragmas instead. (Verilator language support)
That is easy to miss and easy to disbelieve, so here it is measured. The same file, containing $dumpvars(1, tb_self); and an inner submodule holding a register called secret, run under both simulators:
Icarus honoured $dumpvars(1, …) and emitted four signals from the top level only. Verilator emitted eight, descended into u_inner, and dumped secret — the very signal the level argument was there to exclude. If you are porting a testbench from Icarus and wondering why your Verilator traces are enormous, this is why.
The mechanisms that do work in Verilator are two, and they operate at different times:
“Disable waveform tracing for all future signals declared in this module, or instances below this module”
--trace-max-array <depth>
Verilate time
Arrays deeper than N (default 32)
Keeps a 4 KiB memory model out of the trace
--trace-max-width <width>
Verilate time
Signals wider than N bits (default 4096)
Same idea for wide buses
--no-trace-params
Verilate time
Parameters
Removes the constant clutter (WIDTH, BUG, …)
--no-trace-top
Verilate time
Verilator’s synthetic TOP wrapper
Removes the duplicated port copies
tfp.dump() call sites
Run time
Which timestamps get sampled
The only runtime knob; gate it on a cycle window
The last row is the one to reach for when a bug is at cycle 4,000,000. Rather than tracing four million cycles, wrap the dump call:
const long TRACE_FROM = 3'999'000, TRACE_TO = 4'001'000; auto step = [&](int dt) { ctx.timeInc(dt); dut.eval(); if (cycle >= TRACE_FROM && cycle <= TRACE_TO) tfp.dump(ctx.time()); };
Two thousand cycles of trace around a known failure is a few hundred kilobytes, opens instantly, and contains everything you need — because you already know, from the self-checking testbench, which cycle to look at. This is the concrete payoff of putting Testbenches and RTL Verification first: without a cycle number you cannot scope the dump, and without a scoped dump the trace is too big to use.
The practical policy for a small SoC: /*verilator tracing_off*/ on the memory models and anything else that is bulk state rather than logic (a 64 KiB RAM contributes half a million traced bits and tells you nothing you cannot get by printing the transaction), --no-trace-params always, and a windowed dump() for anything past a few hundred thousand cycles.
The Technique — Find the Divergence, Then Walk Backwards
This is the part that is rarely written down, because it is learned by apprenticeship. It is a loop with four steps, and the direction of travel is what makes it work.
flowchart TB
FAIL["self-checking test FAILS<br/>'cycle 1, dut_y=0x6b8b4546,<br/>golden=0x6b8b4561'"]
SCOPE["scope the dump<br/>window around that cycle<br/>· --trace-depth · tracing_off"]
OPEN["open the trace<br/>cursor at the reported time"]
DIVERGE{"is THIS the FIRST<br/>divergent cycle?"}
EARLIER["move the cursor earlier<br/>— an earlier wrong value<br/>caused this one"]
CONE["list the inputs of the<br/>wrong signal at that instant<br/>(read the RTL, not the waveform)"]
CHECK{"are all those inputs<br/>correct at that instant?"}
BACK["one of them is wrong<br/>— make IT the new target"]
FOUND["all inputs correct,<br/>output wrong<br/>➜ the bug is in the logic<br/>between them"]
FIX["form a hypothesis,<br/>state what the trace would show<br/>if it were true"]
TEST["change one thing ·<br/>re-run the self-checking test"]
FAIL --> SCOPE --> OPEN --> DIVERGE
DIVERGE -->|"no"| EARLIER --> DIVERGE
DIVERGE -->|"yes"| CONE --> CHECK
CHECK -->|"no"| BACK --> CONE
CHECK -->|"yes"| FOUND --> FIX --> TEST
TEST -->|"still fails"| DIVERGE
TEST -->|"passes"| DONE["done — and add the case<br/>to the regression suite"]
The waveform debugging loop. What it shows: two nested searches — an outer one backwards in time to the first divergence, and an inner one backwards in space through the cone of logic feeding the wrong value. The insight to take: the two no branches are the whole method, and both point backwards. Debugging forwards — “let me watch what happens next” — is the reflex and it is almost always wasted, because by the time a wrong value is visible at an output, the cause is already several cycles and several signals behind you.
Four things about this loop deserve emphasis because they are where people go wrong.
The first divergence is the only one worth looking at. A wrong value propagates: it gets registered, forwarded, used in an address, branched on. By cycle n+20 a dozen signals are wrong and none of them is the cause. Sorting failures by time and taking the earliest is not a heuristic, it is the definition of the search. This is why the per-cycle checker in Testbenches and RTL Verification reports first at cycle 1 rather than just a count.
Walk backwards through the source, not through the waveform. The waveform tells you a value; the RTL tells you which signals produced it. So the loop alternates: read the RTL to find the inputs of the wrong signal, then read the waveform to find their values at that instant. Trying to do it purely in the viewer means guessing which signal to add next, and guessing badly is how a two-minute debug becomes an hour.
Stop when the inputs are right and the output is wrong. That is the terminal condition and it localises the bug to one expression. If instead you find an input that is also wrong, that input becomes the new target and the loop repeats one level up the cone. The search is a walk up a DAG, and it terminates because the DAG is finite and rooted at registers and primary inputs.
Register boundaries are where the search crosses a cycle. When the wrong signal is a flip-flop output at cycle n, its “inputs” are the flop’s data input at cycle n−1. Forgetting to step the cursor back one clock at every register crossing is the single most common mistake in this procedure — you end up comparing a flop’s output against its own input at the same instant, which of course disagrees, and concluding the flop is broken.
A note on tooling that saves real time here: because a VCD is text, the outer search can be a one-liner. “Which is the first timestamp at which dut_y and the delayed ref_y differ” is a grep/awk question, not a scrolling question, and answering it in the shell before you open a viewer is usually faster than answering it in the viewer. The ASCII waveform renderings in this note were produced by a twenty-line Python VCD parser for exactly that reason.
A Worked Bug, Start to Finish
The design is the two-stage pipelined ALU from Testbenches and RTL Verification, compared cycle by cycle against a single-cycle combinational reference — the same structure the definitely-not-esp32 MOC plans for validating the pipelined core against The Single-Cycle Processor. A parameter plants one defect: with BUG=1, stage 2 reads the input portb instead of the registered b_q.
Step 1 — the automated test gives you a cycle number. Never a waveform first.
$ ./obj_bug1/Vdiff_bug1MISMATCH cycle 1 t= 41: dut_y=0x6b8b4546 golden=0x6b8b4561 (a=0x66334873 b=33 op=7)...FAIL: 2000 cycles, 1631 mismatches, first at cycle 1
Step 2 — open the trace near the reported time and look for the first divergence, which is earlier than the first reported one. The checker could not compare until its delay FIFO had filled, so the earliest cycle it could report is not the earliest cycle that is wrong. Both traces, rendered from the real VCDs:
ASCII waveform of the operand-skew bug, values transcribed verbatim from diff_good.vcd and diff_bug.vcd by a VCD parser. Mermaid has no timing-diagram type that renders in Obsidian, so this is the vault’s sanctioned ASCII fallback. Both builds are shown for the two signals that matter, with a DIFFERS row of ^^^^ under every timestamp where they disagree; all other rows are identical in both runs and are shown once. What it shows: the two builds are bit-identical everywhere until t=23, where the buggy y_comb becomes 6 while the correct one stays 0. The insight to take: the first divergence (t=23) is three cycles and two signals ahead of the first reported mismatch (cycle 1, t=41) — and it is on a combinational signal, not a registered output. Starting the search at the reported failure and walking backwards is what gets you here; starting at the reported failure and reading forwards gets you nothing but more wrong values.
Step 3 — walk backwards through the cone of logic. At t=23 the wrong signal is y_comb, the ALU’s combinational output. Read the RTL to find its inputs:
So y_comb depends on a_q, b_s2 and op_q. Read the waveform at t=23:
Signal
Value at t=23
Correct?
a_q
0x00000000
Yes — reset value, nothing clocked in yet
b_q
0
Yes — same
op_q
0 (= add)
Yes — same
y_comb
0x00000006
No — 0 + 0 is 0
b (the input port)
6
— but this is not supposed to be an input to stage 2
All three declared inputs are correct and the output is wrong. That is the terminal condition of the search, and it identifies the guilty expression: something in the cone is reading a signal that is not on that list.
Step 4 — the giveaway. Look at wheny_comb changed. It changed at t=23, the timestamp at which b changed from 0 to 6, while a_q, b_q and op_q all held still. A combinational output that moves when none of its supposed inputs moved is, by definition, sensitive to something else. In the correct build, y_comb does not move at t=23; it moves at t=26, the clock edge that loads a_q and b_q. That single contrast is the diagnosis.
The same signature repeats at t=33: the buggy y_comb jumps to 0x6b8b4546 when b becomes 33, while a_q still holds 0x6b8b4567. The arithmetic confirms it exactly — op_q is 1 (subtract):
correct: 0x6b8b4567 - 6 = 0x6b8b4561 <-- what ref_y shows
observed: 0x6b8b4567 - 33 = 0x6b8b4546 <-- what dut_y latched at t=36
The difference is 27, which is 33 − 6: this cycle’s b minus last cycle’s b. The DUT combined a from cycle n−1 with b from cycle n. Operand skew, localised to one expression, in four steps.
Step 5 — and the trace file itself already told you. Recall from the VCD header of the buggy build:
$var wire 32 ( b [31:0] $end
$var wire 32 ( b_s2 [31:0] $end <-- same identifier code as `b`
Verilator emitted b_s2 and b under the same VCD identifier because with BUG=1 they are the same net. Two signals that should be distinct sharing an identifier code is a structural statement that they are wired together — and in a viewer it shows up as two waveform rows that are pixel-identical for the whole run. It is worth knowing this signature, because it turns a class of connectivity bug (wrong port in an instantiation, a missing register, a wire that shadows another) into something you can spot without arithmetic.
Correlating a Waveform Against a Disassembly Listing
Once the design under test is a processor, a waveform of raw signals stops being enough. pc = 0x0000001c and instr = 0xfeb51ae3 are correct and useless; what you need to know is which instruction was executing, and that lives in a disassembly listing. Correlating the two is the single most-used skill in CPU bring-up, and the definitely-not-esp32 MOC puts both halves in Stage 0 for exactly that reason: “You now have both halves of every future debug session: a way to see the hardware, and a way to produce input for it.”
Here is the correlation done end to end, with real tools and real output.
Produce the program. A loop that a compiler cannot fold away — an array reduction, which yields a load, an add, an increment and a backward branch:
// sum.cint sum(const int *a, int n) { int acc = 0; for (int i = 0; i < n; i++) acc += a[i]; return acc;}
Fetch it in RTL. A fetch stage — a program counter, a ROM loaded with $readmemh, and a B-type immediate decoder — is enough to produce a correlatable trace long before there is a register file:
Waveform-to-disassembly correlation for one loop iteration. What it shows: the trace’s pc/instr pairs map one-for-one onto the objdump listing, and the hardware-computed branch_target (0x00000010) matches the branch destination objdump printed (<sum+0x10>). The insight to take: this table is how you verify an immediate decoder without a single directed test — the compiler already encoded bne … 0x10 as 0xfeb51ae3, and objdump already decoded it back; if your RTL’s B-type immediate reconstruction produces the same address, your bit-scrambling is right. Note also the three branch_target values on non-branch cycles (0x081c, 0x001e, 0x0024): they are garbage, because immB is computed unconditionally from whatever bits happen to be in instr. That is correct and expected — branch_taken gates it — but a trace full of plausible-looking garbage on don’t-care cycles is a standing invitation to misread a waveform.
Three practical techniques make this scale to a real core.
Print the PC and the instruction from the testbench, every retired instruction. A one-line-per-instruction log — cycle, PC, raw instruction word, and (once you have a decoder) the destination register and its new value — is the artifact you actually diff against a reference simulator’s log. spike --log-commits produces exactly this — the flag is registered in spike_main/spike.cc with the help text --log-commits Generate a log of commits info (riscv-isa-sim source) — and the sail-riscv emulator, generated from the formal specification adopted by RISC-V International, is a second oracle of the same shape; and a textual diff of two instruction logs localises a divergence far faster than any waveform.
Load the ROM contents the same way in simulation and on hardware.$readmemh is supported by Verilator and Icarus alike and takes a text file of hex words; generate that file from the same .bin that goes into the FPGA bitstream so that a bug is never “the simulator ran different code”.
Name your signals so the correlation is mechanical. A trace with pc, instr, instr_valid, retire_valid, rd_addr, rd_wdata at the top level can be read against a listing directly. A trace where the program counter is called r_stage0_addr_q and is buried four levels down cannot.
Viewers — GTKWave and Surfer
Two open-source viewers matter, and they are complementary rather than competing.
GTKWave is the incumbent: a GTK-based viewer that “reads FST, and GHW files as well as standard Verilog VCD/EVCD files” (GTKWave README). Version 3.3.127 is installed here. Its interface is dated and its learning curve is real, but it is complete, it handles enormous files, and — decisively — it has save files. A .gtkw save file records exactly which signals you added, in what order, in what radix, with which markers and zoom level, and reloads them in one command:
$ gtkwave trace.fst debug.gtkw
That matters more than any UI polish. A debugging session is not one look at a waveform; it is dozens of look-change-rebuild-look cycles, and re-adding fifteen signals from a hierarchy tree every time is the actual cost of waveform debugging. Set up the signal list once, save it, and every subsequent iteration is one command. GTKWave also carries the features you eventually need — marker arithmetic (the time between two edges), signal search, pattern search on a bus’s value, and translate filters that map a numeric value onto a symbolic name (a state encoding, an opcode).
Surfer is the newer alternative — “a waveform viewer with a focus on a snappy usable interface, and extensibility” (Surfer README), version 0.7.0 here. It is written in Rust, runs natively on Linux/macOS/Windows, and also runs in a browser at app.surfer-project.org, which makes it unusually good for sharing a trace with someone else. Its command-palette-driven interface is markedly faster to learn, and it has an editor-integration story GTKWave lacks. Its own README is candid that the browser build has worse performance and missing features than the native one, and the user documentation is described as “an extremely early version”.
GTKWave 3.3.127
Surfer 0.7.0
Formats
VCD, EVCD, FST, GHW
VCD, FST, GHW
Interface
GTK menus and trees; dated
Command palette; keyboard-first
Session persistence
.gtkw save files — mature, scriptable
--state-file, plus --command-file scripting
Very large traces
Battle-tested on multi-GB FST
Improving; less mileage
Runs in a browser
No
Yes (app.surfer-project.org)
Maturity
Decades
Young, active
The two open-source viewers. What it shows: GTKWave wins on maturity and session persistence; Surfer wins on ergonomics and shareability. The insight to take: pick on the save-file question, not the looks. Whichever tool you use, the discipline that saves time is committing your signal list to the repository next to the testbench, so that “open the waveform for this failing test” is a single command rather than five minutes of clicking.
Checked locally
Surfer’s format support was confirmed against the installed binary rather than inferred: surfer --help on 0.7.0 documents the positional argument as Waveform file in VCD, FST, or GHW format, and offers -s, --state-file <STATE_FILE> Load previously saved state file plus -c, --command-file <COMMAND_FILE> for replaying a command script after load — the last of which is Surfer’s answer to the .gtkw workflow, with the README’s own caveat that “this feature is not permanent, it will be removed once a solid scripting system is implemented”.
Uncertain #uncertain
Still unverified: Surfer’s large-file performance relative to GTKWave. Reason: no benchmark was run, and neither project publishes one. To resolve: load the 181 MB FST from the trace-cost benchmark above in both viewers and time it.
Two habits are worth more than either tool’s feature list. Group and order signals by pipeline stage, top to bottom, so a value’s progress through the design reads left-to-right, top-to-bottom like a page. And keep the RTL open beside the viewer — the waveform tells you values, the source tells you which values to look at next, and the loop in the previous section alternates between them constantly.
The Honest Limit — Waveforms Cannot Find a Bug You Cannot Localise
Everything above assumes you already know roughly when the design went wrong. Remove that assumption and the technique collapses, for a reason that is arithmetic rather than skill.
A trace is a two-dimensional space: signals across, time down. A modest SoC has a few thousand traceable bits; a ten-million-cycle boot has ten million time points. That is on the order of 10¹⁰ signal-cycle cells. Scrolling through it looking for something wrong is not a slow strategy, it is not a strategy — you would have to know what “wrong” looks like for every signal at every instant, which is precisely the knowledge a reference model encodes and a human does not have.
The trap this creates is specific and common. A test fails with “the program printed the wrong sum”. You open a waveform. You now have every fact about the run and no way to use any of them, so you start reading forwards from the beginning, and what you find is that everything looks plausible — because it is plausible for thousands of cycles before it stops being. Hours later you have learned a lot about your design and nothing about the bug.
The exit is always the same: convert the failure into a cycle number before opening the viewer. In descending order of preference:
A per-cycle differential check against a reference gives you the exact cycle and the exact signal, which is the strongest possible starting point.
A concurrent assertion on an invariant gives you the exact cycle and, better, tells you which property broke — Verilator prints [89] %Error: … Assertion failed in TOP.counter_sva.a_never_eight.
An instruction-log diff against spike --log-commits or a Sail-generated emulator gives you the first divergent instruction, which maps to a cycle via the retirement log.
Bisection on the program itself. If a 10,000-instruction program fails, run the first 5,000 and dump architectural state; compare against the reference. This is git bisect applied to time and it works when nothing else is set up.
A printf funnel. Print one line per bus transaction or per retired instruction, diff two runs (before and after a change), and the first differing line is the cycle. Cheap, ugly, and effective.
Last resort: bound the window by construction. Trace only cycles N to N+2000 and sweep N. Not clever, but a 2,000-cycle window is readable and a 10,000,000-cycle one is not.
The ordering is deliberate: the top of the list is automation you build once and reuse forever, and the bottom is manual effort you spend again on every bug. This is the same argument Testbenches and RTL Verification makes about self-checking tests, arriving from the other direction — there, the point is that a test you must look at does not scale; here, the point is that a waveform without a cycle number cannot be looked at at all.
There is a second, quieter limit worth knowing. A waveform shows you what the model did, not what the hardware will do. Verilator’s trace is a two-state record of the synthesizable subset with no timing: it cannot show you a setup violation, a metastable flop at a clock-domain crossing, a glitch on a combinational path, or the difference between a signal that settles in 2 ns and one that settles in 20. All of those are real failure modes on a Field-Programmable Gate Array and none of them is visible here — which is why Timing Closure and Fmax is a separate discipline reading a synthesis report, not a waveform.
Failure Modes and Common Misunderstandings
The trace disagrees with the printf log. Almost always a dump-timestamp collision. Verilator refuses a second dump() at a time already written and says so once per occurrence:
%Warning: previous dump at t=25, requesting t=25, dump call ignored
The value change you were recording is simply lost, and on a long run the warning scrolls past. The fix is to give every dump its own instant — see the step(2)/step(3)/step(5) clock idiom in Testbenches and RTL Verification.
A stimulus change appears at exactly the same timestamp as the clock edge, and you cannot tell which the flop saw. This is the visible symptom of skipping the settling eval() before toggling the clock, and it makes the trace genuinely ambiguous rather than merely ugly. The comparison of the good and bad counter runs in the sibling note shows the missing timestamp column directly.
A signal you wanted is not in the trace. Several distinct causes, with different fixes. Verilator may have optimised the net away or merged it with another (check the $var list — if two names share an identifier code, they are one net). It may be below --trace-depth. It may be inside a module marked /*verilator tracing_off*/. It may exceed --trace-max-array (default 32) or --trace-max-width (default 4096 bits). Or its name may begin with an underscore, which is excluded unless you pass --trace-underscore.
You ported a testbench from Icarus and the file exploded.$dumpvars’s level and scope arguments are ignored by Verilator; measured above, the same $dumpvars(1, tb_self) produced 4 $var records under Icarus and 8 — including a submodule’s internal register — under Verilator.
Reading a flop’s inputs at the wrong cycle. When a registered signal is wrong at cycle n, its cause is its data input at cycle n−1. Comparing a flop’s output against its own input at the same instant always disagrees and always misleads. Step the cursor back one clock at every register crossing.
Trusting a don’t-care value. Combinational logic computes unconditionally; a branch-target adder produces an address on every cycle, including cycles fetching an addi. The fetch trace above shows branch_target = 0x0000081c on a lw — meaningless, correct, and easy to chase for twenty minutes. Always check the qualifying valid/enable signal before believing a datapath value.
Zeros that are not really zeros. Verilator is two-state: an uninitialised flop reads 0 in the trace where a four-state simulator would show X, and where real hardware might power up as 1. A waveform full of clean zeros before reset is not evidence that reset is complete. Run the design with +verilator+rand+reset+2 and a seed (see the reset-masking measurement in Testbenches and RTL Verification) and look at the trace again.
Trace files consuming the disk. 2.45 GB per million cycles was measured above for a high-activity design in VCD. Put trace outputs on a scratch path, never in the repository, and never leave --trace in the default build target.
Concluding a design is correct because the waveform “looks right”. A waveform can only disconfirm; it cannot confirm. Looking right means you did not spot a discrepancy in the handful of signals you displayed over the handful of cycles you scrolled through — which is a much weaker statement than a passing differential test over two thousand cycles.
Production Notes
Build two binaries from one source tree. One untraced, for the regression suite; one traced, for the failing test. The 3×–126× slowdown measured above is the whole argument, and Verilator makes it trivial because --Mdir puts each build in its own directory:
Default to FST for anything that runs longer than a few thousand cycles. Not because it is faster — measured here, it is not — but because it is 13–21× smaller, opens faster in a viewer, and is the format that keeps a long run’s trace on disk at all.
Make the trace window a runtime argument.--trace-from N --trace-to M parsed by your testbench, defaulting to “everything” for short runs and to a window for long ones, means the traced binary never needs rebuilding to move the window.
Commit the viewer’s save file. A .gtkw (or Surfer state file) alongside the testbench turns “open the waveform” from a five-minute setup into one command, and it encodes real knowledge: which fifteen signals, in which order, in which radix, actually explain this design.
Give the top level a debug port even if nothing uses it. Exposing pc, instr, retire_valid, rd_addr, rd_wdata as top-level outputs costs nothing in simulation, makes them readable from C++ (not just the trace), survives --trace-depth 1, and is the interface an instruction-log diff against a reference simulator is built on. On the Field-Programmable Gate Array the synthesizer will prune whatever is genuinely unused.
Keep a golden trace of a known-good run. When a refactor breaks something, diff-ing the new instruction log against the committed one localises the change in seconds. The traces themselves are too big to commit; the instruction logs are not.
Trace deterministically. Fixed seed, fixed program, fixed reset length. A trace you cannot reproduce is a trace you cannot iterate on, and the entire debug loop above assumes you can re-run and get the same timestamps.
See Also
Testbenches and RTL Verification — the sibling note and the prerequisite: how to get the cycle number that makes a waveform usable
Verilator — the simulator that produces these traces: its compilation model, two-state semantics and language coverage