vault

Operand Forwarding

15 min read

Operand Forwarding

Operand forwarding (also called bypassing, data forwarding, or register bypass) is the hardware trick that lets a pipelined CPU consume the result of an instruction one or two cycles after it was computed, without waiting for that result to be written back to the architectural register file. The Wikipedia article frames it cleanly: “the result of an instruction is forwarded directly to the execute stage of a subsequent instruction” (Wikipedia, Operand forwarding). In a Classic Five-Stage Pipeline the two canonical paths are the EX-to-EX bypass (the just-computed ALU result in the EX/MEM pipeline register is fed back into the ALU’s input mux next cycle) and the MEM-to-EX bypass (the older result, two cycles ahead, sitting in MEM/WB, is also fed back). The Berkeley CS61C course names them “MEM to EX and WB to EX” depending on which end of the pipeline register one names (CS61C Notes, Five-Stage Pipeline). Forwarding turns most Read-After-Write (RAW) data hazards from costly stalls into zero-cost free operations; the one case it cannot fix is the Load-Use Hazard, which always forces a one-cycle bubble. Forwarding is the single most important optimization in any in-order pipelined CPU, including the RV32IMC core built by definitely-not-esp32.

1. The Problem Forwarding Solves

A pipelined CPU runs instructions overlapped: while instruction i is in EX, instruction i+1 is in ID, and so on. If i+1 reads a register that i writes, naively reading that register in i+1’s ID stage gives the old value, because i has not yet reached its WB stage and committed its result to the register file. The CMU 18-447 lecture diagrams this as a write-after-read inversion at the pipeline-register level: the writer (i) is still in EX/MEM when the reader (i+1) reaches the register-file read port (CMU 18-447 Lecture 8).

The naive resolution is to stall i+1 for three cycles, letting i propagate through MEM and WB before i+1’s ID re-reads. Three bubbles per back-to-back dependent pair would inflate CPI catastrophically; in a typical instruction stream perhaps a third of consecutive pairs have a RAW dependence, so unforwarded the average CPI would be near 2.

Forwarding observes a simple fact: even though i’s result is not in the architectural register file yet, it is sitting in a pipeline register, available as a wire signal. The same value the register file would eventually return after WB is already physically present at the output of the ALU (after EX) or at the output of the MEM stage (after MEM). All the hardware has to do is route it. The Wikipedia article on operand forwarding states the mechanism directly: “Instead of waiting for results to be written to registers, the output of one instruction can be used by subsequent instructions before the value is committed to / stored” (Wikipedia, Hazard (computer architecture)).

The cost is modest: two multiplexers at the inputs to the ALU, plus a small comparator network to decide which input each mux should pick. The win is enormous: almost every RAW hazard collapses from three bubbles to zero.

2. The Mental Model: Where Each Result Sits, When

The single most useful mental picture of forwarding is “what is in each pipeline register at the start of each cycle.” For three consecutive cycles centered on the dependent instruction:

flowchart LR
  subgraph Cycle["Cycle N (the consumer is in EX)"]
    direction LR
    ID_REG[("Register File<br/>(old x5)")]
    EXMEM[("EX/MEM<br/>= producer-1 result<br/>(x5 = 10)")]
    MEMWB[("MEM/WB<br/>= producer-2 result<br/>(other reg)")]
    ALU{"ALU"}
    MUX_A{"Mux A"}
    MUX_B{"Mux B"}
    ID_REG -. "stale" .-> MUX_A
    EXMEM -- "EX-to-EX bypass" --> MUX_A
    MEMWB -- "MEM-to-EX bypass" --> MUX_A
    MUX_A --> ALU
    ID_REG -.-> MUX_B
    EXMEM --> MUX_B
    MEMWB --> MUX_B
    MUX_B --> ALU
  end

The two bypass paths and the multiplexers they feed. What it shows: at the start of any cycle, each ALU input is the output of a 3-to-1 mux. Mux input 0 is the value read from the architectural register file (the slow, “official” path). Input 1 is the contents of the EX/MEM pipeline register (the immediately-preceding ALU result). Input 2 is the contents of the MEM/WB pipeline register (the ALU result of the instruction before that, or, for a load, the value just read from data memory). The insight to take: the architectural register file is no longer the only source of operands; the pipeline registers are themselves operand sources, accessed by the forwarding muxes. A small select-signal generator (the forwarding unit) drives the mux selects every cycle based on which destination registers match which source registers.

3. The Forwarding-Unit Decision Logic

The forwarding unit is a small piece of combinational logic that lives in the EX stage. It looks at four pieces of state every cycle:

  • The two source-register indices of the instruction currently in ID/EX (call them rs1_id_ex and rs2_id_ex).
  • The destination-register index of the instruction in EX/MEM (call it rd_ex_mem) plus its “I am going to write a register” control bit (RegWrite_ex_mem).
  • The destination-register index of the instruction in MEM/WB (call it rd_mem_wb) plus its RegWrite_mem_wb bit.

For each source operand (operand A and operand B independently), it picks one of three sources:

  1. EX-to-EX bypass (mux input 1). If RegWrite_ex_mem is set, rd_ex_mem is not the always-zero register (x0 in RISC-V), and rd_ex_mem == rsN_id_ex, then the value the ALU produced last cycle is what this cycle’s ALU wants. Select EX/MEM. This is the highest-priority path because the EX/MEM producer is closer in time than the MEM/WB producer.
  2. MEM-to-EX bypass (mux input 2). Otherwise, if RegWrite_mem_wb is set, rd_mem_wb is not x0, and rd_mem_wb == rsN_id_ex, then the value in MEM/WB is what this cycle’s ALU wants. Select MEM/WB.
  3. Default: register file (mux input 0). Otherwise, take the value the register file produced when ID read it last cycle.

Priority matters because both producers might match the same source register: the closer producer (in EX/MEM) is younger in program order than the farther producer (in MEM/WB), and the architectural semantics demand the youngest value the consumer can see. The CMU 18-447 lecture frames the older-vs-younger ordering as the foundation for any data-hazard analysis (CMU 18-447 Lecture 8).

The full logic for one operand, in pseudo-RTL:

if (RegWrite_ex_mem and rd_ex_mem != x0 and rd_ex_mem == rs_id_ex):
    forward_select = FROM_EX_MEM
elif (RegWrite_mem_wb and rd_mem_wb != x0 and rd_mem_wb == rs_id_ex):
    forward_select = FROM_MEM_WB
else:
    forward_select = FROM_REGFILE

A symmetric block exists for the second operand. The CS61C course notes the realization in hardware: two muxes at the ALU inputs, “wires from the output of the ALU and from the MEM/WB pipelined register that connect to the A/B muxes in the EX stage” (CS61C Notes, Five-Stage Pipeline).

A subtlety: in RISC-V, writes to x0 are silently discarded (because x0 is hardwired to zero in the spec: “Register x0 is hardwired to the constant 0” (RISC-V Unprivileged ISA, src/unpriv/rv32.adoc)). The rd != x0 checks above prevent the forwarding unit from “forwarding” a fake zero in response to a NOP or other x0-destination instruction, which would mask the real producer of the operand further back in the pipeline.

4. Worked Example, Step by Step

Consider this three-instruction RISC-V sequence:

addi x5, x0, 10        # i1: x5 = 10
add  x6, x5, x5        # i2: x6 = x5 + x5 = 20
sub  x7, x6, x5        # i3: x7 = x6 - x5 = 10

i2 reads x5, which i1 writes, distance 1: classic RAW. i3 reads both x6 (from i2, distance 1) and x5 (from i1, distance 2): two RAW dependences at the same time.

Without forwarding, i2 would stall in ID for two cycles waiting for i1’s WB; i3 would stall similarly. With forwarding:

Cycle:    1    2    3    4    5    6    7
i1 addi: IF   ID   EX   MEM  WB
i2 add :      IF   ID   EX   MEM  WB
i3 sub :           IF   ID   EX   MEM  WB

At cycle 4 (i2 enters EX):

  • i2’s rs1 and rs2 are both x5. The forwarding unit looks at the instruction in EX/MEM, which is i1 (rd = x5, RegWrite = 1). Match on both source operands. Forward i1’s ALU result (the value 10) from EX/MEM into both A and B muxes. i2’s ALU computes 10 + 10 = 20. No stall.

At cycle 5 (i3 enters EX):

  • i3’s rs1 is x6, its rs2 is x5.
  • The EX/MEM register now holds i2’s result (x6 = 20).
  • The MEM/WB register now holds i1’s result (x5 = 10).
  • For operand A (rs1 = x6): EX/MEM matches (rd = x6). Forward 20 from EX/MEM into mux A.
  • For operand B (rs2 = x5): EX/MEM does not match (rd = x6 != x5). Fall through to MEM/WB, which matches (rd = x5). Forward 10 from MEM/WB into mux B.
  • ALU computes 20 - 10 = 10. No stall.

Three back-to-back dependent instructions, zero bubbles, CPI of exactly 1. This is the entire point of forwarding.

5. The Internal Register-File Bypass

A subtler third forwarding path is often hidden inside the register file itself: the WB-to-ID bypass (also called the “write-before-read” internal bypass). In cycle t, one instruction is in WB writing register xN, while another instruction is in ID reading xN. Without help, the ID read sees the stale value. The classic fix, described in the Wikipedia Classic RISC pipeline article, is to clock the register file so that “writes happen on the falling edge and reads on the rising edge,” so a write in the first half of cycle t is visible to a read in the second half of cycle t (Wikipedia, Classic RISC pipeline).

This is a form of forwarding, just one local enough that it lives inside the register-file module and is invisible to the EX-stage forwarding unit. From the perspective of the EX-stage forwarding logic, instructions that are now three cycles away from the consumer (in WB) appear to have already written the register file by the time the consumer reads it, so no explicit bypass path is needed. The distance-3 RAW case is therefore handled by the internal register-file bypass, while distances 1 and 2 are handled by the explicit EX/MEM and MEM/WB bypass paths. Distance 4 and beyond is the normal, no-hazard, register-file read.

6. What Forwarding Cannot Fix: The Load-Use Hazard

Forwarding can route any value that already exists somewhere in the pipeline. The Wikipedia Classic RISC pipeline article identifies the exception: loads. “The data read from memory is not present in the data cache until after the Memory Access stage,” so a forwarding path from MEM to EX going backward in time (from end of cycle t to start of cycle t) does not exist; it would require time travel (Wikipedia, Classic RISC pipeline). A load is in MEM in cycle t, finishes the memory access at the end of cycle t, and only then has its result available. If the very next instruction (a dependent ALU op) is in EX in cycle t (start), it needs the load’s data at the start of cycle t, which is impossible.

The minimum fix is one cycle of stall. The dependent instruction is held in ID for an extra cycle (a bubble is pushed into EX), and in the following cycle the load’s data is forwarded from MEM/WB into the dependent ALU op’s input. The Wikipedia article calls this fix a “pipeline interlock” inserting “a NOP bubble” (Wikipedia, Classic RISC pipeline). The full mechanism and the (small) detection logic that fires it lives in Load-Use Hazard.

Similarly, any multi-cycle functional unit has the same time-travel problem: an unpipelined 8-cycle integer multiplier produces its result only at the end of 8 cycles, and any consumer entering EX before then must stall until forwarding becomes possible. The forwarding network sees no special case here; the stall is the same kind of interlock that fires for the load-use hazard, just sized to the functional unit’s latency.

7. Forwarding Paths in Deeper and Wider Pipelines

The two-path forwarding network (EX/MEM and MEM/WB) is specific to the 5-stage pipeline. In a deeper or wider pipeline, the network gets richer.

Deeper pipeline. If EX is split into EX1/EX2 (a common technique to push the ALU off the critical path), then a producer in EX2 in cycle t should forward back to a consumer in EX1 in cycle t+1, and to a consumer in EX1 in cycle t+2 (since the EX2 result then sits in EX2/MEM). New bypass paths emerge between every pair of stages where a producer’s result is materialized and any later stage where a consumer’s input is consumed.

Multi-issue (superscalar). If two instructions enter EX in the same cycle, the forwarding network has to consider intra-issue dependences too: instruction A’s result, computed simultaneously in lane 1, may need to feed lane 2’s consumer in the same cycle. This is usually handled by either issuing dependent pairs serially (forcing one to wait one cycle) or by adding an extra path that forwards within the same cycle, which is timing-critical because it adds an ALU-output-to-mux-select path to the cycle’s critical path.

Out-of-order. The forwarding concept generalizes to the bypass network of an out-of-order back-end: every functional unit’s output is broadcast on the “common data bus” (a generalization of Tomasulo’s CDB), and reservation stations listening on the bus catch the operand when its tag matches. The conceptual model is the same as the 5-stage forwarding (producer-to-consumer routing without round-tripping through the architectural register file), but the implementation is far more elaborate.

For the 4-stage in-order pipeline of the ESP32-C3 (Espressif TRM; SiFive forum thread on the ESP32-C3’s SiFive E2-series core), forwarding is needed too, just with one fewer stage. The principles are unchanged; the topology is a smaller version of the same mux network.

8. Concrete Hardware Cost

The forwarding network in a 5-stage RV32I core costs, roughly:

  • Two 32-bit 3-to-1 multiplexers at the ALU inputs (about 64 LUT-equivalents on a small FPGA).
  • A small forwarding-unit combinational block: four 5-bit comparators (rs1/rs2 vs rd_ex_mem and rd_mem_wb), two AND gates incorporating the RegWrite and rd != x0 checks, and the priority encoder that picks between the two matches. Maybe a dozen LUTs total.
  • Additional wires from EX/MEM and MEM/WB outputs back into EX, plus the routing they consume.

Compared to the alternative of three bubbles per back-to-back RAW pair, the cost is trivial. Forwarding is one of the highest-payoff transistors-per-cycle optimizations in CPU design. Wikipedia’s Hazard article puts the conclusion in one sentence: with forwarding enabled, “the new value is immediately available to dependent instructions,” and the RAW penalty for non-load producers collapses to zero (Wikipedia, Hazard (computer architecture)).

9. Failure Modes and Common Misunderstandings

  • “Forwarding eliminates all data hazards.” No: load-use always needs one bubble, multi-cycle ALU ops need their full latency in bubbles, and the rare WAR/WAW hazards that appear in out-of-order designs are resolved by register renaming, not by forwarding (Wikipedia, Operand forwarding).
  • “The register file is just slow; we forward to bypass it.” The register file is not slow; it just sits in the wrong place in the timeline. ID’s read in cycle t+1 happens before WB’s write in cycle t+4. Forwarding does not replace the register file; it short-circuits the multi-cycle round-trip when the producer and consumer are close in time.
  • “Forwarding is a software optimization.” No, it is purely hardware. The compiler does not have to know that forwarding exists; correct programs run the same. (Some old ISAs exposed pipeline timing to software via load delay slots, but RISC-V deliberately does not, per the Delay slot article.)
  • “Bypassing creates a new hazard.” It can: if the bypass paths add too much combinational delay (output of ALU into mux select into ALU input), they extend the EX-stage critical path. Designers sometimes choose to not implement a particular bypass path (taking the stall) to preserve clock frequency. This is the silicon-equivalent of choosing whether to pay in cycles or in MHz.
  • “The MEM-to-EX bypass forwards the load’s data.” It forwards whatever is in MEM/WB, which for a load is the loaded value, but the bypass path itself is the same as for any other value. The load-use case is special because of when the load’s data becomes available, not how it propagates afterward.
  • “The same forwarding logic works for any pipeline depth.” No: the comparator network and mux fan-in scale with the number of in-flight instructions whose results might still match a source register. Going from 5 stages to 7 typically adds one or two new bypass paths and grows each ALU-input mux by one or two inputs.

10. See Also

Graph View

Backlinks