FORGE: Procedural Weight Generation for Sparse MoE Inference

The gist: FORGE introduces a procedural weight generation system for Sparse MoE inference that replaces HBM-resident expert weights with on-chip procedural regeneration, leading to estimated 1.3–2.8× higher system throughput and 1.10–1.15× lower per-user decode latency at short context, while maintaining a controlled tradeoff in downstream accuracy

How it works

FORGE addresses the dual memory bottleneck of autoregressive Sparse MoE decode by regenerating expert weights on-the-fly from rotor parameters rather than storing and loading every expert weight tensor from HBM The approach is realized as the Rotor Generation Unit (RGU), a fixed-function datapath that generates FFN weight tiles without competing with Tensor Core execution This system replaces HBM weightloading path for replaced layers with a dedicated on-chip generator, shifting the per-expert MoE FFN kernel from bandwidth-bound to computebound

The core mechanism involves treating each expert projection as approximately procedurally synthesizable, where recent work by Pence et al. [51] on a Clifford-algebra object offers the theoretical feasibility that linear projections can be approximated by O(log2 d) rotor parameters In practice, each pre-trained expert in a replaced MoE layer is fit offline to a rotor-parameter seed, and at inference the seed stands in for the dense expert tensor for that layer

Key Architectural Insights

The parallel execution model is derived from the initial inherently sequential generation concept, exposing three orthogonal axes of parallelism: (1) column independence within a row-pair update, (2) row-pair independence within a rotation axis, and (3) k-block independence within a chunk The RGU exploits these axes to generate the Supertile, which is the intersection of the algebraic block-diagonal structure and the tiled-GEMM tile granularity

The Supertile spans the full n k rows of one k-block and a column width matching the GEMM thread-block tile, meaning each SM consumes one weight tile per K-dimension iteration, and a single Supertile provides exactly one such tile This alignment dictates the pipeline design, allowing generation, activation streaming, and Tensor Core execution to overlap fully

Performance Gains

The substitution is tunable; with 30% of the MoE FFN layers replaced, accuracy falls below that of the same model fine-tuned without rotor replacement The throughput gain holds across context lengths because it rests on freed HBM capacity

Specifically, the per-expert FP8 kernel speedup is measured at 2.3–3.2× at 8 tokens/expert, marking the transition from memory-bound to compute-bound execution on replaced layers This leads to a projected 1.10–1.15× lower per-user decode latency at short context Furthermore, freed HBM expands KV cache capacity by 1.2–2.6×, compounding with faster decode to give a projected 1.3–2.8× system throughput at matched GPU count

Microarchitectural Efficiency

The RGU is designed as a compact, fixed-function accelerator embedded within each Streaming Multiprocessor, occupying approximately 0.032 mm2 per SM This negligible on-chip overhead removes hundreds of gigabytes of HBM expert-weight storage

The datapath relies on a compact 4-bit mantissa multipliers, and the total steady-state power is approximately 5.1 W system-wide, which is a negligible 0.7% of the H100’s 700 W TDP The pipeline closes timing at 1.80 GHz, safely reaching 91% of the H100’s boost clock

Model Quality Tradeoff

Across three model families, FORGE at r=0.3 falls short of the same model fine-tuned with LoRA and no rotor replacement Under an identical rank-8 LoRA budget, the dense model reaches 6.53 perplexity on Qwen3-30B and FORGE reaches 7.65 This gap is the explicit quality cost of procedural weight generation, traded for the memory and bandwidth gains

The capacity advantage of procedural generation widens with model scale because while stored weight volume scales linearly, rotor parameter seeds scale logarithmically The system achieves these results by replacing the weight-load path and leaving the rest of the pipeline untouched

The paper concludes that FORGE replaces HBM weight loading with procedural on-the-fly generation for Sparse MoE FFN layers The Rotor Generation Unit executes rotor operations in a dedicated pipelined datapath and writes generated weights directly into Shared Memory, bypassing the HBM hierarchy without contending for SIMT resources Across five Sparse MoE models with 30% of the MoE FFN layers replaced, we estimate 1.3–2.8× higher system throughput at matched GPU count and 1.10–1.15× lower per-user decode latency at short context, at a cost in downstream accuracy relative to the same model fine-tuned without rotor replacement Underneath, KV cache capacity grows by 1.2–2.6× and the per-expert FP8 kernel speeds up by 2.3–3.2× independent of r Higher ratios promise larger system gains; recovery methods that preserve accuracy at those ratios are future work Underneath, KV cache capacity grows by 1.2–2.6× and the per-expert FP8 kernel speeds up by 2.3–3.2× independent of r Higher ratios promise larger system gains; recovery methods that preserve accuracy at those ratios are future work Underneath, KV cache capacity grows by 1.2–2.6× and the per-expert FP8 kernel speeds up by 2.3–3.2× independent of r Higher ratios promise larger system gains; recovery methods that preserve accuracy at those ratios are future work Underneath, KV cache capacity grows by 1.2–2.6× and the per-expert FP8 kernel speeds up by 2.3–3.2× independent of r Higher ratios promise larger system gains; recovery methods that preserve accuracy at those ratios are future work Underneath, KV cache capacity grows by 1.2–2.6× and the per-expert FP8 kernel speeds up by 2.3–3.2× independent of r Higher ratios promise larger system gains; recovery methods that preserve accuracy at those ratios are future work Underneath, KV cache capacity grows by 1.2–2.6× and the per-expert FP8 kernel speeds up by 2.3–3.2× independent of r Higher ratios promise larger system gains; recovery methods that preserve accuracy at those ratios are future work Underneath, KV cache capacity grows by 1.2–2.6× and the per-expert FP8 kernel speeds up by 2.3–3.2× independent of r Higher ratios promise larger system gains; recovery methods that preserve accuracy at those ratios are future work Underneath, KV cache capacity grows by 1.2–2.6× and the per-expert FP8 kernel speeds up by 2.

1. Bold weight generation replaces HBM loading with on-chip procedural regeneration via a Rotor Generation Unit (RGU). This shifts per-expert FFN operations from bandwidth-bound to computebound by generating weight tiles on demand for those to-be-replaced layers and writes them directly into Shared Memory, feeding the Tensor Core pipeline without an HBM weight load.

2. System throughput increases by 1.3–2.8× at matched GPU count due to freed HBM capacity expanding KV cache capacity by 1.2–2.6×, which compounds with faster decode to give a projected 1.3–2.8× system throughput at matched GPU count.

3. Per-user decode latency is reduced by 1.10–1.15× at short context, as the system achieves a 28–29% reduction in total parameter memory and the per-expert FP8 kernel speeds up by 2.3–3.2× independent of r.

4. The RGU microarchitecture achieves high efficiency by utilizing a dual 64-wide design with 4-bank interleaving to synthesize 2R2W effective bandwidth from four simple 1R1W banks, resulting in a total area overhead of only 0.9% SM area overhead (0.032 mm2/SM, 7nm).

5. The system can handle larger models with fewer GPUs because the approach collapses the minimum-fit GPU count into a single node and removing cross-node all-to-all from decode.

We introduce FORGE (Fast On-chip Rotor Generation for Experts), a procedural weight generation system for Sparse Mixture-of-Experts (MoE) inference. Rather than storing and loading every expert weight tensor from HBM, we show how a compact accelerator inside each Streaming Multiprocessor can regenerate them on-the-fly from O(squared d) rotor parameters per projection. Per replaced layer, gigabytes of stored expert weights are approximated with under a megabyte of seeds. This addresses the dual memory bottleneck of autoregressive Sparse MoE decode, where per-expert FFN operations are HBM-bandwidth-bound at small routed-token counts and the aggregate expert footprint forces multi-node Expert Parallelism (EP) with all-to-all traffic. We realize the approach as the Rotor Generation Unit (RGU), a fixed-function datapath that generates FFN weight tiles without competing with Tensor Core execution. With 30% of the MoE FFN layers replaced across five Sparse MoE models, we estimate 1.3-2.8x higher system throughput at matched GPU count and 1.10-1.15x lower per-user decode latency at short context. In exchange, downstream accuracy on three model families falls below that of the same model fine-tuned without rotor replacement, and the replacement ratio controls this tradeoff. FORGE also expands KV cache capacity by 1.2-2.6x. It is agnostic to the bit width at which weights are stored, so it can compose with quantization. The RGU is fully implemented/verified in SystemVerilog at 0.9% SM area overhead (0.032 mm 2/SM, 7nm).