X-Stage: Modeling Post-Issue Backpressure in GPU Communication--Computation Fusion
The episode discusses the paper "X-Stage," which identifies a hidden delay period in GPU communication called the X-Stage. By using a "Burst-Gap" model to account for this post-issue progress, researchers achieved significant speedups in MegaMoE and FlashAttention by interleaving computation and communication more efficiently.