MpFA: Hardware-Efficient Train-Free QK4V8 FlashAttention Kernels on Blackwell GPUs
summary
The gist
MpFA presents MpFA, a training-free FlashAttention kernel optimized for data-center Blackwell GPUs.
In short
MpFA introduces MpFA, a training-free FlashAttention kernel for Blackwell GPUs using mixed precision NVFP4(Q)K⊤ + FP8(P)V. It optimizes prefill and decode stages with asynchronous pipelines and rank-one Tensor Core compensation to boost throughput by up to 2.81x over BF16 baselines across long contexts.
Key concepts
- FlashAttention
- A highly optimized attention mechanism used in large language models that speeds up sequence processing by efficiently managing memory access and computation, crucial for handling long sequences.
- Mixed Precision NVFP4(Q)K⊤ + FP8(P)V
- This is the specific low-bit precision scheme MpFA uses. It quantizes the Query (Q) and Key (K) matrices to NVFP4 and the Position/Value (P, V) matrices to FP8, balancing speed with accuracy.
- RankOne Tensor Core MMA Compensation
- This technique moves a performance correction term away from the softmax calculation and onto Tensor Cores. This reduces overhead on the main softmax path while maintaining accuracy recovery.
Terminology used across episodes
This episode discusses
- MpFA: Hardware-Efficient Train-Free QK4V8 FlashAttention Kernels on Blackwell GPUs · Paper Radio
- Training Verifiers to Solve Math Word Problems
- HiFA4: Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference
- Microbenchmarking NVIDIA's Blackwell Architecture: An in-depth Architectural Analysis
The paper
MpFA: Hardware-Efficient Train-Free QK4V8 FlashAttention Kernels on Blackwell GPUs · Read on arXiv
College of Computer Science and Technology, National University of Defense Technology
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "MpFA: Hardware-Efficient Train-Free QK4V8 FlashAttention Kernels on Blackwell GPUs".
Jane: Here's the gist of the paper. MpFA presents MpFA, a training-free FlashAttention kernel optimized for data-center Blackwell GPUs.
Tom: On an NVIDIA B200 and across 16K–128K contexts, MpFA improves prefill throughput over state-of-the-art BF16/FP8 baselines and increases end-to-end output throughput by 2.81× over BF16 FA4 across Llama-3.1-8B and Qwen3-14B.
Jane: Hardware Constraints on Low-Bit Attention Long-context LLM inference pushes modern GPU serving stacks into an attention-bound regime, where both compute and memory are dominated by the softmax–GEMM pipeline.
Tom: The hardware characterization reveals two challenges in building training-free FP4 FlashAttention for data-center Blackwell GPUs.
Paper summary: Tom: So we're wrapping up this look at MpFA: Hardware-Efficient Train-Free QK4V8 FlashAttention Kernels on Blackwell GPUs. This paper is about a training-free attention kernel designed for data center Blackwell chips and it uses mixed precision to boost speed.
Jane: Right, so what does that actually mean for us? It means this new way of doing attention can help us run much longer conversations without waiting as long for the initial part of the answer to show up.
Lu: The authors are focused on how they designed this kernel using mixed precision, specifically NVFP4 for the Q and K parts and FP8 for the P and V parts, which they say avoids a lot of data movement headaches along that softmax path.
Meng: I gotta ask about the scale here. How much does this add to our running costs when we're actually shipping models to customers?
Tom: They show that across sixteen thousand tokens up to one hundred twenty-eight thousand tokens, you get average prefill speedups of one point three one times over the BF16 FlashAttention four version.
Jane: And they also found a two point eight one times increase in the end-to-end output throughput when comparing this new method to the old BF16 version across both Llama three point one eight billion and Qwen three fourteen billion models.
Lu: The core design principles they landed on involve mixed precision, reducing non-matmul overhead by moving compensation math away from the softmax path, and specializing prefill versus decode stages.
Meng: If I'm looking at deployment right now, what are the actual accuracy numbers? Does this speedup cost us quality?
Tom: They found that this rank one compensation method recovers sixty-two point five percent of the accuracy loss on average across five benchmark suites and two models, with only about a two point zero percent kernel overhead.
Jane: So, in simple terms, MpFA is a way to get massive speed gains on these powerful Blackwell chips for long context inference while keeping the quality fairly close to what you expect.
Lu: The authors confirm that this QK4PV8 approach provides the best balance between arithmetic throughput and those non-matmul overheads on the B200 hardware.
Tom: It's a specialized kernel, tailored specifically for Blackwell architectures, focusing on minimizing those non-matmul costs they mentioned earlier in the paper.
Jane: They addressed what they didn't do by focusing prefill and decode specialization, which means prefill handles the long query sequence while decode focuses on generating the output tokens efficiently.
Lu: The next thing to watch is how this specialized kernel performs when you try to run it on real traffic versus just these clean benchmark numbers they used.
Conclusion: Tom: So we're wrapping up this look at MpFA: Hardware-Efficient Train-Free QK4V8 FlashAttention Kernels on Blackwell GPUs. This paper is about a training-free attention kernel designed for data center Blackwell chips and it uses mixed precision to boost speed.
Jane: Right, so what does that actually mean for us? It means this new way of doing attention can help us run much longer conversations without waiting as long for the initial part of the answer to show up.
Lu: The authors are focused on how they designed this kernel using mixed precision, specifically NVFP4 for the Q and K parts and FP8 for the P and V parts, which they say avoids a lot of data movement headaches along that softmax path.
Meng: I gotta ask about the scale here. How much does that add to our running costs when we're actually shipping models to customers?
Tom: They show that across sixteen thousand tokens up to one hundred twenty-eight thousand tokens, you get average prefill speedups of one point three one times over the BF16 FlashAttention four version.
Jane: And they also found a two point eight one times increase in the end-to-end output throughput when comparing this new method to the old BF16 version across both Llama three point one eight billion and Qwen three fourteen billion models.
Lu: The core design principles they landed on involve mixed precision, reducing non-matmul overhead by moving compensation math away from the softmax path, and specializing prefill versus decode stages.
Meng: If I'm looking at deployment right now, what are the actual accuracy numbers? Does this speedup cost us quality?
Tom: They found that this rank one compensation method recovers sixty-two point five percent of the accuracy loss on average across five benchmark suites and two models, with only about a two point zero percent kernel overhead.
Jane: So, in simple terms, MpFA is a way to get massive speed gains on these powerful Blackwell chips for long context inference while keeping the quality fairly close to what you expect.
Lu: The authors confirm that this QK4PV8 approach provides the best balance between arithmetic throughput and those non-matmul overheads on the B200 hardware.
Tom: It's a specialized kernel, tailored specifically for Blackwell architectures, focusing on minimizing those non-matmul costs they mentioned earlier in the paper.
Jane: They addressed what they didn't do by focusing prefill and decode specialization, which means prefill handles the long query sequence while decode focuses on generating the output tokens efficiently.
Lu: The next thing to watch is how this specialized kernel performs when you try to run it on real traffic versus just these clean benchmark numbers they used.
Tom: So we're wrapping up this look at MpFA: Hardware-Efficient Train-Free QK4V8 FlashAttention Kernels on Blackwell GPUs. This paper is about a training-free attention kernel designed for data center Blackwell chips and it uses mixed precision to boost speed.
Jane: Right, so what does that actually mean for us? It means this new way of doing attention can help us run much longer conversations without waiting as long for the initial part of the answer to show up.
Lu: The authors are focused on how they designed this kernel using mixed precision, specifically NVFP4 for the Q and K parts and FP8 for the P and V parts, which they say avoids a lot of data movement headaches along that softmax path.
Meng: I gotta ask about the scale here. How much does that add to our running costs when we're actually shipping models to customers?
Tom: They show that across sixteen thousand tokens up to one hundred twenty-eight thousand tokens, you get average prefill speedups of one point three one times over the BF16 FlashAttention four version.
Jane: And they also found a two point eight one times increase in the end-to-end output throughput when comparing this new method to the old BF16 version across both Llama three point one eight billion and Qwen three fourteen billion models.
Lu: The core design principles they landed on involve mixed precision, reducing non-matmul overhead by moving compensation math away from the softmax path, and specializing prefill versus decode stages.
Meng: If I'm looking at deployment right now, what are the actual accuracy numbers? Does this speedup cost us quality?
Tom: They found that this rank one compensation method recovers sixty-two point five percent of the accuracy loss on average across five benchmark suites and two models, with only about a two point zero percent kernel overhead.
Jane: So, in simple terms, MpFA is a way to get massive speed gains on these powerful Blackwell chips for long context inference while keeping the quality fairly close to what you expect.
Lu: The authors confirm that this QK4PV8 approach provides the best balance between arithmetic throughput and those non-matmul overheads on the B200 hardware.
Tom: It's a specialized kernel, tailored specifically for Blackwell architectures, focusing on minimizing those non-matmul costs they mentioned earlier in the paper.
Jane: They addressed what they didn't do by focusing prefill and decode specialization, which means prefill handles the long query sequence while decode focuses on generating the output tokens efficiently.
Lu: The next thing to watch is how this specialized kernel performs when you try to run it on real traffic versus just these clean benchmark numbers they used.
More episodes
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines
- 2601.15318-On the closest balanced game