MpFA: Hardware-Efficient Train-Free QK4V8 FlashAttention Kernels on Blackwell GPUs
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "MpFA: Hardware-Efficient Train-Free QK4V8 FlashAttention Kernels on Blackwell GPUs".
Jane: Here's the gist of the paper. MpFA presents MpFA, a training-free FlashAttention kernel optimized for data-center Blackwell GPUs.
Tom: On an NVIDIA B200 and across 16K–128K contexts, MpFA improves prefill throughput over state-of-the-art BF16/FP8 baselines and increases end-to-end output throughput by 2.81× over BF16 FA4 across Llama-3.1-8B and Qwen3-14B.
Jane: Hardware Constraints on Low-Bit Attention Long-context LLM inference pushes modern GPU serving stacks into an attention-bound regime, where both compute and memory are dominated by the softmax–GEMM pipeline.
Tom: The hardware characterization reveals two challenges in building training-free FP4 FlashAttention for data-center Blackwell GPUs.
Paper summary: Tom: So we're wrapping up this look at MpFA: Hardware-Efficient Train-Free QK4V8 FlashAttention Kernels on Blackwell GPUs. This paper is about a training-free attention kernel designed for data center Blackwell chips and it uses mixed precision to boost speed.
Jane: Right, so what does that actually mean for us? It means this new way of doing attention can help us run much longer conversations without waiting as long for the initial part of the answer to show up.
Lu: The authors are focused on how they designed this kernel using mixed precision, specifically NVFP4 for the Q and K parts and FP8 for the P and V parts, which they say avoids a lot of data movement headaches along that softmax path.
Meng: I gotta ask about the scale here. How much does this add to our running costs when we're actually shipping models to customers?
Tom: They show that across sixteen thousand tokens up to one hundred twenty-eight thousand tokens, you get average prefill speedups of one point three one times over the BF16 FlashAttention four version.
Jane: And they also found a two point eight one times increase in the end-to-end output throughput when comparing this new method to the old BF16 version across both Llama three point one eight billion and Qwen three fourteen billion models.
Lu: The core design principles they landed on involve mixed precision, reducing non-matmul overhead by moving compensation math away from the softmax path, and specializing prefill versus decode stages.
Meng: If I'm looking at deployment right now, what are the actual accuracy numbers? Does this speedup cost us quality?
Tom: They found that this rank one compensation method recovers sixty-two point five percent of the accuracy loss on average across five benchmark suites and two models, with only about a two point zero percent kernel overhead.
Jane: So, in simple terms, MpFA is a way to get massive speed gains on these powerful Blackwell chips for long context inference while keeping the quality fairly close to what you expect.
Lu: The authors confirm that this QK4PV8 approach provides the best balance between arithmetic throughput and those non-matmul overheads on the B200 hardware.
Tom: It's a specialized kernel, tailored specifically for Blackwell architectures, focusing on minimizing those non-matmul costs they mentioned earlier in the paper.
Jane: They addressed what they didn't do by focusing prefill and decode specialization, which means prefill handles the long query sequence while decode focuses on generating the output tokens efficiently.
Lu: The next thing to watch is how this specialized kernel performs when you try to run it on real traffic versus just these clean benchmark numbers they used.
Conclusion: Tom: So we're wrapping up this look at MpFA: Hardware-Efficient Train-Free QK4V8 FlashAttention Kernels on Blackwell GPUs. This paper is about a training-free attention kernel designed for data center Blackwell chips and it uses mixed precision to boost speed.
Jane: Right, so what does that actually mean for us? It means this new way of doing attention can help us run much longer conversations without waiting as long for the initial part of the answer to show up.
Lu: The authors are focused on how they designed this kernel using mixed precision, specifically NVFP4 for the Q and K parts and FP8 for the P and V parts, which they say avoids a lot of data movement headaches along that softmax path.
Meng: I gotta ask about the scale here. How much does that add to our running costs when we're actually shipping models to customers?
Tom: They show that across sixteen thousand tokens up to one hundred twenty-eight thousand tokens, you get average prefill speedups of one point three one times over the BF16 FlashAttention four version.
Jane: And they also found a two point eight one times increase in the end-to-end output throughput when comparing this new method to the old BF16 version across both Llama three point one eight billion and Qwen three fourteen billion models.
Lu: The core design principles they landed on involve mixed precision, reducing non-matmul overhead by moving compensation math away from the softmax path, and specializing prefill versus decode stages.
Meng: If I'm looking at deployment right now, what are the actual accuracy numbers? Does this speedup cost us quality?
Tom: They found that this rank one compensation method recovers sixty-two point five percent of the accuracy loss on average across five benchmark suites and two models, with only about a two point zero percent kernel overhead.
Jane: So, in simple terms, MpFA is a way to get massive speed gains on these powerful Blackwell chips for long context inference while keeping the quality fairly close to what you expect.
Lu: The authors confirm that this QK4PV8 approach provides the best balance between arithmetic throughput and those non-matmul overheads on the B200 hardware.
Tom: It's a specialized kernel, tailored specifically for Blackwell architectures, focusing on minimizing those non-matmul costs they mentioned earlier in the paper.
Jane: They addressed what they didn't do by focusing prefill and decode specialization, which means prefill handles the long query sequence while decode focuses on generating the output tokens efficiently.
Lu: The next thing to watch is how this specialized kernel performs when you try to run it on real traffic versus just these clean benchmark numbers they used.
Tom: So we're wrapping up this look at MpFA: Hardware-Efficient Train-Free QK4V8 FlashAttention Kernels on Blackwell GPUs. This paper is about a training-free attention kernel designed for data center Blackwell chips and it uses mixed precision to boost speed.
Jane: Right, so what does that actually mean for us? It means this new way of doing attention can help us run much longer conversations without waiting as long for the initial part of the answer to show up.
Lu: The authors are focused on how they designed this kernel using mixed precision, specifically NVFP4 for the Q and K parts and FP8 for the P and V parts, which they say avoids a lot of data movement headaches along that softmax path.
Meng: I gotta ask about the scale here. How much does that add to our running costs when we're actually shipping models to customers?
Tom: They show that across sixteen thousand tokens up to one hundred twenty-eight thousand tokens, you get average prefill speedups of one point three one times over the BF16 FlashAttention four version.
Jane: And they also found a two point eight one times increase in the end-to-end output throughput when comparing this new method to the old BF16 version across both Llama three point one eight billion and Qwen three fourteen billion models.
Lu: The core design principles they landed on involve mixed precision, reducing non-matmul overhead by moving compensation math away from the softmax path, and specializing prefill versus decode stages.
Meng: If I'm looking at deployment right now, what are the actual accuracy numbers? Does this speedup cost us quality?
Tom: They found that this rank one compensation method recovers sixty-two point five percent of the accuracy loss on average across five benchmark suites and two models, with only about a two point zero percent kernel overhead.
Jane: So, in simple terms, MpFA is a way to get massive speed gains on these powerful Blackwell chips for long context inference while keeping the quality fairly close to what you expect.
Lu: The authors confirm that this QK4PV8 approach provides the best balance between arithmetic throughput and those non-matmul overheads on the B200 hardware.
Tom: It's a specialized kernel, tailored specifically for Blackwell architectures, focusing on minimizing those non-matmul costs they mentioned earlier in the paper.
Jane: They addressed what they didn't do by focusing prefill and decode specialization, which means prefill handles the long query sequence while decode focuses on generating the output tokens efficiently.
Lu: The next thing to watch is how this specialized kernel performs when you try to run it on real traffic versus just these clean benchmark numbers they used.
College of Computer Science and Technology, National University of Defense Technology
cs.DC
Submitted: 2026-09-27
Updated: 2026-11-10
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 85/100
The gist: MpFA presents MpFA, a training-free FlashAttention kernel optimized for data-center Blackwell GPUs.
Key concepts
- FlashAttention
- A highly optimized attention mechanism used in large language models that speeds up sequence processing by efficiently managing memory access and computation, crucial for handling long sequences.
- Mixed Precision NVFP4(Q)K⊤ + FP8(P)V
- This is the specific low-bit precision scheme MpFA uses. It quantizes the Query (Q) and Key (K) matrices to NVFP4 and the Position/Value (P, V) matrices to FP8, balancing speed with accuracy.
- RankOne Tensor Core MMA Compensation
- This technique moves a performance correction term away from the softmax calculation and onto Tensor Cores. This reduces overhead on the main softmax path while maintaining accuracy recovery.
Terminology
Summary
MpFA presents MpFA, a training-free FlashAttention kernel optimized for data-center Blackwell GPUs. p. 0 On an NVIDIA B200 and across 16K–128K contexts, MpFA improves prefill throughput over state-of-the-art BF16/FP8 baselines and increases end-to-end output throughput by 2. p. 081× over BF16 FA4 across Llama-3. p. 01-8B and Qwen3-14B. p. 0
Hardware Constraints on Low-Bit Attention
Long-context LLM inference pushes modern GPU serving stacks into an attention-bound regime, where both compute and memory are dominated by the softmax–GEMM pipeline. The hardware characterization reveals two challenges in building training-free FP4 FlashAttention for data-center Blackwell GPUs. p. 0 First, when fifth-generation Tensor Cores execute block-scaled FP4 matrix multiply-add (MMA), matrix operands must be supplied from shared memory, whereas scale factors must come from tensor memory. p. 0 Second, existing smoothing compensation methods place the compensation term on the softmax path, which further loads the already saturated path. p. 1
MpFA Architecture and Precision Assignment
MpFA adopts a mixed precision NVFP4(QK⊤)+FP8(PV) and optimizes the FA4 pipeline for both inference stages: prefill uses a fine-grained asynchronous pipeline and tensor-memory column reuse to co-locate scale factors with double-buffered accumulators. To reduce smoothing overhead, MpFA introduces a rankone Tensor Core MMA compensation method that moves the correction from the saturated softmax path to otherwise underutilized Tensor Cores. p. 1 The evaluation shows that QK4PV8 provides the best balance between arithmetic throughput and non-matmul overhead on the B200, making it the hardware-efficient choice for FA4. p. 1
Performance Gains Across Workloads
Experiments on the B200 show that, across 16K–128K, MpFA achieves average prefill speedups of 1.31–1.33× over BF16 FA4 and 1. p. 0 p. 015–1.17× over FP8 FlashInfer. p. 1 Across 8K–128K, its decode kernel achieves a 2. p. 108× average speedup over split-KV-enabled BF16 FA4. p. 1 MpFA achieves 2. p. 181× the endto-end output throughput of BF16 FA4 across both models and 16K–128K, and by 1. p. 106× over FP8 FlashInfer across both models and 32K–128K contexts. p. 1
Key Design Principles
The analysis yields three principles for low-bit attention on data-center Blackwell GPUs. Principle 1 is mixed precision, which states that NVFP4(QK⊤)+FP8(PV) accelerates QK⊤ computation without quantizing P to NVFP4, avoiding the associated data-movement and scale management overheads along the softmax path. p. 4 Principle 2 is to reduce non-matmul overhead, where rank-one Tensor Core MMA compensation moves the cost from softmax to Tensor Cores, while a fine-grained pipeline improves utilization. p. 4 Principle 3 is to specialize prefill and decode, as prefill processes the complete query sequence and is primarily constrained by the softmax path. p. 4
Quantization and Compensation Details
MpFA quantizes Q and K to NVFP4 with one e4m3 scale factor per 16 head-dimension elements, and quantizes P and V to FP8. Sequence centering is introduced by subtracting a sequence-level mean for each layer and head before quantization. p. 4 The rank-one Tensor Core MMA compensation term is implemented as the outer product 1M ΔS within a score tile, moving compensation from the softmax path to otherwise idle Tensor Cores without changing online softmax. p. 5
System Integration and Results
MpFA is integrated into SGLang [32] as an attention backend, supporting paged prefill and decode, GQA, masking, and CUDA Graph capture. End-to-end latency improvements show MpFA improves TTFT and TPOT over BF16 by 1. p. 911× and 3.30× in geometric mean across both models and 16K–128K. p. 9 The results confirm that split-KV does not change the end-to-end performance ordering, as MpFA remains faster at every length. p. 9
Accuracy Recovery
Across five benchmark suites and two models, rank-one compensation recovers 62.5% of the accuracy loss on average, with about 2. p. 10% average kernel overhead. p. 1 p. 1 Table 2 compares four backends across five benchmarks and two models, showing that MpFA recovers 49%–77% on Llama-3. p. 01-8B and Qwen3-14B, trailing FP8 by at most 0. p. 8009 on knowledge and commonsense tasks. p. 7 The residual accuracy loss concentrates on RULER-QA and GSM8K, where RULER-QA on the softmax warps (prior work) on the Tensor Core (ours). p. 7
Conclusion
MpFA is a high performance QK4PV8 attention kernel for data-center Blackwell GPUs. MpFA instead adopts mixed-precision NVFP4(QK⊤)+FP8(PV) and addresses its performance challenges through a finegrained asynchronous pipeline, tensor memory reuse, and stage-specific optimizations for prefill and decode. p. 1 On the B200, this compensation recovers 62. p. 05% of the accuracy loss with only 2. p. 00% average kernel overhead. Across 16K–128K contexts, MpFA achieves average prefill speedups of 1. p. 131–1.33× over BF16 FA4 and 1.15–1.17× over FP8 FlashInfer for GQA 32:8 and 40:8, and achieves 2. p. 181× the end-to-end output throughput of BF16 FA4 across both evaluated models. p. 1
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
mixed precision implementation for attention kernels allows QK4PV8 to provide
the best balance between arithmetic throughput and non-matmul overhead on the B200.
This enables faster prefill throughput, achievingaverage prefill speedups of 1.31–1.33x over BF16 FA4
across 16K–128K contexts for models like Llama-3.1-8B and Qwen3-14B. -
rank one Tensor Core MMA compensation moves the cost from the saturated softmax path to
otherwise underutilized Tensor Cores,
reducing the cost of smoothing toabout 2.0% kernel overhead
while recovering62.5% of the accuracy loss.
-
stage specific optimizations for prefill and decode allow MpFA to utilize a
fine-grained asynchronous pipeline, tensor-memory reuse, and adaptive parallel partitioning across prefill and decode,
leading to a2.81x the endto-end output throughput of BF16 FA4
across both models. -
quantization of QK4PV8 uses a fixed scale factor sP = 448 for P, avoiding
runtime amax statistics, per-block scale generation, or calibration,
ensuring that "each unnormalized probability generated by online softmax satisfies 0 < P(t)ij <= 1." -
sequence centering mitigates stable channel outliers by subtracting a mean from Q and K before quantization to obtain centered values Qc and Kbar c, which are then quantized into blocks with recovered means used as compensation factors.
-
the kernel utilizes a
double-buffer
strategy where stage r's score region is read by its softmax warp while stage 1-r's score region is in the gap after softmax and PV have consumed it, hiding softmax latency and enabling reuse of scale factors.
Sources
- Training Verifiers to Solve Math Word Problems
- HiFA4: Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference
- Microbenchmarking NVIDIA's Blackwell Architecture: An in-depth Architectural Analysis
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends