Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training
cs.LG, cs.DC, cs.NA, math.NA
Submitted: 2026-09-28
Updated: 2026-09-28
Code: https://github.com/Dao-AILab/flash-attention
Sources
- Online Pseudo-average Shifting Attention(PASA) for Robust Low-precision LLM Inference: Algorithms and Numerical Analysis
- Scaling Vision Transformers to 22 Billion Parameters
- Scaling FP8 training to trillion-token LLMs
- Is Flash Attention Stable?
- The Llama 3 Herd of Models
- Deep Learning with Limited Numerical Precision
- AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights
- Hardware-Aware FP4 FlashAttention-4
- A Study of BFLOAT16 for Deep Learning Training
- Kimi K3: Open Frontier Intelligence
- Kimi K2: Open Agentic Intelligence
- Muon is Scalable for LLM Training
- Mixed Precision Training
- FP8-LM: Training FP8 Large Language Models
- Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention
- FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
- Small-scale proxies for large-scale Transformer training instabilities
- Efficient Streaming Language Models with Attention Sinks
- MXAttention: Data-Free Optimal Scaling and Pre-Normalization Quantization for MXFP4 Attention
- Revisiting BFloat16 Training
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks