Training Dynamics of Softmax Self-Attention: Fast Global Convergence via Preconditioning
cs.LG, stat.ML
Submitted: 2026-03-02
Updated: 2026-11-11
Terminology
Sources
- Linear attention is (maybe) all you need (to understand transformer optimization)
- Neural Machine Translation by Jointly Learning to Align and Translate
- Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- GPT-4 Technical Report
- Formal Algorithms for Transformers
- Implicit Optimization Bias of Next-Token Prediction in Linear Models
- Transformers as Support Vector Machines
- SOAP: Improving and Stabilizing Shampoo using Adam
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis