Incremental Learning of Sparse Attention Patterns in Transformers
cs.LG, math.OC, stat.ML
Submitted: 2026-02-22
Updated: 2026-11-10
Code: https://github.com/tml-epfl/sparse-attention-dynamics
Terminology
Sources
- Multi-Head Attention Is a Multi-Player Game
- Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality
- Training Dynamics of Softmax Self-Attention: Fast Global Convergence via Preconditioning
- Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsity
- In-context Learning and Induction Heads
- Approximating How Single Head Attention Learns
- Parameter Symmetry Potentially Unifies Deep Learning Theory
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis