The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training
cs.LG, cs.AI, cs.CL, math.OC, stat.ML
Submitted: 2025-02-26
Updated: 2025-06-13
Code: https://github.com/Wongboo/BlockwiseLearningRate
Project page: http://skylion007.github.io/OpenWebTextCorpus
Terminology
Sources
- GPT-4 Technical Report
- Adaptive Gradient Methods at the Edge of Stability
- Understanding Optimization in Deep Learning with Central Flows
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Scaling Exponents Across Parameterizations and Optimizers
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- Training Compute-Optimal Large Language Models
- MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
- The MiniPile Challenge for Data-Efficient Language Models
- Adam: A Method for Stochastic Optimization
- Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models
- DeepSeek-V3 Technical Report
- RoBERTa: A Robustly Optimized BERT Pretraining Approach
- Decoupled Weight Decay Regularization
- What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis
- Training Tips for the Transformer Model
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Does SGD really happen in tiny subspaces?
Related papers
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis
- (Mis)generalization of Helpful-Only Fine-tuning