When is Warmstarting Effective for Scaling Language Models?
cs.LG
Submitted: 2026-05-13
Updated: 2026-11-17
Code: https://github.com/Neeratyoy/warmstarting_exps
Terminology
Sources
- Scaling with Collapse: Efficient and Predictable Training of LLM Families
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Sparse maximal update parameterization: A holistic approach to sparse training dynamics
- Don't be lazy: CompleteP enables compute-efficient deep transformers
- Maintaining Plasticity in Deep Continual Learning
- Practical Efficiency of Muon for Pretraining
- Tune As You Scale: Hyperparameter Optimization For Compute Efficient Training
- Optimal Scaling Needs Optimal Norm
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- Scaling Laws for Neural Language Models
- Landscape-Aware Growing: The Power of a Little LAG
- Reusing Overtrained Language Models Saturates Scaling
- $\mu$pscaling small models: Principled warm starts and hyperparameter transfer
- GPT-4 Technical Report
- Knowledge Inheritance for Pre-trained Language Models
- Scaling Language Models: Methods, Analysis & Insights from Training Gopher
- Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization
- RoFormer: Enhanced Transformer with Rotary Position Embedding
- $\mu$LO: Compute-Efficient Meta-Generalization of Learned Optimizers
- LLaMA: Open and Efficient Foundation Language Models
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis