On the Origin of Algorithmic Progress in AI: Evidence from Language Model Pre-Training
cs.LG, cs.AI
Submitted: 2025-11-26
Updated: 2026-11-10
Code: https://github.com/stanford-cs336/spring2025-lectures
Project page: https://ai-2027.com
Terminology
Sources
- Linear attention is (maybe) all you need (to understand transformer optimization)
- Compute Requirements for Algorithmic Innovation in Frontier AI Models
- Chinchilla Scaling: A replication attempt
- DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
- Bridging the Gap for Tokenizer-Free Language Models
- AI capabilities can be significantly improved without expensive retraining
- Don't be lazy: CompleteP enables compute-efficient deep transformers
- Scaling Laws for Acoustic Models
- Algorithmic progress in computer vision
- GATE: An Integrated Assessment Model for AI Automation
- Mixture of A Million Experts
- Measuring the Algorithmic Efficiency of Neural Networks
- Deep Learning Scaling is Predictable, Empirically
- Training Compute-Optimal Large Language Models
- Scaling Laws for Neural Language Models
- Improving Generalization Performance by Switching from Adam to SGD
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
- Muon is Scalable for LLM Training
- KAN: Kolmogorov-Arnold Networks
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis