Smooth Scaling Laws Hide Stepwise Token Learning
cs.CL
Submitted: 2026-06-29
Updated: 2026-11-07
Code: https://github.com/applewpj/token-learning-spectrum
Terminology
Sources
- Scaling Laws for Neural Language Models
- Learning Curve Theory
- A Solvable Model of Neural Scaling Laws
- Neural Scaling Laws Rooted in the Data Distribution
- Efficient Online Data Mixing For Language Model Pre-Training
- Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules
- Physics of Language Models: Part 1, Learning Hierarchical Language Structures
- On the origin of neural scaling laws: from random graphs to natural language
- Emergent Abilities of Large Language Models
- Deep Learning Scaling is Predictable, Empirically
- DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
- Deriving Neural Scaling Laws from the statistics of natural language
- Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data