Skaling: Chinchilla's Exponents Meet Kaplan's Coupling
Mathurin Videau, Badr Youbi-Idrissi, David Lopez-Paz, Kartik Ahuja
cs.CL
Submitted: 2026-08-07
Updated: 2026-08-10
Code: https://github.com/facebookresearch/lingua
License: http://creativecommons.org/licenses/by/4.0/
The gist: Neural scaling laws are foundational for language model development, yet standard formulations systematically under- and overestimate loss at data-scarce and overtraining extremes.
Terminology
Abstract
Neural scaling laws are foundational for language model development, yet standard formulations systematically under- and overestimate loss at data-scarce and overtraining extremes. This failure originates in the underlying assumption that model size and training data impact the loss independently. To address this, we introduce the Skaling law, a generalized functional form that couples model capacity and data through a single interaction exponent. This simple extension reduces the Mean Absolute Percentage Error (MAPE) by 1.5-3x across both interpolation and extrapolation regimes. When paired with a sparse grid strategy restricted to low-compute regimes, the Skaling law achieves accurate full-grid extrapolation using approximately 10x less compute than uniform sweeps. By enabling reliable performance prediction from small-scale experiments, the Skaling law provides a more robust and resource-efficient framework for allocating compute budgets in next-generation model training.
Sources
- Chinchilla Scaling: A replication attempt
- DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
- Distillation Scaling Laws
- The CMA Evolution Strategy: A Tutorial
- Deep Learning Scaling is Predictable, Empirically
- Training Compute-Optimal Large Language Models
- Scaling Laws for Neural Language Models
- Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models
- Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining
- Reconciling Kaplan and Chinchilla Scaling Laws
- Resolving Discrepancies in Compute-Optimal Scaling of Language Models
- A Constructive Prediction of the Generalization Error Across Scales
- Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering