Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation

Name: Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation
Uploaded: 2026-10-22T07:33:02.046575+00:00
Duration: 629 s
Description: The research investigates how training data and model size are combined to predict LLM scaling laws. It found that training runs restricted to a single line (collinear design) create numerical instability, leading to poor extrapolation. The study proves that using diverse, non-collinear training designs is critical for robust and accurate scaling law estimations.

The research investigates how training data and model size are combined to predict LLM scaling laws. It found that training runs restricted to a single line (collinear design) create numerical instability, leading to poor extrapolation. The study proves that using diverse, non-collinear training designs is critical for robust and accurate scaling law estimations.

Open the full episode