How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Laws
The episode discusses a paper on optimal data scheduling for Large Language Models (LLMs). It identifies a negative correlation between early performance deficits and final gains. The discussion concludes that tasks requiring signal accumulation, such as math or code generation, benefit from dynamic, high-quality input during training to maximize complex reasoning capabilities.