Repetition Mismatch: Why Data Mixture Experiments Don't Scale and How to Fix Them
The episode discusses the paper "Repetition Mismatch," which identifies how small-scale AI experiments fail to scale because they over-rely on limited data, causing high repetition that distorts learning The hosts explore a solution—a repetition-aware subsampling technique—that achieves state-of-the-art results with significantly less computational cost, suggesting a shift away from brute force scaling.