Benchmarks Saturate When The Model Gets Smarter Than The Judge
The study investigated why model performance stops improving when models become very smart. It found that saturation isn't just about model skill; it depends critically on the triplet of dataset, model, and judge. Poor evaluation pipelines and unreliable judges cause errors that limit benchmark accuracy, suggesting benchmarks must be treated as these complete triplets.