Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data
The episode discusses a paper detailing the limitations of using large language models (LLM as Judge) to evaluate advanced AI models. The hosts conclude that when an LLM judge cannot beat the model it is judging, there is a fundamental mathematical limit on improvement, capping sample efficiency at a factor of two.