ROC-n-reroll: How verifier imperfection affects test-time scaling
The episode analyzes the paper "ROC-n-reroll," which mathematically frames how imperfect verifiers limit test-time scaling. The discussion focuses on two methods: Best-of-N and Rejection Sampling, concluding that Rejection Sampling is more compute-efficient than Best-of-N. A critical warning is issued that performance cannot be reliably extrapolated from small sample tests.