Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination
The episode critiques benchmark contamination mitigation, arguing the standard metric G-AP is flawed because it averages per-question gaps, allowing over- and under-suppression to cancel. The hosts discuss the new SA-PPG metric, which stratifies by clean-model solve probability, and RailCap, a decoding-time strategy that caps greedy tokens. Results show prior methods overestimated restoration.