The Judge Knows When It Knows: Calibrated Abstention for LLM-Based A/B-Test Prediction
The episode reviews a paper on using LLMs for A/B test prediction, focusing on 'calibrated abstention.' The hosts discuss how unconditional predictions are weak, but gating calls based on panel confidence reveals real signal. They conclude that the most important finding is the discipline of pre-registration and the warning against shared prior collapse in judge ensembles.