Winning by Peeking: Unenforced Budgets and Test-Set Selection Inflate Short-Budget AutoML Comparisons
This episode dissects a paper by independent researchers showing their AutoML tool Orcetra's apparent dominance over FLAML and AutoGluon was an artifact of protocol flaws: test-set peeking, unenforced budgets, and data merging. Fixing these dropped Orcetra's win rate from 59.4% to 34.3%, with no significant differences remaining.