False Floors: LLM Safety Routing Evaluations Break Under Distribution Shift
The paper examines how safety routing systems fail when tested under distribution shift—where test data differs significantly from training data. It shows that standard evaluations using models trained on in-sample labels create a 'false floor,' making safety routing appear ineffective. Even perfect routing offers minimal harm reduction, and evaluation must strictly use training data baselines for accurate results.