Adaptive auditing of AI systems with anytime-valid guarantees
The authors introduce a hypothesis testing framework using Safe Anytime-Valid Inference (SAVI) to rigorously test generative AI failure modes during adaptive auditing. They formalize this as a duel between the model's null and the auditor's null, allowing for statistically sound conclusions even with limited data and flexible sampling strategies. This method guarantees Type-I error control while outperforming traditional testing methods.