Binary Agreement for Efficient Unsupervised Confidence Calibration of Reasoning LLMs
The method introduces Binary Agreement for Self-consistency Calibration (BASC), an unsupervised technique to calibrate confidence for reasoning LLMs without needing labeled data or expensive test-time sampling. It uses repeated sampling on unlabeled data to create a lightweight predictor that estimates self-consistency as a proxy for correctness, achieving strong calibration across multiple models and tasks.