Jailbreaks for Black-Box Uncertainty Quantification in Large Reasoning Models
cs.AI, cs.CL, cs.LG
Submitted: 2026-09-28
Updated: 2026-09-28
Code: https://github.com/eth-sri/SynthPAI
License: http://creativecommons.org/licenses/by/4.0/
The gist: While Large Reasoning Models (LRMs) excel at complex reasoning, alignment through reinforcement learning often induces systemic overconfidence.
Terminology
Abstract
While Large Reasoning Models (LRMs) excel at complex reasoning, alignment through reinforcement learning often induces systemic overconfidence. In production environments, where logits may be unavailable, robust black-box uncertainty quantification (UQ) is essential for trustworthiness and safety. Focusing on question-answering for LRMs, we show that existing black-box methods, such as paraphrase-based self-consistency and confidence verbalization, offer little to no improvement over simple repeated sampling, suggesting that alignment suppresses useful output variability. We introduce prompt-level relaxation operators that broaden the model's effective output distribution by approximating the effect of an optimal policy obtained with a stronger KL-regularization parameter, hence closer to the reference model. Theoretically, we demonstrate that relaxation improves calibration. We propose Jailbreak for Uncertainty (J4U), a jailbreak-derived technique for UQ that empirically reproduces the behavioral signatures predicted by our relaxation theory. Across 3 datasets and 4 LRMs, including a closed-source production model, J4U's improvement over repeated sampling achieves statistical significance in up to 6 times more LRM-dataset-metric settings than the strongest black-box UQ state-of-the-art baseline we evaluate, with average ECE reductions up to 5 times larger. These results provide a practical tool for UQ in black-box LRM deployment.
Sources
- Training Verifiers to Solve Math Word Problems
- LLMs are Overconfident: Evaluating Confidence Interval Calibration with FermiEval
- Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Setting
- On the Robustness of Verbal Confidence of LLMs in Adversarial Attacks
- Calibration Attacks: A Comprehensive Study of Adversarial Attacks on Model Confidence
- GPT-4 Technical Report
- gpt-oss-120b & gpt-oss-20b Model Card
- Large Language Model Confidence Estimation via Black-Box Access
- Humanity's Last Exam
- Qwen3 Technical Report
- Proximal Policy Optimization Algorithms
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- The 2025 Foundation Model Transparency Index
- Just rephrase it! Uncertainty estimation in closed-source language models via multiple rephrased queries
- On Verbalized Confidence Scores for LLMs
- Uncertainty is Fragile: Manipulating Uncertainty in Large Language Models
- Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
- SeSE: Black-Box Uncertainty Quantification for Large Language Models Based on Structural Information Theory
- Group Sequence Policy Optimization
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems