SVR: Self-Verifying Refinement via Joint Verdict-Confidence Reinforcement Learning for Adaptive Test-Time Compute
Hongyu Chen, Liang Lin, Guangrun Wang
cs.AI, cs.CL, cs.LG
Submitted: 2026-07-30
Comments: 8 pages, 4 figures, 4 tables
License: http://creativecommons.org/publicdomain/zero/1.0/
The gist: Scaling test-time computation can improve language-model reasoning, but uniform budgets waste computation on easy inputs, while verifier-guided refinement relies on external feedback.
Terminology
Abstract
Scaling test-time computation can improve language-model reasoning, but uniform budgets waste computation on easy inputs, while verifier-guided refinement relies on external feedback. We introduce Self-Verifying Refinement (SVR), an oracle-free multi-turn reinforcement learning framework that learns to use self-verification as a compute-control policy. At each turn, the model produces a solution together with a discrete correctness verdict and a confidence score; it retains the current answer only when the verdict is Correct and confidence exceeds a threshold, and otherwise continues refinement using its own self-verification. Ground-truth correctness is used only to construct training rewards and is never exposed to the policy through refinement prompts or required at inference. SVR is trained with GRPO on fixed-horizon trajectories using rewards that promote solution correctness, calibration-aware self-verification, and stop-ready correct states; adaptive stopping is activated only at inference. On seven mathematical reasoning benchmarks with Qwen3.5-2B, SVR achieves a macro-average accuracy of 0.563 with only 2.99 inference turns on average. In the evaluated complete-system comparison, it exceeds standard GRPO, strong multi-turn baselines, and a fixed-budget oracle-guided score-feedback reference while requiring substantially fewer turns than fixed ten-turn inference. These results demonstrate that learned self-verification can serve as an effective internal control signal for answer retention and adaptive test-time compute allocation.
Sources
- Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMs
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs
- MURPHY: Feedback-Aware GRPO with Retrospective Credit Assignment for Multi-Turn Code Generation
- iGRPO: Self-Feedback-Driven LLM Reasoning
- CoRefine: Confidence-Guided Self-Refinement for Adaptive Test-Time Compute
- Language Models (Mostly) Know What They Know
- Efficient PRM Training Data Synthesis via Formal Verification
- ScRPO: From Errors to Insights
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
- Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models
- Process Supervision of Confidence Margin for Calibrated LLM Reasoning
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example
- Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling
- VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks
- Adaptive Test-Time Compute Allocation for Reasoning LLMs via Constrained Policy Optimization
- Group Sequence Policy Optimization
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems