REIN: Bridging the Gap between Reasoning and Reliability via Reflection and Abstention Alignment
Zhengze Huang, Luyang Yu, Di Hong, Xinzhe Huang, Wanyu Lin, Zhixuan Chu, Zhan Qin, Tianhang Zheng
cs.AI
Submitted: 2026-08-08
Updated: 2026-08-11
Comments: 26 pages and 22 figures
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large reasoning models (LRMs) are prone to hallucination, which undermines their reliability and poses challenges for safe deployment.
Terminology
Abstract
Large reasoning models (LRMs) are prone to hallucination, which undermines their reliability and poses challenges for safe deployment. Hallucinations in LRMs arise from two distinct failure sources: reasoning hallucination, where flawed inference steps propagate to an incorrect conclusion, and knowledge hallucination, where the model lacks the requisite factual knowledge to answer the query. To address reasoning hallucination, we propose REIN, an alignment framework that trains LRMs to produce a structured reasoning sequence,<think> to<reflection> to<answer>, enabling explicit self-reflection before committing to a final answer. To address knowledge hallucination, REIN introduces a reward mechanism that encourages explicit abstention (e.g., "I don't know") when none of the sampled reasoning chains yields a correct answer, allowing the model to refrain from unsupported predictions. Extensive evaluations on mathematical and commonsense reasoning benchmarks show that REIN consistently improves selective accuracy, reduces incorrect-but-self-endorsed responses, and maintains high coverage compared with competitive baselines. Notably, REIN achieves these gains within a single forward pass, without requiring process supervision, inference-time controllers, external search, or multi-round critiques. Experiments on multiple backbones show that REIN reduces the hallucination proxy by 58 about72% relative to the base models while maintaining 86 about91% average coverage, and improves selective accuracy on attempted questions by 6.6 about14.2%.
Sources
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Qwen3 Technical Report
- Scaling Physical Reasoning with the PHYSICS Dataset
- R1-Code-Interpreter: LLMs Reason with Code via Supervised and Multi-stage Reinforcement Learning
- FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
- Empowering LLMs with Logical Reasoning: A Comprehensive Survey
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Solving math word problems with process- and outcome-based feedback
- Self-Refine: Iterative Refinement with Self-Feedback
- Reflexion: Language Agents with Verbal Reinforcement Learning
- Large Language Models Cannot Self-Correct Reasoning Yet
- Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback
- Safe RLHF: Safe Reinforcement Learning from Human Feedback
- Scaling Relationship on Learning Mathematical Reasoning with Large Language Models
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- DualBreach: Efficient Dual-Jailbreaking via Target-Driven Initialization and Multi-Target Optimization
- TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning
- KnowRL: Exploring Knowledgeable Reinforcement Learning for Factuality
- DataShield: Uncovering Risky Fine-Tuning Data Across LLMs Through Consensus Subspace Alignment
- BAPO: Boundary-Aware Policy Optimization for Reliable Agentic Search
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems