Visual sensitivity is not claim retractability: persistence-aware credit assignment for multimodal reinforcement learning
cs.AI
Submitted: 2026-09-29
Updated: 2026-10-11
Comments: MLLM,RL
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Reinforcement Learning with Verifiable Rewards (RLVR) has been extended to Large Vision-Language Models (LVLMs), and perception-aware methods further encourage policies to rely on visual evidence.
Terminology
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) has been extended to Large Vision-Language Models (LVLMs), and perception-aware methods further encourage policies to rely on visual evidence. Yet relying on the image does not guarantee that visual claims are supported by it. Before RL training, 27.81% of the correctly answered responses of Qwen2.5-VL-7B on four multimodal reasoning benchmarks contain at least one direct visual claim that the image does not support. Since outcome-level RL rewards each response as a whole, these claims inherit the positive credit of the correct answer. We introduce a fixed-rollout counterfactual diagnostic that re-scores the same response under an intervened image to separate Evidence-Function Sensitivity (EFS), how strongly the model's predictions change, from claim persistence, whether the model keeps supporting the same claim rather than retracting it. The diagnostic reveals Sensitivity-Persistence Decoupling (SPD): under DAPO and VPPO, EFS increases and claims become more retractable overall, yet unsupported claims become significantly more persistent, whereas GRPO raises EFS without this deterioration. We therefore propose Persistence-Aware Credit Gating (PACG), which attenuates positive credit for unusually persistent visual claims and leaves all other credit unchanged. It requires no supported/unsupported labels and adds no inference cost. On Qwen2.5-VL-7B, PACG raises the nine-benchmark average over three seeds from 58.1% to 59.9% with DAPO and from 59.8% to 60.9% with VPPO, while making unsupported claims more retractable. The gains extend to a larger model, a newer backbone, and the accuracy of HallusionBench also improves consistently. These results suggest that visual sensitivity and claim retractability are complementary dimensions of multimodal credit assignment.
Sources
- Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learning
- Explanation-Driven Counterfactual Testing for Faithfulness in Vision-Language Model Explanations
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Evidence-RL: Towards Evidence-intensive Visual Reasoning
- SketchVL: Policy Optimization via Fine-Grained Credit Assignment for Chart Understanding and More
- OpenAI o1 System Card
- Structured Role-Aware Policy Optimization for Multimodal Reasoning
- Credit Where It is Due: Cross-Modality Connectivity Drives Precise Reinforcement Learning for MLLM Reasoning
- PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping
- More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models
- NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation
- Bridging Perception and Reasoning: Token Reweighting for RLVR in Multimodal LLMs
- Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning
- MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learning
- LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RL
- We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Kimi K3: Open Frontier Intelligence
- Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
- VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems