Improving Generalization Robustness of Multimodal RLVR
Pengfei Zhou, Zhiwei Tang, Xiaopeng Peng, Chenrui Zhou, Lama Moukheiber, Yixing Ma, Bin Xu, Jiajun Song, Zhenglin Wan, Wangbo Zhao, Jiasheng Tang, Bohan Zhuang, Fan Wang, Yang You
cs.AI
Submitted: 2026-08-14
Updated: 2026-08-17
Comments: 32 pages, 5 figures
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reinforcement Learning with Verifiable Rewards (RLVR) makes Multimodal Large Language Models more accurate, but the gains are brittle: simply paraphrasing a question or changing the prompt template
Terminology
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) makes Multimodal Large Language Models more accurate, but the gains are brittle: simply paraphrasing a question or changing the prompt template can degrade them, which challenges reliable deployment in high-stakes scenarios like medical VQA. We trace this to two issues of the standard RL objective. First, the binary verifier conflates format with content, so the reward signal cannot tell a wrong answer apart from a misformatted one. Second, the training distribution covers only a thin slice of the real-world prompts that the model might meet at deployment, so policies that perform well on the training distribution can behave differently under unseen prompts during test. Both failures call for a robust post-training method that helps the policy cover a broader distribution of semantically equivalent prompts, and we identify two measures that help achieve this objective: separating format from semantics in the reward, and applying policy invariance across perturbed prompts with equivalent semantics. We therefore propose Prompt-Invariant RLVR (PIRL), consisting of a dynamic trinary reward and a consistency regularizer based on an embedding-space adversary. Under stress testing, PIRL's average accuracy on benchmarks drops by only 1%, where GRPO drops 3%. On dynamic evaluation, PIRL also achieves the smallest performance drop.
Sources
- Qwen3-VL Technical Report
- Qwen2.5-VL Technical Report
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- PathVQA: 30000+ Questions for Medical Visual Question Answering
- REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization
- Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Models
- Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning
- Revisiting Visual Understanding in Multimodal Reasoning through a Lens of Image Perturbation
- R1-Fuzz: Specializing Language Models for Textual Fuzzing via Reinforcement Learning
- Understanding R1-Zero-Like Training: A Critical Perspective
- R-Horizon: How Far Can Your Large Reasoning Model Really Go in Breadth and Depth?
- Information-Theoretic Reward Decomposition for Generalizable RLHF
- MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learning
- Addressing Benchmarking Gaps in Large Language Models for Health and Medicine with Dynamic Red-Teaming
- Proximal Policy Optimization Algorithms
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Causally-Enhanced Reinforcement Policy Optimization
- Adversarial Preference Learning for Robust LLM Alignment
- A Survey of Reinforcement Learning for Large Reasoning Models
- Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Models
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems