DyCo-RL: Dynamic Cross-Modal Coordination for Visual Reasoning
cs.CV
Submitted: 2026-06-06
Updated: 2026-09-27
Code: https://github.com/Sammy20207109/DyCo-RL
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a leading paradigm for enhancing visual reasoning in Multimodal Large Language Models (MLLMs).
Terminology
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a leading paradigm for enhancing visual reasoning in Multimodal Large Language Models (MLLMs). However, existing RLVR methods optimize primarily for the reasoning outcome and overlook how that outcome is produced: as a chain-of-thought unfolds, some tokens must extract evidence from the image while others must anchor to the preceding textual context, yet a single sequence-level reward provides no signal about whether each token draws on the modality its role demands. Token-level analyses and controlled interventions reveal that tokens frequently under-attend to their designated modality, and that this coordination breakdown may contribute to reasoning failures. Motivated by these findings, we propose DyCo-RL, which integrates dynamic cross-modal coordination into RLVR optimization. Specifically, DyCo-RL uses the Fisher-Rao geodesic distance to measure within-modality attention shifts, assigning tokens to either visually-oriented or text-oriented functional roles. It then evaluates the alignment between a token's actual attention allocation and its assigned role, leveraging this score for alignment-guided advantage reweighting during policy optimization. Extensive experiments using 11K examples for RL training demonstrate that the algorithm-agnostic DyCo-RL, when applied to Qwen2.5-VL-3B/7B and Qwen3.5-27B, improves the average performance of all four evaluated RLVR algorithms across seven benchmarks spanning visual-centric and mathematical reasoning.
Sources
- Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin
- MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
- Soft Adaptive Policy Optimization
- Spotlight on Token Perception for Multimodal Reinforcement Learning
- Credit Where It is Due: Cross-Modality Connectivity Drives Precise Reinforcement Learning for MLLM Reasoning
- Explain Before You Answer: A Survey on Compositional Visual Reasoning
- What does RL improve for Visual Reasoning? A Frankenstein-Style Analysis
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
- Fleming-VL: Towards Universal Medical Visual Reasoning with Multimodal LLMs
- SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvement
- Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluation
- Perception-Aware Policy Optimization for Multimodal Reasoning
- VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models
- BioProBench: A Corpus and Benchmark for Biological Protocol Reasoning in Autonomous Science
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale
- Token-level Direct Preference Optimization
- Perceptual-Evidence Anchored Reinforced Learning for Multimodal Reasoning
- Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning
- MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models