ParaVT: Taming the Tool Prior Paradox for Parallel Tool Use in Agentic Video Reinforcement Learning
summary
The gist
Training large multimodal models (LMMs) via reinforcement learning (RL) to natively invoke video-processing tools has become a promising route to long-video understanding However…
In short
Existing methods for training AI models to use video tools sequentially fail due to error propagation and high cost. ParaVT introduces a parallel, multi-agent framework that dispatches multiple video crops simultaneously in one turn for better context and fault tolerance. To fix training instability caused by pre-trained tool knowledge, PARA-GRPO uses targeted rewards and frame budget randomization.
Key concepts
- Tool Prior Paradox
- This paradox describes how the knowledge from pre-trained tools—which helps the model explore new actions—simultaneously destabilizes the model's learned structural format (like correct tagging) and encourages it to skip necessary tool calls during reinforcement learning training.
- Format Fragility
- This refers to when a model, while exploring tools, reverts to its initial pre-trained structure instead of following the desired output format. This manifests as incorrect tags or failing to close necessary structural boundaries in the generated text.
- Tool Necessity Gap
- This occurs when the reward signal for calling a tool is too weak compared to skipping it, especially when a direct answer is possible. The model learns to exploit this gap by skipping tools, leading to poor task accuracy despite the presence of tool-use capabilities.
Terminology used across episodes
This episode discusses
- ParaVT: Taming the Tool Prior Paradox for Parallel Tool Use in Agentic Video Reinforcement Learning · Paper Radio
- Qwen3-VL Technical Report
- VideoChat-M1: Collaborative Policy Planning for Video Understanding via Multi-Agent Reinforcement Learning
- Scaling RL to Long Videos
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- VideoZoomer: Reinforcement-Learned Temporal Focusing for Long Video Reasoning
- Video-R1: Reinforcing Video Reasoning in MLLMs
- LOVE-R1: Advancing Long Video Understanding with an Adaptive Zoom-in Mechanism via Multi-Step Reasoning
- AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models
- GPT-4o System Card
- SAGE: Training Smart Any-Horizon Agents for Long Video Reasoning with Reinforcement Learning
- VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning
- LongVideoAgent: Multi-Agent Reasoning with Long Videos
- MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Grounding
- MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learning
- Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs
- Conan: Progressive Learning to Reason Like a Detective over Multi-Scale Visual Evidence
- LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RL
- Safety Alignment Should Be Made More Than Just a Few Tokens Deep
The paper
ParaVT: Taming the Tool Prior Paradox for Parallel Tool Use in Agentic Video Reinforcement Learning · Read on arXiv
Zuhao Yang, Kaichen Zhang, Sudong Wang, Keming Wu, Zhongyu Yang
MiroMind · NTU · HKU-HKUST(GZ) · THU
Training large multimodal models (LMMs) via reinforcement learning (RL) to natively invoke video-processing tools (e.g., cropping) has become a promising route to long-video understanding. However, most existing agentic Video-LMMs dispatch tool calls sequentially (i.e., one per turn). Sequential tool calling can propagate localization errors, accumulate irrelevant context, and require an additional main-model generation for each successive crop. We introduce ParaVT, a multi-agent RL framework for Parallel Video Tool calling, dispatching multiple time-window crops in a single turn, allowing evidence from multiple temporal windows to be examined concurrently. Yet applying standard Group Relative Policy Optimization (GRPO) to ParaVT reveals an obstacle we term the Tool Prior Paradox: inherited tool-use priors appear to support exploration while competing with the output format learned during supervised fine-tuning. Standard RL can also settle on tool skipping. A comparison with a model with weaker tool-use priors shows stable formatting but no tool calls during RL, consistent with a trade-off between format stability and tool exploration. We propose PARA-GRPO (Parseability-Anchored and Ratio-gAted GRPO), which augments standard RL with two complementary mechanisms: (i) a format reward that encourages tag closure and penalizes unclosed reasoning blocks, and (ii) a per-prompt randomization of the overview-frame budget to create a reward advantage for tool use when the overview provides insufficient evidence. Across six long-video understanding benchmarks, ParaVT improves over the Qwen3-VL baseline by +7.9% on average, with PARA-GRPO raising training-time format reward from 0.13 to a peak of 0.64. These findings highlight the need to account for inherited tool-use priors when designing RL methods for long-video understanding.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "ParaVT: Taming the Tool Prior Paradox for Parallel Tool Use in Agentic Video Reinforcement Learning".
Jane: Here's the gist of the paper. Training large multimodal models (LMMs) via reinforcement learning (RL) to natively invoke video-processing tools has become a promising route to long-video understanding However, existing native-RL methods dispatch tool calls sequentially (i.e., one per turn): a single wrong crop propagates errors without peer correction, multi-turn tool calls corrupt context, and inference cost scales linearly with the number of turns We introduce ParaVT, the first multi-agent end-to-end RL-trained framework for Parallel Video Tool calling, dispatching multiple time-window crops in a single turn for cleaner context and better fault tolerance Yet applying standard RL to ParaVT reveals an obstacle we term the Tool Prior Paradox: the pretrained tool priors that enable tool exploration also destabilize cold-started structural format and expose the skip-tool reward shortcut under temperature sampling We propose PARA-GRPO (Parseability-Anchored and Ratio-gAted GRPO), which augments standard RL with two complementary mechanisms: (i) a targeted format reward applied only at the structural-token positions most prone to collapse, and (ii) a per-prompt frame-budget randomization that creates training prompts where calling the tool yields a measurable reward signal over skipping it Across six longvideo understanding benchmarks, ParaVT improves over the Qwen3-VL baseline by +7.9% on average, with PARA-GRPO lifting training-time format compliance from 0.13 to 0.64 Pruning redundant reasoning tokens The paper identifies the Tool Prior Paradox, which is decomposed into Format Fragility and Tool Necessity Gap Format Fragility manifests in two forms: tag-level reversion (i.e., the policy emitting the pretrained schema in place of) and structural-boundary collapse (i.e., failure to close and).
Tom: The authors verify this diagnosis with a cross-model contrast on a weak-prior LMM, noting that format stays stable but RL elicits zero tool calls, indicating that prior strength is the shared driver of both format collapse and tool exploration.
Jane: How the training signal is built The PARA-GRPO framework introduces two complementary mechanisms to tame the paradox: Exploration Anchoring and nFrames Gating.
Tom: Exploration Anchoring stabilizes the format side via two cooperating mechanisms: a selective reward term targets the few structural-token positions most vulnerable to collapse, and a Constrained Generation hook fixes only the opening reasoning tag.
Paper summary: Tom: So we've talked about the main idea of ParaVT, and now let's look a bit closer at what they actually claim in this paper. The core thesis revolves around moving away from sequential tool calling to parallel tool calling in one step for long videos.
Jane: They are specifically targeting the problem where a single wrong crop error gets passed along without peer correction, and multi-turn calls just mess up the context structure of the model.
Lu: ParaVT’s approach is dispatching multiple time window crops in a single turn, which they claim leads to cleaner context and better fault tolerance compared to traditional sequential methods.
Meng: Cleaner context sounds good for accuracy, but I'm curious about how this parallel dispatch actually impacts inference cost when you have many tools being called simultaneously.
Lalam: The paper also introduces the Tool Prior Paradox, which is the central challenge they identified: pretrained tool priors that enable exploration can destabilize the cold-started structural format and expose a shortcut where skipping tools becomes rewarded under temperature sampling.
Tom: That paradox is tricky because it suggests that the very thing we use to get models to try new things—those pretraining priors—is what causes them to break their basic rules when they start learning via reinforcement learning.
Jane: To fight this, they propose PARA-GRPO, which uses Exploration Anchoring and nFrames Gating. The anchoring part fixes format issues by selectively rewarding structural tokens most prone to collapse, and the gating part creates a curriculum for training by randomizing the overview frame budget per prompt so tools are called when necessary.
Lu: The exploration anchoring mechanism is designed to stabilize the format side by using a selective reward term on those vulnerable structural token positions, and it also includes a constrained generation hook to fix only the opening reasoning tag.
Meng: That sounds like they are trying to keep the rollouts well-formed enough so they can actually be parsed into tool calls, which is crucial for any RL training pipeline.
Lalam: And then there's the nFrames Gating, which tackles the reward signal by randomizing the overview frame budget per prompt, setting up a curriculum where calling a tool yields a measurable reward signal over skipping it.
Conclusion: Tom: So, to wrap up ParaVT: Taming the Tool Prior Paradox for Parallel Tool Use in Agentic Video Reinforcement Learning, what’s the big picture here? It’s about moving from brittle sequential chains to a parallel dispatch system for video tools.
Jane: The authors are essentially showing that just having a model that can use tools isn't enough; you need a way to make those tool uses robust and context-aware in long videos.
Lu: They propose PARA-GRPO as the solution, which combines format anchoring and reward gating to tame the trade-off between exploration and structural integrity. It shows that you can use pretrained knowledge without it breaking the model during reinforcement learning training under temperature sampling.
Meng: From a deployment standpoint, if this works, it suggests we could build more reliable agents for video understanding where we don't have to worry about cascading errors from a single bad tool execution.
Lalam: The main implication is that you can get better performance on those long video benchmarks with models like ParaVT when they are trained with PARA-GRPO, achieving a seven point nine percent average gain over the Qwen3 VL baseline.
Tom: So, in short, it’s about parallel tool use for long videos. It moves us away from the slow, error-prone sequential way and introduces a structured training method to handle the inherent instability of pretrained priors during RL training.
More episodes
- 2603.28113-The multiplicative Lipschitz bound as an inductive bias
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound