CoEvoWhen: Policy-Tool Coevolution for Ultra-Long Video Temporal Grounding

summary

The gist

Ultra-long video temporal grounding requires balancing long-range evidence search with fine-grained event understanding under a limited visual budget, yet existing agentic methods still rely largely

In short

CoEvoWhen is a framework that jointly evolves high-level policies and executable media tools for ultra-long video grounding. It starts with a minimal skill and iteratively refines these components using experience from execution trajectories. This allows a frozen Vision-Language Model (VLM) to autonomously perform complex temporal reasoning by acquiring evidence efficiently without needing separate planning models.

Key concepts

Policy–Tool Coevolution: This is the core mechanism where high-level policies (planning and orchestration) and executable media tools (processing capabilities) improve together. The system learns by distilling experience from how the policy guides tool use, refining both components simultaneously to create a reusable skill.
Skill Representation: The external skill is formally represented as two parts: high-level policies ($\Pi(k)$) that guide task planning and observation orchestration, and executable media tools ($T(k)$) that perform the actual media processing. These components are updated iteratively through experience to become more effective.
Coordinated Observation Strategy: The framework recognizes that tool design and policy are interdependent. It coordinates image-based search for long temporal ranges with video-based verification for local details, allowing the system to strategically choose between modalities based on whether it needs global evidence or fine-grained action order information.
Inference at Test Time: At runtime, the fixed evolved skill is applied to the frozen VLM. The policy guides tool invocation based on the query and accumulated evidence, while tools process that evidence. This allows the VLM to autonomously acquire necessary visual data in a structured way.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 20 episodes
Retrieval-Augmented Generation (RAG) — explained in 19 episodes
Flow Matching — explained in 18 episodes
Federated Learning — explained in 17 episodes
Quantum Fisher Information (QFI) — explained in 17 episodes

This episode discusses

The paper

CoEvoWhen: Policy-Tool Coevolution for Ultra-Long Video Temporal Grounding · Read on arXiv

Yiduo Jia, Muzhi Zhu, Jinchuan Shi, Hao Zhong, Yuling Xi, Ke Liu, Hao Chen

Zhejiang University

Ultra-long video temporal grounding requires balancing long-range evidence search with fine-grained event understanding under a limited visual budget, yet existing agentic methods still rely largely on predefined policies and tool capabilities. Motivated by this, we propose a novel policy-tool coevolution framework that jointly evolves high-level policies and executable media tools from the agentic reasoning trajectories of a VLM, forming a reusable skill without updating model parameters. During evolution, an external skill updater distills transferable task experience in long-video temporal grounding, accordingly refining the orchestration of long-range image-based and fine-grained video-based observations. Alongside these policy updates, the updater employs its coding capabilities to upgrade existing tools or create new ones, adapting the tools to long-video evidence acquisition. Equipped with the evolved skill, the VLM autonomously orchestrates tools under the guidance of the evolved policy, coordinating image and video observations for agentic inference without relying on a separate, stronger planning model. Extensive experiments spanning five benchmarks and three VLMs show that policy-tool coevolution consistently improves temporal grounding accuracy in ultra-long videos while reducing visual token cost at inference, and that the evolved skill yields substantial performance gains on general long-video QA without additional task-specific evolution, demonstrating the effectiveness and generalizability of our framework for long-video understanding.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "CoEvoWhen: Policy-Tool Coevolution for Ultra-Long Video Temporal Grounding".

Jane: Ultra-long video temporal grounding requires balancing long-range evidence search with fine-grained event understanding under a limited visual budget,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Let's talk about the title and authors of this paper, "CoEvoWhen: Policy-Tool Coevolution for Ultra-Long Video Temporal Grounding." It clearly tells us the core mechanism is about evolving both policies and tools together for ultra-long video grounding.

Jane: The authors are Yiduo Jia, Muzhi Zhu, Jinchuan Shi, Hao Zhong, Yuling Xi, Ke Liu, and Hao Chen from Zhejiang University and the State Key Lab of CAD and CG. They're the ones driving this policy-tool coevolution framework.

Lu: I find it compelling that they’re focusing on ultra-long video; that's where current methods really struggle with balancing searching far out while still getting fine details right.

Meng: It suggests a departure from just tuning a single model, as they are building this reusable skill externally using trajectories to refine the system.

Lalam: This approach points toward building skills that can be modularly plugged into larger systems, which is something we've been thinking about for improving cultural understanding of complex data.

The paper's summary: Tom: So, in terms of what this paper summarizes, CoEvoWhen introduces a framework where an external skill evolves by distilling experience from the VLM's reasoning paths. It starts with a minimal base skill and then uses an external skill updater to refine both the high-level policies and the executable media tools.

Jane: The summary emphasizes that for policy evolution, they distill transferable experience to improve how the agent plans its search and organizes its observations, while for tool evolution, they use coding capabilities to upgrade or create new media processing tools.

Lu: It’s interesting how they pair these two updates; the idea is that the policy determines what evidence is needed while the tools dictate exactly what kind of evidence can be presented back to the system during observation.

Meng: They explicitly show this interdependence, stating that tool-use policies and tool design are linked, which makes sense because you need good tools to execute a good plan for long video tasks.

Lalam: This structure suggests that the skill isn't just a set of instructions but a dynamic capability that learns how to balance searching globally with zooming in on specific details.

The paper's improvements: Tom: Now, looking at the improvements they propose, CoEvoWhen shows how this coevolution actually works by refining the policies and tools based on execution trajectories. They show that policy updates refine strategies for planning and observation orchestration, while tool updates adapt or expand media processing capabilities.

Jane: The core mechanism is formalized as updating the skill components: Policy (k+one) is refined by (k), and Tools T(k+one) are refined by T(k), which is how they manage that iterative improvement process.

Lu: The paper highlights a crucial insight about coordination, suggesting that comparing temporally distant candidates benefits from image-based observations for compact coverage, whereas judging action order needs local video-based observations to check continuity.

Meng: This points toward a coordinated observation strategy where the system knows exactly when to switch between global image search and detailed video verification based on what the query demands.

Lalam: The result of this coordination is a skill that supports image-based search, candidate refinement, and selective video verification, which seems like a very targeted way to use visual resources.

Conclusion: Tom: To wrap things up with the conclusion of "CoEvoWhen," it really boils down to how this policy–tool coevolution framework successfully evolves an external skill from a minimal base into something capable of ultra-long video temporal grounding.

Jane: The paper concludes that this method improves grounding accuracy while simultaneously reducing the visual token cost required for those complex tasks, showing a trade-off that is quite efficient.

Lu: The implication here is that by treating the skill as an evolving entity rather than just a static prompt, we can get better performance on challenging temporal reasoning without needing to update the underlying VLM parameters.

Meng: From a practical standpoint, this means we can deploy agents with much more sophisticated temporal understanding on longer video sequences without having to overhaul the massive model architecture itself.

Lalam: This reusable skill concept is powerful because it means we build expertise once and reuse it across different long-video tasks, which really makes scaling AI applications more feasible.

More episodes

← Home