Video-Browser: Towards Open-web Agentic Video Browsing

arXiv:2512.23044 · cs.CV · Submitted 2025-12-28 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Video-Browser: Towards Open-web Agentic Video Browsing".

Jane: The gist:

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper, "Video-Browser: Towards Open-web Agentic Video Browsing." It sounds like they’re trying to figure out how an AI agent can actually search the entire open web using video, which is pretty big.

Jane: I mean, it tackles that huge gap we talked about earlier between just getting text summaries and actually seeing what’s in the video. The authors are Zhengyang Liang, Yan Shu, Xiangrui Liu, Minghao Qin, Nicu Sebe and Zheng Liu from Singapore Management University and other places.

Lu: They're coming from some strong AI backgrounds at institutions like Beijing Academy of Artificial Intelligence and Hong Kong Polytechnic University. They have a deep understanding of how these models work on the training side.

Meng: I haven't seen much work directly tackling open-web video browsing in production yet, so seeing a formal paper on this is interesting for understanding the theoretical limits we might hit later at scale.

Lalam: This paper sets up a framework to evaluate if agents can actually do this, moving beyond just simple retrieval tasks.

The paper's summary: Tom: Okay, so they introduce VideoBrowseComp, which is this benchmark designed specifically to make sure any agent we test has to use videos as a mandatory part of its work.

Jane: That's key because it forces the agent out of that easy path where it can just grab a Wikipedia plot instead of actually looking at the video content. They structured it into three difficulty levels, from simple retrieval up to really complex reasoning across different sources.

Lu: The design principles they use for this benchmark are mandatory video dependency, hard-to-find and easy-to-verify questions, and answer uniqueness. That’s a rigorous way to test the agent's perception versus just its text search skills.

Meng: I worry about the "hard-to-find" part. If the queries are too complex, we might just get models that fail because they can't handle that kind of cross-source reasoning yet.

Lalam: The benchmark design is smart because it's not just asking for a general answer; it demands specific spatio-temporal attributes for verification, which keeps things objective.

The paper's improvements: Tom: The core idea they propose is this novel agent called Video-Browser, which uses something they call Pyramidal Perception to handle the video data.

Jane: So the main improvement is treating the web videos like a pyramid instead of trying to process everything at once. They use cheap metadata first to filter out stuff they don't need to see visually.

Lu: The architecture breaks down into three stages for perception: Semantic Filter, Sparse Localization, and then Zoom-in. Stage one uses just metadata with zero visual inference cost, which is a smart way to save resources early on.

Meng: Saving costs sounds good on paper, but what happens when that filter misses something important? That initial pruning stage is where the agent could lose critical context before it even gets to the visual part.

Lalam: The results show that this approach works: Video-Browser achieves a thirty-seven point five percent relative improvement in accuracy while cutting token consumption by fifty-eight point three percent compared to direct video inference baselines, which confirms that our method successfully balances the scalability required for open-web video research with precision needed for fine-grained verification <ref:2512.23044#pg2,confirms that our method successfully balances the scalability required for open-web>

Conclusion: Tom: So, to wrap up, the main point of "Video-Browser: Towards Open-web Agentic Video Browsing" is that we can balance the need for scale with needing high precision by only paying for visual perception when it's absolutely necessary.

Jane: It suggests that instead of just feeding everything to a massive model, we should use cheap metadata to prune the search space and only zoom in on specific windows when needed. It’s about smart compute allocation, not just brute force.

Lu: The ablation studies confirm that Stage one the semantic filter using metadata, is necessary because without it, the agent wastes its perceptual budget on irrelevant content <ref:2512.23044#pg1>

Zhengyang Liang, Yan Shu, Xiangrui Liu, Minghao Qin, Nicu Sebe, Zheng Liu, Lizi Liao

Singapore Management University · University of Trento · Beijing Academy of Artificial Intelligence · Hong Kong Polytechnic University

cs.CV

Submitted: 2025-12-28

Updated: 2026-10-13

Code: https://github.com/chrisx599/VideoBrowser

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: The gist: Video-Browser proposes a novel agent leveraging Pyramidal Perception to address the modality gap in open-web video browsing by filtering with cheap metadata and zooming in with expensive

Key concepts

Pyramidal Perception
This is a three-stage filtering process for videos. Stage I uses cheap metadata to filter out irrelevant content without needing visual understanding. Stage II uses sparse scanning to find relevant time windows, and Stage III performs detailed visual decoding only on those prioritized clips.
VideoBrowseComp
This is a new benchmark designed to test agents specifically for video tasks. It forces agents to use videos for their answers and scales difficulty from simple retrieval (Level 1) to complex reasoning across multiple, hard-to-find videos (Level 3).
Modality Gap
This refers to the difficulty in processing the vast amount of dynamic information found in video compared to text. Traditional models struggle because they either ignore visual details or require too much context, leading to poor performance when dealing with open-ended video research.
Planner (P), Watcher (W), Analyst (A)
These are the three specialized modules of the Video-Browser agent. The Planner creates search queries, the Watcher uses Pyramidal Perception to filter and scan videos, and the Analyst handles final reasoning based on the gathered evidence.

Terminology

Summary

The gist: Video-Browser proposes a novel agent leveraging Pyramidal Perception to address the modality gap in open-web video browsing by filtering with cheap metadata and zooming in with expensive visual perception only when necessary.

Introduction and Problem Statement

The evolution of autonomous agents is redefining information seeking, transitioning from passive retrieval to proactive, open-ended web research However, a significant modality gap remains in processing the web’s most dynamic and information-dense modality: video Current paradigms struggle to reconcile the scale of open-ended video exploration with the need for fine-grained visual verification Direct visual inference (e.g., RAG) maximizes perception but incurs prohibitive context costs, while text-centric summarization optimizes efficiency but often misses critical visual details required for accurate grounding The field should move towards Agentic Video Browsing to address these problems.

Benchmark and Task Definition

To operationalize this task, the paper introduces VideoBrowseComp, a challenging benchmark designed to evaluate open-ended agentic browsing tasks that enforce a mandatory dependency on videos The benchmark is stratified into three difficulty levels (Level 1 to Level 3), scaling from explicit retrieval to complex, cross-source reasoning that demands aggregating fragmentary evidence across disparate videos. The principles of the benchmark include Mandatory Video Dependency, Hard-to-Find, Easy-to-Verify questions, and Answer Uniqueness.

Video-Browser Architecture

VideoBrowse is formalized as an agentic system S = ⟨P, W, A,M⟩ comprising three specialized modules: Planner P, Watcher W, and Analyst A. The Planner P decomposes the query into sequential sub-tasks and generates specific search keywords qt tailored to each sub-task based on the interaction history Ht. The Watcher W addresses the PerceptionContext Trade-off by employing Pyramidal Perception, which involves three stages: Semantic Filter (Stage I), Sparse Localization (Stage II), and Zoom-in (Stage III). Stage I functions as a metadata-driven filter to prune the search space at zero visual inference cost. Stage II uses a sparse scanning strategy to identify temporal windows using an MLLM, compressing hourlong videos into focused “glimpses”. Stage III performs Targeted Visual Decoding on prioritized windows at high frequency to resolve fine-grained visual details.

Experimental Results and Efficiency

Experiments demonstrate that Video-Browser achieves a 37.5% relative improvement in accuracy while reducing token consumption by 58.3% compared to direct visual inference baselines The framework successfully balances the scalability required for open-web video research with precision needed for fine-grained verification. In efficiency analysis, Pyramidal Perception reduces the Total Tokens to 32,627, a 58.3% (GPT5.1) reduction compared to Direct Visual Inference. This confirms that our architecture successfully identifies where to spend compute, minimizing consumption while maximizing visual perception.

Ablation and Conclusion

Ablation studies confirm the necessity of each component, showing that removing Stage 1 yields the lowest performance, indicating that without effective initial filtering, the agent wastes its perceptual budget on irrelevant content. Removing Stage 2 causes a catastrophic spike in computational cost and context load explosion. The results establish a foundation for verifiable open-web video research Video-Browser successfully navigates from zero knowledge to precise verification through iterative feedback, as demonstrated in the end-to-end workflow analysis.

Limitations

The paper acknowledges that Video-BrowseComp operates at a modest scale, prioritizing annotation rigor over computational accessibility. Furthermore, the modality gap is empirically validated by showing that search-augmented models fail when visual evidence is not indexed in text search engines. The work respects copyright and platform policies by not redistributing video content or collecting non-public sources.

References

Video-Browser: Towards Agentic Open-web Video Browsing, Page 1

Video-Browser: Towards Agentic Open-web Video Browsing, Page 2

Video-Browser: Towards Agentic Open-web Video Browsing, Page 3

Video-Browser: Towards Agentic Open-web Video Browsing, Page 4

Video-Browser: Towards Agentic Open-web Video Browsing, Page 5

Video-Browser: Towards Agentic Open-web Video Browsing, Page 7

Video-Browser: Towards Agentic Open-web Video Browsing, Page 8

Limitations, Page 9

Ethical Considerations, Page 9

References, Page 10

References, Page 10

Appendix Contents, Page 12

Appendix A.5.2 Cost-Efficiency Analysis (Token Usage), Page 9

Appendix B.3.2 Paradigm Comparison, Page 14

Appendix B.3.3 Failure Case Analysis, Page 15

Appendix A Video-Browsecomp A.6 Benchmark Case Studies, Page 16

Appendix B Video-Browser B.2 Prompts, Page 23

Appendix B Workflow of the Video-Browser, Page 25

Appendix A Video-Browsecomp A.4 Evaluation Prompts, Page 17

Appendix B Experiment Settings B.1 Experiment Settings, Page 28

Appendix B Case Studies B.3 Case Studies, Page 29

Appendix A Video-Browsecomp A.5 Benchmark Analysis, Page 13

Appendix A Video-Browsecomp A.3 Evaluation Metrics Details, Page 18

Appendix B.1 Experiment Settings B.1 Experiment Settings, Page 28

Appendix B Case Studies B.3 Case Studies, Page 29

Appendix A Video-Browsecomp A.5 Benchmark Analysis, Page 13

Appendix A Video-Browsecomp A.6 Benchmark Case Studies, Page 16

Appendix B.3 Failure Case Analysis B.3 Failure Case Analysis, Page 15

Appendix B Experiment Settings B.1 Experiment Settings, Page 28

Appendix A Video-Browsecomp A.4 Evaluation Prompts, Page 17

Appendix B Case Studies B.3 Case Studies, Page 29

Appendix A Video-Browsecomp A.5 Benchmark Analysis, Page 16

Appendix B Experiment Settings B.1 Experiment Settings, Page 28

Appendix B Case Studies B.3 Case Studies, Page 29

Appendix A Video-Browsecomp A.6 Benchmark Case Studies, Page 16

Appendix B Experiment Settings B.1 Experiment Settings, Page 28

Appendix B Case Studies B.3 Case Studies, Page 29

Appendix A Video-Browsecomp A.4 Evaluation Prompts, Page 17

Appendix B Experiment Settings B.1 Experiment Settings, Page 28

Appendix B Case Studies B.3 Case Studies, Page 29

Appendix A Video-Browsecomp A.6 Benchmark Case Studies, Page 16

Appendix B Experiment Settings B.1 Experiment Settings, Page 28

Appendix B Case Studies B.3 Case Studies, Page 29

Appendix A Video-Browsecomp A.5 Benchmark Analysis, Page 16

Appendix B Experiment Settings B.1 Experiment Settings, Page 28

Appendix B Case Studies B.3 Case Studies, Page 29

Appendix A Video-Browsecomp A.6 Benchmark Case Studies, Page 16

Appendix B Experiment Settings B.1 Experiment Settings, Page 28

Appendix B Case Studies B.3 Case Studies, Page 29

Appendix A Video-Browsecomp A.5 Benchmark Analysis, Page 16

Appendix B Experiment Settings B.1 Experiment Settings, Page 28

Appendix B Case Studies B.3 Case Studies, Page 29

Appendix A Video-Browsecomp A.6 Benchmark Case Studies, Page 16

Appendix B Experiment Settings B.1 Experiment Settings, Page 28

Appendix B Case Studies B.

Improvements for AI systems

  1. Agentic Video Browsing via Pyramidal Perception: The proposed Video-Browser architecture addresses scalability by filtering with cheap metadata and zooming in with expensive visual perception only when necessary, achieving a 37.5% relative improvement while reducing token consumption by 58.3% compared to Direct visual inference.

  2. Benchmark Rigor for Agentic Limits: The introduction of VideoBrowseComp, which enforces a mandatory dependency on videos and stratifies difficulty from Level 1 to Level 3, allows for the systematic assessment of agentic limits across explicit retrieval, implicit retrieval, and complex multi-source reasoning.

  3. Adaptive Reasoning Loop: The iterative process where the Planner evaluates Watcher's observations and adaptively refines its search strategy—either generating new queries to explore alternative perspectives or adjusting keywords for improved precision ensures that agents can handle knowledge gaps effectively without succumbing to failure when initial searches fail.

Sources

Related papers