OneSearch-VL: Unified Multimodal Deep Research Agent for Image and Video

arXiv:2610.12419 · cs.CV · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "OneSearch-VL: Unified Multimodal Deep Research Agent for Image and Video".

Jane: The gist OneSearch-VL introduces a unified agent for deep research over single-image, multi-image,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Now that we know the basics of OneSearch-VL, let's look at what the paper actually calls its improvements. They’re really pushing beyond just adding a new model and trying to build a fundamentally better way to handle multimodal research.

Jane: They focus on three main areas here. First, they have the VGEG-Centered Data Engine for Unified Research Agents, which is about building that central reference structure we talked about earlier.

Lu: That’s the core idea—the VGEG acts as a unified task-level reference structure that lets them link localized visual anchors to real-world entities, record multi-hop relational paths and source-supported facts.

Meng: So it's not just about seeing things; it's about encoding the entire research workflow into this single graph to support question–answer construction and verification across all three visual inputs simultaneously.

Tom: Right. Then they have the Evidence-aware Visual-Grounded Rubric reward, or EVGR, which is for fine-grained supervision during reinforcement learning.

Jane: That reward specifically checks evidence traceability—whether the required facts are supported by tool observations—and visual grounding—whether the correct objects or regions are used in the solution.

Lu: This moves supervision beyond just getting a right final answer; it forces the agent’s observations to align with the evidence required by that specific task, leading to higher quality reasoning.

Meng: It sounds like they're making sure the agent is not just guessing correctly, but is actually using the right visual information in a verifiable way.

Tom: And finally, they have this operation-oriented benchmarking for fine-grained analysis using OneSearch-MI-Bench and OneSearchVideoBench.

Jane: These benchmarks organize questions by the specific research operations encoded in their VGEGs, which capture those entity–relation paths and fact–composition dependencies required to get the answer.

Lu: This allows researchers to analyze *how* an agent retrieves and composes evidence instead of just looking at what’s in the visual input, enabling targeted improvements for things like multi-anchor arithmetic or multi-hop retrieval.

Meng: So they aren't just testing if the model can answer; they’re testing if it can perform specific kinds of reasoning steps required for different research problems.

Tom: It’s about defining the complexity of the task by looking at these operations, which gives us a much finer lens on where the agent is succeeding or failing.

Jane: So to summarize, they are not just proposing a bigger model; they are proposing a new way to structure multimodal research through this graph and using targeted rewards and benchmarks to guide that structure.

Lu: It’s about building an entire research pipeline as a verifiable, traceable process within the AI system itself.

Meng: That makes the system much more transparent about its reasoning steps, which is crucial for any AI application that needs to be reliable.

Tom: So we’ve seen how they set up this sophisticated framework for deep research, and now we’re moving into how they actually conclude their work and where they might go next.

Jane: We’ll wrap up by summarizing the main implications of OneSearch-VL and what that means for the future of multimodal agents.

Lu: And we’ll hear from our team members on the potential impact and any remaining questions about their work.

Meng: I’ll talk about how this level of structured reasoning could translate into more reliable, complex agentic applications in the real world.

Lalam: I can talk about how this structured approach to evidence and grounding fundamentally improves the quality of information synthesis that we can expect from multimodal systems moving forward.

Tom: Let's get into that summary for OneSearch-VL now.

The paper's summary: Jane: So, to recap what we’ve discussed so far, this paper introduces OneSearch-VL as a unified agent designed for deep research across single images, multiple images, and videos. The central concept is the Visually Grounded Evidence Graph or VGEG.

Lu: The VGEG links those localized visual anchors to real-world entities and records the multi-hop relational paths and source-supported facts needed to build an answer.

Meng: It’s essentially a shared task-level reference structure that encodes all the dependencies between what’s seen visually, what we find externally, and how we combine those facts into a final answer.

Tom: They show that this VGEG-based data engine constructs and verifies multi-image and video questions by filtering expert trajectories.

Jane: They also detail how they assemble the OneSearch-VL-SFT-110K and OneSearch-VL-RL-10K datasets using this engine to train the agent <ref:2610.12419#pg1,OneSearch-VL-SFT-110K and OneSearch-VL-RL-10K>.

Lu: This whole system is about creating a unified policy that selects visual inspection, temporal localization, and web retrieval operations based on the input and question.

Meng: They are developing this policy to condition on the interaction history, generating an action at which is either a tool call or the final response based on what’s happening visually and what we’ve retrieved so far.

Tom: So it boils down to a system that learns how to perform deep research by weaving together visual grounding, external knowledge retrieval, and fact composition into one cohesive process.

Jane: It means the agent isn't just looking at pictures; it's performing active inspection, retrieving information from the web, and synthesizing a supported answer dynamically.

Lu: It’s about establishing end-to-end provenance for every step in that research process across all three visual modalities.

Meng: The implication here is that we are moving toward agents that perform these complex, multi-step tasks with more explicit oversight on how they manage their multimodal inputs and external knowledge.

Tom: So, essentially, OneSearch-VL is about making the connection between perception and knowledge synthesis a single, verifiable agentic workflow.

Jane: Right. It’s about making sure the agent’t just sees things correctly but also knows exactly how to use that visual information to build an answer supported by evidence.

Lu: It establishes this end-to-end provenance for every step in that research process across all three visual modalities, which is a big step forward.

Meng: It suggests that future agents will need this kind of explicit structure to handle the complexity of real-world data interactions effectively.

Tom: So, we've got a solid foundation now on what OneSearch-VL is and how it works internally before we look at the specific improvements they made to make it better.

Jane: That leads us perfectly into the improvements section where they detail how they enhanced this core structure.

Lu: They’ll show us exactly how EVGR adds supervision for evidence traceability and visual grounding during reinforcement learning.

Meng: And they'll also show us how those operation-oriented benchmarks are designed to test the reasoning steps themselves.

Tom: This is where we get into the specifics of how they fine-tune the agent to make sure it learns robust reasoning behaviors.

Jane: We’ll unpack those details and see how much better these specific mechanisms make the system perform compared to older methods.

The paper's improvements: Tom: Now we’re talking about the specifics of what they did to enhance the VGEG-Centered Data Engine, like the EVGR and the benchmarking. They are focusing on how these pieces improve supervision and analysis.

Jane: They introduced that Evidence-aware Visual-Grounded Rubric reward or EVGR derived from VGEG annotations to supervise evidence traceability and visual grounding during reinforcement learning.

Lu: The EVGR is built from VGEG annotations specifically to evaluate evidence traceability, which checks if the required facts are supported by tool observations.

Meng: And it also evaluates visual grounding, checking whether the relevant objects or regions in the frame are correctly identified and used in the solution.

Tom: This moves supervision beyond just getting a right final answer by ensuring that the agent’s observations align with what is required by the task.

Jane: That’s a significant step because it ensures that reasoning is supported by actual evidence, not just some abstract correctness score.

Lu: It means the agent learns to be more deliberate about which visual anchors and facts it relies on during its search process.

Meng: That deliberate selection of inputs based on evidence makes the agent's search behavior much more grounded in reality.

Tom: And they also introduced Operation-oriented benchmarking with OneSearch-MI-Bench and OneSearchVideoBench to organize questions by the research operations encoded in their VGEGs.

Jane: These benchmarks capture those entity–relation paths and fact–composition dependencies needed to derive the answer, allowing researchers to analyze how an agent retrieves evidence rather than just what the visual input contains.

Lu: It lets researchers target improvements for specific reasoning capabilities, like multi-anchor arithmetic or multi-anchor comparison.

Meng: So instead of general performance metrics, they can pinpoint exactly which step in the process is causing a failure, which is very actionable information for development.

Tom: This moves us from broad performance metrics to deep analysis of the agent’s internal reasoning structure and how it handles different types of research tasks.

Jane: It’s about giving researchers tools to probe the agent's capabilities at a much more granular level than before, which is essential for understanding these complex systems.

Lu: So they are essentially providing a way to measure the success of the VGEG structure itself by seeing how well it guides the agent through those defined research operations.

Meng: That’s a powerful way to validate that their proposed framework isn't just theoretical; it’s guiding the actual learning process effectively.

Tom: So we have seen how they use these specific tools to make the system smarter, and now we can talk about where the overall picture goes in the conclusion of this paper.

Jane: Let’s wrap up with a final summary of what OneSearch-VL does and why it matters for AI research today.

Conclusion: Tom: Wrapping up, the main point here is that OneSearch-VL provides a unified agent approach to deep research over single images, multiple images, and videos by centering everything on the VGEG.

Jane: It successfully establishes a workflow where visual grounding, external retrieval, and fact composition are tightly linked through this graph for both data construction and evaluation.

Lu: This means we have a system that can actively inspect visuals and use that inspection to build verifiable answers supported by evidence from all three modalities.

Meng: It’s about making the connection between perception and knowledge synthesis a single, verifiable agentic workflow instead of separate pipelines.

Tom: The performance gains they report on the OneSearch-VL paper are significant when compared to previous models with tool access.

Jane: So, this unified approach is what drives those reported improvements across various benchmarks and training configurations.

Lu: The VGEG is the central mechanism that ties everything together into a single reference for data construction, process supervision, and operation evaluation.

Meng: It’s a massive piece of infrastructure for building more reliable and complex agentic systems.

Tom: This paper introduces OneSearch-VL as the unified multimodal deep research agent for image and video.

Jane: So it’s about making sure that the dependencies between visual grounding, external retrieval, and fact composition are explicitly managed through this VGEG structure.

Lu: It sets a new standard for how we approach structuring these complex multimodal tasks.

Meng: This has real implications for building agents that can handle research tasks with high fidelity and verifiable evidence.

Tom: That’s the big picture on OneSearch-VL, and we’re all really excited about this work and where it points next.

Hongyu Li, Manyuan Zhang, Kaituo Feng, Shu Chen, Dian Zheng, Hao Li, Hao Yu, Zhangquan Chen, Zoey Guo, Ray Zhang

BUAA 2CUHK 3NTU 4THU 5HFUT

cs.CV

Submitted: 2026-10-08

Updated: 2026-10-08

Code: https://github.com/appletea233/OneSearch-VL

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: The gist OneSearch-VL introduces a unified agent for deep research over single-image, multi-image, and video inputs by centering its Visually Grounded Evidence Graph (VGEG) to encode dependencies

Key concepts

Visually Grounded Evidence Graph (VGEG)
The VGEG is the core structure that maps visual elements in an input (like an image or video) to real-world entities and external facts. It records the relationships between these elements, tracks multi-hop paths, and specifies exactly which operations are needed to combine these pieces into a final answer.
Data Engine
This engine is built around the VGEG to construct and verify complex research questions for images and videos. It extracts visual entities, retrieves supporting web facts through tools like image search, verifies this information, and synthesizes expert tool-use paths while keeping track of all visual locations and source references.
Evidence-aware Visual-Grounded Rubric Reward (EVGR)
EVGR is a reward system derived from the VGEG annotations used to train the agent. It supervises two critical aspects: evidence traceability, ensuring facts come from supported tool observations, and visual grounding, confirming that the correct objects or frames are accurately identified and used in the solution.

Terminology

Summary

The gist OneSearch-VL introduces a unified agent for deep research over single-image, multi-image, and video inputs by centering its Visually Grounded Evidence Graph (VGEG) to encode dependencies across visual grounding, external retrieval, and fact composition.

How it works

  1. VGEG is the core structure that encodes dependencies as a shared task-level reference for data construction, process supervision, and operation-level evaluation The VGEG links localized visual anchors to real-world entities, records multi-hop relational paths and source-supported facts, and specifies the operations that compose these facts into an answer.

  2. A data engine is developed based on VGEG to construct and verify multi-image and video questions and filters expert trajectories. This engine extracts visual entities, retrieves and verifies web facts, generates compositional questions, and synthesizes expert tool-use trajectories while retaining visual locations, source provenance, and evidence dependencies.

  3. The Evidence-aware Visual-Grounded Rubric reward (EVGR) is derived from VGEG annotations to supervise evidence traceability and visual grounding during Reinforcement Learning. This reward combines evidence traceability, which verifies whether the required facts are supported by tool observations, and visual grounding, which verifies whether the relevant objects, regions, or frames are correctly identified and used in the solution.

  4. The agent is trained in two stages: Supervised Fine-Tuning (SFT) on OneSearch-VL-SFT-110K and Reinforcement Learning (RL) on OneSearch-VL-RL-10K. SFT learns a unified multimodal toolinteraction policy from expert trajectories, while RL optimizes answer quality, search behavior, and evidence use through online interaction with the tool environment.

  5. For fine-grained evaluation, OneSearch-MI-Bench and OneSearch-VideoBench are constructed to organize questions by the research operations encoded in their VGEGs. These benchmarks focus on organizing questions according to the research operations represented in their corresponding VGEGs, capturing the entity-relation paths and fact-composition dependencies required to derive the answer.

Key Components and Data Engine

The data engine pipeline involves five stages starting from raw visual sources. Stage 3, Web Evidence Graph Construction, connects visual anchors to external knowledge through image search, OCR, and text retrieval. This results in an input-level evidence graph GX that preserves the correspondence among visual anchors, real-world entities, external facts, and supporting sources. Stage 4 constructs candidate research tasks and their task-level references from GX through a task-conditioned projection Φi that selects the required visual anchors and facts from GX and augments them with explicit answer-producing operations.

Evaluation Framework

The paper introduces two complementary dimensions for evaluation: evidence traceability and visual grounding, which are captured by EVGR. Evidence traceability measures whether tool observations establish the answer-core entities and required fact hops, and whether subsequent reasoning follows those observations without unsupported substitutions. Visual grounding measures whether the correct visible entities, regions, or video frames are identified and used to drive subsequent evidence retrieval.

Benchmark Organization

OneSearch-MI-Bench and OneSearch-VideoBench organize questions by the principal research operation required to answer each question. The six principal research operations include Single-anchor lookup, Multi-hop retrieval, Knowledge-conditioned count, Multi-anchor join, Multi-anchor arithmetic, and Multi-anchor comparison. The structural difficulty score is derived from the principal research operation, reasoning depth (hi), and number of required facts (Fi).

Performance Results

Experiments show that OneSearch-VL-8B improves over Qwen3-VL-8B with tool access by 20.2 and 17.6 percentage points on the two new benchmarks, respectively, while also achieving substantial gains across 7 image benchmarks and VideoDR. Joint SFT on all three visual input types achieves a six-benchmark average of 55.8, versus 51.7-55.1 for single-type training. The full reward performs best on five benchmarks and ties for best on the remaining one.

Limitations and Future Work

OneSearch-VL depends on external search tools such as TextSearch and ImageSearch (Table 1). Future work should preserve retrieval snapshots and evaluate robustness to tool failures. Automated VGEG construction and model-based rewards may also introduce annotation errors or judging biases. Human calibration and open multimodal process judges could improve the reliability of trajectory-level assessment.

The paper introduces OneSearch-VL, a unified agent for deep research over single-image, multi-image, and videos by centering its Visually Grounded Evidence Graph (VGEG) to encode dependencies across visual grounding, external retrieval, and fact composition.

How it works

The VGEG links localized visual anchors to real-world entities, records multi-hop relational paths and source-supported facts, and specifies the operations that compose these facts into an answer. The data engine constructs and verifies multi-image and video questions and filters expert trajectories.

Key Components and Data Engine

The data engine pipeline involves five stages starting from raw visual sources. Stage 3, Web Evidence Graph Construction, connects visual anchors to external knowledge through image search, OCR, and text retrieval. This results in an input-level evidence graph GX that preserves the correspondence among visual anchors, real-world entities, external facts, and supporting sources.

Performance Results

Experiments show that OneSearch-VL-8B improves over Qwen3-VL-8B with tool access by 20.2 and 17.6 percentage points on the two new benchmarks, respectively, while also achieving substantial gains across 7 image benchmarks and VideoDR. Joint SFT on all three visual input types achieves a six-benchmark average of 55.8, versus 51.7-55.1 for single-type training.

Limitations and Future Work

OneSearch-VL depends on external search tools such as TextSearch and ImageSearch (Table 1). Future work should preserve retrieval snapshots and evaluate robustness to tool failures. Automated VGEG construction and model-based rewards may also introduce annotation errors or judging biases.

Improvements for AI systems

  1. textbfVGEG-Centered Data Engine for Unified Research Agents: Improved Evidence Linkage and Task Construction. The VGEG, a unified task-level reference structure, allows for the construction of data that links localized visual anchors to real-world entities, records multi-hop relational paths and source-supported facts, and specifies the operations that compose these facts into an answer. This enables the system to generate data that supports question–answer construction and verification across single-image, multi-image, and video inputs simultaneously.

  2. textbfEvidence-aware Visual-Grounded Rubric Reward (EVGR) for Fine-grained Supervision. The EVGR is constructed from VGEG annotations to supervise evidence traceability and visual grounding during RL, evaluating "evidence traceability, which verifies whether the required facts are supported by tool observations, and visual grounding, which verifies whether the relevant objects, regions, or frames are correctly identified and used in the solution. This moves beyond simple answer correctness by ensuring that the agent’s observations align with the evidence required by the task," leading to higher-quality reasoning.

  3. textbfOperation-oriented Benchmarking for Fine-grained Analysis. The introduction of OneSearch-MI-Bench and OneSearchVideoBench organizes questions according to the research operations represented in their corresponding VGEGs, which capture the entity–relation paths and fact–composition dependencies required to derive the answer. This allows researchers to analyze how an agent retrieves and composes evidence rather than what the visual input contains, enabling targeted improvements in specific reasoning capabilities like multi-anchor arithmetic or multi-hop retrieval.

  4. textbfUnified Agent for Cross-Modal Deep Research. OneSearch-VL is developed as a unified agent trained on data spanning single image, multi-image, and video inputs through SFT and RL, allowing a single policy to select input–appropriate tools within a unified action space. This enables the system to perform deep research across all three visual modalities by sharing an action–observation protocol while retaining input-specific behaviors.

  5. textbfProcess-Level Reward Optimization via GRPO. The system optimizes the policy using Group Relative Policy Optimization (GRPO), where the reward is composed as: R(τ) = Rfmt(τ) (λaccRacc(τ) + λqueryRquery(τ) + λEVGRREVGR(τ)) and gated by format validity. This method allows for optimization across multiple complementary dimensions—answer correctness, query quality, evidence traceability, and visual grounding—as shown by the finding that Using both Trace and Ground yields the best average of 61.1.

Sources

Related papers