SimpleSearch-VL: A Simple Recipe for Multimodal Agentic Deep Search

arXiv:2606.31504 · cs.CV · Submitted 2026-06-30 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "SimpleSearch-VL: A Simple Recipe for Multimodal Agentic Deep Search".

Jane: Here's the gist of the paper. SimpleSearch-VL presents an efficient, reliable, and practical framework for multimodal agentic search Its core idea is to improve the agent’s own search-and-verification process rather than scaling data, tools, or auxiliary model components.

Tom: The SimpleSearch-VL-30B-A3B model further achieves performance competitive with agentic Gemini-3-Pro.

Jane: Factorized Adaptive Rollout Factorized Adaptive Rollout (FAR) improves sampling efficiency by forming more informative training groups while using redundant samples to mitigate long-tail latency and expose hard samples.

Tom: This strategy combines Prompt Expansion, which introduces new prompt groups when more useful signal is needed, and Rollout Allocation, which gives additional attempts to groups that still lack reward contrast and skips redundant tail rollouts once a signal is obtained.

Paper summary: Jane: So, going back to the start of "SimpleSearch-VL: A Simple Recipe for Multimodal Agentic Deep Search," what's the core thesis they’re pushing? What is this framework fundamentally trying to achieve?

Tom: The paper presents SimpleSearch-VL as an efficient, reliable, and practical framework specifically designed for multimodal agentic search. The main idea isn't to scale up the data or the tools or any auxiliary model components.

Lu: It claims that the core idea is to improve the agent’s own internal search-and-verification process instead of scaling those other things. They want efficiency and reliability through smarter sampling and better verification steps.

Jane: Can you unpack what that means in plain terms for a listener? What kind of problem does this solve that existing methods aren't handling well?

Tom: It means they are focusing on making the agent’s search-and-verification loop more effective on its own, rather than just throwing more compute at it. They want to be smart about how they explore and check information.

Meng: If the focus is internal process improvement, what kind of performance gains are we actually seeing when you compare this approach to scaling up the model size or adding external tools?

Lalam: The results show that with only 5K supervised tool-interleaved trajectories and 2K RL data, SimpleSearch-VL improves Qwen3-VL agentic baselines by fifteen point eight for the 8B variant and sixteen point zero for the larger one.

Jane: So they’re showing that this method can be competitive with state-of-the-art agentic models like Gemini-three Pro, even on their smaller model variants, which is a strong point <ref:2606.31504#pg3>.

Tom: Yeah, the paper validates that the training and tool-use design itself can be more important than just making the model bigger. The performance competitive with Gemini-three Pro for the 30B variant really validates this practical utility of their design choices.

Lu: It proves that optimizing rollout allocation, making visual evidence explicitly verifiable, and keeping webpage summarization internal are the keys to achieving these results without needing a massive data budget.

Conclusion: Jane: So wrapping up this discussion on "SimpleSearch-VL: A Simple Recipe for Multimodal Agentic Deep Search," what’s the final word on this paper and where does it leave us?

Tom: The authors are presenting this as an efficient, reliable, and practical framework for multimodal agentic search. The key contribution is shifting the focus from scaling up resources to improving the agent's internal search-and-verification mechanism.

Lu: It’s a recipe that prioritizes efficiency by using Factorized Adaptive Rollout to handle hard samples intelligently, and reliability through explicit chain-of-thought verification of visual and textual cues.

Meng: For deployment, it suggests a model can achieve strong multimodal search performance with substantially smaller training data and a much lighter training budget than methods like OpenSearch-VL. That’s the practical part we need to focus on for production.

Lalam: The implications are that we don't need huge data sets or massive tool suites; instead, optimizing how the agent allocates its search steps and verifying what it finds is what drives the performance gains.

Jane: It’s about making the agent smarter in its decision-making process, which makes sense for a system that has to navigate complex evidence gathering tasks.

Southeast University · Ant Group

cs.CV

Submitted: 2026-06-30

Updated: 2026-11-12

Comments: Technical Report

Code: https://github.com/AQ-MedAI/SimpleSearch-VL

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: SimpleSearch-VL presents an efficient, reliable, and practical framework for multimodal agentic search Its core idea is to improve the agent’s own search-and-verification process rather than…

Key concepts

Factorized Adaptive Rollout (FAR)
This technique enhances sampling efficiency by grouping training data intelligently. It uses Prompt Expansion to introduce new prompts when needed and Rollout Allocation to give extra attempts to groups that lack reward contrast, ensuring hard samples are explored without wasting time on redundant ones.
Evidence Verification Mechanisms
The system performs explicit reasoning by verifying retrieved visual and textual cues. When using the image_search tool, it checks if the returned thumbnails and URLs match the original query before relying on titles or links, improving accuracy through direct verification.
Self-Summarized Visit and Tool Simplicity
SimpleSearch-VL keeps a minimal tool interface and performs webpage summarization internally. The policy model generates a concise goal summary appended to the next state, eliminating the need for external summarization services during training or inference, keeping all understanding within the agent.
Performance Gains and Efficiency
The framework achieves significant performance boosts (15.8% to 16.0 points) on agentic baselines using relatively small amounts of supervised data and RL data compared to other methods like OpenSearch-VL, proving efficiency gains come from process improvement.

Terminology

Summary

SimpleSearch-VL presents an efficient, reliable, and practical framework for multimodal agentic search Its core idea is to improve the agent’s own search-and-verification process rather than scaling data, tools, or auxiliary model components. p. 0 The SimpleSearch-VL-30B-A3B model further achieves performance competitive with agentic Gemini-3-Pro. p. 0

Factorized Adaptive Rollout

Factorized Adaptive Rollout (FAR) improves sampling efficiency by forming more informative training groups while using redundant samples to mitigate long-tail latency and expose hard samples. This strategy combines Prompt Expansion, which introduces new prompt groups when more useful signal is needed, and Rollout Allocation, which gives additional attempts to groups that still lack reward contrast and skips redundant tail rollouts once a signal is obtained. p. 2 The mechanism uses an expand-only-all-wrong rule to mask later expanded slots for a group if all observed rollouts are answer-incorrect, thereby spending extra attempts on hard groups that still lack an answer-correct trajectory.

Evidence Verification Mechanisms

SimpleSearch-VL performs evidence-verified reasoning, explicitly using chain-of-thought verification to assess the relevance of retrieved visual and textual cues to the original context. It exposes this by having the image search tool return matched thumbnails, webpage titles, and source URLs, enabling the model to verify whether the candidate matches the queried visual content before using its title or URL. p. 2 Furthermore, it employs a region-level cache for image search to reuse results for nearby predicted boxes over the same visual target when IoU is above a threshold. p. 6

Self-Summarized Visit and Tool Simplicity

SimpleSearch-VL keeps a lightweight tool interface and performs webpage self-summary within the agent, requiring no additional external dependencies. The policy model itself produces a concise goal-conditioned summary appended to the next Markov state in Eq. p. 6 1, which avoids deploying an additional summarization service during training and inference. p. 6 This design keeps webpage understanding inside the agent while reducing tool orchestration and model dependencies. p. 2

Performance Gains and Efficiency

With only 5K supervised tool-interleaved trajectories and 2K RL data, SimpleSearch-VL improves Qwen3-VL agentic baselines by 15.8 and 16.0 average points for the 8B and 30B-A3B variants, respectively. p. 0 The training cost comparison shows that SimpleSearch-VL reaches strong multimodal search performance with substantially smaller training data and a much lighter training budget compared to OpenSearch-VL. p. 2 This suggests that the gains come not from scaling data or cluster size, but from improving the efficiency of rollout allocation, making visual evidence explicitly verifiable, and keeping webpage summarization inside the agent itself.

Benchmark Validation

SimpleSearch-VL validates its approach on six multimodal search benchmarks, where the 8B and 30B-A3B variants improve agentic baselines by 15.8 and 16.0 average points. p. 0 The results demonstrate that the training and tool-use design can be more important than increasing model size alone. p. 8 The performance competitive with agentic Gemini-3-Pro for the 30B variant further validates its practical utility. p. 2

Tool Usage Analysis

The training-time tool behavior shows that SFT+RL keeps a longer and more stable search process, with average reasoning rounds changing only from 5.6 to 5.4 between steps 10 and 100. p. 8 The sample-level image-search ratio in Fig. p. 6 9a shows that Direct RL increases image-search invocation on all three reported benchmarks, from 59. p. 71% to 84.2% on MMS, 33. p. 80% to 74.0% on BC-VL, and 74.0% to 93.0 on FVQA. This suggests that SFT provides a stronger tool-use prior, whereas Direct RL more readily expands reverse image search as a broadly useful evidence-gathering action. The inference rounds and Pass@1 scores show that SimpleSearch-VL uses about seven rounds on the harder MMSearch+ benchmark and about four rounds on MMSearch, indicating it adjusts its search depth to the task difficulty.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Factorized Adaptive Rollout (FAR) improves sampling efficiency by forming more informative training groups while using redundant samples to mitigate long-tail latency and expose hard samples. This allows agents to spend extra attempts on hard groups that lack reward contrast, speeding up convergence without increasing system complexity.

  2. Evidence verification is made explicit by using a thumbnail-aware reverse image search, enabling the agent to check whether retrieved evidence matches the queried visual content before using it. This prevents reliance on visually similar but incorrect candidates by comparing thumbnails with the queried region.

  3. SimpleSearch-VL replaces external summarization with a self-summary mechanism that allows webpage understanding inside the agent while reducing tool orchestration and model dependencies. This keeps webpage understanding local to the agent's generation process, avoiding additional model deployments.

  4. The framework is trained using only 5K supervised tool-interleaved trajectories and 2K RL data, demonstrating that improvements come from improving the efficiency of rollout allocation, making visual evidence explicitly verifiable, and keeping webpage summarization inside the agent itself rather than scaling data or tools.

  5. The system can adapt its search depth to task difficulty, as it uses about seven rounds on the harder MMSearch+ benchmark and about four rounds on MMSearch, indicating an adaptive reasoning process rather than a fixed search template.

Sources

Related papers