ToolVision: Learning When and How to Use Visual Tools with Capability-Aligned Supervision

arXiv:2608.08907 · cs.CV, cs.AI · Submitted 2026-08-09 · Read on arXiv

Delin Mao, Chenghao Sun, Jingwei Song, Chishui Chen, Linfeng Zhang

cs.CV, cs.AI

Submitted: 2026-08-09

Updated: 2026-08-11

Comments: 18 pages, 13 figures

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 44× in this paper · explained in 2 episodes
• Accuracy — 16× in this paper · explained in 1 episode
• Localization — 14× in this paper · explained in 2 episodes
• Planner — 14× in this paper · explained in 3 episodes
• Grounding — 9× in this paper · explained in 3 episodes
• Reinforcement Learning — 9× in this paper · explained in 4 episodes
• Computer Vision and Pattern Recognition — 7× in this paper
• MLLMs — 6× in this paper · explained in 2 episodes
• Verification — 4× in this paper · explained in 2 episodes
• Ablation — 3× in this paper · explained in 1 episode
• Semantic Equivalence — 3× in this paper · explained in 2 episodes
• Transformer — 3× in this paper · explained in 2 episodes
• Transformers — 3× in this paper · explained in 2 episodes
• Vision-language models — 3× in this paper · explained in 2 episodes
• Actor — 2× in this paper · explained in 3 episodes
• CVF International Conference on Computer Vision — 2× in this paper
• Composition — 2× in this paper · explained in 2 episodes
• Core — 2× in this paper · explained in 2 episodes
• Fine-Tuning — 2× in this paper · explained in 5 episodes
• LLM judge — 2× in this paper · explained in 2 episodes
• Pruning — 2× in this paper · explained in 4 episodes
• Winter Conference on Applications of Computer Vision — 2× in this paper
• Clarity — 1× in this paper · explained in 3 episodes
• Consistency — 1× in this paper · explained in 3 episodes
• Evaluation Protocol — 1× in this paper · explained in 2 episodes
• Failure Modes — 1× in this paper · explained in 3 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Foundation Models — 1× in this paper · explained in 8 episodes
• Generator — 1× in this paper · explained in 2 episodes
• Group Sequence Policy Optimization — 1× in this paper
• Group Sequence Policy Optimization (GSPO) — 1× in this paper · explained in 2 episodes
• Harness — 1× in this paper · explained in 2 episodes
• Multimodal Reasoning — 1× in this paper · explained in 5 episodes
• Papers — 1× in this paper · explained in 3 episodes
• Vision Foundation Models — 1× in this paper · explained in 2 episodes
• Visual Grounding — 1× in this paper · explained in 2 episodes

Sources

• Qwen3-VL Technical Report
• Qwen2.5-VL Technical Report
• PaddleOCR 3.0 Technical Report
• Chain-of-Spot: Interactive Reasoning Improves Large Vision-Language Models
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
• Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers
• CountQA: How Well Do MLLMs Count in the Wild?
• Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
• Group Sequence Policy Optimization
• InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models

Related papers

• Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
• AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
• Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
• MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
• TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
• A Survey on Efficient Vision-Language-Action Models