Train the Agent, Not the Expert: Learning to Harness Heterogeneous Experts for Multi-Turn Visual Reasoning
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Train the Agent, Not the Expert".
Jane: Here's the gist of the paper. VisHarness is a trainable visual agent that learns to harness heterogeneous visual experts for multi-turn interactions, decoupling high-level perception and reasoning from low-level execution to solve complex visual tasks <ref:2605.29894#pg11>.
Tom: This approach allows the agent to preserve general intelligence while leveraging specialized models' precision in concrete tasks <ref:2605.29894#pg4>.
Jane: Decoupling Perception and Execution The core design principle of VisHarness is to decouple the perception, reasoning, and decision-making abilities of a high-level agent from the execution of low-level vision tasks <ref:2605.29894#pg3>.
Tom: Specifically, the agent learns general perception, reasoning, and decision-making abilities while delegating the execution of specific vision tasks to specialized visual models.
Title and authors: Tom: The paper is titled "Train the Agent, Not the Expert: Learning to Harness Heterogeneous Experts for Multi-Turn Visual Reasoning," and it's written by Yaowu Fan, Tao Han, Dazhao Du, Andy J. Ma, and Jia Wan from Sun Yat-sen University and HKUST.
Jane: That’s a pretty standard set of authors for this kind of work in vision and language research. The title really captures the essence: focusing on how the agent learns to use those experts rather than just training one giant model to solve the whole problem at once.
Lu: What I find interesting is that they are tackling something that existing methods struggle with, which is combining multiple capabilities—like first understanding instructions, then localizing something, then segmenting it, and finally reasoning based on all of that intermediate information.
Meng: It sounds like they're addressing the limitation where multimodal large language models are good at understanding but often lack the precision for actual pixel-level execution tasks like accurate segmentation or counting.
Tom: They frame this as a way to use MLLMs for high-level perception and reasoning, but then delegate the low-level visual execution to these specialized experts, which is where the precision comes from.
Jane: So instead of one giant model trying to do everything poorly, you have a general agent that decides which expert does what at each step, which makes sense if you think about how complex real-world vision tasks are.
The paper's summary: Tom: VisHarness learns this harnessing policy through multi-turn interactions where it uses both the image and text to make decisions, ultimately producing results denoted as (Bj, Mj). It's about a high-level agent making the calls.
Jane: So, it’s not just one shot; it’s an ongoing conversation between the agent and these visual experts until the task is done. The key here is that at each turn, VisHarness perceives the current image and text, reasons to make a decision, and then decides whether to refine a previous result or call a new expert.
Lu: The paper introduces this system built upon six experts: three fundamental visual models for executing specific vision tasks and three auxiliary tools designed to extend the capabilities of those base models. That's the set they are learning to harness.
Meng: From an engineering standpoint, that sounds like a lot of components to manage in a live system. How do you keep track of all those expert inputs and outputs while the agent is reasoning?
Tom: That's where they introduce this dynamic visual memory archiving mechanism, which is crucial because it deals with the memory overhead from these continuous interactions during reinforcement learning training.
Jane: It’s smart because it says that instead of keeping every single thing the agent has seen or done in its entire history, it dynamically archives outdated visual memories while only keeping the results from the most recent expert.
The paper's improvements: Tom: The major improvement they highlight is how this setup allows VisHarness to solve a variety of fundamental vision tasks naturally, like Generalized Referring Expression Segmentation and reasoning segmentation.
Jane: They show that because of this decoupling, VisHarness can handle complex tasks that are too difficult for a single visual expert model to solve on its own, especially when those tasks involve dense small-object perception.
Lu: The training setup is also quite efficient; they train this generalizable harnessing policy with only a small amount of data and through on-policy reinforcement learning in a live environment where the visual experts are already deployed.
Meng: On the deployment side, I worry about that memory archiving thing you mentioned earlier. If it’s constantly changing what it archives, how stable is that during actual deployment at scale? What happens if the reasoning chain gets too long?
Tom: They addressed that by decomposing the multi-turn Markov decision process into multiple single-turn Markov decision processes to keep things consistent between training and inference. That's a neat trick for making the reinforcement learning setup work.
Conclusion: Jane: So, to wrap up, VisHarness is this trainable visual agent that decouples high-level perception and reasoning from the low-level execution of vision tasks by harnessing heterogeneous experts.
Tom: And they’ve introduced this dynamic visual memory archiving mechanism to keep the training efficient by reducing context overhead during those multi-turn interactions.
Lu: The results on benchmarks like Generalized Referring Expression Segmentation, where they only used about zero point seven percent of the gRefCOCO training set, show that VisHarness performs comparably to or even better than task-specific models.
Meng: I still see the challenge in scaling that live environment; it’s good for learning a policy, but deploying something with this kind of dynamic memory management needs very solid testing against real traffic.
Lalam: From my perspective, based on the literature I've processed, VisHarness successfully adapts and refines expert outputs through multi-turn reasoning to solve complex visual tasks involving dense small-object perception and language understanding.
Jane: It sounds like they’ve provided a powerful framework for building general-purpose visual intelligence by letting an agent adaptively select and coordinate different visual tools based on what the current situation demands.
Tom: And the quantitative experiments, like showing it substantially increasing the use of SplitImageIntoPatches and SuperResolution on the Dense200 dataset, give concrete evidence that this approach leads to better performance than other models.
Lu: Overall, this paper shows a path for building agents that can tackle complex visual challenges by focusing on learning a harness policy over diverse experts rather than just training one monolithic expert.
Yaowu Fan, Tao Han, Dazhao Du, Andy J. Ma, Jia Wan
Sun Yat-sen University · HKUST · Harbin Institute of Technology
cs.CV
Submitted: 2026-05-28
Updated: 2026-10-29
Comments: This work has been accepted to NeurIPS 2026
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 100/100
The gist: VisHarness is a trainable visual agent that learns to harness heterogeneous visual experts for multi-turn interactions…
Key concepts
- Decoupling Perception and Execution
- This design separates the agent's high-level thinking (perception, reasoning, decision-making) from the actual low-level visual work (execution). The agent learns *how* to think and decide which tool to use, while specialized models handle the precise pixel-level tasks. This keeps general intelligence intact while gaining expert accuracy.
- Harnessing Heterogeneous Experts
- Instead of training one model for every specific task, VisHarness learns a policy to coordinate a set of diverse visual experts. These experts include fundamental vision models and auxiliary tools. The agent dynamically selects and combines their outputs across multiple turns to solve complex problems.
- Dynamic Visual Memory Archiving Mechanism
- During training, tracking all visual history creates too much memory overhead for reinforcement learning. This mechanism solves this by only keeping the results from the most recent expert call while archiving older memories dynamically. This reduces context load while still allowing the agent to maintain necessary historical context for reasoning.
- Multi-turn Interaction and Self-Correction
- VisHarness operates in turns, where it reasons based on its perception and then decides whether to refine a previous result or call a new expert. This iterative process allows the agent to self-correct errors by using feedback from visual experts, enabling it to handle complex instructions requiring detailed reasoning.
Terminology
Summary
VisHarness is a trainable visual agent that learns to harness heterogeneous visual experts for multi-turn interactions, decoupling high-level perception and reasoning from low-level execution to solve complex visual tasks <ref:2605. p. 829894#pg11>. This approach allows the agent to preserve general intelligence while leveraging specialized models' precision in concrete tasks <ref:2605.29894#pg4>.
Decoupling Perception and Execution
The core design principle of VisHarness is to decouple the perception, reasoning, and decision-making abilities of a high-level agent from the execution of low-level vision tasks <ref:2605. p. 129894#pg3>. Specifically, the agent learns general perception, reasoning, and decision-making abilities while delegating the execution of specific vision tasks to specialized visual models. p. 1 This design preserves the general intelligence of the agent while fully leveraging the precision advantages of specialized visual models in concrete visual tasks. p. 1
Harnessing Heterogeneous Experts
VisHarness learns to harness a set of carefully designed heterogeneous visual experts instead of training a model to solve a specific task <ref:2605.29894#pg11>. The agent performs multi-turn reasoning based on both the image and text, ultimately producing results denoted as (Bj, Mj). p. 3 The heterogeneous visual expert suite consists of six experts, including three fundamental visual models for executing specific vision tasks and three auxiliary visual tools for extending the capabilities of these models. p. 4
Dynamic Memory Archiving Mechanism
To enable VisHarness to learn a generalizable visual expert-harnessing policy, it is trained with on-policy reinforcement learning in a live environment where visual expert models are deployed. This introduces substantial memory and context overhead during multi-turn interactions with visual experts. p. 2 To mitigate this, the dynamic visual memory archiving mechanism dynamically archives outdated visual memories while only retaining the visual results produced by the most recent visual expert. p. 2 This mechanism preserves necessary historical information while substantially reducing the context overhead introduced by historical visual memory. p. 2
Performance and Generalization Gains
Experimental results on four representative benchmarks demonstrate that VisHarness substantially outperforms existing general-purpose models and achieves competitive or superior performance compared to task-specific models <ref:2605.29894#pg11>. For Generalized Referring Expression Segmentation, VisHarness is trained with only about 0. p. 17% of the gRefCOCO training set yet achieves results comparable to or even better than those specialized models. p. 6 Furthermore, online reinforcement learning effectively reduces redundant visual tool invocations. p. 2 The superior performance on the Dense200 dataset is attributed to VisHarness substantially increasing the use of SplitImageIntoPatches and SuperResolution compared to other models. p. 7
Conclusion
In conclusion, VisHarness provides a trainable visual agent that learns to harness heterogeneous visual experts for complex fundamental vision tasks. Through multi-turn interactions with visual experts, VisHarness can adaptively select, coordinate, and refine expert outputs to solve visual tasks involving complex language understanding and dense small-object perception. p. 8 The dynamic visual memory archiving mechanism enables efficient reinforcement learning in a live environment with deployed visual experts, further improving the learning of generalizable policies. p. 8 Quantitative experiments show that VisHarness achieves comparable or superior performance to specialized models across tasks <ref:2605. p. 829894#pg11>. The qualitative analysis demonstrates the superiority of VisHarness in adaptively selecting different visual expert-harnessing strategies for different data and correcting errors through multi-turn interactions. p. 8
--- Page 1 ---
Train the Agent, Not the Expert: Learning to Harness Heterogeneous Experts for Multi-Turn Visual Reasoning <ref:2605.29894#pg2>
Abstract Recent progress in computer vision has produced a wide range of powerful specialized models for detection, segmentation, counting, and other visual tasks <ref:2605.29894#pg2>. However, these models are usually optimized for isolated task formulations, making it difficult to directly support general-purpose visual intelligence <ref:2605. p. 029894#pg2>. In this paper, we propose VisHarness, a trainable visual agent that decouples high-level perception, reasoning, and decision-making from low-level task execution <ref:2605. p. 029894#pg2>. Instead of training a model to solve a specific visual task, VisHarness learns to harness a set of carefully designed heterogeneous visual experts <ref:2605. p. 029894#pg2>. This paradigm preserves the general intelligence of the agent while fully leveraging the precision advantages of specialized visual models in concrete visual tasks <ref:2605. p. 029894#pg2>. With only lightweight training, VisHarness learns a generalizable visual expert-harnessing policy and can solve common fundamental vision tasks under various complex conditions through multi-turn interactions with visual expert models <ref:2605. p. 029894#pg2>. To enable efficient on-policy reinforcement learning training in a live environment, we introduce dynamic visual memory archiving, which mitigates the rapidly accumulating visual-token overhead caused by multi-turn interactions with visual expert models <ref:2605. p. 029894#pg2>. Experiments on four representative benchmarks covering reasoning segmentation, generalized referring segmentation, dense small-object detection, and referring counting demonstrate that VisHarness substantially outperforms existing general-purpose models and achieves competitive or superior performance compared to task-specific models <ref:2605. p. 029894#pg2>.
--- Page 2 ---
The Expert-training Era The Agent-training Era Perception, Reasoning and Decision-Making Visual Experts for Execution Task Data Task Data Task Data Detect Net Segment Net Count Loss Seg Loss Count Loss Task-specific Training Duplicate Effort High Cost Limited Generalization Complementary Strengths Scalable Strong Generalization … (a) to solve the entire problem (see Fig. 1(a)) Visual tasks in real-world scenarios usually require the composition of multiple capabilities, such as first understanding the textual instruction, then localizing the target, segmenting the target region, and finally verifying or further reasoning based on intermediate results <ref:2605. p. 129894#pg3>. In recent years, multimodal large language models (MLLMs) have demonstrated strong vision-language understanding and reasoning capabilities, offering new possibilities for building general-purpose visual intelligence <ref:2605. p. 129894#pg2>. However, MLLMs themselves are not always well suited for precise low-level visual execution, such as pixel-level segmentation, dense object detection, or accurate counting <ref:2605. p. 129894#pg2>. Achieving reliable performance on these tasks often requires dedicated architectural modifications and careful design, as well as large-scale and high-quality training data to ensure strong performance on downstream visual tasks <ref:2605. p. 129894#pg2>. On the other hand, although integrating MLLMs with external visual models can further extend their capabilities, existing methods typically rely on manually designed prompts or single-turn decision-making <ref:2605. p. 129894#pg2>. As a result, they struggle to self-correct based on intermediate visual results in complex tasks and have difficulty to effectively select and coordinate among different visual tools <ref:2605. p. 129894#pg2>.
--- Page 3 ---
To address the above issues, we propose VisHarness, a trainable high-level agent for harnessing visual experts <ref:2605.29894#pg2>. Built upon a carefully designed set of heterogeneous visual expert models, VisHarness can learn a generalizable visual expert-harnessing policy from only a small amount of data, enabling it to solve various common fundamental vision tasks <ref:2605. p. 129894#pg2>. The core design principle of VisHarness is to decouple the perception, reasoning, and decision-making abilities of a high-level agent from the execution of low-level vision tasks <ref:2605.29894#pg2>. Specifically, the agent learns general perception, reasoning, and decision-making abilities while delegating the execution of specific vision tasks to specialized visual models <ref:2605. p. 129894#pg2>. This design preserves the general intelligence of the agent while fully leveraging the precision advantages of specialized visual models in concrete visual tasks <ref:2605. p. 129894#pg2>. In contrast, existing methods usually modify the original architecture of MLLMs and train them with large-scale data into specialized models that can only handle a particular type of vision task <ref:2605. p. 129894#pg2>. Based on this design principle, VisHarness can naturally support a variety of fundamental vision tasks, including Generalized Referring Expression Segmentation (GRES), Reasoning Segmentation, and referring expression detection and counting for dense small objects <ref:2605. p. 1 p. 129894#pg2>. Through multi-turn interactions with visual experts, VisHarness can further handle complex visual tasks that are difficult for a single visual expert model to solve <ref:2605. p. 1 p. 129894#pg2>. At each turn, VisHarness first perceives the current image and text content, then reasons to make a decision <ref:2605. p. 1 p. 129894#pg2>. After a specific visual expert model returns the execution result in visualized and textual form, VisHarness can continue reasoning based on this feedback and decide in the next turn whether to refine the previous result or invoke a new visual expert model, until the task is completed <ref:2605. p. 1 p. 129894#pg2>.
--- Page 4 ---
In contrast, existing methods usually modify the original architecture of MLLMs and train them with large-scale data into specialized models that can only handle a particular type of vision task <ref:2605.29894#pg2>. Based on this design principle, VisHarness can naturally support a variety of fundamental vision tasks, including Generalized Referring Expression Segmentation (GRES), Reasoning Segmentation, and referring expression detection and counting for dense small objects <ref:2605.29894#pg2>. Through multi-turn interactions with visual experts, VisHarness can further handle complex visual tasks that are difficult for a single visual expert model to solve <ref:2605.29894#pg2>. At each turn, VisHarness first perceives the current image and text content, then reasons to make a decision <ref:2605.29894#pg2>. After a specific visual expert model returns the execution result in visualized and textual form, VisHarness can continue reasoning based on this feedback and decide in the next turn whether to refine the previous result or invoke a new visual expert model, until the task is completed <ref:2605.29894#pg2>.
--- Page 5 ---
To enable VisHarness to learn a generalizable visual expert-harnessing policy, we train it with on-policy reinforcement learning in a live environment where visual expert models are deployed <ref:2605.29894#pg2>. However, during multi-turn interactions with visual experts, visual feedback continuously accumulates <ref:2605. p. 129894#pg2>. This introduces substantial memory and context overhead, making RL training difficult to conduct <ref:2605. p. 229894#pg2>. To this end, we further introduce a dynamic visual memory archiving mechanism <ref:2605. p. 229894#pg2>. At each reasoning step of VisHarness, this mechanism dynamically archives outdated visual memories, while only the visual results produced by the most recent visual expert is retained <ref:2605. p. 229894#pg2>. This mechanism preserves necessary historical information while substantially reducing the context overhead introduced by historical visual memory <ref:2605. p. 229894#pg2>. Since dynamic visual memory archiving causes the historical memory to change at each turn, we decompose the multi-turn Markov decision process into multiple single-turn Markov decision processes to maintain consistency between training and inference during the stage of RL training <ref:2605. p. 229894#pg2>.
--- Page 6 ---
In summary, our main contributions are as follows: We introduce VisHarness, a trainable visual agent that decouples high-level visual perception, reasoning, and decision-making from low-level visual task execution <ref:2605.29894#pg2>. We propose a dynamic visual memory archiving mechanism which substantially reduces the visual overhead in multi-turn interactions while preserving the coherence of historical memory <ref:2605. p. 229894#pg2>. Experimental results on four benchmarks covering Generalized Referring Expression Segmentation (GRES), Reasoning Segmentation, and referring expression detection and counting for dense small objects show that VisHarness outperforms general-purpose models and achieves comparable or even better performance than specialized models <ref:2605. p. 229894#pg2>. Further analysis demonstrates that online reinforcement learning effectively reduces redundant visual tool invocations <ref:2605. p. 229894#pg2>.
--- Page 7 ---
Experiments on four representative benchmarks covering reasoning segmentation, generalized referring segmentation, dense small-object detection, and referring counting demonstrate that VisHarness substantially outperforms existing general-purpose models and achieves competitive or superior performance compared to task-specific models <ref:2605.29894#pg2>.
--- Page 8 ---
In reasoning segmentation [13], the text does not explicitly specify the object to be segmented in the image. Instead, the model needs to first infer the target object from the implicit textual description <ref:2605. p. 629894#pg2>. The perception–reasoning–decision–execution capability of VisHarness naturally supports this task paradigm: the VLM first infers the target object from the given text and then passes it as an argument to the visual expert model for execution <ref:2605. p. 629894#pg2>. As shown in Table 2, VisHarness significantly outperforms other state-of-the-art models specifically designed for reasoning segmentation <ref:2605. p. 729894#pg2>. In particular, VisHarness clearly outperforms SAM3-Agent [4], which simply combines an MLLM with SAM3 <ref:2605. p. 729894#pg2>.
--- Page 9 ---
PhraseToBoxMask PhraseToPoint PointToBoxMask SplitImageIntoPatches SuperResolution Visual Expert Usage Distribution of Qwen3-VL-8B on Dense200 (a) Visual Expert Usage Distribution of VisHarness on Dense200 (b) Visual Expert Usage Distribution of VisHarness-SFT on gRefCOCO (c) Visual Expert Usage Distribution of VisHarness on gRefCOCO (d) Figure 4: Distribution of visual expert calling by different model variants across different datasets <ref:2605.29894#pg2>.
--- Page 10 ---
Table 1 Generalized Referring Expression Segmentation on gRefCOCO Bold: best. Method Training Data Ratio Validation Set Test Set A Test Set B Avg. p. 6 gIoU cIoU gIoU cIoU gIoU cIoU Avg. p. 6 (a) Traditional Specialized Segmentation Models LTS [10] 100% 52. p. 670 52.30 62.64 61.87 50.42 49.96 <ref:2605. p. 029894#pg2>.
--- Page 11 ---
Table 3 Comparison with MLLMs and proprietary models on the dense-object detection dataset Dense200 Type Method F1@IoU 0.5 F1@IoU 0. p. 595 mIoU mIoU Open-set Grounding DINO-Swin-T [17] 36. p. 79 19.7 33.1 Open-set MLLM DeepSeek-VL2-Tiny 2. p. 72 0.3 1.5 DeepSeek-VL2-Small 16. p. 70 3.9 12.7 Agent Qwen3-VL-8B (w/ tools) 83. p. 73 14.5 mIoU Qwen3-VL-8B (w/ tools) 67. p. 79 VisHarness (Qwen3-VL-8B) 85. p. 75 14.7 mIoU Qwen2. p. 75-VL-7B 29.7 0.4 15.9 DeepSeek-VL2-Small Agent Qwen3-VL-8B (w/ tools) 60. p. 72 10.6 mIoU VisHarness (Qwen3-VL-8B) 85. p. 75 mIoU
--- Page 12 ---
Table 4 Comparison with specialized methods and general-purpose MLLMs on the REC-8K dataset Type Method Training Data Ratio Validation Set Test set MAE↓ RMSE↓ MAE↓ RMSE↓ ZSC (ResNet-50) [37] 14.28 27.75 13.75 25.91 ZSC (Swin-T) [37] 12. p. 796 26.74 13.00 29.07 TFOC [24] 16. p. 708 31.61 17.27 32.68 CounTX [1] 1,888, RMSE↓ MAE↓ RMSE↓ CountGD [2] 9. p. 751 22.91 1,333, CountGD [2] p. 7
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
General purpose visual intelligence through heterogeneous expert harnessing: VisHarness learns
a generalizable visual expert-harnessing policy from only a small amount of data
to solve complex tasks without needing task-specific training for every new scenario. -
Decoupled perception, reasoning, and decision-making: The agent
decouples high-level perception, reasoning, and decision-making from low-level task execution,
preserving general intelligence while leveraging theprecision advantages of specialized visual models in concrete visual tasks.
-
Robust handling of complex language instructions: VisHarness supports tasks where phrase understanding requires complex reasoning, enabling it to handle requests specifying
complex attributes such as relative spatial relationships, object material, and pose or state
through multi-turn interactions. -
Efficient on-policy reinforcement learning: The introduction of
dynamic visual memory archiving
mitigates therapidly accumulating visual-token overhead,
allowing for efficient training in a live environment by only retaining thevisual results produced by the most recent visual expert.
-
Adaptive and error-correcting reasoning: Through multi-turn reasoning, VisHarness can adaptively select and refine expert outputs; for instance, it can
identify and correct this error
when a tool returns an incorrect result, demonstrating its ability to perform self-correction.
Sources
- Open-world Text-specified Object Counting
- Qwen3-VL Technical Report
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Kimi K2.5: Visual Agentic Intelligence
- Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement
- ToolRL: Reward is All Tool Learning Needs
- SAM 2: Segment Anything in Images and Videos
- YOLO26: Key Architectural Enhancements and Performance Benchmarking for Real-Time Object Detection
- OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learning
- UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interface
- Acting Less is Reasoning More! Teaching Model to Act Efficiently
- Zero-Shot Object Counting with Language-Vision Models
- Empowering Segmentation Ability to Multi-modal Large Language Models
- Deformable DETR: Deformable Transformers for End-to-End Object Detection
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs