Train the Agent, Not the Expert: Learning to Harness Heterogeneous Experts for Multi-Turn Visual Reasoning

summary

The gist

VisHarness is a trainable visual agent that learns to harness heterogeneous visual experts for multi-turn interactions…

In short

VisHarness is a trainable visual agent that learns to use different specialized visual models as experts for complex tasks. It separates high-level reasoning from low-level execution, allowing the agent to preserve general intelligence while using the precision of task-specific models. It uses dynamic memory archiving to efficiently learn this expert harnessing policy through reinforcement learning.

Key concepts

Decoupling Perception and Execution
This design separates the agent's high-level thinking (perception, reasoning, decision-making) from the actual low-level visual work (execution). The agent learns *how* to think and decide which tool to use, while specialized models handle the precise pixel-level tasks. This keeps general intelligence intact while gaining expert accuracy.
Harnessing Heterogeneous Experts
Instead of training one model for every specific task, VisHarness learns a policy to coordinate a set of diverse visual experts. These experts include fundamental vision models and auxiliary tools. The agent dynamically selects and combines their outputs across multiple turns to solve complex problems.
Dynamic Visual Memory Archiving Mechanism
During training, tracking all visual history creates too much memory overhead for reinforcement learning. This mechanism solves this by only keeping the results from the most recent expert call while archiving older memories dynamically. This reduces context load while still allowing the agent to maintain necessary historical context for reasoning.
Multi-turn Interaction and Self-Correction
VisHarness operates in turns, where it reasons based on its perception and then decides whether to refine a previous result or call a new expert. This iterative process allows the agent to self-correct errors by using feedback from visual experts, enabling it to handle complex instructions requiring detailed reasoning.

Terminology used across episodes

This episode discusses

The paper

Train the Agent, Not the Expert: Learning to Harness Heterogeneous Experts for Multi-Turn Visual Reasoning · Read on arXiv

Yaowu Fan, Tao Han, Dazhao Du, Andy J. Ma, Jia Wan

Sun Yat-sen University · HKUST · Harbin Institute of Technology

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Train the Agent, Not the Expert".

Jane: Here's the gist of the paper. VisHarness is a trainable visual agent that learns to harness heterogeneous visual experts for multi-turn interactions, decoupling high-level perception and reasoning from low-level execution to solve complex visual tasks <ref:2605.29894#pg11>.

Tom: This approach allows the agent to preserve general intelligence while leveraging specialized models' precision in concrete tasks <ref:2605.29894#pg4>.

Jane: Decoupling Perception and Execution The core design principle of VisHarness is to decouple the perception, reasoning, and decision-making abilities of a high-level agent from the execution of low-level vision tasks <ref:2605.29894#pg3>.

Tom: Specifically, the agent learns general perception, reasoning, and decision-making abilities while delegating the execution of specific vision tasks to specialized visual models.

Title and authors: Tom: The paper is titled "Train the Agent, Not the Expert: Learning to Harness Heterogeneous Experts for Multi-Turn Visual Reasoning," and it's written by Yaowu Fan, Tao Han, Dazhao Du, Andy J. Ma, and Jia Wan from Sun Yat-sen University and HKUST.

Jane: That’s a pretty standard set of authors for this kind of work in vision and language research. The title really captures the essence: focusing on how the agent learns to use those experts rather than just training one giant model to solve the whole problem at once.

Lu: What I find interesting is that they are tackling something that existing methods struggle with, which is combining multiple capabilities—like first understanding instructions, then localizing something, then segmenting it, and finally reasoning based on all of that intermediate information.

Meng: It sounds like they're addressing the limitation where multimodal large language models are good at understanding but often lack the precision for actual pixel-level execution tasks like accurate segmentation or counting.

Tom: They frame this as a way to use MLLMs for high-level perception and reasoning, but then delegate the low-level visual execution to these specialized experts, which is where the precision comes from.

Jane: So instead of one giant model trying to do everything poorly, you have a general agent that decides which expert does what at each step, which makes sense if you think about how complex real-world vision tasks are.

The paper's summary: Tom: VisHarness learns this harnessing policy through multi-turn interactions where it uses both the image and text to make decisions, ultimately producing results denoted as (Bj, Mj). It's about a high-level agent making the calls.

Jane: So, it’s not just one shot; it’s an ongoing conversation between the agent and these visual experts until the task is done. The key here is that at each turn, VisHarness perceives the current image and text, reasons to make a decision, and then decides whether to refine a previous result or call a new expert.

Lu: The paper introduces this system built upon six experts: three fundamental visual models for executing specific vision tasks and three auxiliary tools designed to extend the capabilities of those base models. That's the set they are learning to harness.

Meng: From an engineering standpoint, that sounds like a lot of components to manage in a live system. How do you keep track of all those expert inputs and outputs while the agent is reasoning?

Tom: That's where they introduce this dynamic visual memory archiving mechanism, which is crucial because it deals with the memory overhead from these continuous interactions during reinforcement learning training.

Jane: It’s smart because it says that instead of keeping every single thing the agent has seen or done in its entire history, it dynamically archives outdated visual memories while only keeping the results from the most recent expert.

The paper's improvements: Tom: The major improvement they highlight is how this setup allows VisHarness to solve a variety of fundamental vision tasks naturally, like Generalized Referring Expression Segmentation and reasoning segmentation.

Jane: They show that because of this decoupling, VisHarness can handle complex tasks that are too difficult for a single visual expert model to solve on its own, especially when those tasks involve dense small-object perception.

Lu: The training setup is also quite efficient; they train this generalizable harnessing policy with only a small amount of data and through on-policy reinforcement learning in a live environment where the visual experts are already deployed.

Meng: On the deployment side, I worry about that memory archiving thing you mentioned earlier. If it’s constantly changing what it archives, how stable is that during actual deployment at scale? What happens if the reasoning chain gets too long?

Tom: They addressed that by decomposing the multi-turn Markov decision process into multiple single-turn Markov decision processes to keep things consistent between training and inference. That's a neat trick for making the reinforcement learning setup work.

Conclusion: Jane: So, to wrap up, VisHarness is this trainable visual agent that decouples high-level perception and reasoning from the low-level execution of vision tasks by harnessing heterogeneous experts.

Tom: And they’ve introduced this dynamic visual memory archiving mechanism to keep the training efficient by reducing context overhead during those multi-turn interactions.

Lu: The results on benchmarks like Generalized Referring Expression Segmentation, where they only used about zero point seven percent of the gRefCOCO training set, show that VisHarness performs comparably to or even better than task-specific models.

Meng: I still see the challenge in scaling that live environment; it’s good for learning a policy, but deploying something with this kind of dynamic memory management needs very solid testing against real traffic.

Lalam: From my perspective, based on the literature I've processed, VisHarness successfully adapts and refines expert outputs through multi-turn reasoning to solve complex visual tasks involving dense small-object perception and language understanding.

Jane: It sounds like they’ve provided a powerful framework for building general-purpose visual intelligence by letting an agent adaptively select and coordinate different visual tools based on what the current situation demands.

Tom: And the quantitative experiments, like showing it substantially increasing the use of SplitImageIntoPatches and SuperResolution on the Dense200 dataset, give concrete evidence that this approach leads to better performance than other models.

Lu: Overall, this paper shows a path for building agents that can tackle complex visual challenges by focusing on learning a harness policy over diverse experts rather than just training one monolithic expert.

More episodes

← Home