ActiveMedAgent: Cost-Aware Trajectory Learning for Multimodal Medical Diagnosis
cs.LG, cs.AI, cs.CL, cs.CV, cs.MM
Submitted: 2026-10-08
Updated: 2026-10-08
Comments: EMNLP 2026
Code: https://github.com/VisualReasoner/activeagent
License: http://creativecommons.org/licenses/by/4.0/
The gist: Clinical diagnosis is inherently sequential: clinicians escalate from cheap to costly tests only when additional evidence is expected to resolve diagnostic uncertainty.
Terminology
Abstract
Clinical diagnosis is inherently sequential: clinicians escalate from cheap to costly tests only when additional evidence is expected to resolve diagnostic uncertainty. We present ActiveMedAgent, a framework that brings this cost-aware sequential logic to multimodal medical AI. Given a frozen, API-accessed vision-language model, ActiveMedAgent tracks probability distributions over candidate diagnoses and scores each acquisition by its per-step diagnostic utility minus cost. A lightweight MLP controller is then trained offline on these scored trajectories, learning when to request additional evidence and when to commit. Across three commonly used benchmarks, trajectory-based policy learning consistently outperforms both unguided acquisition and full-modality baselines. Notably, we identify an information overload effect. In 175 cases, the agent produces a correct diagnosis with fewer channels while the full-modality baseline fails, showing that learning what to omit can be as important as learning what to acquire.
Sources
- GPT-4 Technical Report
- ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?
- BIRD: A Trustworthy Bayesian Inference Framework for Large Language Models
- MedPAIR: Measuring Physicians and AI Relevance Alignment in Medical Question Answering
- Measuring Massive Multitask Language Understanding
- GPT-4o System Card
- Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?
- Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens
- Extracting Probabilistic Knowledge from Large Language Models for Bayesian Network Parameterization
- Sequential Diagnosis with Language Models
- Capabilities of GPT-4 on Medical Challenge Problems
- What Are the Odds? Language Models Are Capable of Probabilistic Reasoning
- Reflexion: Language Agents with Verbal Reinforcement Learning
- Disentangling Reasoning and Knowledge in Medical Large Language Models
- LLaMA: Open and Efficient Foundation Language Models
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks