MedRouter: Demystifying Knowledge Differences Across Medical LLMs for Routing-Based Reasoning
cs.CL, cs.AI
Submitted: 2026-09-27
Updated: 2026-09-27
License: http://creativecommons.org/licenses/by/4.0/
The gist: Medical question answering spans diverse specialties and modalities, and individual medical large language models (LLMs) exhibit distinct strengths across tasks and domains.
Terminology
Abstract
Medical question answering spans diverse specialties and modalities, and individual medical large language models (LLMs) exhibit distinct strengths across tasks and domains. This heterogeneity suggests that combining specialists may enable broader coverage of medical questions than relying on any single model. However, existing LLM routing methods primarily seek to balance answer quality and inference cost, leaving open how to exploit differences in specialist competence to improve medical reasoning. In this paper, we introduce MedRouter, an agentic system that uses an embedding-based multi-label router to select and query specialist LLMs, then passes their responses to a generator to produce the final answer. We further propose SCALE (Specialist Competence-Aware Learning), a two-stage training framework that first trains the Router with specialist correctness supervision and then optimizes its selections through reinforcement learning. The second stage uses a Performance Gain Reward (PGR) that measures how specialist information affects the generator's answer correctness relative to answering without that information. Experiments on eight text-based and multimodal medical QA benchmarks show that MedRouter outperforms the strongest routing baseline by 8% in average accuracy. Our analysis of specialist outputs further reveals distinct strengths and complementary question-level coverage, motivating learned routing to combine these capabilities for more comprehensive medical reasoning.
Sources
- Formula-R1: Incentivizing LLM Reasoning over Complex Tables with Numerical Computation via Formula-Driven Reinforcement Learning
- EHR-RAG: Bridging Long-Horizon Structured Electronic Health Records and Large Language Models via Enhanced Retrieval-Augmented Generation
- GuideSkill: Evolving Executable LLM Agent Skills for Guideline-Grounded Clinical Reasoning
- FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
- Cost-Effective Online Multi-LLM Selection with Versatile Reward Models
- GraphRouter: A Graph-based Router for LLM Selections
- PathVQA: 30000+ Questions for Medical Visual Question Answering
- DeepRetrieval: Hacking Real Search Engines and Retrievers with Large Language Models via Reinforcement Learning
- Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
- What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams
- Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking
- Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement Learning
- GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
- SuperRL: Reinforcement Learning with Supervision to Boost Language Model Reasoning
- Training language models to follow instructions with human feedback
- xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learning
- Proximal Policy Optimization Algorithms
- MedGemma Technical Report
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- MedGuideX: Internalizing Decision Logic from Executable Guidelines into Large Language Models for Clinical Reasoning
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering