Conditional Rank Allocation for Taxonomy-Aware Medical Language Model Adaptation
cs.AI
Submitted: 2026-10-05
Updated: 2026-10-05
Comments: Accepted at IEEE BIBM 2026
License: http://creativecommons.org/licenses/by/4.0/
The gist: Medical question answering spans specialties and clinical operations that may benefit from different adaptation directions.
Terminology
Abstract
Medical question answering spans specialties and clinical operations that may benefit from different adaptation directions. We propose ARBOR, a parameter-efficient method that selects rank-one components from a shared low-rank basis for each question. An additive gate combines question representations, specialty tags, operation tags, and their interaction; a learned coefficient scales the adapter residual. An illustrative separation under orthogonal, equiprobable subtasks shows how conditional selection can avoid an approximation floor faced by a fixed update with the same active rank. This result motivates the design without asserting a corresponding bound for medical corpora. On Qwen3-8B across CMB, CMExam, MedQA, and MedMCQA, five-seed experiments yield 69.69% mean accuracy across benchmarks, exceeding LoRA r16 and MoELoRA by 1.26 and 1.30 percentage points, respectively. The reported advantage over LoRA r16 increases from 0.08 to 1.94 points as training expands from one to seven specialties. Tag perturbations and atom masking support the usefulness of clinical routing, while atom clusters align with the supplied specialty labels (adjusted Rand index 0.62). Calibration, transfer, and measured costs further characterize the method. These findings support structured conditional adaptation for medical QA, while leaving clinical safety and broader deployment untested.
Sources
- MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Models
- Aligning LLMs with Biomedical Knowledge using Balanced Fine-Tuning
- MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models
- LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routing
- MedAdapter: Efficient Test-Time Adaptation of Large Language Models towards Medical Reasoning
- Enhancing Biomedical Named Entity Recognition via Multiple Programming Languages Instruction Tuning and Ensemble Method
- A Guideline-Augmented Multi-Agent Framework for Schema-as-Code Biomedical Named Entity Recognition
- Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Qwen3 Technical Report
- CMB: A Comprehensive Medical Benchmark in Chinese
- Dual-Pathway Circuits of Object Hallucination in Vision-Language Models
- The Llama 3 Herd of Models
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems