Modular Expert Merging for Biomedical Retrieval

arXiv:2602.04731 · cs.CL, cs.LG · Submitted 2026-02-04 · Read on arXiv

cs.CL, cs.LG

Submitted: 2026-02-04

Updated: 2026-09-02

Comments: Accepted to Findings of EMNLP 2026

License: http://creativecommons.org/licenses/by/4.0/

The gist: Adapting general-purpose LLMs into domain-specialized dense retrievers typically requires large-scale training on mixed-domain data.

Terminology

Abstract

Adapting general-purpose LLMs into domain-specialized dense retrievers typically requires large-scale training on mixed-domain data. We show that merging independently trained domain-specialized experts consistently exceeds this approach across four decoder-only LLM families (0.6B-7B), four merging methods, and twelve medical and general retrieval tasks from MTEB, suggesting that parameter-space composition captures complementary domain strengths that large-scale mixed-domain training averages out. To further maximize expert quality, we introduce Synthesize-Train-Merge (STM), a modular framework that synthesizes hard negatives with a top-tier LLM and fine-tunes domain-specialized experts via LoRA before merging them, without continual pre-training. Synthesized hard negatives yield the largest gains for smaller models, and STM achieves strong performance on biomedical retrieval tasks while maintaining competitive general-domain results across all four backbone families.

Sources

Related papers