Listwise Cross-Encoder Fine-Tuning vs. Agentic Instruction Tuning for LLM Rerankers: A Systematic Study in Medical Procedure Reranking

This paper presents a systematic comparison of two reranking paradigms for medical procedure alignment in a health-insurance information retrieval system: (1) small cross-encoders (MedCPT, MiniLM-L12) fine-tuned with listwise learning-to-rank objectives across layer-freezing configurations, and (2) Qwen3-Reranker-4B, a 4B-parameter instruction reranker whose prompt is iteratively refined via an agentic optimization loop driven by GPT-4.1.

The task is defined as: "Given a user query q describing a medical need and a set of n candidate procedures P = p1,..., pn, the task is to learn a scoring function f (q, pi) → R that induces a ranking π over P such that procedures most clinically relevant to q are ranked highest. The task is complicated by a significant lexical gap: patients describe needs using layman terms and symptom descriptions (e.g., 'my knee hurts when I walk'), while procedures are catalogued in clinical nomenclature (e.g., 'arthrocentesis of the knee joint')."

The authors construct a purpose-built dataset using a two-phase LLM synthesis pipeline: Phase 1 generates queries using GPT-4o with temperature 0.7, with queries diversified along intent type (coverage / symptom / direct request), formality, and synonym usage; Phase 2 uses a separate GPT-4o call (temperature 0.0) to rank all candidate procedures for each query. Quality filtering retains only rows where the teacher model places the target procedure within the top 3 positions, yielding a final dataset of 2,647 query-procedure lists across 708 insurance services. Human validation showed Expert A accepted the top-1 placement in 92% of cases, Expert B in 97%. Top-3 acceptance rates were 92% and 97% respectively. Raw inter-annotator agreement was 95% (Cohen's κ = 0.52).

The methodology includes fine-tuning two cross-encoder backbones (MedCPT-Cross-Encoder and MiniLM-L-12-v2) with three listwise loss functions (LambdaLoss, ListNet, PListMLE) across three layer-freezing strategies (no freeze, freeze embeddings, freeze 6 layers), comprising an 18-cell experiment grid. For the LLM baseline, the authors propose an agentic prompt optimization framework using GPT-4.1 to iteratively refine the task instruction for Qwen3-Reranker-4B, starting from a domain-agnostic instruction Rank the candidates by relevance to the query. The optimizer converged within 3 iterations and terminated at iteration 9 via early stopping, discovering the optimized instruction: Order the procedures from most to least relevant based on how thoroughly each aligns with the stated symptoms, diagnostic needs, or procedural requests in the query.

Key results: The best configuration, MedCPT with ListNet and no freezing, achieves 0.961 NDCG@3 (±0.002), a +4.6pp improvement over the base model MedCPT baseline. The largest gains appear on Spearman correlation (+19.4pp), indicating that listwise training particularly improves full-list ordering beyond the top position.

In the cross-encoder vs. LLM comparison: The best fine-tuned MedCPT (ListNet, no freeze) outperforms the prompt-optimized 4B-parameter Qwen3-Reranker on all metrics: +1.6pp on NDCG@1, +2.6pp on NDCG@3, and +13.3pp on Spearman - while using 37× fewer parameters. Even the weakest fine-tuned configuration leads Qwen3 by +2.1pp on NDCG@3. The paper notes "Qwen3's Spearman correlation (0.646) is notably closer to the MedCPT baseline (0.585) than to any fine-tuned model (≥0.748), suggesting that prompt optimization alone cannot match the full-list ordering calibration that listwise training provides."

Practical takeaways include: fine-tuned cross-encoders as production rerankers (109M-parameter models can be served on commodity CPU hardware, whereas 4B-parameter models require dedicated GPUs); scalable dataset construction via LLM synthesis; and agentic prompt optimization as a principled way to establish strong LLM baselines.

Limitations acknowledged: the dataset is constructed from a single organization's service-to-procedure mappings; relevance rankings are produced by GPT-4o as a teacher model, so any systematic biases of this teacher are inherited by the training labels; only two cross-encoder backbones and one instruction reranker are evaluated; and the freezing strategies did not explore freezing upper layers while keeping embeddings trainable.

The conclusion states: "we found that 109M-parameter MedCPT models fine-tuned with listwise objectives outperform a prompt-optimized 4B-parameter Qwen3-Reranker by 2.6pp on NDCG@3 and 13.3pp on Spearman correlation, with tight seed variance (σ̂ < 0.003), despite using 37× fewer parameters."

Improvements to AI Systems:

1. Hybrid Reranking Architecture – Combine a fine-tuned 109M cross-encoder (MedCPT + ListNet) as the primary reranker with a lightweight LLM (e.g., 1–3B) used only for edge cases where lexical overlap is low (e.g., symptom-to-procedure mismatch). This yields high NDCG@3 (0.961) while reducing GPU dependency by 90% compared to a 4B reranker.

2. Listwise-Aware Instruction Tuning – Instead of prompt optimization alone, fine-tune a small LLM (e.g., 1B) with listwise loss (ListNet/PListMLE) on the synthesized dataset. This bridges the gap between prompt-optimized LLMs (Spearman 0.646) and cross-encoders (≥0.748), improving full-list ordering without needing a 4B model.

3. Agentic Prompt Optimization as a Pre-Training Step – Use the GPT-4.1-driven iterative prompt refinement (converging in 3 iterations) to generate domain-specific instructions for any reranker. Apply this optimized instruction (e.g., Order procedures by alignment with symptoms, diagnostic needs, or procedural requests) as a universal prefix for zero-shot LLM reranking in medical IR, reducing hallucination and improving relevance.

4. Layer-Freezing as a Regularization Strategy – Adopt the finding that no-freezing yields best results (0.961 NDCG@3) but use freezing of lower layers (embeddings) as a regularization technique when training data is scarce or noisy, preventing overfitting to teacher biases. This enables robust fine-tuning on smaller datasets.

5. Teacher-Bias Mitigation via Multi-Teacher Ensemble – Since GPT-4o labels inherit systematic biases, train the cross-encoder on an ensemble of rankings from GPT-4o and a second teacher (e.g., GPT-4.1 or a clinical ontology-based scorer). Use the top-3 agreement filter (92–97% human acceptance) as a confidence threshold to discard ambiguous labels, improving label quality.

6. Scalable Synthetic Data Generation for Niche Domains – Replicate the two-phase LLM synthesis (temperature 0.7 for query diversity, 0.0 for ranking) to create training data for other low-resource domains (e.g., legal, finance, technical support). The 2,647-list dataset with 708 services demonstrates feasibility for domains lacking human-annotated relevance data.

7. Deployment-Aware Model Selection – Use the paper’s cost-performance trade-off to build an auto-selector: if serving on CPU-only hardware, choose MedCPT+ListNet (0.961 NDCG@3); if GPU is available and full-list ordering is critical (e.g., for recommendation), use a fine-tuned cross-encoder with Spearman >0.75; avoid 4B LLMs unless zero-shot generalization across unseen domains is required.

What the Improved AI System Can Do:

• Medical Procedure Search in Insurance Portals: Accurately rank procedures for layperson queries (e.g., knee pain when walking → arthrocentesis) with 96.1% NDCG@3, running on commodity CPUs, enabling real-time deployment in resource-constrained health systems.

• Cross-Domain Reranking with Minimal Data: Generate high-quality training lists for any specialized vocabulary (legal clauses, financial products) using the LLM synthesis pipeline, then fine-tune a small cross-encoder to achieve near-human relevance (92–97% top-3 acceptance).

• Zero-Shot LLM Reranking with Optimized Prompts: Use the agentically discovered instruction to improve any off-the-shelf LLM’s ranking quality by +13.3pp Spearman over naive prompts, without additional fine-tuning.

• Robust Ranking Under Noisy Labels: Apply layer-freezing and multi-teacher filtering to maintain performance even when training data is synthetically generated, reducing risk of propagating teacher biases.

• Adaptive Deployment: Automatically choose between CPU-friendly cross-encoders and GPU-based LLMs based on latency, cost, and required ranking granularity, optimizing for both accuracy and operational efficiency.

Reranking medical procedures against patient queries is a critical component of health insurance information retrieval, complicated by a substantial lexical gap between patient language and clinical nomenclature. We present a systematic comparison of two reranking paradigms for this production task: (1) small cross-encoders (MedCPT, MiniLM-L12) fine-tuned with listwise learning-to-rank objectives across layer freezing configurations, and (2) Qwen3-Reranker-4B, a 4B-parameter instruction reranker whose prompt is iteratively refined via an agentic optimization loop driven by GPT-4.1. On a purpose-built dataset of 2,647 queries across 708 insurance services, we find that a 109M-parameter cross-encoder fine-tuned with ListNet outperforms the 4B-parameter model by 2.6 percentage points on NDCG@3 and 13.3 points on Spearman correlation - at 37x fewer parameters. We report practical findings, a scalable LLM based dataset construction pipeline, and deployment trade-offs relevant to production reranking systems. We release our code and a sample dataset to support reproducibility and adaptation to other domains.