Locally Deployable Small Language Models for Emergency Department Decision Support: A Systematic Benchmark of Fine-Tuning Strategies

Deploying large language models (LLMs) for decision support in emergency departments (EDs) faces two major challenges: privacy risks of transmitting patient data to closed-source commercial LLMs and the lack of systematic evaluation of fine-tuning strategies for locally deployable open-source small language models (SLMs). We benchmarked eight open-source SLMs using zero-shot prompting, prefix tuning, Low-Rank Adaptation (LoRA), and full fine-tuning on three ED tasks: triage level prediction, specialist referral recommendation, and diagnosis prediction. Using 2,083 MIMIC-IV-ED cases and Claude Haiku 4.5 and Claude Sonnet 4.5 as baselines, we found that LoRA fine-tuned open-source SLMs outperform commercial baselines on triage level prediction and specialist referral recommendation, while diagnosis prediction remains challenging for open-source SLMs. Confusion matrix analysis further shows that fine-tuned open-source SLMs can detect highest-severity patients missed by the commercial baselines. These results demonstrate that locally deployable SLMs can achieve clinically competitive performance for ED decision support.

Improvements to AI Systems:

1. Privacy-Preserving Clinical Triage System: Replace reliance on closed-source commercial LLMs with LoRA fine-tuned open-source SLMs (e.g., 7B-13B parameters) that run entirely on local hospital infrastructure. This eliminates transmission of protected health information (PHI) to external servers, ensuring HIPAA/GDPR compliance while maintaining or exceeding baseline accuracy.

2. High-Severity Patient Detection Enhancer: Integrate the fine-tuned SLM’s confusion matrix insights to build an ensemble alert system. The SLM specifically catches highest-severity patients (e.g., ESI level 1) that commercial baselines (Claude Haiku/Sonnet) miss. The improved system can trigger immediate human review for these flagged cases, reducing missed critical admissions.

3. Task-Specific Specialist Referral Optimizer: Deploy the LoRA fine-tuned model as a dedicated referral recommender, achieving higher precision than Claude Sonnet 4.5. The system can output ranked specialist suggestions with confidence scores, and—when combined with triage prediction—provide a unified decision-support dashboard for ED physicians, reducing cognitive load and referral errors.

4. Adaptive Fine-Tuning Pipeline for Resource-Limited EDs: Implement a reusable training framework that takes any open-source SLM and fine-tunes it with LoRA on local ED data (e.g., 2,083 MIMIC-IV-ED cases). The improved system can automatically update itself with new patient outcomes, continuously improving triage and referral accuracy without cloud dependency.

5. Diagnosis Prediction Fallback Mechanism: Since diagnosis prediction remains weak for open-source SLMs, the improved system uses a hybrid approach: the fine-tuned SLM handles triage and referral, while a separate rule-based or retrieval-augmented module (or a larger local model) handles diagnosis. This prevents over-reliance on the SLM for tasks where it underperforms, ensuring clinical safety.

What the Improved AI System Can Do:

• Operate fully offline in an ED, processing patient vitals, symptoms, and history to output: (a) triage level (e.g., ESI 1-5), (b) top-3 specialist referrals with rationale, and (c) a provisional diagnosis list (with a warning if confidence is low).

• Flag highest-severity patients with higher recall than commercial LLMs, enabling earlier intervention.

• Adapt to a specific hospital’s patient population via local fine-tuning, improving over time.

• Provide explainable outputs (via attention weights or LoRA adapter analysis) for clinician trust.

• Reduce operational costs and latency by avoiding API calls to external LLMs, while ensuring data never leaves the hospital network.