Balancing Chain-of-Thought and Answer Supervision for Small Language Models on Medical Question Answering
cs.CL, cs.AI
Submitted: 2026-11-02
Updated: 2026-11-02
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Small language models (<=9B parameters) can support local medical question answering (QA) while reducing privacy and latency concerns, but effective fine-tuning remains challenging.
Terminology
Abstract
Small language models (<=9B parameters) can support local medical question answering (QA) while reducing privacy and latency concerns, but effective fine-tuning remains challenging. This paper compares two chain-of-thought (CoT) supervised fine-tuning methods: Uniform-across-Sequence (UaS), which applies equal loss weighting to the full CoT-and-answer sequence, and Segment-weighted (SwS), which separately weights CoT and answer losses. We evaluate six general- and medical-domain models on multiple-choice QA (MedQA and EMS-MCQA) and free-text QA (Med-O1). Results show that neither method is universally best: on multiple-choice QA (MedQA and EMS-MCQA), SwS generally performs better, with general-purpose models benefiting most; notably, Qwen3.5-4B improves by up to 44.8% over its zero-shot baseline on MedQA. In contrast, medical-domain models show smaller or even negative gains on MCQs. On free-text QA (Med-O1), UaS performs best for nearly all models and provides consistent improvements across both model types, with the 3B Llama-3.2 model approaching the performance of much larger 7B medical-domain models. We also find that excessively long CoT generation is associated with lower performance in some models on either multiple-choice or free-text QA tasks, suggesting that limiting the CoT length during training may benefit selected models. Overall, the results show that CoT loss weighting should be matched to the task format and the knowledge already encoded in the base model, rather than using a single fine-tuning strategy for all clinical QA tasks.
Sources
- Capabilities of GPT-4 on Medical Challenge Problems
- Training Verifiers to Solve Math Word Problems
- MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data
- Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning
- The Llama 3 Herd of Models
- HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs
- Decoupled Weight Decay Regularization
- Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMs
- Overclocking LLM Reasoning: Monitoring and Controlling Thinking Path Lengths in LLMs
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data