When Confidence Fails: Overconfidence in LLMs under Uncertainty and Missing Clinical Information

This paper investigates the reliability of Large Language Models (LLMs) in medical question answering, specifically focusing on their behavior under conditions of clinical uncertainty and missing information. The authors propose an evaluation framework based on the MedMCQA dataset, consisting of two complementary experimental settings: (1) uncertainty framing, where linguistic cues (C0–C3) are injected into prompts to simulate ambiguous clinical contexts, and (2) answer removal, where the correct answer is deliberately excluded and replaced with an explicit abstention option (Insufficient information to determine the correct answer). Five LLMs were evaluated—GPT-5, GPT-4o, GPT-4o-mini, LLaMA-3-8B-Instruct, and Mistral-7B-Instruct-v0.3—across 500 medical questions, using metrics including accuracy, calibration gap, Expected Calibration Error (ECE), and Unsafe Confident Error Rate (UCER).

The results reveal a consistent failure mode: although accuracy degrades under increasing uncertainty, model confidence remains misaligned with accuracy, leading to a substantial increase in unsafe confident errors. Specifically, in the uncertainty framing setting, predictive accuracy remains relatively stable across all conditions (e.g., GPT-5 ranges from 0.82 to 0.84), but calibration behavior differs substantially. GPT-5 is the only model demonstrating comparatively well-calibrated behavior with small negative calibration gaps, while GPT-4o and GPT-4o-mini consistently exhibit overconfidence, and open-weight models show severe miscalibration—Mistral achieves accuracy ≈ 0.50 but maintains confidence close to 1, with the highest ECE and UCER among all models. Importantly, uncertainty framing does not lead to any meaningful reduction in confidence across any model, indicating that LLMs fail to internalize uncertainty signals introduced at the prompt level.

In the answer removal setting, abstention rates remain consistently low across all models. GPT-4o achieves the highest abstention rate of 0.382, followed by GPT-4o-mini with 0.368, while LLaMA abstains in only 15.6% of cases and Mistral in 23.8%. When models fail to abstain, they generate hallucinated answers by selecting incorrect options, with hallucination rates remaining high across all models—GPT-5 exhibits the highest hallucination rate of 0.808, followed by LLaMA with 0.798 and Mistral with 0.762. Critically, these hallucinated predictions are frequently associated with high confidence: LLaMA exhibits the highest unsafe rate of 0.828, Mistral 0.760, and GPT-4o and GPT-4o-mini produce substantial unsafe predictions (0.428 and 0.424, respectively). The analysis also reveals substantial discrepancies between self-reported confidence and logit-based probabilities—for instance, LLaMA reports a very high average confidence of 0.970 while its logit-based confidence remains considerably lower at 0.280, and Mistral reports near-maximal confidence of 0.998 despite only moderate logit probabilities of 0.657.

Statistical analysis using pairwise McNemar tests with Holm–Bonferroni correction confirms that performance differences between proprietary and open-weight models are statistically robust, while differences within each group are less pronounced. The authors conclude that current LLMs, despite strong predictive performance, exhibit fundamental limitations in uncertainty awareness, with confidence estimates not reliably tracking underlying uncertainty. They highlight that prompt-based linguistic uncertainty framing does not meaningfully improve model behavior, and that models remain biased toward producing answers even in the absence of sufficient evidence. The paper emphasizes that predictive accuracy alone is insufficient as a reliability metric for medical AI systems, and future work should focus on developing calibration-aware training objectives, incorporating explicit abstention mechanisms, and designing benchmarks that systematically evaluate model behavior under incomplete or ambiguous clinical information.

Improvements to AI Systems:

1. Implement Uncertainty-Aware Training Objectives: Train models with calibration-aware loss functions that penalize overconfidence on ambiguous inputs, directly optimizing for alignment between confidence and accuracy (e.g., using focal calibration or temperature scaling during fine-tuning).

2. Integrate Explicit Abstention Mechanisms: Add a dedicated abstain output token or decision layer that the model can trigger when evidence is insufficient, trained via reinforcement learning or supervised learning on datasets with deliberately missing answers.

3. Develop Confidence Calibration Post-Processing: Apply post-hoc calibration techniques (e.g., Platt scaling, isotonic regression) specifically on logit-based probabilities to reduce the gap between self-reported confidence and true predictive accuracy, especially for open-weight models like LLaMA and Mistral.

4. Introduce Uncertainty Signal Internalization: Fine-tune models on prompts with linguistic uncertainty cues (e.g., possibly, unclear, insufficient data) so they learn to reduce confidence and increase abstention rates when such cues are present, rather than ignoring them.

5. Create Reliability-Aware Evaluation Metrics: Replace accuracy-only benchmarks with composite metrics that penalize unsafe confident errors (UCER) and reward correct abstention, and use these during model selection and deployment in high-stakes domains like medicine.

What the Improved AI System Can Do:

• Accurately gauge its own uncertainty: It will produce confidence scores that closely match its true probability of being correct, even under ambiguous or incomplete clinical information.

• Know when to say I don't know: It will abstain from answering when the correct answer is absent or evidence is insufficient, reducing hallucinated medical advice.

• Remain calibrated under linguistic ambiguity: It will lower its confidence when prompts contain uncertainty markers (e.g., possibly, not clear), avoiding overconfident but wrong diagnoses.

• Minimize unsafe confident errors: It will drastically reduce cases where it gives a wrong answer with high confidence, making it safer for clinical decision support.

• Provide reliable logit-based confidence: It will align its verbalized confidence with its internal probability estimates, eliminating discrepancies like reporting 0.97 confidence while logits indicate 0.28.

• Support trustworthy deployment in medicine: It will meet stricter reliability standards, enabling use in triage, differential diagnosis, and patient education where overconfidence is dangerous.