Popular but Wrong: Understanding and Mitigating LLM Overconfidence through Knowledge Popularity
cs.CL
Submitted: 2025-05-23
Updated: 2026-08-31
Comments: EMNLP2026 Main
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language models (LLMs) often produce incorrect answers with high confidence, yet the factors associated with such overconfidence remain insufficiently understood.
Terminology
Abstract
Large language models (LLMs) often produce incorrect answers with high confidence, yet the factors associated with such overconfidence remain insufficiently understood. We study this problem through the lens of knowledge popularity. Using entity-centric factual QA derived from Wikidata triplets, we characterize popularity through question entity popularity, answer popularity, and question-answer co-occurrence. We find two consistent patterns. First, hallucinated answers are far from random: compared with ground-truth answers, they tend to be more popular or more frequently associated with the question entity. Second, confidence is strongly tied to the popularity of generated answers: even among incorrect predictions, more popular answers or those with higher question-answer co-occurrence receive higher confidence. Together, these findings suggest that popular but wrong alternatives may contribute to overconfidence. We further show that popularity-related signals can mitigate overconfidence and improve overall confidence estimation. Across six models and three datasets, incorporating knowledge popularity reduces average confidence on incorrect answers from 0.765 to 0.254 and overall ECE from 0.356 to 0.050, while improving Alignment from 77.08% to 83.72%.
Sources
- GPT-4 Technical Report
- INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection
- Calibration of Pre-trained Transformers
- Rowen: Adaptive Retrieval-Augmented Generation for Hallucination Mitigation in LLMs
- The Llama 3 Herd of Models
- Language Models (Mostly) Know What They Know
- Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation
- Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models
- Teaching Models to Express Their Uncertainty in Words
- SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
- When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentation
- Towards Fully Exploiting LLM Internal States to Enhance Knowledge Boundary Perception
- Are Large Language Models More Honest in Their Probabilistic or Verbalized Confidence?
- Prompting GPT-3 To Be Reliable
- Unsupervised Real-Time Hallucination Detection based on the Internal States of Large Language Models
- Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs
- Qwen2 Technical Report
- Alignment for Honesty
- Do Large Language Models Know What They Don't Know?
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering