Prompting the Unknown: Understanding Response Uncertainty in Large Language Models
cs.LG, cs.CL
Submitted: 2024-07-20
Updated: 2026-09-01
Comments: Accepted at ACL Findings 2026
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language models (LLMs) are widely used in decision-making across diverse domains.
Terminology
Abstract
Large language models (LLMs) are widely used in decision-making across diverse domains. Ensuring the generation of safe and reliable responses is critical for the effective deployment of LLM-based applications, particularly in high-stakes domains such as healthcare and finance. Most of these applications typically use carefully crafted prompts to guide response generation; however, the relationship between prompts and the reliability of LLM-generated responses is not yet fully understood. To address this gap, we propose a novel prompt-response concept model that explains the relationship between the amount of task-relevant information (informativeness) provided in the prompt and the LLM-generated response uncertainty by identifying four sources of response uncertainty: prompt underspecification, model quality, task variability, and semantic redundancy. We prove that response uncertainty decreases as prompt informativeness or model quality increases, mirroring the behavior of epistemic uncertainty in probabilistic models. Our experimental results on real-world datasets further validate our proposed model and corroborate the theoretical results.
Sources
- GPT-4 Can't Reason
- Language models show human-like content effects on reasoning tasks
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- A Survey on In-context Learning
- Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
- Rethinking Large Language Models in Mental Health Applications
- Multiple Choice Questions: Reasoning Makes Large Language Models (LLMs) More Self-Confident, Especially When They are Wrong
- What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams
- Language Models (Mostly) Know What They Know
- Scaling and evaluating sparse autoencoders
- PaLM 2 Technical Report
- Structured Attention Networks
- A Theory of Emergent In-Context Learning as Implicit Structure Induction
- Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation
- Large Language Models Cannot Self-Correct Reasoning Yet
- DEUP: Direct Epistemic Uncertainty Prediction
- A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
- Look Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language Models
- Think Twice Before Trusting: Self-Detection for Large Language Models through Comprehensive Answer Reflection
- Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks