Screen Before You Interpret: A Portable Validity Screen for Benchmark-Based LLM Confidence Signals
cs.CL, cs.AI
Submitted: 2026-04-20
Updated: 2026-11-13
Code: https://github.com/synthiumjp/validity-scaling-llm
Terminology
Sources
- Rescaling confidence: What scale design reveals about LLM metacognition
- Language Models (Mostly) Know What They Know
- How Overconfidence in Initial Choices and Underconfidence Under Criticism Modulate Change of Mind in Large Language Models
- Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancement
- BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidence
- Nontrivial solutions to the relative overdetermined torsion problem in a cylinder
- LLMs as Signal Detectors: Sensitivity, Bias, and the Temperature-Criterion Analogy
- Do LLMs Know What They Know? Measuring Metacognitive Efficiency with Signal Detection Theory
- The Metacognitive Monitoring Battery: A Cross-Domain Benchmark for LLM Self-Monitoring
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data