The Metacognitive Profile Atlas: Domain-level confidence discrimination in 33 frontier LLMs on MMLU
cs.CL, cs.AI, cs.LG
Submitted: 2026-04-21
Updated: 2026-11-13
Code: https://github.com/synthiumjp/metacognitive-profile-atlas
Terminology
Sources
- LLMs as Signal Detectors: Sensitivity, Bias, and the Temperature-Criterion Analogy
- Do LLMs Know What They Know? Measuring Metacognitive Efficiency with Signal Detection Theory
- The Metacognitive Monitoring Battery: A Cross-Domain Benchmark for LLM Self-Monitoring
- Before You Interpret the Profile: Validity Scaling for LLM Metacognitive Self-Report
- Screen Before You Interpret: A Portable Validity Screen for Benchmark-Based LLM Confidence Signals
- Concurrent Criterion Validation of a Validity Screen for LLM Confidence Signals via Selective Prediction
- A Neuropsychologically Grounded Evaluation of LLM Cognitive Abilities
- Language Models (Mostly) Know What They Know
- How do LLMs Compute Verbal Confidence
- Closing the Confidence-Faithfulness Gap in Large Language Models
- BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidence
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data