Self-Aware Knowledge Probing: Evaluating Language Models' Relational Knowledge through Confidence Calibration
This research developed a framework to assess how well language models know facts by looking at their confidence scores. The study found that most models are overconfident, meaning they think they are more certain than they actually are. The best way to measure reliable knowledge is through structural consistency estimates and accounting for how statements can be rephrased.