When Guessing is Rewarded: Rethinking Language Model Evaluation with Distributional Uncertainty Scoring
Traditional model evaluation fails by only scoring single predictions, missing how models distribute their beliefs. This work introduces the Distributional Correctness Score (DCS), a new metric that evaluates a model's entire probability distribution over answers, including abstention. DCS rewards genuine uncertainty and punishes confident wrong answers, incentivizing models to express epistemic humility.