A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation
cs.AI
Submitted: 2026-08-25
Updated: 2026-08-25
Comments: 39 pages, 10 figures, 11 tables
Code: https://github.com/Airjiannan05/TriDeHall
License: http://creativecommons.org/licenses/by/4.0/
The gist: LLM-as-a-judge evaluation is usually assessed by agreement and robustness to surface perturbations, but reliability does not establish construct validity.
Terminology
Abstract
LLM-as-a-judge evaluation is usually assessed by agreement and robustness to surface perturbations, but reliability does not establish construct validity. We formalize construct validity for an evaluator as a two-dimensional profile: invariance S, the probability that a verdict is unchanged under construct-preserving edits, and construct sensitivity R, the probability that it changes under minimal construct-changing edits. We show that S and R are independent and that no scalar summary preserves all relevant comparisons. We measure the profile across 7 judges and 4 domains using 7 construct-changing intervention types and 5 register-only controls, with intervention direction determined by human annotators and generation, verification, and judging assigned to disjoint model families. At matched invariance S >= 0.90, judges average S = 0.945 but R = 0.319. Sensitivity also differs between scope and strength edits: R scope = 0.383 versus R strength = 0.262, a +0.121 gap with the same sign for all 7 judges. We further audit five public label sets and find that surface-only predictors reproduce 55%-67% of labels in paired mode, including 67.4% of MT-Bench human votes. These results show that high judge agreement can coexist with weak sensitivity to changes in the construct being evaluated, motivating joint reporting of invariance and sensitivity and auditing the validation set itself.
Sources
- Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
- Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinations
- Conformal Linguistic Calibration: Trading-off between Factuality and Specificity
- RewardBench: Evaluating Reward Models for Language Modeling
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
- A Question Answering Framework for Decontextualizing User-facing Snippets from Scientific Documents
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Rectifying Shortcut Behaviors in Preference-based Reward Learning
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems