Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias
cs.CL
Submitted: 2026-06-17
Updated: 2026-11-03
Comments: arXiv admin note: This submission has been withdrawn because it does not meet arXiv's research content quality standards
License: http://creativecommons.org/licenses/by/4.0/
The gist: LLM-as-a-Judge has become the dominant evaluation paradigm for language models, but judge validation in practice relies on exact-match agreement…
Terminology
Abstract
LLM-as-a-Judge has become the dominant evaluation paradigm for language models, but judge validation in practice relies on exact-match agreement, a metric that does not correct for chance and systematically overstates discriminative ability. We present the largest systematic evaluation of LLM-as-a-Judge to date: 21 judges from nine providers across MT-Bench, JudgeBench, and RewardBench, evaluated under three protocols (agreement, consistency, bias audit) over 118 runs and approximately 541,000 individual judgments. Four findings emerge, consistent across the full cohort, including the April 2026 frontier: kappa deflation between exact match and Cohen's kappa is universal (33--41 pp on MT-Bench), judge rankings shift by up to 14 positions across benchmarks, high test--retest reliability (>0.95) coexists with severe position bias (>0.10) in two production-deployed judges (instantiating a consistency--bias paradox), and verbosity bias is small (<0.011) across our cohort under a single pairwise rubric. We distill these into a Minimum Viable Validation Protocol.
Sources
- Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory
- Balanced Accuracy: The Right Metric for Evaluating LLM Judges -- Explained through Youden's J statistic
- A Survey on LLM-as-a-Judge
- Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreement
- CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks
- LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
- Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge
- Large Language Models are Inconsistent and Biased Evaluators
- Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models
- J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data