LLM Judges Have Dark Current: Stratified Null and Utility Tests for LLM Judge Reliability
cs.CL, astro-ph.IM, cs.AI, cs.LG
Submitted: 2026-06-14
Updated: 2026-10-12
Comments: 21 pages, 4 figures, 9 tables. Substantially revised: input-stratified null and target-selection utility audits, additional fixed-configuration studies, and an operational audit protocol. Supersedes v1, originally titled "LLM Judges Have Dark Current: A Psychometric Datasheet for LLM-as-a-Judge Evaluation."
Code: https://github.com/openai/evals
License: http://creativecommons.org/licenses/by/4.0/
The gist: Passing an exact-clone null gate does not entail passing a matched target-selection utility gate.
Terminology
Abstract
Passing an exact-clone null gate does not entail passing a matched target-selection utility gate. We call a non-tie preference between exact duplicates dark current. On 102 prespecified fresh RewardBench 2 rows, all four configuration--criterion cells report 0/102 clone preferences. Both neutral utility arms pass, but strict utility lower bounds of.539 and.745 miss the.75 working threshold. Conversely, re-evaluation of the same rows on two additional configurations yields a neutral gpt-oss utility pass with 2/102 clone preferences, leaving low-null behavior uncertified. A separately frozen 485-row extension passes both aggregate working gates under both criteria on one local raw-text configuration, yet strict Precise Instruction Following (Precise IF) alignment is 58/154 (.377). Thus matched null and utility results can diverge, and aggregate passage can coexist with weak subset alignment. Interpretation is conditional on the declared configurations, fixed source mixtures, source-label targets, and study-specific working models. The bounds are marginal, the subset rate is descriptive, and non-certification does not prove a defect.
Sources
- Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
- LLMs as Signal Detectors: Sensitivity, Bias, and the Temperature-Criterion Analogy
- Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge
- Quantifying and Mitigating Self-Preference Bias of LLM Judges
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering