LLM Judges Have Dark Current: Stratified Null and Utility Tests for LLM Judge Reliability

arXiv:2606.15610 · cs.CL, astro-ph.IM, cs.AI, cs.LG · Submitted 2026-06-14 · Read on arXiv

cs.CL, astro-ph.IM, cs.AI, cs.LG

Submitted: 2026-06-14

Updated: 2026-10-12

Comments: 21 pages, 4 figures, 9 tables. Substantially revised: input-stratified null and target-selection utility audits, additional fixed-configuration studies, and an operational audit protocol. Supersedes v1, originally titled "LLM Judges Have Dark Current: A Psychometric Datasheet for LLM-as-a-Judge Evaluation."

Code: https://github.com/openai/evals

License: http://creativecommons.org/licenses/by/4.0/

The gist: Passing an exact-clone null gate does not entail passing a matched target-selection utility gate.

Terminology

Abstract

Passing an exact-clone null gate does not entail passing a matched target-selection utility gate. We call a non-tie preference between exact duplicates dark current. On 102 prespecified fresh RewardBench 2 rows, all four configuration--criterion cells report 0/102 clone preferences. Both neutral utility arms pass, but strict utility lower bounds of.539 and.745 miss the.75 working threshold. Conversely, re-evaluation of the same rows on two additional configurations yields a neutral gpt-oss utility pass with 2/102 clone preferences, leaving low-null behavior uncertified. A separately frozen 485-row extension passes both aggregate working gates under both criteria on one local raw-text configuration, yet strict Precise Instruction Following (Precise IF) alignment is 58/154 (.377). Thus matched null and utility results can diverge, and aggregate passage can coexist with weak subset alignment. Interpretation is conditional on the declared configurations, fixed source mixtures, source-label targets, and study-specific working models. The bounds are marginal, the subset rate is descriptive, and non-certification does not prove a defect.

Sources

Related papers