Opening LLM Judges: Recovering Preference Signals Beyond the Final Verdict
cs.AI
Submitted: 2026-09-26
Updated: 2026-09-26
License: http://creativecommons.org/licenses/by/4.0/
The gist: LLM judges are widely used to evaluate model outputs, but their verdicts can be unreliable: a judge may favor the worse answer for its position, length, or other surface features.
Terminology
Abstract
LLM judges are widely used to evaluate model outputs, but their verdicts can be unreliable: a judge may favor the worse answer for its position, length, or other surface features. When a judge is wrong, is the information needed to judge correctly absent from the model, or present in its internal representations but not reflected in the output? We study this across 64 open-weight evaluators and 14 datasets, including causal interventions on 41 judges (editing activations mid-run to see whether the verdict changes). On LLMBar, built so the superficially better answer is the worse one, the verdicts of 50 judges agree with human labels only 0.456 of the time, even after averaging both answer orders. Yet a small probe on the same judges' activations, with no weight updates, reaches 0.846, and 0.686 once surface features such as length and position are residualized out (0.507 with shuffled labels). The gap holds across eight benchmarks and model families, but is not universal: a score of how well surface features alone predict the human label, computed before any probe is trained, predicts the size of the gain (Spearman rho = 0.90). On rubric tasks that score one answer at a time, leaving no surface cue to exploit, reading the internals gives no advantage. The interventions also show that editing activations mid-network already changes the verdict, before it can be read off directly, and locate the pathways carrying position and length bias. At the same label budget, the recovered signal lets a judge flag cases where it is likely wrong and yields better labels for preference learning. A wrong verdict, then, does not mean the judge lacks the information, and a simple diagnostic shows when it is worth recovering.
Sources
- Eliciting Latent Predictions from Transformers with the Tuned Lens
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
- Judge Circuits Explain Format-Induced Inconsistency in LLM-as-a-Judge
- Reference-Free Rating of LLM Responses via Latent Information
- Localizing Model Behavior with Path Patching
- Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations
- Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents
- The Geometry of LLM-as-Judge: Why Inter-LLM Consensus Is Not Human Alignment
- Verbosity Bias in Preference Labeling by Large Language Models
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems