Agreement Without Alignment: Label-Free Rubric Clarification Increases LLM-Judge Consensus but Not Agreement with Humans
cs.CL, cs.AI
Submitted: 2026-10-09
Updated: 2026-10-09
Comments: 12 pages, 12 tables
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- HealthBench: Evaluating Large Language Models Towards Improved Human Health
- Judging LLM-as-a-Judge: Concerning Rubric Artifacts in LLM-based Automated Text Generation Evaluation
- Decomposing Physician Disagreement in HealthBench
- CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation
- Quantifying the Statistical Effect of Rubric Modifications on Human-Autorater Agreement
- The Geometry of LLM-as-Judge: Why Inter-LLM Consensus Is Not Human Alignment
- Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why
- PReMISE: Policy Rubrics as Measurement Specifications for LLM Judges
- ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents
- Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judge
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering