RADAR: Rubric-Aware Dependency and Redundancy Analysis for LLM-as-Judge Evaluation
Divyansh Singh, Reza Davari, Afra Mashhadi
cs.CL
Submitted: 2026-08-03
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Rubric-based LLM-as-judge pipelines often assume that evaluation criteria provide independent signals.
Terminology
Abstract
Rubric-based LLM-as-judge pipelines often assume that evaluation criteria provide independent signals. In practice, however, criteria can be behaviorally coupled: improving one criterion may systematically change scores on another, distorting aggregate scores used in model-release or product-update decisions. We introduce RADAR, a lightweight preflight diagnostic framework for estimating such coupling before large-scale evaluation. Given a rubric, RADAR generates targeted synthetic probes, scores each probe on all criteria, and produces a directional coupling matrix that shows which criteria co-score and how. We validate RADAR on three industry-relevant evaluation settings: NVIDIA HelpSteer2, SumPubMed, and the Yale-Salesforce SummEval benchmark. Using only a small number of probes per criterion, RADAR recovers human inter-criterion correlation structure (Pearson r > 0.84) and provides practitioners with concrete audit signals about redundancy, hierarchy, and aggregation sensitivity before committing to large-scale judging.
Sources
- JudgeLRM: Large Reasoning Models as a Judge
- RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation
- When Judgment Becomes Noise: How Design Failures in LLM Judge Benchmarks Silently Undermine Validity
- G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
- Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks
- Large Language Models are Inconsistent and Biased Evaluators
- Large Language Models are not Fair Evaluators
- HelpSteer2: Open-source dataset for training top-performing reward models
- Self-Preference Bias in LLM-as-a-Judge
- FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets
- Through the Judge's Eyes: Inferred Thinking Traces Improve Reliability of LLM Raters
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering