From Judgment Quality to Downstream Utility: Rethinking LLM-as-a-Judge for Open-Ended Tasks
cs.AI
Submitted: 2026-09-29
Updated: 2026-09-29
Terminology
Sources
- HealthBench: Evaluating Large Language Models Towards Improved Human Health
- Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning
- Are We on the Right Way to Assessing LLM-as-a-Judge?
- Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Text
- Reinforcement Learning with Rubric Anchors
- Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training
- Inference-Time Scaling for Generalist Reward Modeling
- DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks
- Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubrics
- Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judge
- Co-Evolving LLM Evaluators and Policies via DynamicRubric
- Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates
- ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubrics
- Grad2Reward: From Sparse Judgment to Dense Rewards for Improving Open-Ended LLM Reasoning
- Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems