JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment
cs.CL, cs.AI, cs.CY
Submitted: 2026-05-24
Updated: 2026-11-06
Code: https://github.com/liftlab-SLS/JudgmentBench
Terminology
Sources
- HealthBench: Evaluating Large Language Models Towards Improved Human Health
- Configurable Preference Tuning with Rubric-Guided Synthetic Data
- LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
- Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
- AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following
- Reinforcement Learning with Rubric Anchors
- CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling
- OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment
- Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generation
- GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks
- Benchmark^2: Systematic Evaluation of LLM Benchmarks
- Online Rubrics Elicitation from Pairwise Comparisons
- PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice
- Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation
- Checklists Are Better Than Reward Models For Aligning Language Models
- PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance
- Self-Rewarding Language Models
- Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data