CARE: Contrastive Anchor-based Rubric Evolution for Large Language Model Post-Training
cs.AI
Submitted: 2026-09-01
Updated: 2026-09-01
Comments: EMNLP 2026 MainConference
License: http://creativecommons.org/licenses/by/4.0/
The gist: Rubric-based reinforcement learning decomposes open-ended instructions into prompt-specific, flexible rubrics, making it better suited than reinforcement learning with verifiable rewards for
Terminology
Abstract
Rubric-based reinforcement learning decomposes open-ended instructions into prompt-specific, flexible rubrics, making it better suited than reinforcement learning with verifiable rewards for post-training LLMs on open-ended tasks. However, static rubrics are inevitably hacked as the policy evolves, and existing dynamic approaches introduce new problems: undirected rubric extraction, unreliable hack detection, and unbounded rubric proliferation. We propose CARE (C ontrastive A nchor-based R ubric E volution), which grounds every rubric evolution step in a high-quality anchor response generated by a frontier model conditioned on the prompt and its rubrics. At each training step, CARE contrasts the highest-scoring rollout against the anchor, enabling two complementary mechanisms: an Adaptive branch that reactively repairs reward misspecification; and a Chase branch that proactively converts frontier-level quality gaps into sharper rubrics. Together, the two branches maintain discriminative accuracy in the high-reward region ---the precise region where reward over-optimization mostly originates. Experiments on WildChecklist-9K with Qwen2.5-7B-Base and Qwen2.5-7B-Instruct show that CARE achieves state-of-the-art performance on Arena-Hard-2.0, InfoBench, and FollowBench, and is the only method whose win rate against GPT-4.1 anchor responses shows sustained improvement throughout 300 training steps; additional results on Llama-3.1-8B-Instruct and Qwen3-8B further indicate that CARE generalizes across model families.
Sources
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
- Critique-out-Loud Reward Models
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- Reinforcement Learning with Rubric Anchors
- The Hitchhikers Guide to Rubric Quality Understanding and Enrichment
- Inference-Time Scaling for Generalist Reward Modeling
- Online Rubrics Elicitation from Pairwise Comparisons
- GPT-4 Technical Report
- DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research
- DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Checklists Are Better Than Reward Models For Aligning Language Models
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems