Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria
cs.AI
Submitted: 2026-05-08
Updated: 2026-11-07
Comments: 32 pages, 10 figures
Code: https://github.com/OpenEnvision/AutoRubric-as-Reward
License: http://creativecommons.org/licenses/by/4.0/
The gist: Aligning multimodal generative models with human preferences demands reward signals that respect the compositional, multi-dimensional structure of human judgment.
Terminology
Abstract
Aligning multimodal generative models with human preferences demands reward signals that respect the compositional, multi-dimensional structure of human judgment. Prevailing RLHF approaches reduce this structure to scalar or pairwise labels, collapsing nuanced preferences into opaque parametric proxies and exposing vulnerabilities to reward hacking. While recent Rubrics-as-Reward (RaR) methods attempt to recover this structure through explicit criteria, generating rubrics that are simultaneously reliable, scalable, and data-efficient remains an open problem. We introduce Auto-Rubric as Reward (ARR), a framework that reframes reward modeling from implicit weight optimization to explicit, criteria-based decomposition. Before any pairwise comparison, ARR externalizes a VLM's internalized preference knowledge as prompt-specific rubrics, translating holistic intent into independently verifiable quality dimensions. This conversion of implicit preference structure into inspectable, interpretable constraints substantially suppresses evaluation biases including positional bias, enabling both zero-shot deployment and few-shot conditioning on minimal supervision. To extend these gains into generative training, we propose Rubric Policy Optimization (RPO), which distills ARR's structured multi-dimensional evaluation into a robust binary reward, replacing opaque scalar regression with rubric-conditioned preference decisions that stabilize policy gradients. On text-to-image generation and image editing benchmarks, ARR-RPO outperforms pairwise reward models and VLM judges, demonstrating that explicitly externalizing implicit preference knowledge into structured rubrics achieves more reliable, data-efficient multimodal alignment, revealing that the bottleneck is the absence of a factorized interface, not a deficit of knowledge.
Sources
- Critique-out-Loud Reward Models
- Qwen3-VL Technical Report
- Moderate Deviations for the Capacity of the Random Walk range in dimension four
- Training Diffusion Models with Reinforcement Learning
- BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset
- ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generation
- Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
- ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment
- Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Image
- FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space
- Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedback
- Step1X-Edit: A Practical Framework for General Image Editing
- Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training
- EditScore: Unlocking Online RL for Image Editing via High-Fidelity Reward Modeling
- RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Following
- OpenAI GPT-5 System Card
- Emu3: Next-Token Prediction is All You Need
- Unified Reward Model for Multimodal Understanding and Generation
- Qwen-Image Technical Report
- OmniGen2: Towards Instruction-Aligned Multimodal Generation
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents