Verifiable Visual Rewards Transfer from Synthetic Scenes to Natural Prompts
cs.AI, cs.CV
Submitted: 2026-09-28
Updated: 2026-09-28
Comments: 33 pages, 10 figures, 18 tables
Code: https://github.com/stellalisy/VVRBenchhttps:
License: http://creativecommons.org/licenses/by/4.0/
The gist: Precise instruction following in image generation, such as satisfying object counts and spatial relations, remains an open challenge at least in part because it is learned using unreliable reward
Terminology
Abstract
Precise instruction following in image generation, such as satisfying object counts and spatial relations, remains an open challenge at least in part because it is learned using unreliable reward models such as object detectors and vision-language models. We introduce Verifiable Visual Rewards (VVR), the first framework for programmatically verifiable image rewards, and show that training on it generalizes to natural prompts. Each VVR task is a scene of geometric objects and relations among them, from which we derive both the prompt and a deterministic verifier, so tasks can be generated in any number and at any chosen complexity. We release VVRBench, with 10,000 tasks over 32 constraint types, and VVRBench-Challenge, with 720 more complex tasks; the strongest model we evaluate---GPT-Image-2.5---solves 21.4% of VVRBench-Challenge. Using VVR scores as rewards for reinforcement learning (RLVVR) raises the accuracy of Stable Diffusion 3.5 Medium on VVRBench from 2.8% to 28.3% and demonstrates consistent easy-to-hard generalization. These gains extend to out-of-domain benchmarks, and mixing VVR into existing objectives further improves overall performance and human preference, motivating the adoption of VVR into standard image generation post-training recipes.
Sources
- DeTikZify: Synthesizing Graphics Programs for Scientific Figures and Sketches with TikZ
- Training Diffusion Models with Reinforcement Learning
- HiDream-I1: A High-Efficient Image Generative Foundation Model with Sparse Diffusion Transformer
- Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles
- MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?
- Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation
- Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation
- Directly Fine-Tuning Diffusion Models on Differentiable Rewards
- Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
- DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- CLIPScore: A Reference-free Evaluation Metric for Image Captioning
- Understanding Reward Hacking in Text-to-Image Reinforcement Learning
- ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment
- TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering
- T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generation
- AlphaGRPO: Unlocking Self-Reflective Multimodal Generation in UMMs via Decompositional Verifiable Reward
- CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
- Evaluating Numerical Reasoning in Text-to-Image Models
- GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluation
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems