Diffusion-DRF: Free, Rich, and Differentiable Reward for Video Diffusion Fine-Tuning
cs.CV
Submitted: 2026-01-07
Updated: 2026-11-03
Comments: Published in NeurIPS 2026. Webpage: https://yfwang1999.github.io/Diffusion-DRF/
Code: https://github.com/hpcaitech/Open-Sora
Project page: https://snap-research.github.io/diffusion-drf
License: http://creativecommons.org/licenses/by/4.0/
The gist: Video diffusion alignment has been heavily relied on scalar rewards.
Terminology
Abstract
Video diffusion alignment has been heavily relied on scalar rewards. These rewards are typically derived from learned reward models in human preference datasets, requiring additional training and extensive collection. Moreover, scalar rewards provide coarse, global supervision, offering limited prompt-generation mismatch credit assignment and making models prone to reward exploitation and unstable optimization. We propose Diffusion-DRF, a free, rich, and differentiable reward framework for video diffusion fine-tuning. Diffusion-DRF employs a frozen, off-the-shelf Vision-Language Model (VLM) as the critic, eliminating the need for reward model training. Instead of relying on a single scalar reward, it decomposes each user prompt into multi-dimensional questions with freeform dense VQA explanation queries, yielding information-rich feedback. By direct differentiable optimization over this rich feedback, Diffusion-DRF achieves stable reward-based tuning without preference datasets collection. Diffusion-DRF achieves significant gains both quantitatively and qualitatively, outperforming state-of-the-art Flow-GRPO by 4.74% in overall performance on unseen VBench-2.0.
Sources
- Qwen2.5-VL Technical Report
- Constitutional AI: Harmlessness from AI Feedback
- Training Diffusion Models with Reinforcement Learning
- RealDPO: Real or Not Real, that is the Preference
- Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedback
- Scalable Ranked Preference Optimization for Text-to-Image Generation
- Adam: A Method for Stochastic Optimization
- HunyuanVideo: A Systematic Framework For Large Video Generative Models
- Flow-GRPO: Training Flow Matching Models via Online RL
- Improving Video Generation with Human Feedback
- Dual-Process Image Generation
- OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation
- Video Diffusion Alignment via Reward Gradients
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Wan: Open and Advanced Large-Scale Video Generative Models
- LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment
- H3AE: High Compression, High Speed, and High Quality AutoEncoder for Video Diffusion Models
- DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Models
- VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation
- DanceGRPO: Unleashing GRPO on Visual Generation
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs