Reinforcing the Generation Order of Multimodal Masked Diffusion Models
Yidong Ouyang, Zhe Wang, Sourav Bhabesh, Dmitriy Bespalov
cs.LG, cs.AI, stat.ML
Submitted: 2026-07-09
License: http://creativecommons.org/licenses/by/4.0/
The gist: Diffusion Language Models (DLMs) have recently achieved substantial progress in natural language generation tasks.
Terminology
Abstract
Diffusion Language Models (DLMs) have recently achieved substantial progress in natural language generation tasks. Recent research demonstrates that adaptive token generation ordering can significantly improve performance in mathematical reasoning and code synthesis applications. In this work, we investigate the optimization of generation order for both text-to-image synthesis and multimodal understanding. We first establish that, unlike structured problems in language generation such as Sudoku puzzles, model logits alone are insufficient for determining optimal generation sequences in text-to-image generation and multimodal understanding. To address this challenge, we introduce a learnable control module trained via Group Relative Policy Optimization (GRPO) to determine the generation order. Our results demonstrate that learning this control block substantially improves both text-to-image alignment and multimodal understanding in DLMs. In particular, it enhances the model's ability to capture fine-grained spatial relationships in generated images while also strengthening performance on multimodal reasoning and comprehension tasks. We evaluate our framework on GenEval, an object-focused benchmark for text-to-image alignment, where it achieves 4.08% relative improvements. In addition, experiments on VLMEvalKit confirm 4.85% relative improvements in multimodal understanding, highlighting the broad effectiveness of our approach.
Sources
- Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models
- Structured Denoising Diffusion Models in Discrete State-Spaces
- A Continuous Time Framework for Discrete Denoising Models
- SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models
- Discrete Flow Matching
- SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation
- GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment
- TempFlow-GRPO: When Timing Matters for GRPO in Flow Models
- Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Models
- Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions
- Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation
- MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE
- Flow-GRPO: Training Flow Matching Models via Online RL
- Large Language Diffusion Models
- Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data
- Training language models to follow instructions with human feedback
- Choosing to Rank
- Simple and Effective Masked Diffusion Language Models
- Proximal Policy Optimization Algorithms
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks