Diff-Instruct with Diffused Reward: Towards Principled One-step Generator RL

arXiv:2605.24001 · cs.CV, cs.AI, cs.LG · Submitted 2026-05-18 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Diff-Instruct with Diffused Reward".

Jane: Here's the gist of the paper. Diff-Instruct with Diffused Reward: Towards Principled One-step Generator RL is a data-free trajectory-level alignment framework that propagates RLHF-optimal reward signals across all noise levels of a diffusion trajectory to create a principled objective for one-step text-to-image generators The core insight is to systematically propagate reward signals defined on clean images to every noise level of the diffusion trajectory This method resolves the structural mismatch in prior methods where rewards are applied only at the clean image output while regularization spans all noise levels, which leads to terminal reward domination Pruning redundant reasoning tokens The authors propose Diff-Instruct with Diffused Reward (Didr) as a principled reinforcement learning framework for one-step image generators without using image data The framework starts from the KL-regularized RLHF objective, identifying its optimal clean-image target as the reward-tilted density q∗(x0c) ∝ q0(x0c) exp(r(x0, c)/τ) This optimal target is then diffused through the reference process to obtain trajectory-level reward-tilted marginals q∗ t(xtc) Minimizing the Integral KL (IKL) divergence against these marginals yields a target score that decomposes into the reference score plus a reward-induced correction, the Diffused Reward Score (DRS), defined at every noise level through reference posterior How the training signal is built The training objective for Didr is defined as minimizing the IKL divergence between its forward-noised marginals p θ,t and the reward-tilted trajectory q∗ t This objective is LDIDR(θ) = Ec∼C ∫ T 0 w(t) DKLp θ,t (·c)∥q∗ t (·c) The target score decomposition shows that the trajectory target score is the reference score plus a reward-induced correction, DRS(xt, t, c) = sref + sr This DRS term is defined as s r(xt, t, c) = ∇xt log Ex0∼q(x0xt,c) exp(r(x0,c)/τ) Where the gains come from The Diffused Reward Proxy (DRP) is introduced as an efficient estimator of DRS based on differentiable short-step denoising This allows for stable reward-gradient propagation across the entire trajectory by approximating the intractable posterior expectation using Monte Carlo estimators The practical generator update is driven by the mismatch between its own score and the reward-tilted target score sref + sr Extensive experiments demonstrate that Didr consistently Pareto-dominates existing one-step SDXL baselines Ablation studies and results Empirically, when transferred to a 6B DiT backbone (Z-Image), Didr surpasses its 50-step teacher in preference alignment while requiring only a single generation step Standard Didr Pareto-dominates all one-step alignment baselines on preference vs.

Tom: fidelity Furthermore, Didr generalizes to DiT architectures at larger scale, where zimage-Didr surpasses the 50-step Z-Image base on all preference metrics in a single step The temperature hyperparameter τ=0.01 is selected as the operating point balancing both preference–fidelity trade-off We identified terminal reward domination as a fundamental failure mode of trajectory distillation with terminal rewards, and proposed Didr to resolve it The core idea is to lift the RLHF-optimal clean-image target q∗ to a principled trajectory objective via the Diffused Reward Score, and approximate it tractably with the Diffused Reward Proxy through differentiable short-step denoising—requiring no image training data Limitations include potential amplification of reward model biases and increased training cost relative to endpoint-only methods The paper concludes that principled trajectory-level reward propagation is an effective alignment strategy across architectures and scales References Stability AI.

Jane: Stable Diffusion 3.5.

Tom: https://stability.ai/news/stable-diffusion-3-5, 2024 Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine.

Paper summary: Tom: So we're diving into "Diff-Instruct with Diffused Reward: Towards Principled One-step Generator RL." This paper tackles a problem where existing one-step text-to-image generators have this structural mismatch between optimizing the final clean image and what happens throughout the entire noise process.

Jane: That sounds like a really deep issue for someone trying to get high quality images quickly. So, what's the main idea here? What are they proposing with this Diff-Instruct with Diffused Reward framework?

Tom: Basically, they propose this data-free trajectory-level alignment framework that takes the RLHF optimal reward signals defined on clean images and propagates them across every single noise level of a diffusion trajectory to create a principled objective for one-step text-to-image generators.

Lu: The core insight is systematically propagating reward signals defined on clean images to every noise level of the diffusion trajectory. This resolves the structural mismatch where rewards are only applied at the final clean image output, while regularization spans all noise levels, which leads to terminal reward domination.

Meng: Terminal reward domination sounds like a big headache for deployment. So, they are saying their method avoids that by focusing on the whole path of the generation process?

Lalam: Yes, they start from the KL-regularized RLHF objective to identify the optimal clean image target as a reward-tilted density q star of x zero given c, which is proportional to q zero of x zero given c times exp of r of x zero and c over tau.

Jane: That sounds like they are taking the known best way to optimize for a clean image and using that as a starting point for something else. Can you tell us more about how they turn that clean image target into something useful along the noise path?

Tom: They take this optimal target and diffuse it through the reference process to obtain trajectory-level reward-tilted marginals, q star of xt given c, which is the next step in their objective.

Lu: Minimizing the integral KL divergence against these marginals yields a target score that decomposes into the reference score plus a reward induced correction called the Diffused Reward Score, or DRS, defined at every noise level through reference posterior.

Meng: So they derived this DRS term analytically because it breaks down neatly into two parts: the reference score and this reward-induced correction. What does that mean for actually training something?

Lalam: The reward induced correction term is defined as s r of xt, t, c which is the gradient of xt with respect to x zero sampled from q zero given xt and c times exp of r of x zero and c over tau.

Jane: That means they're defining a specific score at every noise step that tells the generator how to move in a way that incorporates both the standard reference guidance and this extra reward information. How do they make sense of this correction term practically?

Tom: They introduce the Diffused Reward Proxy, or DRP, as an efficient estimator of this DRS based on differentiable short-step denoising. This lets them approximate the intractable posterior expectation using Monte Carlo estimators for stable reward gradient propagation across the entire trajectory.

Lu: The practical generator update is driven by the mismatch between its own score and this target score which is that reference score plus s r of xt, t, c.

Meng: So they're using a proxy to estimate something complicated so they can actually train the generator without needing huge amounts of image data for every noise level?

Lalam: Exactly. And empirically, when transferred to a 6B DiT backbone called Z-Image, Didr surpasses its fifty-step teacher in preference alignment while only requiring a single generation step <ref:2605.24001#pg3>.

Jane: That's a significant jump from needing fifty steps down to just one step for the same kind of alignment. What about the trade-off between what they get and what they sacrifice?

Tom: Standard Didr achieves the best pick score fidelity trade-off among one-step methods, and when scaling to larger DiT architectures, zimage Didr surpasses the fifty-step base on all preference metrics in a single step <ref:2605.24001#pg3,trade-off among one-step methods>.

Lu: They also showed that the temperature hyperparameter tau equals zero point zero zero one is selected as the operating point balancing both preference fidelity trade-off.

Meng: So it's working across different architectures and scales, which is good for production environments where you might have to choose a model based on available resources. But what about potential issues? What are the limitations they flag?

Lalam: They do acknowledge that there's potential for amplification of reward model biases and increased training cost relative to endpoint only methods.

Jane: So, while this framework is principled because it propagates the reward correctly, you have to be careful about inheriting any biases from the initial human preference rewards?

Tom: That's true. But they conclude that principled trajectory-level reward propagation is an effective alignment strategy across different architectures and scales. This paper on Diff-Instruct with Diffused Reward lays out a way to lift the RLHF optimal clean image target to a principled trajectory objective via the Diffused Reward Score, approximated tractably with the DRP through differentiable short step denoising.

Lu: It’s about finding a way to make sure we don't just optimize for the final pixel but that every step along the way is guided by those human preferences.

Conclusion: Tom: So we're wrapping up this look at "Diff-Instruct with Diffused Reward: Towards Principled One-step Generator RL." This paper tackles how to make one step text-to-image generators actually follow human preferences across their entire generation process.

Jane: It sounds like they took the standard way of training a model to make good images and tried to spread that guidance out over every single noise level in the diffusion. What's the main takeaway from this?

Tom: The core idea is lifting that optimal clean image target up into a trajectory objective. They call this the Diffused Reward Score, or DRS, which they calculate at every noise level. It’s not just optimizing the final picture anymore; it’s guiding every step.

Lu: What's interesting is how they make this work practically. They introduce something called the Diffused Reward Proxy to estimate that complex score without needing a ton of extra image data for every single noise step. This lets you propagate the reward signal stably across the whole trajectory.

Meng: So, it’s about getting a principled objective without relying on huge datasets for every noise level? That's what I need to know if I'm trying to deploy this kind of thing at scale.

Lalam: The authors say that they identified terminal reward domination as a problem with older methods. Their solution is this propagation mechanism, and they show it works well across different model sizes and architectures. It’s been tested on things like DiT backbones, where it outperformed longer training methods in just one step.

Jane: So what does this mean for the people using these models? If you use a one-step generator with this Diff-Instruct method, what's the real benefit compared to just using the standard ones?

Tom: The benefit is that you get alignment with human preferences while still keeping it efficient. They show it Pareto-dominates existing one-step alignment baselines on both preference and fidelity metrics. It’s a solid way to get better results faster than waiting for longer training cycles.

Lu: They also point out that they found a sweet spot for the temperature parameter, tau, around zero point zero zero one. It balances the quality of the image with how well it matches what humans want to see.

Meng: But they did mention some downsides, right? What's the catch if you actually try to run this on a production server?

Lalam: They do flag potential issues like reward model biases being amplified, and training costs might be higher compared to just optimizing for the final image alone. That’s the caveat.

Jane: It sounds like it’s a powerful alignment strategy because it solves that structural mismatch between how we optimize and how diffusion models work. So where does this leave us?

Tom: They conclude that principled trajectory-level reward propagation is an effective way to align these generators across different sizes and architectures. It moves the goalpost from just a single output to guiding the entire path of creation.

Junyi Wu, Weijian Luo, Haoyang Zheng, Ruizhe Zhang, Guang Lin

Purdue University

cs.CV, cs.AI, cs.LG

Submitted: 2026-05-18

Updated: 2026-11-10

Comments: 33 pages. Updated manuscript and author information

Code: https://github.com/black-forest-labs/flux

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: Diff-Instruct with Diffused Reward: Towards Principled One-step Generator RL is a data-free trajectory-level alignment framework that propagates RLHF-optimal reward signals across all noise levels of…

Key concepts

KL-regularized RLHF objective
This is the starting point for training, using a KL divergence regularization term. It aims to make the learned generation process stay close to a reference model while simultaneously optimizing for human preferences defined by the reward function.
Diffused Reward Score (DRS)
The DRS is a score calculated at every noise level of the diffusion trajectory. It combines a reference score with a reward-induced correction term derived from the optimal clean-image target, allowing rewards to influence generation throughout the entire denoising process.
Diffused Reward Proxy (DRP)
The DRP is an efficient estimator used to approximate the complex DRS term. It uses differentiable short-step denoising to stably propagate reward gradients across all noise levels, enabling practical training without needing full image data.

Terminology

Summary

Diff-Instruct with Diffused Reward: Towards Principled One-step Generator RL is a data-free trajectory-level alignment framework that propagates RLHF-optimal reward signals across all noise levels of a diffusion trajectory to create a principled objective for one-step text-to-image generators The core insight is to systematically propagate reward signals defined on clean images to every noise level of the diffusion trajectory This method resolves the structural mismatch in prior methods where rewards are applied only at the clean image output while regularization spans all noise levels, which leads to terminal reward domination

Pruning redundant reasoning tokens

The authors propose Diff-Instruct with Diffused Reward (Didr) as a principled reinforcement learning framework for one-step image generators without using image data The framework starts from the KL-regularized RLHF objective, identifying its optimal clean-image target as the reward-tilted density q∗(x0c) ∝ q0(x0c) exp(r(x0, c)/τ) This optimal target is then diffused through the reference process to obtain trajectory-level reward-tilted marginals q∗ t(xtc) Minimizing the Integral KL (IKL) divergence against these marginals yields a target score that decomposes into the reference score plus a reward-induced correction, the Diffused Reward Score (DRS), defined at every noise level through reference posterior

How the training signal is built

The training objective for Didr is defined as minimizing the IKL divergence between its forward-noised marginals p θ,t and the reward-tilted trajectory q∗ t This objective is LDIDR(θ) = Ec∼C ∫ T 0 w(t) DKLp θ,t (·c)∥q∗ t (·c) The target score decomposition shows that the trajectory target score is the reference score plus a reward-induced correction, DRS(xt, t, c) = sref + sr This DRS term is defined as s r(xt, t, c) = ∇xt log Ex0∼q(x0xt,c) exp(r(x0,c)/τ)

Where the gains come from

The Diffused Reward Proxy (DRP) is introduced as an efficient estimator of DRS based on differentiable short-step denoising This allows for stable reward-gradient propagation across the entire trajectory by approximating the intractable posterior expectation using Monte Carlo estimators The practical generator update is driven by the mismatch between its own score and the reward-tilted target score sref + sr Extensive experiments demonstrate that Didr consistently Pareto-dominates existing one-step SDXL baselines

Ablation studies and results

Empirically, when transferred to a 6B DiT backbone (Z-Image), Didr surpasses its 50-step teacher in preference alignment while requiring only a single generation step Standard Didr Pareto-dominates all one-step alignment baselines on preference vs. fidelity Furthermore, Didr generalizes to DiT architectures at larger scale, where zimage-Didr surpasses the 50-step Z-Image base on all preference metrics in a single step The temperature hyperparameter τ=0. p. 7

Where the paper says it

Didr generalizes across architectures and model scales. Didr generalizes to DiT architectures at larger scale: applied to the 6B-parameter Z-Image backbone, zimage-Didr surpasses the 50-step Z-Image base on all preference metrics in a single step.

01 is selected as the operating point balancing both preference–fidelity trade-off

Conclusion and Limitations

We identified terminal reward domination as a fundamental failure mode of trajectory distillation with terminal rewards, and proposed Didr to resolve it The core idea is to lift the RLHF-optimal clean-image target q∗ to a principled trajectory objective via the Diffused Reward Score, and approximate it tractably with the Diffused Reward Proxy through differentiable short-step denoising—requiring no image training data Limitations include potential amplification of reward model biases and increased training cost relative to endpoint-only methods The paper concludes that principled trajectory-level reward propagation is an effective alignment strategy across architectures and scales

References

Stability AI. Stable Diffusion 3. p. 2

Where the paper says it

• We analytically derive DRS, which exactly decomposes the trajectory target score into a reference score plus a reward-induced correction. To make this tractable, we introduce DRP, a zero-data estimator that computes this correction via differentiable short-step posterior denoising through a frozen reference model, enabling stable reward-gradient propagation across the entire trajectory.

5. https://stability.ai/news/stable-diffusion-3-5, 2024

Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training Diffusion Models with Reinforcement Learning. p. 0

Where the paper says it

, 2024b) by minimizing an Integral Kullback-Leibler divergence. Later works based on concepts of distribution matching explored the reinforcement learning post-training of one-step text-to-image generators by combining image-space rewards with latent space distribution matching (Luo, 2024; Luo et al.

In Proc. of the International Conference on Learning Representation (ICLR), 2024

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences. p. 3

Where the paper says it

To construct a mathematically sound trajectory objective, we must first establish the correct alignment target at the clean-image level. Standard Reinforcement Learning from Human Feedback (RLHF) (Christiano et al.

In Advances in Neural Information Processing Systems (NeurIPS), 2017

Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Marc L Klasky, and Jong Chul Ye. Diffusion Trajectories and One-step Generators. p. 2

Where the paper says it

Furthermore, Didr successfully transfers to a 6B DiT-based Z-Image backbone, surpassing its 50-step teacher on preference metrics in a single step. Preliminaries Diffusion Trajectories and One-step Generators.

In Advances in Neural Information Processing Systems (NeurIPS), 2017

Kevin Clark, Paul Vicol, Kevin Swersky, and David J Fleet. Posterior Sampling for General Noisy Inverse Problems. p. 5

Where the paper says it

) The DRS term 𝑠𝑟 adds a reward-driven correction to the reference score, steering the generator toward the RLHF optimum at every noise level. Because 𝑠𝑟 is defined through the posterior 𝑞(𝑥0|𝑥𝑡 , 𝑐), it propagates preference information only where the clean image is identifiable from the noisy observation, naturally attenuating reward guidance at high noise.

In Proc. of the International Conference on Learning Representation (ICLR), 2023

Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack. p. 0

Where the paper says it

Moreover, when transferred to a 6B DiT backbone (Z-Image), Didr surpasses its 50-step teacher in preference alignment while requiring only a single generation step. Introduction Deep generative models have achieved remarkable progress in text-to-image (T2I) synthesis (Esser et al.

In arXiv preprint arXiv:2309. p. 0
Where the paper says it

We refer to the resulting tendency to over-tilt toward high-reward outputs as terminal reward domination, and make it precise in a tractable example (§3. In contrast, Didr derives a principled trajectory-level target directly from the KL-regularized RLHF optimum, ensuring arXiv:2605.

15807, 2023

Prafulla Dhariwal and Alexander Nichol. Diffusion Models Beat GANs on Image Synthesis. p. 0

Where the paper says it

Moreover, when transferred to a 6B DiT backbone (Z-Image), Didr surpasses its 50-step teacher in preference alignment while requiring only a single generation step. Introduction Deep generative models have achieved remarkable progress in text-to-image (T2I) synthesis (Esser et al.

In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 8780–8794, 2021

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Aichele, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. p. 2

Where the paper says it

While the reference model generates high-quality images, its iterative sampling process is computationally expensive. To achieve real-time synthesis, a one-step generator (Goodfellow et al.

In Proc. of the International Conference on Machine Learning (ICML), 2024

Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models. p. 9

Where the paper says it

Related Work RLHF and Preference Alignment for Diffusion Models. Preference alignment for multi-step diffusion models uses trajectory-accessible signals via supervised fine-tuning (Dai et al.

In Advances in Neural Information Processing Systems (NeurIPS), 2023

Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment. p. 7

Where the paper says it

Didr also maintains text alignment, achieving the highest DPG-Bench (75. 06) and GenEval (0.

In Advances in Neural Information Processing Systems (NeurIPS), 2023

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. p. 0

Where the paper says it

Previous reinforcement learning methods for one-step generators combines the image space reward optimization and the diffusion noisy space distribution matching. This paradigm brings challenges due to a mismatch between terminal reward optimization and the underlying generative dynamics.

In Advances in Neural Information Processing Systems (NeurIPS), 2014

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A ReferenceFree Evaluation Metric for Image Captioning. p. 7

Where the paper says it

Didr generalizes to DiT architectures at larger scale: applied to the 6B-parameter Z-Image backbone, zimage-Didr surpasses the 50-step Z-Image base on all preference metrics in a single step. Against the 8-step Z-Image-Turbo, it leads on ImageReward, Aesthetic Score, CLIPScore, and DPG-Bench with lower FID, while trailing on PickScore, HPSv2.

In Proc. of the Conference on Empirical Methods for Natural Language Processing (EMNLP), 2021

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. p. 7

Where the paper says it

As the reward model 𝑟(𝑥0, 𝑐), we use PickScore (Kirstain et al. , 2023), a widely accepted human preference model trained on large-scale pairwise human annotations that scores how well a generated image matches human preference given a text prompt.

In Advances in Neural Information Processing Systems (NeurIPS), 2017

Jiwoo Hong, Sayak Paul Lee, Ji-hoon Moon, Junhyuk Kim, Jaemin Ham, Moontae Jeon, et al. MaPO: Margin-aware Preference Optimization for Aligning Diffusion Models without Reference. p. 3

Where the paper says it

This structural mismatch creates a fundamental optimization loophole: because forward noise naturally masks image details, the regularization penalty against mode divergence drastically weakens at high noise levels. Optimizers readily exploit this trajectory vulnerability, abandoning the reference distribution to collapse onto high-reward regions without incurring sufficient KL penalty.

In arXiv preprint arXiv:2406. p. 0
Where the paper says it

We refer to the resulting tendency to over-tilt toward high-reward outputs as terminal reward domination, and make it precise in a tractable example (§3. In contrast, Didr derives a principled trajectory-level target directly from the KL-regularized RLHF optimum, ensuring arXiv:2605.

06424, 2024

Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment. p. 9

Where the paper says it

01 as the operating point balancing both (Table 3, Figure 7). Related Work RLHF and Preference Alignment for Diffusion Models.

In arXiv preprint arXiv:2403. p. 0
Where the paper says it

We refer to the resulting tendency to over-tilt toward high-reward outputs as terminal reward domination, and make it precise in a tractable example (§3. In contrast, Didr derives a principled trajectory-level target directly from the KL-regularized RLHF optimum, ensuring arXiv:2605.

05135, 2024

Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up GANs for Text-to-Image Synthesis. p. 0

Where the paper says it

Diff-Instruct with Diffused Reward: Towards Principled One-step Generator RL Junyi Wu1 , Weijian Luo2 , Haoyang Zheng1 , Ruizhe Zhang1 and Guang Lin1 1Purdue University, 2hi-lab, Xiaohongshu Inc. Recent advances in one-step text-to-image generation have enabled real-time synthesis with remarkable efficiency and quality.

In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023

Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and Improving the Image Quality of StyleGAN. p. 0

Where the paper says it

This paradigm brings challenges due to a mismatch between terminal reward optimization and the underlying generative dynamics. As a result, optimization tends to exploit stochastic degrees of freedom, often improving reward at the expense of image fidelity.

In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020

Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. p. 0

Where the paper says it

This paradigm brings challenges due to a mismatch between terminal reward optimization and the underlying generative dynamics. As a result, optimization tends to exploit stochastic degrees of freedom, often improving reward at the expense of image fidelity.

In Proc. of the International Conference on Learning Representation (ICLR), 2015

Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Peng, and Omer Levy. Pick-aPic: An Open Dataset of User Preferences for Text-to-Image Generation. p. 0

Where the paper says it

Diff-Instruct with Diffused Reward: Towards Principled One-step Generator RL Junyi Wu1 , Weijian Luo2 , Haoyang Zheng1 , Ruizhe Zhang1 and Guang Lin1 1Purdue University, 2hi-lab, Xiaohongshu Inc. Recent advances in one-step text-to-image generation have enabled real-time synthesis with remarkable efficiency and quality.

In Advances in Neural Information Processing Systems (NeurIPS), 2023

Cindy Xiangrui Kong, Yueqi Wang, Haoyang Zheng, Weijian Luo, and Guang Lin. Ultra Fast PDE Solving via Physics Guided Few-step Diffusion. arXiv preprint arXiv:2602. p. 0

Where the paper says it

We refer to the resulting tendency to over-tilt toward high-reward outputs as terminal reward domination, and make it precise in a tractable example (§3. In contrast, Didr derives a principled trajectory-level target directly from the KL-regularized RLHF optimum, ensuring arXiv:2605.

03627, 2026

Black Forest Labs. FLUX.1 Models. https://github.com/black-forest-labs/flux, 2024

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In Proc. of the European Conference on Computer Vision (ECCV), 2014

Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. p. 5

Where the paper says it

We train the generator by minimizing the IKL divergence between its forward-noised marginals 𝑝𝜃,𝑡 and the reward-tilted trajectory {𝑞 ∗ 𝑡 }: LDIDR(𝜃) = 𝔼𝑐∼C ∫ 𝑇 0 𝑤(𝑡) DKL𝑝𝜃,𝑡 (·|𝑐) ∥𝑞 ∗ 𝑡 (·|𝑐) d𝑡 . (5) By minimizing IKL against 𝑞 ∗ 𝑡 rather than the bare reference marginals, Didr encodes the full RLHF signal inside the trajectory without a separate terminal reward term, inheriting the broad distribution support and stable gradient flow of the IKL formulation while correctly injecting reward signals at every noise level.

arXiv preprint arXiv:2209. p. 0
Where the paper says it

We refer to the resulting tendency to over-tilt toward high-reward outputs as terminal reward domination, and make it precise in a tractable example (§3. In contrast, Didr derives a principled trajectory-level target directly from the KL-regularized RLHF optimum, ensuring arXiv:2605.

03003, 2022

Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, and Qiang Liu. InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation. p. 0

Where the paper says it

Recent advances in one-step text-to-image generation have enabled real-time synthesis with remarkable efficiency and quality. Previous reinforcement learning methods for one-step generators combines the image space reward optimization and the diffusion noisy space distribution matching.

In Proc. of the International Conference on Learning Representation (ICLR), 2024

Weijian Luo, A Comprehensive Survey on Knowledge Distillation of Diffusion Models. arXiv preprint arXiv:2304. p. 0

Where the paper says it

We refer to the resulting tendency to over-tilt toward high-reward outputs as terminal reward domination, and make it precise in a tractable example (§3. In contrast, Didr derives a principled trajectory-level target directly from the KL-regularized RLHF optimum, ensuring arXiv:2605.

04262, 2023

Weijian Luo, Diff-Instruct++: Training One-step Text-to-image Generator Model to Align with Human Preferences. Transactions on Machine Learning Research (TMLR), 2024

Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff-Instruct: A Universal Approach for Transferring Knowledge from Pre-trained Diffusion Models. p. 0

Where the paper says it

While these models achieve real-time synthesis, they frequently exhibit suboptimal aesthetic quality and poor alignment with human preferences—a central requirement for real-world deployment. Diff-Instruct(Luo et al.

p. 0
Where the paper says it

While these models achieve real-time synthesis, they frequently exhibit suboptimal aesthetic quality and poor alignment with human preferences—a central requirement for real-world deployment. Diff-Instruct(Luo et al.

In Advances in Neural Information Processing Systems (NeurIPS), 2023a

Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff-Instruct: A Universal Approach for Transferring Knowledge from Pre-trained Diffusion Models. In Advances in Neural Information Processing Systems (NeurIPS), 2023b

Weijian Luo, Colin Zhang, Debing Zhang, and Zhengyang Geng. David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training. p. 0

Where the paper says it

, 2024b) by minimizing an Integral Kullback-Leibler divergence. Later works based on concepts of distribution matching explored the reinforcement learning post-training of one-step text-to-image generators by combining image-space rewards with latent space distribution matching (Luo, 2024; Luo et al.

In Proc. of the International Conference on Machine Learning (ICML), 2025

Alexander Quinn Nichol and Prafulla Dhariwal. Improved Denoising Diffusion Probabilistic Models. p. 9

Where the paper says it

01 as the operating point balancing both (Table 3, Figure 7). Related Work RLHF and Preference Alignment for Diffusion Models.

In Proc. of the International Conference on Machine Learning (ICML), 2021

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Kelsey Slama, Alex Ray, et al. Training Language Models to Follow Instructions with Human Feedback. p. 3

Where the paper says it

To construct a mathematically sound trajectory objective, we must first establish the correct alignment target at the clean-image level. Standard Reinforcement Learning from Human Feedback (RLHF) (Christiano et al.

Advances in Neural Information Processing Systems (NeurIPS), 2022

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Peng, and Robin Rombach. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. p. 7

Where the paper says it

, 2024) at 1024 × 1024 resolution, initializing the generator from the DMD2- SDXL-1step checkpoint (Yin et al. , 2024a) and using the pretrained SDXL diffusion model as both the frozen reference 𝑠ref and the TA initialization.

In Proc. of the International Conference on Learning Representation (ICLR), 2024

Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning Text-to-Image Diffusion Models with Reward Backpropagation. p. 0

Where the paper says it

Recent advances in one-step text-to-image generation have enabled real-time synthesis with remarkable efficiency and quality. Previous reinforcement learning methods for one-step generators combines the image space reward optimization and the diffusion noisy space distribution matching.

In arXiv preprint arXiv:2310. p. 0
Where the paper says it

We refer to the resulting tendency to over-tilt toward high-reward outputs as terminal reward domination, and make it precise in a tractable example (§3. In contrast, Didr derives a principled trajectory-level target directly from the KL-regularized RLHF optimum, ensuring arXiv:2605.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Diffusion trajectory-level alignment allows one-step generators to map latent noise directly to images while ensuring that reward and regularization are balanced at every noise level. This resolves terminal reward domination by propagating RLHF-optimal reward signals across all noise levels, rather than just at the clean image endpoint.

  2. A differentiable estimator called the Diffused Reward Proxy (DRP) approximates the intractable Diffused Reward Score (DRS) using differentiable short-step denoising chains from x t using the frozen reference model, enabling stable reward-gradient propagation back through the trajectory to update weights.

  3. The system can achieve a superior PickScore–FID trade-off compared to existing one-step methods, as shown by Didr Pareto-dominating all existing one-step alignment baselines on preference vs. fidelity metrics at 1024x1024 resolution.

  4. The model can generalize to different architectures and scales; specifically, the Z-Image backbone (6B DiT) surpasses its 50-step teacher in preference alignment while requiring only a single generation step.

Abstract

Recent advances in one-step text-to-image generation have enabled real-time synthesis with remarkable efficiency and quality. Previous reinforcement learning methods for one-step generators combine image-space reward optimization with diffusion noisy-space distribution matching. This paradigm brings challenges due to a mismatch between terminal reward optimization and the underlying generative dynamics. As a result, optimization tends to exploit stochastic degrees of freedom, often improving reward at the expense of image fidelity. To address this issue, we propose Diff-Instruct with Diffused Reward (DIDR), a data-free trajectory-level alignment framework derived from Integral KL minimization. DIDR propagates the RLHF-optimal reward-tilted clean-image distribution across all noise levels along the diffusion trajectory. We show that this objective admits the same minimizer as clean-image RLHF, while naturally inducing the Diffused Reward Score (DRS), which acts as a reward-driven correction to the reference score function. To make this practical, we further introduce the Diffused Reward Proxy (DRP), an efficient estimator of DRS based on differentiable short-step denoising. Extensive experiments demonstrate that DIDR consistently Pareto-dominates existing one-step SDXL baselines. Moreover, when transferred to a 6B DiT backbone (Z-Image), DIDR surpasses its 50-step teacher in preference alignment while requiring only a single generation step. Code: https://github.com/Junyi-W/DIDR

Sources

Related papers