Diff-Aid: Inference-time Adaptive Interaction Denoising for Rectified Text-to-Image Generation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Diff-Aid: Inference-time Adaptive Interaction Denoising for Rectified Text-to-Image Generation".
Jane: Here's the gist of the paper. Diff-Aid is a lightweight inference-time method that adaptively adjusts per-token text and image interactions across transformer blocks and denoising timesteps The paper proposes Diff-Aid to provide a more flexible and efficient solution to the problem of insufficient interactions between textual conditions and image features during text-to-image generation Problem Context Recent text-to-image (T2I) diffusion models have achieved remarkable advancement, yet faithfully following complex textual descriptions remains challenging due to insufficient interactions between textual and visual features Prior approaches enhance such interactions via architectural design or handcrafted textual condition weighting, but lack flexibility and overlook the dynamic interactions across different blocks and denoising stages Proposed Mechanism Diff-Aid introduces two key innovations: 1) an Aid module, which explicitly adjusts textual conditions according to the varied blocks, timesteps, and tokens; and 2) a gated sparsity and stabilized regularization mechanism that respectively ensure attaching more attention to important blocks and avoiding training collapse The core of this is the proposed Adaptive interaction denoising (AiD) module, which learns the coefficient αl t ∈ R N to modulate text-image interactions based on the current diffusion timestep, transformer block, and text features This modulation is applied as ec l t = c l t + c l t ⊙ α l t, where ⊙ is element-wise multiplication Learning Objectives The learning objective for the Aid module is defined by the combined loss L = Ldif f + λdpoLdpo + λregLreg The diffusion loss Ldif f minimizes the squared error between predicted velocity and target velocity, while DPO loss Ldpo optimizes the model based on human preference data to align outputs with human judgments Furthermore, training is stabilized by randomly skipping the Aid modules during training with a probability of p Empirical Validation Experiments on strong baselines (SD 3.5 and FLUX) demonstrate consistent improvements in prompt adherence, visual quality, and human preference across various metrics The learned parameters for adjustment are interpretable with respect to different blocks, varied timesteps, and textual tokens Ablation studies confirm that the boundedness and sparsity constraints are necessary for optimal performance, showing that using Tanh activation yields better results than simply clipping the α values The sparse enhancement strategy, which selects a subset of transformer blocks based on learned α values, achieves similar outputs as the full method Application Versatility Diff-Aid is designed as a plug-in and generalizable module that can be seamlessly integrated into homologous models for controllable image generation (b), styled LoRA generation (c), and zero-shot editing (d) It has been shown to enhance performance when combined with existing LoRAs from the community, exhibiting richer details and more vivid textures The method also shows consistent improvements on widely-used metrics such as HPSv2, ImageReward, and Aesthetic Score when applied to other evaluation benchmarks The method can be flexibly applied to different applications while obtaining competitive performance improvements for both style LoRAs and control conditions generation We presented Diff-Aid, a plug-in module for rectified textto-image diffusion transformers that learns to adaptively adjust block-wise interactions between token-level textual conditions across denoising trajectories Beyond improving the generation quality, Diff-Aid provides insights into the dynamic interplay of cross-modal signals in transformer blocks and can be seamlessly applied to various tasks The work aims to improve diffusion models for text-to-image generation, with potential applications in creative fields like design and content creation Page 1 --- Diff-Aid: Inference-time Adaptive Interaction Denoising for Rectified Text-to-Image Generation Binglei Li 1 2 Mengping Yang 1 3 † Zhiyu Tan 1 3 Junping Zhang 1 Hao Li 1,2,3 a photo of a yellow stop sign and a blue potted plant a photo of a purple sheep and a pink banana FLUX FLUX + Ours Man from Namibia, hexagon, plain blue-pink FLUX FLUX + Ours Canny input FLUX Canny FLUX Canny + Ours Depth input FLUX Depth InputFLUX Depth + Ours Wolf WallpapersFor Tablets fishing boatson the Danube Canal Vienna Poster Original output FLUX SketchFLUX Sketch + Ours Original outputFLUX GlassFLUX Glass + Ours A male druid wearing a tattered spiderweb hood standing next to a bear A pug dog with a cat beautiful fluffy with big eyes, white cloud, blue sky.
Tom: Source image FLUX KontextFLUX Kontext + Ours make the woman to an elf Add a tiger to the painting (a) Text-to image (b) Conditional control (c) Lora integration (d) Instructional Editing Figure 1.
Jane: We present Diff-Aid to adaptively improve the interaction between textual conditions and image latents during test time.
Tom: Abstract Recent text-to-image (T2I) diffusion models have achieved remarkable advancement, yet faithfully following complex textual descriptions remains challenging due to insufficient interactions be1 Fudan University 2 Shanghai Innovation Institute 3 Shanghai Academy of AI for Science, † Project Lead.
Title and authors: Tom: So we’re looking at the paper titled "Diff-Aid: Inference-time Adaptive Interaction Denoising for Rectified Text-to-Image Generation." It seems like this whole thing is about fixing a problem where text and images don't talk to each other well enough during the generation process.
Jane: Right, so before this paper, models were great at making pictures but they often struggled to follow really complicated text descriptions accurately because the connection between what you asked for and what the model was actually drawing wasn't tight enough.
Lu: Exactly. They mention that previous approaches tried to fix this by changing the architecture or adding some manual weighting for text conditions, but those methods lacked flexibility because they didn't account for how things change across different blocks or different stages of the denoising process.
Meng: So what Diff-Aid is doing is trying to be more dynamic than those fixed methods. It’s not just a one-size-fits-all fix; it adapts based on where it is in the generation steps and what text tokens are involved.
Tom: That sounds like they're introducing something that learns how to adjust the text and image interactions on the fly during inference, which is pretty interesting from a mechanics standpoint.
Jane: The core of their idea is this "Aid module," which explicitly adjusts the textual conditions based on the specific transformer blocks, the denoising timesteps, and even individual tokens in your prompt.
Lu: They introduce something called Adaptive interaction denoising, or AiD, which learns these coefficients called alpha values that modulate those text-image interactions based on the current timestep, block, and text features.
Meng: The formula they use is pretty straightforward: the adjusted interaction is the original interaction multiplied element by element by this alpha value.
Tom: And to make sure this isn't just a wild guess, they have two mechanisms for that: a gated sparsity and stabilized regularization mechanism to make sure it focuses on important parts and doesn't collapse during training.
Jane: They also define the learning goal using a combination of diffusion loss, which minimizes the error between predicted velocities and the actual target velocities, plus a DPO loss to align outputs with human preferences.
Lu: And they stabilize training by randomly skipping these Aid modules during training with some probability, which is a way to ensure it learns robust patterns instead of just overfitting to one specific interaction path.
Title and authors: Meng: So, what’s the actual result they show? The paper claims that experiments on strong baselines like SD three point five and FLUX show consistent improvements in prompt adherence, visual quality, and human preference across various metrics <ref:2602.13585#pg2,consistent improvements in prompt adherence, visual quality, and human preference across various>.
Tom: It sounds like the numbers back up the concept of better following of instructions. I'm curious how much better it is compared to just using the baseline models without this adjustment module.
Jane: They show that by using Diff-Aid, you get much tighter semantic alignment during generation, and they make sure these learned adjustment patterns are interpretable regarding which blocks and tokens are most important for the interaction.
Lu: The ablation studies confirm that having those boundedness and sparsity constraints is necessary for the best performance; they found that using a tanh activation yields better results than just clipping those alpha values.
Meng: And this leads to their sparse enhancement strategy, where they select a subset of transformer blocks based on the learned alpha values, and they show that this achieves similar outputs as using the full method.
Tom: That’s good validation; it means the learned coefficients are actually pointing toward what matters in the model's structure. It moves beyond just saying "it works" to showing *why* it works at a structural level.
Jane: Beyond just generation quality, they show this module is designed as a plug-in, meaning you can easily integrate it into different things like LoRAs or instructional image editing tasks without having to completely rebuild the entire model pipeline.
Lu: That versatility is key; because it's a generalizable module, they’ve shown it enhances performance when combined with existing community LoRAs, leading to richer details and more vivid textures in the generated images.
Meng: From a deployment standpoint, I’m interested in how easy this is to integrate into production. They show consistent improvements on metrics like HPSv2 and ImageReward when applied to other evaluation benchmarks, which suggests it’s not just a lab curiosity.
Tom: So, the main thing for me is that they managed to find this dynamic interplay between cross-modal signals in those transformer blocks during the denoising process. That kind of insight into the dynamics is what excites me about how these models actually work.
Jane: And it opens up possibilities for creative fields like design and content creation because it lets you control these generated images with much finer detail based on your text instructions.
Lu: It provides insights into the dynamic interplay of signals, which is valuable because it shows that the way information flows through those layers during denoising has specific patterns that we can learn to exploit.
Title and authors: Meng: But they did flag a limitation, and they mention that while it works well with strong baselines like SD three point five and FLUX, the performance might depend on how well those baselines are already conditioned initially <ref:2602.13585#pg2,SD 3.5 and FLUX>.
Tom: That’s a fair caveat; if you start with a very weak model, the adaptation module might struggle to find good patterns to latch onto.
Jane: So, we're looking at Diff-Aid as this adaptive module that learns to adjust per-token text and image interactions across transformer blocks and denoising timesteps during inference.
Lu: It’s a lightweight approach because it’s designed for inference time, not retraining the whole model from scratch.
Meng: And its plug-in nature means it can be used for things like style LoRAs, controllable generation, and zero-shot editing with minimal effort to integrate.
Tom: So that’s the picture: an adaptive adjustment layer that makes text and image features interact much more intelligently during the final steps of image creation.
Jane: Before we wrap up on this paper about Diff-Aid, Lu wants to say something about what this means for how we view these diffusion models in general.
Lu: It shows that learning meaningful and interpretable interaction patterns aligned with the denoising dynamics is a valid way to understand *why* some models follow instructions better than others.
Meng: From my side, it means that if you want to push production quality, you need mechanisms that can dynamically tune the conditioning based on where the generation is happening in real time.
Lalam: I'm confident in this direction because as a model trained on vast amounts of data, I see these patterns emerge naturally when we give the model a way to learn how to better map text concepts onto visual latents during sampling.
Tom: So, Diff-Aid is definitely a tool that gives us more control and better results without needing massive retraining efforts.
Jane: We’ve covered the title, the summary of what it is doing, and the improvements they found regarding its design.
Lu: We also touched on how this mechanism helps us see inside the dynamics of these diffusion models during inference.
Meng: And we brought up how it can be integrated into real-world applications like LoRAs and controllable generation to get practical results.
Lalam: It’s a solid contribution because it offers a concrete, measurable way to improve prompt adherence using learned parameters that are actually sparse and bounded.
The paper's summary: Tom: So, Diff-Aid is basically this new tool that lets text and images interact adaptively while you’re actually generating something during inference.
Jane: Right, so it's not just one fixed way to connect the prompt to the picture; it learns how to adjust those connections on the fly across different parts of the model and at different moments during image creation.
Lu: The core idea is this Aid module that learns coefficients, which we call alpha values, that change depending on which transformer block you’re in, what denoising step you’re on, and even what specific words are in your text prompt.
Meng: It sounds like it's making fine-grained adjustments to the text-image interaction during the entire denoising trajectory. I want to know if this means we can get much better control over subtle things, like textures or specific lighting, when we run inference on production models.
Tom: Well, the paper shows that by learning these alpha values, it really tightens up how well the text conditions match the image latents during generation. It’s about making sure the right words influence the right visual features at every step.
Jane: And they show this isn't just a random adjustment; they use a gated sparsity and a regularization mechanism to keep those alpha values focused on what matters, so you don't get noise or instability during training or generation.
Lu: They define the learning goal using diffusion loss, which measures how far off the predicted image velocity is from the real one, and they also use DPO loss to align outputs with what humans actually prefer.
Meng: That human preference alignment part is important. It means if the model gets a visually pleasing result that people like, it's getting rewarded during training, which is a good sign for practical deployment.
Tom: And the results on SD three and FLUX show consistent improvements in prompt adherence and overall visual quality compared to just using the baseline models without this added module.
Jane: They even showed that by picking only the most important transformer blocks based on those learned alpha values, it achieves similar performance to using every single block, which is a very efficient way to do things.
Lu: The authors also did some ablation studies showing that putting constraints on how big those alpha values can get and making them sparse is necessary for the best results; they found using a tanh activation works better than just cutting off the values.
Meng: That sparsity strategy is interesting because it suggests we don't need to tune every single interaction point in the model structure, just focus on the most influential ones.
Tom: It’s designed as a plug-in, which means you can drop this module into existing models for things like style LoRAs or even zero-shot editing tasks without having to rewrite the whole diffusion transformer.
Jane: That versatility is huge because it lets researchers test how this dynamic interaction works across completely different kinds of generation tasks, not just standard text-to-image.
Lu: They also showed that combining Diff-Aid with community LoRAs can actually make the generated images richer and have more detailed textures than using either one alone.
Meng: So, for production, this means we could use it to fine-tune our existing models to better follow complex instructions or handle specific style edits more effectively without needing a massive retraining budget.
Tom: It shows that understanding the dynamic interplay between text and image signals during the denoising process gives us a new way to control generation quality right at inference time.
Jane: This opens up possibilities for creative work, like design, because you get much more precise control over how those textual concepts are translated into the final visual output.
Lu: It's a good example of learning meaningful interaction patterns based on the dynamics of the process itself rather than just looking at static features.
Meng: If we can reliably tune these parameters, it moves us closer to having models that are truly controllable and not just good at generating pretty pictures by default.
The paper's improvements: Tom: So, we're looking at how Diff-Aid actually improves things—it’s not just about having an idea; it’s about what they proved with their numbers and experiments.
Jane: Right, so they show that when you use this method, the images generated by models like SD three and FLUX have noticeably better prompt adherence and higher visual quality compared to using those models without the Diff-Aid module.
Lu: They provide quantitative comparisons on benchmarks like HPSv3 and other image quality metrics, showing that Diff-Aid consistently wins in those areas because it improves how well the model follows what you ask for.
Meng: I’m looking at the results for instruction editing tasks on EditBench, and it confirms that this method works well when you're trying to make specific changes to an existing image based on new text instructions.
Tom: They also have some interesting ablation studies showing that the sparsity constraints and boundedness constraints are necessary; they found that using a tanh activation is better than just clipping those alpha values.
Jane: That tells us that the way they learned those coefficients is robust, and it’s not just a fluke; it’s based on how the model actually behaves across different timesteps and blocks.
Lu: And their sparse enhancement strategy really shows that by picking only a subset of transformer blocks based on those learned alpha values, you can get results very close to using every single block in the full method.
Meng: That makes sense for deployment because it means we don't have to run everything at full computational cost just to get good results; we can be smart about which parts are actually doing the heavy lifting.
Tom: It’s also shown that this plug-in works across different tasks, like style LoRAs and control condition generation, where it still gives competitive performance improvements.
Jane: The versatility is key here because it means you can use this same module to enhance different types of controllable generation without having to reinvent the wheel for each one.
Lu: They also show that when combined with existing community LoRAs, Diff-Aid actually leads to richer details and more vivid textures in the final output, which is pretty cool.
Meng: That’s good because it means we can take models that have already been trained for styles and make them perform even better with this layer on top.
Tom: So, the main point is that this isn't just a theoretical idea; they have shown consistent improvements across various metrics when applied to strong baselines.
Jane: They also include failure cases where baseline models struggle, and Diff-Aid still manages to capture some of the key elements from the prompt even when it doesn't fully comply.
Lu: The authors also clearly state that this is designed as a plug-in for rectified text-to-image diffusion transformers, which is important because it shows how applicable this idea is to existing model architectures.
Meng: It’s a practical addition; if we can integrate something like this into our production pipeline, it means better control over the output quality without needing to overhaul the entire generation system.
Tom: So, Diff-Aid gives us a concrete way to get better prompt following and visual fidelity by learning how text and image features interact during the denoising process.
Jane: And it gives us insight into that dynamic interplay, which is valuable for understanding *why* some models follow instructions better than others.
Lu: This work pushes the idea that learning meaningful interaction patterns aligned with the denoising dynamics is a valid way to improve model behavior in generation.
Conclusion: Tom: So, to wrap up on Diff-Aid, we’re looking at how this paper tackles the problem of models not really connecting textual conditions to image features during generation by introducing adaptive adjustments during inference.
Jane: It boils down to learning coefficients that change based on where the model is in the denoising process and what text tokens are involved, which significantly improves prompt following and visual quality on models like SD three and FLUX.
Lu: The core contribution is this Adaptive interaction denoising module that learns alpha values, modulated by the current timestep, transformer block, and text features to adjust those interactions element by element.
Meng: From a deployment standpoint, this means we can integrate this as a plug-in for style LoRAs or instructional editing without having to retrain the entire model architecture from scratch.
Tom: They also showed that they don't need every single interaction point active, because the sparse enhancement strategy actually achieves similar performance to using the full method.
Jane: That sparsity is a big deal because it makes the system more efficient and focused on what matters most for those cross-modal signals.
Lu: And they proved that this approach is robust, showing that constraints on how big those alpha values can get are necessary for optimal performance, with tanh activation being better than just clipping them.
Meng: I still wonder about the computational cost of running this adaptation module in real-time across all those blocks and timesteps; if it’s too slow, it defeats the purpose of inference time efficiency.
Tom: That’s a fair point, Meng, but they frame their learning objective with diffusion loss and DPO loss to keep the training stable, suggesting they optimized for a workable balance.
Jane: So even with those computational costs in mind, this method provides powerful insights into the dynamic interplay of signals inside transformer blocks during denoising trajectories.
Lu: It confirms that learning meaningful interaction patterns based on the dynamics of the process itself is a valid way to improve model behavior in generation tasks.
Meng: If we can reliably tune these parameters, it means we can push models toward better controllability and more precise results with less heavy retraining overhead.
Tom: Overall, this paper on Diff-Aid shows how to make text-to-image generation smarter by making the interactions between text and image features dynamically adjustable during the actual image creation process.
Jane: It gives us a concrete tool for improving prompt adherence while opening up possibilities for more creative control in design and content creation fields.
Lu: This work sets a precedent for using learned, dynamic adjustments to fine-tune how generative models map textual concepts onto visual latents during sampling.
Meng: I'm still watching how they handle the scaling of those alpha values across different model sizes; if that doesn't scale well, it won't be practical for larger models at scale.
Tom: It’s a solid piece of work on inference-time adaptation for text-to-image generation, Diff-Aid.
Jane: And next time we talk about how AI agents are learning to synthesize scientific conclusions or how we can better handle emergent misalignment in other areas.
Fudan University
cs.CV
Submitted: 2026-02-14
Updated: 2026-11-11
Comments: NeurIPS2026, 35 pages
Code: https://github.com/black-forest-labs/flux
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 89/100
The gist: Diff-Aid is a lightweight inference-time method that adaptively adjusts per-token text and image interactions across transformer blocks and denoising timesteps The paper proposes Diff-Aid to provide…
Key concepts
- Adaptive Interaction Denoising (AiD) Module
- This is the core innovation that learns a coefficient $\alpha_{l}^{t}$ to modulate text-image interactions. It dynamically adjusts these interactions based on the current diffusion timestep, transformer block, and text features. This allows the model to tailor how much attention it pays to specific parts of the text and image during each step of denoising.
- Gated Sparsity and Stabilized Regularization
- This mechanism ensures that the learned modulation coefficients ($\alpha_{l}^{t}$) are sparse, meaning only important blocks or tokens receive significant adjustment. A regularization term penalizes large $\alpha$ values, forcing the model to focus its interaction on the most relevant parts of the input, which improves efficiency and prevents training collapse.
- Adaptive Weights ($\alpha_{l}^{t}$)
- These are learned coefficients that determine how strongly textual conditions influence image features at specific points in the generation process. The system learns these weights across different transformer blocks and denoising timesteps, allowing for a flexible and context-aware adjustment of the text-image relationship throughout the entire generation trajectory.
- Plug-in Design Versatility
- Diff-Aid is designed as a general module that can be easily integrated into many different tasks, such as style LoRA generation or zero-shot editing. This plug-in nature allows it to enhance existing models and improve performance in diverse applications by adapting its interaction strategy to the specific requirements of the downstream task.
Terminology
Summary
Diff-Aid is a lightweight inference-time method that adaptively adjusts per-token text and image interactions across transformer blocks and denoising timesteps The paper proposes Diff-Aid to provide a more flexible and efficient solution to the problem of insufficient interactions between textual conditions and image features during text-to-image generation
Problem Context
Recent text-to-image (T2I) diffusion models have achieved remarkable advancement, yet faithfully following complex textual descriptions remains challenging due to insufficient interactions between textual and visual features Prior approaches enhance such interactions via architectural design or handcrafted textual condition weighting, but lack flexibility and overlook the dynamic interactions across different blocks and denoising stages
Proposed Mechanism
Diff-Aid introduces two key innovations: 1) an Aid module, which explicitly adjusts textual conditions according to the varied blocks, timesteps, and tokens; and 2) a gated sparsity and stabilized regularization mechanism that respectively ensure attaching more attention to important blocks and avoiding training collapse The core of this is the proposed Adaptive interaction denoising (AiD) module, which learns the coefficient αl t ∈ R N to modulate text-image interactions based on the current diffusion timestep, transformer block, and text features This modulation is applied as ec l t = c l t + c l t ⊙ α l t, where ⊙ is element-wise multiplication
Learning Objectives
The learning objective for the Aid module is defined by the combined loss L = Ldif f + λdpoLdpo + λregLreg The diffusion loss Ldif f minimizes the squared error between predicted velocity and target velocity, while DPO loss Ldpo optimizes the model based on human preference data to align outputs with human judgments Furthermore, training is stabilized by randomly skipping the Aid modules during training with a probability of p
Empirical Validation
Experiments on strong baselines (SD 3.5 and FLUX) demonstrate consistent improvements in prompt adherence, visual quality, and human preference across various metrics The learned parameters for adjustment are interpretable with respect to different blocks, varied timesteps, and textual tokens Ablation studies confirm that the boundedness and sparsity constraints are necessary for optimal performance, showing that using Tanh activation yields better results than simply clipping the α values The sparse enhancement strategy, which selects a subset of transformer blocks based on learned α values, achieves similar outputs as the full method
Application Versatility
Diff-Aid is designed as a plug-in and generalizable module that can be seamlessly integrated into homologous models for controllable image generation (b), styled LoRA generation (c), and zero-shot editing (d) It has been shown to enhance performance when combined with existing LoRAs from the community, exhibiting richer details and more vivid textures The method also shows consistent improvements on widely-used metrics such as HPSv2, ImageReward, and Aesthetic Score when applied to other evaluation benchmarks The method can be flexibly applied to different applications while obtaining competitive performance improvements for both style LoRAs and control conditions generation
Conclusion
We presented Diff-Aid, a plug-in module for rectified textto-image diffusion transformers that learns to adaptively adjust block-wise interactions between token-level textual conditions across denoising trajectories Beyond improving the generation quality, Diff-Aid provides insights into the dynamic interplay of cross-modal signals in transformer blocks and can be seamlessly applied to various tasks The work aims to improve diffusion models for text-to-image generation, with potential applications in creative fields like design and content creation
--- Page 1 ---
Diff-Aid: Inference-time Adaptive Interaction Denoising for Rectified Text-to-Image Generation
Binglei Li 1 2 Mengping Yang 1 3 † Zhiyu Tan 1 3 Junping Zhang 1 Hao Li 1,2,3
a photo of a yellow stop sign and a blue potted plant a photo of a purple sheep and a pink banana FLUX FLUX + Ours
Man from Namibia, hexagon, plain blue-pink FLUX FLUX + Ours
Canny input FLUX Canny FLUX Canny + Ours Depth input FLUX Depth InputFLUX Depth + Ours Wolf WallpapersFor Tablets fishing boatson the Danube Canal Vienna Poster Original output FLUX SketchFLUX Sketch + Ours Original outputFLUX GlassFLUX Glass + Ours A male druid wearing a tattered spiderweb hood standing next to a bear A pug dog with a cat beautiful fluffy with big eyes, white cloud, blue sky. Source image FLUX KontextFLUX Kontext + Ours make the woman to an elf Add a tiger to the painting (a) Text-to image (b) Conditional control (c) Lora integration (d) Instructional Editing Figure 1. p. 0 cn
FLUX FLUX + Ours
a photo of a yellow stop sign and a blue potted plant a photo of a purple sheep and a pink banana
FLUX FLUX + Ours
Man from Namibia, hexagon, plain blue-pink
FLUX FLUX + Ours
Canny input FLUX Canny FLUX Canny + Ours Depth input FLUX Depth FLUX Depth + Ours
Wolf WallpapersFor Tablets fishing boatson the Danube Canal Vienna Poster
Original output FLUX Sketch FLUX Sketch + Ours Original output FLUX Glass FLUX Glass + Ours
A male druid wearing a tattered spiderweb hood standing next to a bear A pug dog with a cat beautiful fluffy with big eyes, white cloud, blue sky. Source image FLUX Kontext FLUX Kontext + Ours Source image FLUX Kontext FLUX Kontext + Ours
make the woman to an elf Add a tiger to the painting
(a) Text
-to image (b) Conditional control (c) Lora integration (d) Instructional Editing
Figure 1. Source image FLUX Kontext FLUX Kontext + Ours Source image FLUX Kontext FLUX Kontext + Ours
make the woman to an elf Add a tiger to the painting
(a) Text
-to image (b) Conditional control (c) Lora integration (d) Instructional Editing
Figure 1. We present Diff-Aid to adaptively improve the interaction between textual conditions and image latents during test time.Where the paper says it
Where the paper says it
Abstract Recent text-to-image (T2I) diffusion models have achieved remarkable advancement, yet faithfully following complex textual descriptions remains challenging due to insufficient interactions be1 Fudan University 2 Shanghai Innovation Institute 3 Shanghai Academy of AI for Science, † Project Lead. p. 0 Benefited from our plug-in design, Diff-Aid enables improved prompts following and synthesis quality for (a) Text-to-image generation,
(b) adding conditional input for controllable generation, (c) integrating off-the-shelf LoRAs and (d) zero-shot instructional image editing. Abstract
Recent text-to-image (T2I) diffusion models have
achieved remarkable advancement, yet faithfully
following complex textual descriptions remains
challenging due to insufficient interactions be1
Fudan University 2
Shanghai Innovation Institute 3
Shanghai
Academy of AI for Science, †
Project Lead. Correspondence to: Hao Li <lihao lh@fudan. cn>, Junping Zhang
<jpzhang@fudan. arXiv preprint
arXiv:2601. 02211, 2026. Beyond improving generation quality,
1
arXiv:2602. 13585v2 [cs.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 10 ---
Figure 4 Distributional visualization of the learned adaptive weights α across various (a) diffusion transformer blocks, (b) textual tokens and (c) denoising timesteps. (d) attention norm across various timesteps Our Diff-Aid learns to adaptively adjust block-wise interactions between token-level textual conditions across denoising trajectories, while also capturing the intrinsic attributes of the baseline models to enable tighter and more faithful conditioning during generation. p. 4 (d) Visualization of the attention norm across various timesteps. Our Diff-Aid learns to adaptively
adjust block-wise interactions between token-level textual conditions across denoising trajectories, while also capturing the intrinsic
attributes of the baseline models to enable tighter and more faithful conditioning during generation.Where the paper says it
--- Page 11 ---
Figure 10 Visualization of the α across different timesteps and blocks for the models.
--- Page 12 ---
Figure 5 Qualitative comparisons of images generated by baseline FLUX and SD 3.5 with and without our proposed Diff-Aid. p. 5 Qualitative comparisons of images generated by baseline FLUX and SD 3. 5 with and without our proposed Diff-Aid. Qualitative comparisons of images generated by baseline FLUX and SD 3. 5 with and without our proposed Diff-Aid. The quantitative results below each image shows the scores of HPSv3, Image Reward and Aesthetic. Our method demonstrates superior
prompt adherence and image quality. The quantitative results below each image shows the scores of HPSv3, Image Reward and Aesthetic. Our method demonstrates superior
prompt adherence and image quality. The quantitative results below each image shows the scores of HPSv3, Image Reward and Aesthetic. Our method demonstrates superior
prompt adherence and image quality.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 13 ---
Table 1 Quantitative comparisons of strong baselines and our proposed Diff-Aid on the HPSv3 benchmark.
--- Page 14 ---
Table 2 Quantitative comparisons of more image quality metrics.
--- Page 15 ---
Figure 9 Results of sparse enhancement strategy compared to the full method and baseline FLUX. Our sparse enhancement achieves similar performance as the full method, showing the effectiveness of our learned α values in identifying important blocks for text-image interactions. p. 4 Additionally, there are noticeable
zero regions in several blocks, showing the effectiveness of
our sparsity strategies. The α learned by Diff-Aid effectively
captures the varying importance of different transformer
blocks in text-image interactions.Where the paper says it
--- Page 16 ---
Figure 12 More Qualitative comparisons of images generated by baseline FLUX with and without our proposed Diff-Aid. The quantitative results below each image show the scores of HPSv3, Image Reward, and Aesthetic Score. p. 5 5 with and without our proposed Diff-Aid. The quantitative results below each image shows the scores of HPSv3, Image Reward and Aesthetic. 5 with and without our proposed Diff-Aid. The quantitative results below each image shows the scores of HPSv3, Image Reward and Aesthetic.Where the paper says it
Where the paper says it
--- Page 17 ---
Figure 13 More Qualitative comparisons of images generated by baseline SD 3.5 with and without our proposed Diff-Aid. The quantitative results below each image show the scores of HPSv3, Image Reward, and Aesthetic Score. Our method demonstrates superior prompt adherence and image quality.
--- Page 18 ---
Figure 14 Some failure cases of baseline models with and without our proposed Diff-Aid. Although our generated images do not fully comply with the prompts, they still contain some key elements mentioned in the prompts. p. 5 Main Results
Qualitative results. To demonstrate the effectiveness of
our proposed Diff-Aid, we present qualitative comparisons
of images generated by different models given the same
prompts and seeds in Fig.Where the paper says it
--- Page 19 ---
Table 3 Ablation study on key components of our proposed method, namely boundedness constraints, sparsity constraints, and optimization strategies.
--- Page 20 ---
Table 4 Quantitative comparison of instructional image editing tasks on EditBench benchmark.
--- Page 21 ---
Table 5 Default inference hyper-parameters for both SD 3.5 and FLUX. p. 0 cn, {jpzhang, lihao lh}@fudan. cn
FLUX FLUX + Ours
a photo of a yellow stop sign and a blue potted plant a photo of a purple sheep and a pink banana
FLUX FLUX + Ours
Man from Namibia, hexagon, plain blue-pink
FLUX FLUX + Ours
Canny input FLUX Canny FLUX Canny + Ours Depth input FLUX Depth FLUX Depth + Ours
Wolf WallpapersFor Tablets fishing boatson the Danube Canal Vienna Poster
Original output FLUX Sketch FLUX Sketch + Ours Original output FLUX Glass FLUX Glass + Ours
A male druid wearing a tattered spiderweb hood standing next to a bear A pug dog with a cat beautiful fluffy with big eyes, white cloud, blue sky.Where the paper says it
--- Page 22 ---
Table 6 Quantitative results of sparse enhancement strategy compared to the full method and baseline FLUX.
--- Page 23 ---
Table 7 Quantitative results of applying our Diff-Aid to style LoRAs and control conditions generation.
--- Page 24 ---
Table 8 More ablation studies on different dropout probabilities.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Adaptive interaction denoising for text-to-image generation allows models to
adaptively adjust per-token text and image interactions across transformer blocks and denoising timesteps.
This enables improved prompt following by learningadaptive importance weights for textual tokens across blocks and timesteps,
leading to better semantic alignment. -
Plug-in design allows Diff-Aid to be seamlessly integrated into downstream applications such as
style LoRAs, controllable generation, and zero-shot editing.
This versatility means the system can be used for improved prompt adherence, adding conditional input for controllable generation, integrating off-the-shelf LoRAs, and performing zero-shot instructional image editing. -
The gated sparsity mechanism and regularization term ensure that the learned modulation coefficients are
sparse
by penalizing large values of alpha l t: Lreg = Et,l α l t2.This helps the model
focus on the most relevant tokens and transformer blocks," making the system more efficient and preventing training collapse. -
The system learns dynamic timestep awareness because it visualizes
the distribution of α values throughout the denoising process in Fig. 10a and Fig. 10b of appendix,
indicating it canimplicitly learn to adjust text-image interactions dynamically across timesteps.
This results in consistent behavior regardless of the number of sampling steps. -
The system can be applied to new conditional generation tasks, such as using
Canny
andDepth
conditions or integratingSketch
andGlass LoRA modules.
This demonstrates versatility by enhancing performance in these specific applications, showing improvements in prompt adherence and image quality for control conditions.
Abstract
Recent text-to-image (T2I) diffusion models have achieved remarkable advancement, yet faithfully following complex textual descriptions remains challenging due to insufficient interactions between textual and visual features. Prior approaches enhance such interactions via architectural design or handcrafted textual condition weighting, but lack flexibility and overlook the dynamic interactions across different blocks and denoising stages. To provide a more flexible and efficient solution to this problem, we propose Diff-Aid, a lightweight inference-time method that adaptively adjusts per-token text and image interactions across transformer blocks and denoising timesteps. Beyond improving generation quality, Diff-Aid yields interpretable modulation patterns that reveal how different blocks, timesteps, and textual tokens contribute to semantic alignment during denoising. As a plug-and-play module, Diff-Aid can be seamlessly integrated into downstream applications for further improvement, including style LoRAs, controllable generation, and zero-shot editing. Experiments on strong baselines (SD 3.5 and FLUX) demonstrate consistent improvements in prompt adherence, visual quality, and human preference across various metrics. Our code and models will be released.
Sources
- FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space
- Seedream 2.0: A Native Chinese-English Bilingual Image Generation Foundation Model
- Prompt-to-Prompt Image Editing with Cross Attention Control
- Classifier-Free Diffusion Guidance
- In-Context LoRA for Diffusion Transformers
- TexTailor: Inference-Time Textual Guidance Tailoring for Multimodal Diffusion Transformers
- Latte: Latent Diffusion Transformer for Video Generation
- Wan: Open and Advanced Large-Scale Video Generative Models
- Apollo: Unified Multi-Task Audio-Video Joint Generation
- Qwen-Image Technical Report
- Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis
- UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactions
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs