Bokeh Diffusion: Defocus Blur Control in Text-to-Image Diffusion Models

arXiv:2503.08434 · cs.GR, cs.CV · Submitted 2025-03-11 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Bokeh Diffusion: Defocus Blur Control in Text-to-Image Diffusion Models".

Jane: The gist The authors propose Bokeh Diffusion, a scene-consistent bokeh control framework that explicitly conditions a diffusion model on a physical defocus blur parameter to enable precise,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, wrapping up this talk on "Bokeh Diffusion: Defocus Blur Control in Text-to-Image Diffusion Models," the authors have put forward a scene-consistent framework that allows you to control defocus blur physically.

Jane: It’s a method that conditions the diffusion model on an explicit physical defocus parameter, and they trained it using both real images and synthetic data to handle the scarcity of perfect pairs.

Lu: The core of their innovation is this grounded self-attention mechanism, which uses a pivot image to anchor the structural layout when changing blur strength.

Meng: It’s a tool that lets users make precise, lens-like adjustments to the visual aesthetic while keeping the scene content exactly where it should be.

Tom: The implication here is that we can move beyond just describing what we want in text and start giving models more explicit control over the photographic quality of their output.

Jane: It makes creative control much more intuitive for people who understand photography, even if they aren't prompt engineers.

Lu: This work shows that combining in-the-wild data with synthetic augmentations gives a really robust way to learn how to disentangle scene structure from lens effects.

Meng: For practical use, it means generating images where you can precisely dial in the depth of field without the model accidentally changing your subject’s position or background elements.

Tom: It’s a versatile tool because it works on both Stable Diffusion and FLUX architectures, which is a big plus for adoption across different AI tools.

Jane: Overall, Bokeh Diffusion gives us flexible, lens-like bokeh control that stays consistent with the scene, which they show surpasses existing methods in accuracy and consistency tests.

Conclusion: Tom: So we're wrapping up this talk on Bokeh Diffusion, which is basically about how to physically control that blurry background effect in text-to-image models.

Jane: Right, so it’s this framework by the authors that lets you condition a diffusion model directly on a physical defocus parameter.

Lu: It’s really interesting because they tackle the problem of scene consistency when you try to change the blur level.

Meng: I mean, most methods just mess up the whole composition when you adjust those lens settings, so having something that keeps the subject exactly where it is important for practical use.

Lalam: From my perspective in terms of cultural impact, this means we can build tools that let people make highly precise artistic choices without losing their intended scene layout.

Tom: So the title itself is pretty straightforward: Bokeh Diffusion, Defocus Blur Control in Text-to-Image Diffusion Models. The authors are showing us how to do this with something called a grounded self-attention mechanism.

Jane: That mechanism is the key part that replaces the standard attention operation with something that anchors the structure using a random pivot image.

Lu: It’s clever because they trained it on pairs of images from the same scene but with different bokeh levels, which lets it learn how blur strength affects things without destroying what's already there.

Meng: And they did this by mixing real photos with synthetic blur augmentations to get data that actually helps the model separate the content from the lens effect.

Tom: So, for someone just listening, what does this mean? It means we’re moving closer to having a way to tell an AI exactly how much background blur you want without it just making things look messy.

Jane: It gives users flexible, lens-like control over the depth of field that respects the actual structure of the image scene.

Lu: They showed it works across different architectures too, like Stable Diffusion and FLUX, which makes it pretty adaptable for future AI tools.

Meng: The limitation they pointed out is that adding grounded self-attention increases runtime by about seventy-six percent per sample, so we have to consider the computational cost when deploying this.

Lalam: But thinking about the future, this level of physical control opens up so many possibilities for how we use generative models in design and photography.

Tom: Definitely. So next up, we’re going to look at some of those specific results where they show exactly how much better it is compared to the older methods.

Nanyang Technological University

cs.GR, cs.CV

Submitted: 2025-03-11

Updated: 2026-06-05

Comments: SIGGRAPH Asia 2025. Project page: https://atfortes.github.io/projects/bokeh-diffusion/

Journal ref: Proceedings of the SIGGRAPH Asia 2025 Conference Papers (SA '25), Hong Kong, ACM, 2025, Article 160, pp. 1-11

DOI: 10.1145/3757377.3763906

Project page: https://atfortes.github.io/projects/bokeh-diffusion

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 88/100

The gist: The gist The authors propose Bokeh Diffusion, a scene-consistent bokeh control framework that explicitly conditions a diffusion model on a physical defocus blur parameter to enable precise,

Key concepts

Bokeh Diffusion
This is a new framework designed to give text-to-image models explicit control over defocus blur. Instead of relying on vague text prompts, it conditions the model directly on a measurable physical parameter representing how blurry the background is, allowing for scene-consistent and precise lens effects.
Hybrid Training Pipeline
The training method combines two types of data: real photographs taken with various camera settings (in-the-wild) and synthetic images where blur has been artificially added. This approach helps the model learn to separate the actual content of a scene from the visual effect of lens blur, making it robust to diverse inputs.
Grounded Self-Attention Mechanism
This is a specific attention mechanism trained on image pairs that share the same scene but have different bokeh levels. It acts as an anchor, ensuring that when the model adjusts the blur strength, it maintains the correct spatial layout and structure of the underlying scene rather than distorting it.
Decoupled Cross-Attention
This mechanism adds a new set of attention projections to the standard diffusion pipeline. It is tied to a lightweight feature representing bokeh. This allows the model to process information related to blur independently, separating it from the main image generation task while still influencing the output.

Terminology

Summary

The gist The authors propose Bokeh Diffusion, a scene-consistent bokeh control framework that explicitly conditions a diffusion model on a physical defocus blur parameter to enable precise, scene-consistent bokeh control in text-to-image diffusion models.

Introduction and Motivation

Traditional methods for reproducing camera effects in text-to-image generation typically rely on prompt engineering, which often results in crude approximations and inadvertently alters the scene content. Recent works have started to explore controlling camera intrinsics in image generation via camera tokens, but often still entangle depth-of-field with content changes. This approach allows the model to learn from both authentic lens blur across diverse settings and structured focus–defocus pairs.

Data Curation and Training Pipeline

The hybrid training pipeline is built upon two complementary data sources. First, the authors curate a collection of in-the-wild photographs enriched with lens metadata such as EXIF information and estimated depth maps.

Conditioning Mechanisms

The framework employs two primary mechanisms for conditioning the diffusion process on a physical bokeh parameter. First is decoupled cross-attention, where a new pair of key-value projections W'k and W'v tied to a lightweight bokeh feature cb are added to the existing T2I diffusion pipeline. Second is the grounded self-attention mechanism which is specifically trained with image pairs to ensure that blur changes respect the scene of a reference image. This grounded self-attention replaces the usual self-attention operation for h tgt with Attention Q piv, [K tgt, K piv], [V tgt,V tgt] to anchor the structural layout using a random pivot image. Quantitative evaluation benchmarks the method against baselines on Accuracy (measured by LVCorr and GPT-4o estimations), Consistency (measured by LPIPS, DINOv2, and DreamSim), and Quality (evaluated using ImageReward). The results show that Bokeh Diffusion consistently outperforms baselines across accuracy, consistency, and perceptual quality metrics. Qualitative evaluation confirms that CSaT often alters the scene layout when changing bokeh levels. Furthermore, a user study showed that users preferred Bokeh Diffusion for Bokeh Control (91.00%) and Scene Consistency (92.67%).

Conclusion

In summary, Bokeh Diffusion is a scene-consistent bokeh control framework for T2I diffusion that allows physically motivated defocus blur changes without altering scene content. The method integrates in-the-wild images with synthetic augmentations to balance realism and robustness. The framework is versatile, supporting both the UNet-based Stable Diffusion and the MMDiT-based FLUX architectures. The paper concludes that Bokeh Diffusion surpasses baselines in both accuracy and scene consistency, offering flexible, lens-like bokeh adjustments. The work inspires further research into fine-grained photographic controls in generative models.

Ablation Study Insights

The ablation study on training data composition indicates that the mixed setting achieves the best overall accuracy while also improving consistency compared to training solely on SYN data. Ablating key components reveals that adding grounded self-attention significantly enhances scene consistency, ensuring that spatial layout remains aligned as the blur strength varies. The color transfer step is optional but further improves visual coherence by harmonizing lighting and tone without interfering with the defocus itself. The runtime analysis shows that adding grounded self-attention increases runtime per sample by ∼76% and peak memory usage by ∼33%. The framework is robust to a range of grounding configurations during sampling, with 18 grounding steps providing a good tradeoff between scene consistency and flexible defocus control <ref:2503.

Improvements for AI systems

  1. A scene-consistent bokeh control framework that explicitly conditions a diffusion model on a physical defocus blur parameter, enabling flexible, lens-like blur control that supports smooth transitions across the bokeh range.

  2. The system can perform direct bokeh control during generation, allowing users to specify blur strength without postprocessing, and support adjusting the bokeh strength in both directions—either increasing or decreasing it—while preserving the underlying scene.

  3. The framework supports real image editing via inversion, where a user can adjust the bokeh strength in both directions, demonstrating capability by applying a new bokeh setting to an inverted target latent while keeping the pivot image as the structural anchor.

  4. The model can be deployed on both Stable Diffusion and FLUX architectures, achieving strong results across both, indicating versatility in supporting state-of-the-art T2I models without multi-frame inference overhead.

  5. The system can leverage a hybrid training pipeline combining in-the-wild photographs enriched with lens metadata and synthetic defocus blur augmentations, allowing it to learn from both authentic lens blur and structured focus–defocus pairs to better disentangle scene content from depth-of-field.

  6. The system can maintain structural fidelity across different defocus levels through a grounded self-attention mechanism where the pivot image acts as a structural anchor for the other images during training, ensuring that blur changes respect the scene of a reference image.

  7. The system incorporates decoupled cross-attention to inject defocus conditioning, and in MMDiT architectures, it integrates both bokeh conditioning and pivot-based grounding directly into the joint attention module to ensure scene-consistent bokeh control entirely within the T2I paradigm.

  8. The system can be used for advanced image editing by choosing a pivot image with a given bokeh level to anchor the scene whose bokeh is to be adjusted, demonstrating that using either a sharper or blurrier pivot yields results matching the requested bokeh parameter without changing its contents.

Sources

Related papers