CLUE: Controllable Latent space of Unprompted Embeddings for Diversity Management in Text-to-Image Synthesis

arXiv:2511.10993 · cs.CV · Submitted 2025-11-14 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "CLUE: Controllable Latent space of Unprompted Embeddings for Diversity Management in Text-to-Image Synthesis".

Jane: Here's the gist of the paper. Text-to-image synthesis models require the ability to generate diverse images while maintaining stability through fixed-format prompts without requiring any additional data.

Tom: CLUE (Controllable Latent space of Unprompted Embeddings) is a generative model framework that achieves diverse generation while maintaining stability through fixed-format prompts without requiring any additional data.

Jane: Problem Statement Text-to-image synthesis models face challenges when applied to specialized fields such as medicine, where only limited types and insufficient amounts of data are available.

Tom: The medical domain presents distinct challenges for text-to-image synthesis, including a significant visual and structural gap between medical and general domain images.

Paper summary: Tom: So, CLUE proposes this framework to fix that by integrating a new component into the Stable Diffusion architecture.

Jane: They introduce something called a Style Encoder that learns visual attributes directly from paired images and prompts.

Lu: This encoder takes the image and the prompt and generates style embeddings which are then fed into a second attention layer of the U-Net architecture >

Meng: That sounds like more complexity when you're trying to deploy something at scale, how much overhead is that adding?

Tom: The authors use Kullback-Leibler divergence to make sure these features stay within Gaussian regions, which keeps the representation continuous and independent of the prompts themselves.

Jane: And for training, they use a two-part objective: a denoising loss to guide the generation, and a KL regularization loss to keep things consistent with a standard Gaussian prior >

Lalam: From my perspective in analyzing this literature, it seems like they are trying to decouple the visual style from the prompt distribution observed during training >

Lu: They argue that much of the domain knowledge is implicitly encoded in the training data, and this is inaccessible through prompts alone, which becomes especially problematic when you need precise generation in a clinical setting >

Tom: So, they're not just adding another layer; they're fundamentally changing how the model understands style and prompt interaction >

Conclusion: Jane: So, looking at the title, CLUE is all about controlling the latent space of embeddings when you don't have extra data available.

Tom: It seems to be tackling that problem head-on by focusing on managing diversity through these controllable latent spaces without needing massive amounts of supplemental information.

Lu: The authors are essentially showing how to improve fidelity and control diversity in image synthesis by combining the strengths of variational autoencoders with diffusion models >

Meng: I'm interested in the results they show, because if this actually works well, it could mean we don't need huge proprietary datasets just to get good medical image generation.

Jane: They showed substantial improvements on the otitis media dataset, reducing the FID score from forty-six point eight one down to nine point three zero and boosting recall to seventy point twenty-nine percent >

Tom: And when they combined synthetic images with real data, they got an F1 score of eighty-three point twenty-one percent, and when using one hundred percent synthetic data with real data, that hit ninety-four point seventy-six percent >

Lalam: It suggests that this framework has a strong ability to improve performance on downstream classifiers when you mix the synthetic and real data in a balanced way >

Jane: The limitation they mention is that the style embedding space doesn't offer explicit clinical interpretability yet, which means we don't know exactly *why* it made a certain stylistic choice >

Tom: So, while this is an improvement in generation quality and diversity without requiring new data for every task, we still need to figure out how to make those style embeddings more understandable clinically.

University of Ulsan College of Medicine · Asan Medical Center

cs.CV

Submitted: 2025-11-14

Updated: 2025-11-14

DOI: 10.1038/s41598-026-79220-8

Code: https://github.com/rwightman/pytorch-image-models

License: http://creativecommons.org/publicdomain/zero/1.0/

Importance score: 82/100

The gist: Text-to-image synthesis models require the ability to generate diverse images while maintaining stability through fixed-format prompts without requiring any additional data.

Key concepts

Style Encoder
This component learns visual attributes by processing pairs of images and their corresponding text prompts. It creates 'style embeddings' that capture the visual characteristics described by the prompt, which are then used to guide the image generation process within the model.
U-Net Architecture Augmentation
The Style Encoder's style embeddings are fed into a new attention layer within the U-Net structure of Stable Diffusion. This allows the model to incorporate learned visual style information directly into its denoising process, enhancing control and diversity during image creation.
KL Regularization Loss
This loss function measures the difference between the model's approximate posterior distribution and a standard Gaussian prior. It ensures that the learned feature representations remain continuous within Gaussian regions, independent of specific prompts, promoting stable generation.

Terminology

Summary

Text-to-image synthesis models require the ability to generate diverse images while maintaining stability through fixed-format prompts without requiring any additional data. p. 0

Where the paper says it

CLUE: Controllable Latent space of Unprompted Embeddings for Diversity Management in Text-to-Image Synthesis Keunwoo Parka , Jihye Chaea , Joong Ho Ahnb and Jihoon Kweona,∗ aDepartments of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil, Songpa-gu, Seoul, 05505, Korea bDepartment of Otorhinolaryngology-Head and Neck Surgery, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil, Songpa-gu, Seoul, 05505, Korea A R T I C L E I N F O Keywords: Deep learning Text-to-image synthesis Medical image generation Diffusion Style embedding Data augmentation A B S T R A C T Text-to-image synthesis models require the ability to generate diverse images while maintaining stability. To overcome this challenge, a number of methods have been proposed, including the collection of prompt-image datasets and the integration of additional data modalities during training.

CLUE (Controllable Latent space of Unprompted Embeddings) is a generative model framework that achieves diverse generation while maintaining stability through fixed-format prompts without requiring any additional data. p. 0
Where the paper says it

Although these methods have shown promising results in general domains, they face limitations when applied to specialized fields such as medicine, where only limited types and insufficient amounts of data are available. We present CLUE (Controllable Latent space of Unprompted Embeddings), a generative model framework that achieves diverse generation while maintaining stability through fixed-format prompts without requiring any additional data.

Problem Statement

Text-to-image synthesis models face challenges when applied to specialized fields such as medicine, where only limited types and insufficient amounts of data are available. The medical domain presents distinct challenges for text-to-image synthesis, including a significant visual and structural gap between medical and general domain images. p. 0

Where the paper says it

As illustrated in Figure 1, the medical domain presents distinct challenges for text-to-image synthesis. First, there exists a significant visual and structural gap between medical and general domain images [9].

Furthermore, diagnostic reports use heterogeneous phrasing conventions which introduces semantic variability that can confuse generative models. p. 0
Where the paper says it

First, there exists a significant visual and structural gap between medical and general domain images [9]. Second, diagnostic reports, although clinically equivalent, are written using heterogeneous phrasing conventions, which introduces semantic variability that can confuse generative models.

This constraint renders both the diversity and precise control of generated images particularly challenging when image synthesis is conducted solely through the use of identical prompts. p. 1
Where the paper says it

When fine-tuned on report A, the model generates accurate images with A-style descriptions, but may produce inappropriate outputs when encountering an unfamiliar phrasing style such as report B. and precise control of generated images particularly challenging when image synthesis is conducted solely through the use of identical prompts.

CLUE Architecture

CLUE augments the Stable Diffusion architecture with a Style Encoder that learns visual attributes from paired images and prompts. This Style Encoder processes images and prompts to generate style embeddings which are subsequently fed into a new second attention layer of the U-Net architecture. p. 0

Where the paper says it

We present CLUE (Controllable Latent space of Unprompted Embeddings), a generative model framework that achieves diverse generation while maintaining stability through fixed-format prompts without requiring any additional data. Based on the Stable Diffusion architecture, CLUE employs a Style Encoder that processes images and prompts to generate style embeddings, which are subsequently fed into a new second attention layer of the U-Net architecture.

The framework employs Kullback-Leibler divergence to achieve a continuous representation of image features within Gaussian regions, independent of prompts. p. 0
Where the paper says it

Based on the Stable Diffusion architecture, CLUE employs a Style Encoder that processes images and prompts to generate style embeddings, which are subsequently fed into a new second attention layer of the U-Net architecture. Through Kullback-Leibler divergence, the latent space achieves continuous representation of image features within Gaussian regions, independent of prompts.

Training Objective

The training objective for CLUE comprises two terms: the denoising loss and the KL regularization loss. The denoising loss is defined as the expected mean squared error between the model’s predicted velocity and the true velocity target. p. 4

Where the paper says it

Training Objective Loss (𝜃, 𝜙) comprises two terms. The first term, the denoising loss, is defined as the expected mean squared error between the model’s predicted velocity 𝑣𝜃 (𝑥𝑡 , 𝑡, 𝑐, 𝑠) and the true velocity target 𝑣𝑡 .

The KL regularization loss computes the Kullback-Leibler divergence between the approximate posterior and a standard Gaussian prior. p. 4
Where the paper says it

This expectation is taken over the data distribution of original images 𝑥 and text conditions 𝑐, a sampled timestep 𝑡, standard Gaussian noise 𝜖 used to produce the noised image 𝑥𝑡 , and the style latent 𝑠 sampled from the encoder 𝑆𝐸𝜙 . The second term, the KL regularization loss, computes the Kullback-Leibler divergence between the approximate posterior 𝑆𝐸𝜙 (𝑠|𝑥, 𝑐) and the standard Gaussian prior (0, 𝐼).

Experimental Validation

Performance on the otitis media dataset showed substantial improvements, reducing FID from 46.81 to 9.30 and improving recall to 70. p. 1

Where the paper says it

Our model achieves a substantial improvement in generation quality, reducing the Fréchet Inception Distance (FID) score from 46. 30 and improving recall from 49.

29%. Synthetic images generated by CLUE significantly improve downstream classifier performance on the real data, achieving an F1 score of 83. p. 1
Where the paper says it

30 and improving recall from 49. Synthetic images generated by CLUE, when used as training data, significantly improve downstream classifier performance on the real data, achieving an 𝐹1 score of 83.

21% when combined with synthetic data. p. 0
Where the paper says it

61%) at 1000% scale. The combined approach achieved an 𝐹1 score of 85.

Furthermore, combining 100% scale synthetic data with equal amounts of real data achieved a F1 score of 94. p. 1
Where the paper says it

83% with reference synthetic data. Combining 100% scale synthetic data with equal amounts of real data further achieved 94.

76%, higher than when using only real data. p. 0
Where the paper says it

Combining synthetic data with equal amounts of real data achieved an 𝐹1 score of 94. 76%, higher than when using only real data.

Generalization and Scalability

CLUE demonstrates robust generalization beyond the institution, as models trained exclusively on CLUE-generated synthetic data achieved an F1 score of 72.37% on the AI-HUB otoscopic image dataset at 1000% scale. p. 1

Where the paper says it

Combining 100% scale synthetic data with equal amounts of real data further achieved 94. For evaluate external generalization, we additionally tested our framework on the AI-HUB otoscopic image dataset.

The framework's ability to improve cross-institution performance through scaling positions it as a practical solution for scenarios where acquiring domain-specific training data is challenging or expensive. CLUE with high sigma values can address the limitation that insufficient diversity in synthetic data generation limits the ability to establish a clear relationship between training volume and performance improvements. p. 0
Where the paper says it

CLUE: Controllable Latent space of Unprompted Embeddings for Diversity Management in Text-to-Image Synthesis Keunwoo Parka , Jihye Chaea , Joong Ho Ahnb and Jihoon Kweona,∗ aDepartments of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil, Songpa-gu, Seoul, 05505, Korea bDepartment of Otorhinolaryngology-Head and Neck Surgery, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil, Songpa-gu, Seoul, 05505, Korea A R T I C L E I N F O Keywords: Deep learning Text-to-image synthesis Medical image generation Diffusion Style embedding Data augmentation A B S T R A C T Text-to-image synthesis models require the ability to generate diverse images while maintaining stability. To overcome this challenge, a number of methods have been proposed, including the collection of prompt-image datasets and the integration of additional data modalities during training.

Limitations and Future Work

A limitation is that the style embedding space lacks explicit clinical interpretability. Future work should explore extending CLUE to a wider range of medical imaging modalities and assess its effectiveness in class imbalanced settings. The framework’s ability to generate diverse yet diagnostically consistent images without requiring additional conditioning inputs addresses key challenges in medical AI development. p. 0

Where the paper says it

We present CLUE (Controllable Latent space of Unprompted Embeddings), a generative model framework that achieves diverse generation while maintaining stability through fixed-format prompts without requiring any additional data. Based on the Stable Diffusion architecture, CLUE employs a Style Encoder that processes images and prompts to generate style embeddings, which are subsequently fed into a new second attention layer of the U-Net architecture.

References

[1] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. p. 2

Where the paper says it

Related Work The proposed CLUE framework integrates the complementary strengths of variational autoencoders [16] and diffusion models to improve both fidelity and control diversity in the image synthesis. In this section, we present a comprehensive review of the theoretical foundations and key principles underlying these generative modeling paradigms.

ACM computing surveys 56(4):1–39 2023.

[2] Yong Xuan Tan, Chin Poo Lee, Mai Neo, Kian Ming Lim, Jit Yan Lim, and Ali Alqahtani. Recent advances in text-to-image synthesis: Approaches, datasets and future research prospects. p. 0

Where the paper says it

Introduction Generative deep learning has achieved remarkable progress in recent years and has attracted significant interest in various application domains [1]. Of these advances, text-to-image synthesis—the generation of visually coherent and semantically consistent images from textual descriptions—stands out as both compelling and challenging [2, 3].

IEEE Access 11:88099–88115 2023.

[3] Md Ahsan Habib, Md Anwar Hussen Wadud, Md Fazlul Karim Patwary, Mohammad Motiur Rahman, M Firoz Mridha, Yuichi Okuyama, and Jungpil Shin. Exploring progress in text-to-image synthesis: An in-depth survey on the evolution of generative adversarial networks. p. 0

Where the paper says it

Introduction Generative deep learning has achieved remarkable progress in recent years and has attracted significant interest in various application domains [1]. Of these advances, text-to-image synthesis—the generation of visually coherent and semantically consistent images from textual descriptions—stands out as both compelling and challenging [2, 3].

IEEE Access 2024.

[4] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. p. 0

Where the paper says it

CLUE: Controllable Latent space of Unprompted Embeddings for Diversity Management in Text-to-Image Synthesis Keunwoo Parka , Jihye Chaea , Joong Ho Ahnb and Jihoon Kweona,∗ aDepartments of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil, Songpa-gu, Seoul, 05505, Korea bDepartment of Otorhinolaryngology-Head and Neck Surgery, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil, Songpa-gu, Seoul, 05505, Korea A R T I C L E I N F O Keywords: Deep learning Text-to-image synthesis Medical image generation Diffusion Style embedding Data augmentation A B S T R A C T Text-to-image synthesis models require the ability to generate diverse images while maintaining stability. To overcome this challenge, a number of methods have been proposed, including the collection of prompt-image datasets and the integration of additional data modalities during training.

In International conference on machine learning pages 8821–8831 Pmlr 2021. p. 4
Where the paper says it

This study adhered to the Declaration of Helsinki and was approved with a waiver of informed consent by the Institutional Review Board of Seoul Asan Medical Center (IRB No. 2021-0837).

[5] Gary Marcus, Ernest Davis, and Scott Aaronson. A very preliminary analysis of dall-e 2. p. 0

Where the paper says it

Of these advances, text-to-image synthesis—the generation of visually coherent and semantically consistent images from textual descriptions—stands out as both compelling and challenging [2, 3]. Recent models such as DALL·E [4, 5], Imagen [6] and Stable Diffusion [7, 8] demonstrate impressive general purpose image generation capabilities.

arXiv preprint arXiv:2204. p. 0
Where the paper says it

Park et al. : Preprint submitted to Elsevier Page 1 of 24 arXiv:2511.

13807 2022.

[6] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans et al. Photorealistic text-to-image diffusion models with deep language understanding. p. 3

Where the paper says it

𝑡 = ||𝑣𝑡 − 𝑣𝜃 (𝑥𝑡 , 𝑡, 𝑐)||2 = ||√ ̄𝛼𝑡 𝜖 − √ 1 − ̄𝛼𝑡𝑥0 − 𝑣𝜃 (𝑥𝑡 , 𝑡, 𝑐)||2 (8) Latent diffusion models (LDMs) operate in a lower dimensional latent space obtained via an autoencoder, substantially reducing computational cost while preserving semantic structure. Stable Diffusion exemplifies this approach by integrating powerful text encoders with efficient latent-space diffusion, achieving photorealistic synthesis from natural language prompts with high fidelity and diversity.

Advances in neural information processing systems 35:36479–36494 2022. p. 2
Where the paper says it

This v-prediction parameterization has been shown to yield improved numerical stability for few step sampling scenarios. 𝑥𝑡 = √ ̄𝛼𝑡𝑥0 + √ 1 − ̄𝛼𝑡 𝜖, 𝑣𝑡 = √ ̄𝛼𝑡 𝜖 − √ 1 − ̄𝛼𝑡𝑥0 , ̄𝛼𝑡 = ∏𝑡 𝑖=1 (1 − 𝛽𝑖 ) (4) The relationships between these parameterizations enable flexible conversion: ̂𝑣 = 𝑣𝜃 (𝑥𝑡 , 𝑡, 𝑐), ̂𝑥𝑡 0 = √ ̄𝛼𝑡𝑥𝑡 − √ 1 − ̄𝛼𝑡 ̂𝑣, ̂𝜖𝑡 = √ 1 − ̄𝛼𝑡𝑥𝑡 + √ ̄𝛼𝑡 ̂𝑣 (5) 𝑣𝜃 is the neural network predicting velocity, 𝑐 is the conditioning information, and ̂𝑥 𝑡 0 , ̂𝜖𝑡 are the estimated clean image and noise at timestep 𝑡, respectively.

[7] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. p. 3

Where the paper says it

𝑡 = ||𝑣𝑡 − 𝑣𝜃 (𝑥𝑡 , 𝑡, 𝑐)||2 = ||√ ̄𝛼𝑡 𝜖 − √ 1 − ̄𝛼𝑡𝑥0 − 𝑣𝜃 (𝑥𝑡 , 𝑡, 𝑐)||2 (8) Latent diffusion models (LDMs) operate in a lower dimensional latent space obtained via an autoencoder, substantially reducing computational cost while preserving semantic structure. Stable Diffusion exemplifies this approach by integrating powerful text encoders with efficient latent-space diffusion, achieving photorealistic synthesis from natural language prompts with high fidelity and diversity.

In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition pages 10684–10695 2022. p. 4
Where the paper says it

dev0) for diffusion-based components and employ the timm library (0. 12) [21] for vision backbones and utility routines.

[8] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer et al. Scaling rectified flow transformers for high-resolution image synthesis. p. 0

Where the paper says it

This requirement is particularly pronounced in medical imaging, where applications demand not only high visual fidelity but also robust stability and precision due to clinical decision-making. As illustrated in Figure 1, the medical domain presents distinct challenges for text-to-image synthesis.

In Forty-first international conference on machine learning 2024. p. 0
Where the paper says it

As illustrated in Figure 1, the medical domain presents distinct challenges for text-to-image synthesis. First, there exists a significant visual and structural gap between medical and general domain images [9].

[9] Jee Seok Yoon, Kwanseok Oh, Yooseung Shin, Maciej A Mazurowski, and Heung-Il Suk. Domain generalization for medical image analysis: A review. p. 4

Where the paper says it

All video sequences were originally acquired for clinical diagnostic purposes; from each sequence, a single frame exhibiting the entire tympanic membrane was anonymized and exported for analysis. This study adhered to the Declaration of Helsinki and was approved with a waiver of informed consent by the Institutional Review Board of Seoul Asan Medical Center (IRB No.

Proceedings of the IEEE 2024.

[10] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. p. 2

Where the paper says it

Diffusion Models Diffusion models define a forward noising process that gradually corrupts data and a learned reverse process that recovers clean samples [10, 18]. This framework unites the stability of probabilistic modeling with the ability to produce high-quality samples, and has achieved state-of-the-art results across image, audio, and text domains.

In International conference on machine learning pages 8162–8171 PMLR 2021. p. 4
Where the paper says it

This study adhered to the Declaration of Helsinki and was approved with a waiver of informed consent by the Institutional Review Board of Seoul Asan Medical Center (IRB No. 2021-0837).

[11] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. p. 0

Where the paper says it

CLUE: Controllable Latent space of Unprompted Embeddings for Diversity Management in Text-to-Image Synthesis Keunwoo Parka , Jihye Chaea , Joong Ho Ahnb and Jihoon Kweona,∗ aDepartments of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil, Songpa-gu, Seoul, 05505, Korea bDepartment of Otorhinolaryngology-Head and Neck Surgery, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil, Songpa-gu, Seoul, 05505, Korea A R T I C L E I N F O Keywords: Deep learning Text-to-image synthesis Medical image generation Diffusion Style embedding Data augmentation A B S T R A C T Text-to-image synthesis models require the ability to generate diverse images while maintaining stability. To overcome this challenge, a number of methods have been proposed, including the collection of prompt-image datasets and the integration of additional data modalities during training.

arXiv preprint arXiv:2308. p. 0
Where the paper says it

Park et al. : Preprint submitted to Elsevier Page 1 of 24 arXiv:2511.

06721 2023.

[12] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. p. 1

Where the paper says it

and precise control of generated images particularly challenging when image synthesis is conducted solely through the use of identical prompts. Instability is rooted in the text-to-image models; their generative behavior is tightly coupled to the prompt distribution observed during training.

In Proceedings of the IEEE/CVF international conference on computer vision pages 3836–3847 2023. p. 4
Where the paper says it

dev0) for diffusion-based components and employ the timm library (0. 12) [21] for vision backbones and utility routines.

[13] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. p. 0

Where the paper says it

CLUE: Controllable Latent space of Unprompted Embeddings for Diversity Management in Text-to-Image Synthesis Keunwoo Parka , Jihye Chaea , Joong Ho Ahnb and Jihoon Kweona,∗ aDepartments of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil, Songpa-gu, Seoul, 05505, Korea bDepartment of Otorhinolaryngology-Head and Neck Surgery, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil, Songpa-gu, Seoul, 05505, Korea A R T I C L E I N F O Keywords: Deep learning Text-to-image synthesis Medical image generation Diffusion Style embedding Data augmentation A B S T R A C T Text-to-image synthesis models require the ability to generate diverse images while maintaining stability. To overcome this challenge, a number of methods have been proposed, including the collection of prompt-image datasets and the integration of additional data modalities during training.

In Proceedings of the AAAI conference on artificial intelligence volume 38 pages 4296–4304 2024.

[14] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. p. 0

Where the paper says it

Based on the Stable Diffusion architecture, CLUE employs a Style Encoder that processes images and prompts to generate style embeddings, which are subsequently fed into a new second attention layer of the U-Net architecture. Through Kullback-Leibler divergence, the latent space achieves continuous representation of image features within Gaussian regions, independent of prompts.

In International Conference on Medical image computing and computer-assisted intervention pages 234–241 Springer 2015. p. 0
Where the paper says it

CLUE: Controllable Latent space of Unprompted Embeddings for Diversity Management in Text-to-Image Synthesis Keunwoo Parka , Jihye Chaea , Joong Ho Ahnb and Jihoon Kweona,∗ aDepartments of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil, Songpa-gu, Seoul, 05505, Korea bDepartment of Otorhinolaryngology-Head and Neck Surgery, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil, Songpa-gu, Seoul, 05505, Korea A R T I C L E I N F O Keywords: Deep learning Text-to-image synthesis Medical image generation Diffusion Style embedding Data augmentation A B S T R A C T Text-to-image synthesis models require the ability to generate diverse images while maintaining stability. To overcome this challenge, a number of methods have been proposed, including the collection of prompt-image datasets and the integration of additional data modalities during training.

[15] Solomon Kullback and Richard A Leibler. p. 0

Where the paper says it

Based on the Stable Diffusion architecture, CLUE employs a Style Encoder that processes images and prompts to generate style embeddings, which are subsequently fed into a new second attention layer of the U-Net architecture. Through Kullback-Leibler divergence, the latent space achieves continuous representation of image features within Gaussian regions, independent of prompts.

On information and sufficiency. p. 2
Where the paper says it

This v-prediction parameterization has been shown to yield improved numerical stability for few step sampling scenarios. 𝑥𝑡 = √ ̄𝛼𝑡𝑥0 + √ 1 − ̄𝛼𝑡 𝜖, 𝑣𝑡 = √ ̄𝛼𝑡 𝜖 − √ 1 − ̄𝛼𝑡𝑥0 , ̄𝛼𝑡 = ∏𝑡 𝑖=1 (1 − 𝛽𝑖 ) (4) The relationships between these parameterizations enable flexible conversion: ̂𝑣 = 𝑣𝜃 (𝑥𝑡 , 𝑡, 𝑐), ̂𝑥𝑡 0 = √ ̄𝛼𝑡𝑥𝑡 − √ 1 − ̄𝛼𝑡 ̂𝑣, ̂𝜖𝑡 = √ 1 − ̄𝛼𝑡𝑥𝑡 + √ ̄𝛼𝑡 ̂𝑣 (5) 𝑣𝜃 is the neural network predicting velocity, 𝑐 is the conditioning information, and ̂𝑥 𝑡 0 , ̂𝜖𝑡 are the estimated clean image and noise at timestep 𝑡, respectively.

The annals of mathematical statistics 22(1):79–86 1951. p. 2
Where the paper says it

It enables the generation of images that satisfy specified conditions within the domain on which the model was trained. 𝑝𝜃 (𝑥|𝑐) = ∫ 𝑝𝜃 (𝑥|𝑧, 𝑐)𝑝𝜃 (𝑧|𝑐)𝑑𝑧 (1) This conditional likelihood is maximized by optimizing the following evidence lower bound (ELBO): log 𝑝𝜃 (𝑥|𝑐) ≥ 𝔼𝑧∼𝑞𝜙(𝑧|𝑥,𝑐) [ log 𝑝𝜃 (𝑥|𝑧, 𝑐) ] − 𝐷KL(𝑞𝜙 (𝑧|𝑥, 𝑐)‖𝑝𝜃 (𝑧|𝑐)) (2) The approximate posterior is reparametrized using Gaussian statistics: 𝑞𝜙 (𝑧|𝑥, 𝑐) = (𝜇𝜙 (𝑥, 𝑐), diag(𝜎𝜙 (𝑥, 𝑐) 2 )), 𝑧 = 𝜇𝜙 (𝑥, 𝑐) + 𝜎𝜙 (𝑥, 𝑐) ⊙ 𝜖, 𝜖 ∼ (0, 𝐼) (3) where ⊙ denotes element-wise multiplication.

[16] Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes 2013. p. 3

Where the paper says it

The clean image and prompt are sent into the style encoder to parameterize a Gaussian posterior from which a style latent 𝑠 ∈ ℝ1024 is sampled. Style Encoding and Sampling Specifically, we reuse the SD 2.

[17] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. p. 6

Where the paper says it

CLUE: Controllable Latent space of Unprompted Embeddings Figure 3: Overview of the four stage preprocessing applied to each endoscopic frame Table 3 Training configurations for generative models Parameter Value Base model SD 2. 1 Diffusion scheduler DDIMScheduler Diffusion timesteps 1,000 Prediction type v-prediction Resolution 512 × 512 × 3 Normalization [-1, +1] Precision FP32 Batch 4 Optimizer AdamW Learning rate 1e-5 Learning scheduler Constant Maximum steps 10,000 Using these exact settings, we trained our proposed CLUE model and, for comparison, fine-tuned a Stable Diffusion 2.

Advances in neural information processing systems 28 2015. p. 2
Where the paper says it

This v-prediction parameterization has been shown to yield improved numerical stability for few step sampling scenarios. 𝑥𝑡 = √ ̄𝛼𝑡𝑥0 + √ 1 − ̄𝛼𝑡 𝜖, 𝑣𝑡 = √ ̄𝛼𝑡 𝜖 − √ 1 − ̄𝛼𝑡𝑥0 , ̄𝛼𝑡 = ∏𝑡 𝑖=1 (1 − 𝛽𝑖 ) (4) The relationships between these parameterizations enable flexible conversion: ̂𝑣 = 𝑣𝜃 (𝑥𝑡 , 𝑡, 𝑐), ̂𝑥𝑡 0 = √ ̄𝛼𝑡𝑥𝑡 − √ 1 − ̄𝛼𝑡 ̂𝑣, ̂𝜖𝑡 = √ 1 − ̄𝛼𝑡𝑥𝑡 + √ ̄𝛼𝑡 ̂𝑣 (5) 𝑣𝜃 is the neural network predicting velocity, 𝑐 is the conditioning information, and ̂𝑥 𝑡 0 , ̂𝜖𝑡 are the estimated clean image and noise at timestep 𝑡, respectively.

[18] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. p. 0

Where the paper says it

CLUE: Controllable Latent space of Unprompted Embeddings for Diversity Management in Text-to-Image Synthesis Keunwoo Parka , Jihye Chaea , Joong Ho Ahnb and Jihoon Kweona,∗ aDepartments of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil, Songpa-gu, Seoul, 05505, Korea bDepartment of Otorhinolaryngology-Head and Neck Surgery, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil, Songpa-gu, Seoul, 05505, Korea A R T I C L E I N F O Keywords: Deep learning Text-to-image synthesis Medical image generation Diffusion Style embedding Data augmentation A B S T R A C T Text-to-image synthesis models require the ability to generate diverse images while maintaining stability. To overcome this challenge, a number of methods have been proposed, including the collection of prompt-image datasets and the integration of additional data modalities during training.

In International conference on machine learning pages 2256–2265 pmlr 2015. p. 0
Where the paper says it

CLUE: Controllable Latent space of Unprompted Embeddings for Diversity Management in Text-to-Image Synthesis Keunwoo Parka , Jihye Chaea , Joong Ho Ahnb and Jihoon Kweona,∗ aDepartments of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil, Songpa-gu, Seoul, 05505, Korea bDepartment of Otorhinolaryngology-Head and Neck Surgery, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil, Songpa-gu, Seoul, 05505, Korea A R T I C L E I N F O Keywords: Deep learning Text-to-image synthesis Medical image generation Diffusion Style embedding Data augmentation A B S T R A C T Text-to-image synthesis models require the ability to generate diverse images while maintaining stability. To overcome this challenge, a number of methods have been proposed, including the collection of prompt-image datasets and the integration of additional data modalities during training.

[19] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. p. 2

Where the paper says it

In addition to the common noise prediction and clean sample parameterization, diffusion models may be trained to predict a velocity term 𝑣𝑡 , which combines the clean signal and noise in a single target [19]. This v-prediction parameterization has been shown to yield improved numerical stability for few step sampling scenarios.

arXiv preprint arXiv:2202. p. 0
Where the paper says it

Park et al. : Preprint submitted to Elsevier Page 1 of 24 arXiv:2511.

00512 2022.

[20] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell et al. Learning transferable visual models from natural language supervision. p. 3

Where the paper says it

Stable Diffusion exemplifies this approach by integrating powerful text encoders with efficient latent-space diffusion, achieving photorealistic synthesis from natural language prompts with high fidelity and diversity. Method We introduce CLUE (Controllable Latent of Unprompted Embedding), a novel architecture that enhances Stable Diffusion’s controllability by learning prompt disentangled visual representations.

In International conference on machine learning pages 8748–8763 PmLR 2021. p. 4
Where the paper says it

This study adhered to the Declaration of Helsinki and was approved with a waiver of informed consent by the Institutional Review Board of Seoul Asan Medical Center (IRB No. 2021-0837).

[21] Ross Wightman. Pytorch image models. p. 4

Where the paper says it

Our implementation is based on Python 3. Core model training and inference leverage PyTorch 2.

https://github.com/rwightman/pytorch-image-models 2019. p. 7
Where the paper says it

This domain adapted model better captures endoscopic image characteristics compared to generic pre-trained models. For each variant, we randomly sampled 1,500 images from each diagnostic category, normal, OME, and COM, yielding 4,500 images per variant.

[22] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. p. 3

Where the paper says it

CLUE: Controllable Latent space of Unprompted Embeddings Substituting the conversion formulas: 𝑥𝑡−1 = √ ̄𝛼𝑡−1 (√ ̄𝛼𝑡𝑥𝑡 − √ 1 − ̄𝛼𝑡 ̂𝑣 ) + √ 1 − ̄𝛼𝑡−1 (√ 1 − ̄𝛼𝑡𝑥𝑡 + √ ̄𝛼𝑡 ̂𝑣 ) (7) This reverse process enables sampling by iteratively denoising from 𝑥𝑇 ∼ (0, 𝐼) to 𝑥0 . Conditional diffusion models extend this framework by incorporating additional information to guide the reverse process through the conditioning variable 𝑐 in 𝑣𝜃 (𝑥𝑡 , 𝑡, 𝑐).

arXiv preprint arXiv:2010. p. 0
Where the paper says it

Park et al. : Preprint submitted to Elsevier Page 1 of 24 arXiv:2511.

02502 2020. p. 4
Where the paper says it

12) [21] for vision backbones and utility routines. Dataset Asan Medical Center (AMC) endoscopic dataset is retrospectively collected from patients who visited the Department of Otolaryngology at Seoul Asan Medical Center between January 2018 and December 2020.

[23] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. p. 4

Where the paper says it

dev0) for diffusion-based components and employ the timm library (0. 12) [21] for vision backbones and utility routines.

In Proceedings of the IEEE conference on computer vision and pattern recognition pages 2818–2826 2016. p. 4
Where the paper says it

dev0) for diffusion-based components and employ the timm library (0. 12) [21] for vision backbones and utility routines.

[24] Karl Pearson. Liii. on lines and planes of closest fit to systems of points in space. p. 0

Where the paper says it

CLUE: Controllable Latent space of Unprompted Embeddings for Diversity Management in Text-to-Image Synthesis Keunwoo Parka , Jihye Chaea , Joong Ho Ahnb and Jihoon Kweona,∗ aDepartments of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil, Songpa-gu, Seoul, 05505, Korea bDepartment of Otorhinolaryngology-Head and Neck Surgery, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil, Songpa-gu, Seoul, 05505, Korea A R T I C L E I N F O Keywords: Deep learning Text-to-image synthesis Medical image generation Diffusion Style embedding Data augmentation A B S T R A C T Text-to-image synthesis models require the ability to generate diverse images while maintaining stability. To overcome this challenge, a number of methods have been proposed, including the collection of prompt-image datasets and the integration of additional data modalities during training.

The London, Edinburgh, and Dublin philosophical magazine and journal of science 2(11):559–572 1901.

[25] Harold Hotelling. Analysis of a complex of statistical variables into principal components. p. 7

Where the paper says it

Examples of synthetic datasets are shown in Appendix C. Feature Space Analysis Principal Component Analysis (PCA) [24, 25] was performed using features extracted from an InceptionV3 model fine-tuned on Dataset 1A.

Journal of educational psychology 24(6):417 1933.

[26] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. p. 0

Where the paper says it

81) and improved recall to 70. A classifier trained on synthetic-only data at 1000% scale achieved an 𝐹1 score of 83.

Advances in neural information processing systems 30 2017. p. 2
Where the paper says it

This v-prediction parameterization has been shown to yield improved numerical stability for few step sampling scenarios. 𝑥𝑡 = √ ̄𝛼𝑡𝑥0 + √ 1 − ̄𝛼𝑡 𝜖, 𝑣𝑡 = √ ̄𝛼𝑡 𝜖 − √ 1 − ̄𝛼𝑡𝑥0 , ̄𝛼𝑡 = ∏𝑡 𝑖=1 (1 − 𝛽𝑖 ) (4) The relationships between these parameterizations enable flexible conversion: ̂𝑣 = 𝑣𝜃 (𝑥𝑡 , 𝑡, 𝑐), ̂𝑥𝑡 0 = √ ̄𝛼𝑡𝑥𝑡 − √ 1 − ̄𝛼𝑡 ̂𝑣, ̂𝜖𝑡 = √ 1 − ̄𝛼𝑡𝑥𝑡 + √ ̄𝛼𝑡 ̂𝑣 (5) 𝑣𝜃 is the neural network predicting velocity, 𝑐 is the conditioning information, and ̂𝑥 𝑡 0 , ̂𝜖𝑡 are the estimated clean image and noise at timestep 𝑡, respectively.

[27] Thomas Cover and Peter Hart. Nearest neighbor pattern classification. p. 5

Where the paper says it

Preprocessing Each raw RGB endoscopic frame is first converted to grayscale and binarized with a fixed intensity threshold of 26 to isolate the bright circular field of view, after which the largest contour is extracted and subjected to a Euclidean distance transform, its maximum distance point defining the center and radius of the maximal inscribed circle. A binary mask of this circle is then applied to the original color image to remove all pixels outside the endoscopic aperture, the masked region is cropped via its axis aligned bounding rectangle, and the result is resized to 512 × 512 pixels using bilinear interpolation with nearest neighbor resizing for the mask.

IEEE transactions on information theory 13(1):21–27 1967. p. 2
Where the paper says it

This v-prediction parameterization has been shown to yield improved numerical stability for few step sampling scenarios. 𝑥𝑡 = √ ̄𝛼𝑡𝑥0 + √ 1 − ̄𝛼𝑡 𝜖, 𝑣𝑡 = √ ̄𝛼𝑡 𝜖 − √ 1 − ̄𝛼𝑡𝑥0 , ̄𝛼𝑡 = ∏𝑡 𝑖=1 (1 − 𝛽𝑖 ) (4) The relationships between these parameterizations enable flexible conversion: ̂𝑣 = 𝑣𝜃 (𝑥𝑡 , 𝑡, 𝑐), ̂𝑥𝑡 0 = √ ̄𝛼𝑡𝑥𝑡 − √ 1 − ̄𝛼𝑡 ̂𝑣, ̂𝜖𝑡 = √ 1 − ̄𝛼𝑡𝑥𝑡 + √ ̄𝛼𝑡 ̂𝑣 (5) 𝑣𝜃 is the neural network predicting velocity, 𝑐 is the conditioning information, and ̂𝑥 𝑡 0 , ̂𝜖𝑡 are the estimated clean image and noise at timestep 𝑡, respectively.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Style embedding injection enables disentanglement of implicit features by injecting style embeddings into a new second attention layer of the U-Net architecture, allowing for precise control over image characteristics not captured in prompts.

  2. The double cross-attention mechanism modulates latent features first by a cross-attention layer attending to the prompt embedding and then by a second cross-attention layer attending to the sampled style latent, enabling both semantic alignment through text and stylistic control through the learned style representation.

  3. The training objective combines a denoising loss between predicted velocity and true velocity with a KL regularization loss between the approximate posterior of the style encoder and a standard Gaussian prior, ensuring both high fidelity generation and adherence to a controllable latent space.

  4. Increasing style variance sigma from 0.0 to 0.5 expands the feature distributions in the PCA space, successfully introducing controlled diversity that better approximates the spread and coverage of real endoscopic image features without compromising diagnostic integrity.

  5. The framework can generate diverse yet stable images from limited datasets by achieving a substantial improvement in generation quality, reducing FID from 46.81 to 9.30 on otitis media data and improving recall to 70.29%.

  6. CLUE-generated synthetic data can serve as an effective data augmentation method for domainspecific applications, maintaining high classification performance even when training exclusively on synthetic data at 1000% scale, achieving an F1 score of 83.21% in one configuration.

  7. CLUE demonstrates superior cross-institution generalization capability; models trained exclusively on CLUE-generated synthetic data at 1000% scale achieved a F1 score of 76.77%, compared to the vanilla model’s 60.61%.

Abstract

Text-to-image synthesis models require the ability to generate diverse images while maintaining stability. To overcome this challenge, a number of methods have been proposed, including the collection of prompt-image datasets and the integration of additional data modalities during training. Although these methods have shown promising results in general domains, they face limitations when applied to specialized fields such as medicine, where only limited types and insufficient amounts of data are available. We present CLUE (Controllable Latent space of Unprompted Embeddings), a generative model framework that achieves diverse generation while maintaining stability through fixed-format prompts without requiring any additional data. Based on the Stable Diffusion architecture, CLUE employs a Style Encoder that processes images and prompts to generate style embeddings, which are subsequently fed into a new second attention layer of the U-Net architecture. Through Kullback-Leibler divergence, the latent space achieves continuous representation of image features within Gaussian regions, independent of prompts. Performance was assessed on otitis media dataset. CLUE reduced FID to 9.30 (vs. 46.81) and improved recall to 70.29% (vs. 49.60%). A classifier trained on synthetic-only data at 1000% scale achieved an F1 score of 83.21% (vs. 73.83%). Combining synthetic data with equal amounts of real data achieved an F1 score of 94.76%, higher than when using only real data. On an external dataset, synthetic-only training achieved an F1 score of 76.77% (vs. 60.61%) at 1000% scale. The combined approach achieved an F1 score of 85.78%, higher than when using only the internal dataset. These results demonstrate that CLUE enables diverse yet stable image generation from limited datasets and serves as an effective data augmentation method for domain-specific applications.

Sources

Related papers