CLUE: Controllable Latent space of Unprompted Embeddings for Diversity Management in Text-to-Image Synthesis
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "CLUE: Controllable Latent space of Unprompted Embeddings for Diversity Management in Text-to-Image Synthesis".
Jane: Here's the gist of the paper. Text-to-image synthesis models require the ability to generate diverse images while maintaining stability through fixed-format prompts without requiring any additional data.
Tom: CLUE (Controllable Latent space of Unprompted Embeddings) is a generative model framework that achieves diverse generation while maintaining stability through fixed-format prompts without requiring any additional data.
Jane: Problem Statement Text-to-image synthesis models face challenges when applied to specialized fields such as medicine, where only limited types and insufficient amounts of data are available.
Tom: The medical domain presents distinct challenges for text-to-image synthesis, including a significant visual and structural gap between medical and general domain images.
Paper summary: Tom: So, CLUE proposes this framework to fix that by integrating a new component into the Stable Diffusion architecture.
Jane: They introduce something called a Style Encoder that learns visual attributes directly from paired images and prompts.
Lu: This encoder takes the image and the prompt and generates style embeddings which are then fed into a second attention layer of the U-Net architecture >
Meng: That sounds like more complexity when you're trying to deploy something at scale, how much overhead is that adding?
Tom: The authors use Kullback-Leibler divergence to make sure these features stay within Gaussian regions, which keeps the representation continuous and independent of the prompts themselves.
Jane: And for training, they use a two-part objective: a denoising loss to guide the generation, and a KL regularization loss to keep things consistent with a standard Gaussian prior >
Lalam: From my perspective in analyzing this literature, it seems like they are trying to decouple the visual style from the prompt distribution observed during training >
Lu: They argue that much of the domain knowledge is implicitly encoded in the training data, and this is inaccessible through prompts alone, which becomes especially problematic when you need precise generation in a clinical setting >
Tom: So, they're not just adding another layer; they're fundamentally changing how the model understands style and prompt interaction >
Conclusion: Jane: So, looking at the title, CLUE is all about controlling the latent space of embeddings when you don't have extra data available.
Tom: It seems to be tackling that problem head-on by focusing on managing diversity through these controllable latent spaces without needing massive amounts of supplemental information.
Lu: The authors are essentially showing how to improve fidelity and control diversity in image synthesis by combining the strengths of variational autoencoders with diffusion models >
Meng: I'm interested in the results they show, because if this actually works well, it could mean we don't need huge proprietary datasets just to get good medical image generation.
Jane: They showed substantial improvements on the otitis media dataset, reducing the FID score from forty-six point eight one down to nine point three zero and boosting recall to seventy point twenty-nine percent >
Tom: And when they combined synthetic images with real data, they got an F1 score of eighty-three point twenty-one percent, and when using one hundred percent synthetic data with real data, that hit ninety-four point seventy-six percent >
Lalam: It suggests that this framework has a strong ability to improve performance on downstream classifiers when you mix the synthetic and real data in a balanced way >
Jane: The limitation they mention is that the style embedding space doesn't offer explicit clinical interpretability yet, which means we don't know exactly *why* it made a certain stylistic choice >
Tom: So, while this is an improvement in generation quality and diversity without requiring new data for every task, we still need to figure out how to make those style embeddings more understandable clinically.
University of Ulsan College of Medicine · Asan Medical Center
cs.CV
Submitted: 2025-11-14
Updated: 2025-11-14
DOI: 10.1038/s41598-026-79220-8
Code: https://github.com/rwightman/pytorch-image-models
License: http://creativecommons.org/publicdomain/zero/1.0/
Importance score: 82/100
The gist: Text-to-image synthesis models require the ability to generate diverse images while maintaining stability through fixed-format prompts without requiring any additional data.
Key concepts
- Style Encoder
- This component learns visual attributes by processing pairs of images and their corresponding text prompts. It creates 'style embeddings' that capture the visual characteristics described by the prompt, which are then used to guide the image generation process within the model.
- U-Net Architecture Augmentation
- The Style Encoder's style embeddings are fed into a new attention layer within the U-Net structure of Stable Diffusion. This allows the model to incorporate learned visual style information directly into its denoising process, enhancing control and diversity during image creation.
- KL Regularization Loss
- This loss function measures the difference between the model's approximate posterior distribution and a standard Gaussian prior. It ensures that the learned feature representations remain continuous within Gaussian regions, independent of specific prompts, promoting stable generation.
Terminology
Summary
Text-to-image synthesis models require the ability to generate diverse images while maintaining stability through fixed-format prompts without requiring any additional data. p. 0 CLUE: Controllable Latent space of Unprompted Embeddings for
Diversity Management in Text-to-Image Synthesis
Keunwoo Parka
, Jihye Chaea
, Joong Ho Ahnb
and Jihoon Kweona,∗
aDepartments of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil,
Songpa-gu, Seoul, 05505, Korea
bDepartment of Otorhinolaryngology-Head and Neck Surgery, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro
43-gil, Songpa-gu, Seoul, 05505, Korea
A R T I C L E I N F O
Keywords:
Deep learning
Text-to-image synthesis
Medical image generation
Diffusion
Style embedding
Data augmentation
A B S T R A C T
Text-to-image synthesis models require the ability to generate diverse images while maintaining
stability. To overcome this challenge, a number of methods have been proposed, including the
collection of prompt-image datasets and the integration of additional data modalities during
training. Although these methods have shown promising results in general domains, they face
limitations when applied to specialized fields such as medicine, where only limited types and
insufficient amounts of data are available. We present CLUE (Controllable Latent space of
Unprompted Embeddings), a generative model framework that achieves diverse generation while
maintaining stability through fixed-format prompts without requiring any additional data.Where the paper says it
Where the paper says it
Problem Statement
Text-to-image synthesis models face challenges when applied to specialized fields such as medicine, where only limited types and insufficient amounts of data are available. The medical domain presents distinct challenges for text-to-image synthesis, including a significant visual and structural gap between medical and general domain images. p. 0 As illustrated in Figure 1, the medical domain presents distinct challenges for text-to-image synthesis. First, there
exists a significant visual and structural gap between medical and general domain images [9]. First, there
exists a significant visual and structural gap between medical and general domain images [9]. Second, diagnostic
reports, although clinically equivalent, are written using heterogeneous phrasing conventions, which introduces
semantic variability that can confuse generative models. When fine-tuned on report A, the model generates accurate images with A-style descriptions, but
may produce inappropriate outputs when encountering an unfamiliar phrasing style such as report B. and precise control of generated images particularly challenging when image synthesis is conducted solely through the
use of identical prompts.Where the paper says it
Where the paper says it
Where the paper says it
CLUE Architecture
CLUE augments the Stable Diffusion architecture with a Style Encoder that learns visual attributes from paired images and prompts. This Style Encoder processes images and prompts to generate style embeddings which are subsequently fed into a new second attention layer of the U-Net architecture. p. 0 We present CLUE (Controllable Latent space of
Unprompted Embeddings), a generative model framework that achieves diverse generation while
maintaining stability through fixed-format prompts without requiring any additional data. Based
on the Stable Diffusion architecture, CLUE employs a Style Encoder that processes images and
prompts to generate style embeddings, which are subsequently fed into a new second attention
layer of the U-Net architecture. Based
on the Stable Diffusion architecture, CLUE employs a Style Encoder that processes images and
prompts to generate style embeddings, which are subsequently fed into a new second attention
layer of the U-Net architecture. Through Kullback-Leibler divergence, the latent space achieves
continuous representation of image features within Gaussian regions, independent of prompts.Where the paper says it
Where the paper says it
Training Objective
The training objective for CLUE comprises two terms: the denoising loss and the KL regularization loss. The denoising loss is defined as the expected mean squared error between the model’s predicted velocity and the true velocity target. p. 4 Training Objective
Loss (𝜃, 𝜙) comprises two terms. The first term, the denoising loss, is defined as the expected mean squared
error between the model’s predicted velocity 𝑣𝜃
(𝑥𝑡
, 𝑡, 𝑐, 𝑠) and the true velocity target 𝑣𝑡
. This expectation is taken
over the data distribution of original images 𝑥 and text conditions 𝑐, a sampled timestep 𝑡, standard Gaussian noise 𝜖
used to produce the noised image 𝑥𝑡
, and the style latent 𝑠 sampled from the encoder 𝑆𝐸𝜙
. The second term, the KL
regularization loss, computes the Kullback-Leibler divergence between the approximate posterior 𝑆𝐸𝜙
(𝑠|𝑥, 𝑐) and the
standard Gaussian prior (0, 𝐼).Where the paper says it
Where the paper says it
Experimental Validation
Performance on the otitis media dataset showed substantial improvements, reducing FID from 46.81 to 9.30 and improving recall to 70. p. 1 Our model achieves a
substantial improvement in generation quality, reducing the Fréchet Inception Distance (FID) score from 46. 30 and improving recall from 49. 30 and improving recall from 49. Synthetic images generated by CLUE, when used as training data,
significantly improve downstream classifier performance on the real data, achieving an 𝐹1
score of 83. 61%) at 1000%
scale. The combined approach achieved an 𝐹1
score of 85. 83% with reference synthetic data. Combining 100% scale synthetic data with equal amounts of real data further
achieved 94. Combining synthetic data with equal amounts
of real data achieved an 𝐹1
score of 94. 76%, higher than when using only real data.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Generalization and Scalability
CLUE demonstrates robust generalization beyond the institution, as models trained exclusively on CLUE-generated synthetic data achieved an F1 score of 72.37% on the AI-HUB otoscopic image dataset at 1000% scale. p. 1 Combining 100% scale synthetic data with equal amounts of real data further
achieved 94. For evaluate external generalization, we additionally tested our framework on the AI-HUB otoscopic
image dataset. CLUE: Controllable Latent space of Unprompted Embeddings for
Diversity Management in Text-to-Image Synthesis
Keunwoo Parka
, Jihye Chaea
, Joong Ho Ahnb
and Jihoon Kweona,∗
aDepartments of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil,
Songpa-gu, Seoul, 05505, Korea
bDepartment of Otorhinolaryngology-Head and Neck Surgery, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro
43-gil, Songpa-gu, Seoul, 05505, Korea
A R T I C L E I N F O
Keywords:
Deep learning
Text-to-image synthesis
Medical image generation
Diffusion
Style embedding
Data augmentation
A B S T R A C T
Text-to-image synthesis models require the ability to generate diverse images while maintaining
stability. To overcome this challenge, a number of methods have been proposed, including the
collection of prompt-image datasets and the integration of additional data modalities during
training.Where the paper says it
Where the paper says it
Limitations and Future Work
A limitation is that the style embedding space lacks explicit clinical interpretability. Future work should explore extending CLUE to a wider range of medical imaging modalities and assess its effectiveness in class imbalanced settings. The framework’s ability to generate diverse yet diagnostically consistent images without requiring additional conditioning inputs addresses key challenges in medical AI development. p. 0 We present CLUE (Controllable Latent space of
Unprompted Embeddings), a generative model framework that achieves diverse generation while
maintaining stability through fixed-format prompts without requiring any additional data. Based
on the Stable Diffusion architecture, CLUE employs a Style Encoder that processes images and
prompts to generate style embeddings, which are subsequently fed into a new second attention
layer of the U-Net architecture.Where the paper says it
References
[1] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. p. 2 Related Work
The proposed CLUE framework integrates the complementary strengths of variational autoencoders [16] and
diffusion models to improve both fidelity and control diversity in the image synthesis. In this section, we present
a comprehensive review of the theoretical foundations and key principles underlying these generative modeling
paradigms.Where the paper says it
[2] Yong Xuan Tan, Chin Poo Lee, Mai Neo, Kian Ming Lim, Jit Yan Lim, and Ali Alqahtani. Recent advances in text-to-image synthesis: Approaches, datasets and future research prospects. p. 0 Introduction
Generative deep learning has achieved remarkable progress in recent years and has attracted significant interest
in various application domains [1]. Of these advances, text-to-image synthesis—the generation of visually coherent
and semantically consistent images from textual descriptions—stands out as both compelling and challenging [2, 3].Where the paper says it
[3] Md Ahsan Habib, Md Anwar Hussen Wadud, Md Fazlul Karim Patwary, Mohammad Motiur Rahman, M Firoz Mridha, Yuichi Okuyama, and Jungpil Shin. Exploring progress in text-to-image synthesis: An in-depth survey on the evolution of generative adversarial networks. p. 0 Introduction
Generative deep learning has achieved remarkable progress in recent years and has attracted significant interest
in various application domains [1]. Of these advances, text-to-image synthesis—the generation of visually coherent
and semantically consistent images from textual descriptions—stands out as both compelling and challenging [2, 3].Where the paper says it
[4] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. p. 0 CLUE: Controllable Latent space of Unprompted Embeddings for
Diversity Management in Text-to-Image Synthesis
Keunwoo Parka
, Jihye Chaea
, Joong Ho Ahnb
and Jihoon Kweona,∗
aDepartments of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil,
Songpa-gu, Seoul, 05505, Korea
bDepartment of Otorhinolaryngology-Head and Neck Surgery, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro
43-gil, Songpa-gu, Seoul, 05505, Korea
A R T I C L E I N F O
Keywords:
Deep learning
Text-to-image synthesis
Medical image generation
Diffusion
Style embedding
Data augmentation
A B S T R A C T
Text-to-image synthesis models require the ability to generate diverse images while maintaining
stability. To overcome this challenge, a number of methods have been proposed, including the
collection of prompt-image datasets and the integration of additional data modalities during
training. This study adhered to the Declaration of
Helsinki and was approved with a waiver of informed consent by the Institutional Review Board of Seoul Asan Medical
Center (IRB No. 2021-0837).Where the paper says it
Where the paper says it
[5] Gary Marcus, Ernest Davis, and Scott Aaronson. A very preliminary analysis of dall-e 2. p. 0 Of these advances, text-to-image synthesis—the generation of visually coherent
and semantically consistent images from textual descriptions—stands out as both compelling and challenging [2, 3]. Recent models such as DALL·E [4, 5], Imagen [6] and Stable Diffusion [7, 8] demonstrate impressive general purpose
image generation capabilities. Park et al. : Preprint submitted to Elsevier Page 1 of 24
arXiv:2511.Where the paper says it
Where the paper says it
[6] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans et al. Photorealistic text-to-image diffusion models with deep language understanding. p. 3 𝑡 = ||𝑣𝑡 − 𝑣𝜃
(𝑥𝑡
, 𝑡, 𝑐)||2 = ||√
̄𝛼𝑡
𝜖 −
√
1 − ̄𝛼𝑡𝑥0 − 𝑣𝜃
(𝑥𝑡
, 𝑡, 𝑐)||2
(8)
Latent diffusion models (LDMs) operate in a lower dimensional latent space obtained via an autoencoder,
substantially reducing computational cost while preserving semantic structure. Stable Diffusion exemplifies this
approach by integrating powerful text encoders with efficient latent-space diffusion, achieving photorealistic synthesis
from natural language prompts with high fidelity and diversity. This v-prediction
parameterization has been shown to yield improved numerical stability for few step sampling scenarios. 𝑥𝑡 =
√
̄𝛼𝑡𝑥0 +
√
1 − ̄𝛼𝑡
𝜖, 𝑣𝑡 =
√
̄𝛼𝑡
𝜖 −
√
1 − ̄𝛼𝑡𝑥0
, ̄𝛼𝑡 =
∏𝑡
𝑖=1
(1 − 𝛽𝑖
) (4)
The relationships between these parameterizations enable flexible conversion:
̂𝑣 = 𝑣𝜃
(𝑥𝑡
, 𝑡, 𝑐), ̂𝑥𝑡
0
=
√
̄𝛼𝑡𝑥𝑡 −
√
1 − ̄𝛼𝑡
̂𝑣, ̂𝜖𝑡 =
√
1 − ̄𝛼𝑡𝑥𝑡 +
√
̄𝛼𝑡
̂𝑣 (5)
𝑣𝜃
is the neural network predicting velocity, 𝑐 is the conditioning information, and ̂𝑥
𝑡
0
, ̂𝜖𝑡
are the estimated clean
image and noise at timestep 𝑡, respectively.Where the paper says it
Where the paper says it
[7] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. p. 3 𝑡 = ||𝑣𝑡 − 𝑣𝜃
(𝑥𝑡
, 𝑡, 𝑐)||2 = ||√
̄𝛼𝑡
𝜖 −
√
1 − ̄𝛼𝑡𝑥0 − 𝑣𝜃
(𝑥𝑡
, 𝑡, 𝑐)||2
(8)
Latent diffusion models (LDMs) operate in a lower dimensional latent space obtained via an autoencoder,
substantially reducing computational cost while preserving semantic structure. Stable Diffusion exemplifies this
approach by integrating powerful text encoders with efficient latent-space diffusion, achieving photorealistic synthesis
from natural language prompts with high fidelity and diversity. dev0) for diffusion-based components and employ the timm
library (0. 12) [21] for vision backbones and utility routines.Where the paper says it
Where the paper says it
[8] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer et al. Scaling rectified flow transformers for high-resolution image synthesis. p. 0 This requirement is particularly pronounced in medical imaging, where applications demand not only high visual
fidelity but also robust stability and precision due to clinical decision-making. As illustrated in Figure 1, the medical domain presents distinct challenges for text-to-image synthesis. As illustrated in Figure 1, the medical domain presents distinct challenges for text-to-image synthesis. First, there
exists a significant visual and structural gap between medical and general domain images [9].Where the paper says it
Where the paper says it
[9] Jee Seok Yoon, Kwanseok Oh, Yooseung Shin, Maciej A Mazurowski, and Heung-Il Suk. Domain generalization for medical image analysis: A review. p. 4 All video
sequences were originally acquired for clinical diagnostic purposes; from each sequence, a single frame exhibiting
the entire tympanic membrane was anonymized and exported for analysis. This study adhered to the Declaration of
Helsinki and was approved with a waiver of informed consent by the Institutional Review Board of Seoul Asan Medical
Center (IRB No.Where the paper says it
[10] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. p. 2 Diffusion Models
Diffusion models define a forward noising process that gradually corrupts data and a learned reverse process that
recovers clean samples [10, 18]. This framework unites the stability of probabilistic modeling with the ability to produce
high-quality samples, and has achieved state-of-the-art results across image, audio, and text domains. This study adhered to the Declaration of
Helsinki and was approved with a waiver of informed consent by the Institutional Review Board of Seoul Asan Medical
Center (IRB No. 2021-0837).Where the paper says it
Where the paper says it
[11] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. p. 0 CLUE: Controllable Latent space of Unprompted Embeddings for
Diversity Management in Text-to-Image Synthesis
Keunwoo Parka
, Jihye Chaea
, Joong Ho Ahnb
and Jihoon Kweona,∗
aDepartments of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil,
Songpa-gu, Seoul, 05505, Korea
bDepartment of Otorhinolaryngology-Head and Neck Surgery, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro
43-gil, Songpa-gu, Seoul, 05505, Korea
A R T I C L E I N F O
Keywords:
Deep learning
Text-to-image synthesis
Medical image generation
Diffusion
Style embedding
Data augmentation
A B S T R A C T
Text-to-image synthesis models require the ability to generate diverse images while maintaining
stability. To overcome this challenge, a number of methods have been proposed, including the
collection of prompt-image datasets and the integration of additional data modalities during
training. Park et al. : Preprint submitted to Elsevier Page 1 of 24
arXiv:2511.Where the paper says it
Where the paper says it
[12] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. p. 1 and precise control of generated images particularly challenging when image synthesis is conducted solely through the
use of identical prompts. Instability is rooted in the text-to-image models; their generative behavior is tightly coupled to the prompt
distribution observed during training. dev0) for diffusion-based components and employ the timm
library (0. 12) [21] for vision backbones and utility routines.Where the paper says it
Where the paper says it
[13] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. p. 0 CLUE: Controllable Latent space of Unprompted Embeddings for
Diversity Management in Text-to-Image Synthesis
Keunwoo Parka
, Jihye Chaea
, Joong Ho Ahnb
and Jihoon Kweona,∗
aDepartments of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil,
Songpa-gu, Seoul, 05505, Korea
bDepartment of Otorhinolaryngology-Head and Neck Surgery, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro
43-gil, Songpa-gu, Seoul, 05505, Korea
A R T I C L E I N F O
Keywords:
Deep learning
Text-to-image synthesis
Medical image generation
Diffusion
Style embedding
Data augmentation
A B S T R A C T
Text-to-image synthesis models require the ability to generate diverse images while maintaining
stability. To overcome this challenge, a number of methods have been proposed, including the
collection of prompt-image datasets and the integration of additional data modalities during
training.Where the paper says it
[14] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. p. 0 Based
on the Stable Diffusion architecture, CLUE employs a Style Encoder that processes images and
prompts to generate style embeddings, which are subsequently fed into a new second attention
layer of the U-Net architecture. Through Kullback-Leibler divergence, the latent space achieves
continuous representation of image features within Gaussian regions, independent of prompts. CLUE: Controllable Latent space of Unprompted Embeddings for
Diversity Management in Text-to-Image Synthesis
Keunwoo Parka
, Jihye Chaea
, Joong Ho Ahnb
and Jihoon Kweona,∗
aDepartments of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil,
Songpa-gu, Seoul, 05505, Korea
bDepartment of Otorhinolaryngology-Head and Neck Surgery, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro
43-gil, Songpa-gu, Seoul, 05505, Korea
A R T I C L E I N F O
Keywords:
Deep learning
Text-to-image synthesis
Medical image generation
Diffusion
Style embedding
Data augmentation
A B S T R A C T
Text-to-image synthesis models require the ability to generate diverse images while maintaining
stability. To overcome this challenge, a number of methods have been proposed, including the
collection of prompt-image datasets and the integration of additional data modalities during
training.Where the paper says it
Where the paper says it
[15] Solomon Kullback and Richard A Leibler. p. 0 Based
on the Stable Diffusion architecture, CLUE employs a Style Encoder that processes images and
prompts to generate style embeddings, which are subsequently fed into a new second attention
layer of the U-Net architecture. Through Kullback-Leibler divergence, the latent space achieves
continuous representation of image features within Gaussian regions, independent of prompts. This v-prediction
parameterization has been shown to yield improved numerical stability for few step sampling scenarios. 𝑥𝑡 =
√
̄𝛼𝑡𝑥0 +
√
1 − ̄𝛼𝑡
𝜖, 𝑣𝑡 =
√
̄𝛼𝑡
𝜖 −
√
1 − ̄𝛼𝑡𝑥0
, ̄𝛼𝑡 =
∏𝑡
𝑖=1
(1 − 𝛽𝑖
) (4)
The relationships between these parameterizations enable flexible conversion:
̂𝑣 = 𝑣𝜃
(𝑥𝑡
, 𝑡, 𝑐), ̂𝑥𝑡
0
=
√
̄𝛼𝑡𝑥𝑡 −
√
1 − ̄𝛼𝑡
̂𝑣, ̂𝜖𝑡 =
√
1 − ̄𝛼𝑡𝑥𝑡 +
√
̄𝛼𝑡
̂𝑣 (5)
𝑣𝜃
is the neural network predicting velocity, 𝑐 is the conditioning information, and ̂𝑥
𝑡
0
, ̂𝜖𝑡
are the estimated clean
image and noise at timestep 𝑡, respectively. It enables the generation of images that satisfy specified
conditions within the domain on which the model was trained. 𝑝𝜃
(𝑥|𝑐) = ∫
𝑝𝜃
(𝑥|𝑧, 𝑐)𝑝𝜃
(𝑧|𝑐)𝑑𝑧 (1)
This conditional likelihood is maximized by optimizing the following evidence lower bound (ELBO):
log 𝑝𝜃
(𝑥|𝑐) ≥ 𝔼𝑧∼𝑞𝜙(𝑧|𝑥,𝑐)
[
log 𝑝𝜃
(𝑥|𝑧, 𝑐)
]
− 𝐷KL(𝑞𝜙
(𝑧|𝑥, 𝑐)‖𝑝𝜃
(𝑧|𝑐)) (2)
The approximate posterior is reparametrized using Gaussian statistics:
𝑞𝜙
(𝑧|𝑥, 𝑐) = (𝜇𝜙
(𝑥, 𝑐), diag(𝜎𝜙
(𝑥, 𝑐)
2
)), 𝑧 = 𝜇𝜙
(𝑥, 𝑐) + 𝜎𝜙
(𝑥, 𝑐) ⊙ 𝜖, 𝜖 ∼ (0, 𝐼) (3)
where ⊙ denotes element-wise multiplication.Where the paper says it
Where the paper says it
Where the paper says it
[16] Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes 2013. p. 3 The clean
image and prompt are sent into the style encoder to parameterize a Gaussian posterior from which a style latent 𝑠 ∈ ℝ1024
is sampled. Style Encoding and Sampling
Specifically, we reuse the SD 2.Where the paper says it
[17] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. p. 6 CLUE: Controllable Latent space of Unprompted Embeddings
Figure 3: Overview of the four stage preprocessing applied to each endoscopic frame
Table 3
Training configurations for generative models
Parameter Value
Base model SD 2. 1
Diffusion scheduler DDIMScheduler
Diffusion timesteps 1,000
Prediction type v-prediction
Resolution 512 × 512 × 3
Normalization [-1, +1]
Precision FP32
Batch 4
Optimizer AdamW
Learning rate 1e-5
Learning scheduler Constant
Maximum steps 10,000
Using these exact settings, we trained our proposed CLUE model and, for comparison, fine-tuned a Stable Diffusion
2. This v-prediction
parameterization has been shown to yield improved numerical stability for few step sampling scenarios. 𝑥𝑡 =
√
̄𝛼𝑡𝑥0 +
√
1 − ̄𝛼𝑡
𝜖, 𝑣𝑡 =
√
̄𝛼𝑡
𝜖 −
√
1 − ̄𝛼𝑡𝑥0
, ̄𝛼𝑡 =
∏𝑡
𝑖=1
(1 − 𝛽𝑖
) (4)
The relationships between these parameterizations enable flexible conversion:
̂𝑣 = 𝑣𝜃
(𝑥𝑡
, 𝑡, 𝑐), ̂𝑥𝑡
0
=
√
̄𝛼𝑡𝑥𝑡 −
√
1 − ̄𝛼𝑡
̂𝑣, ̂𝜖𝑡 =
√
1 − ̄𝛼𝑡𝑥𝑡 +
√
̄𝛼𝑡
̂𝑣 (5)
𝑣𝜃
is the neural network predicting velocity, 𝑐 is the conditioning information, and ̂𝑥
𝑡
0
, ̂𝜖𝑡
are the estimated clean
image and noise at timestep 𝑡, respectively.Where the paper says it
Where the paper says it
[18] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. p. 0 CLUE: Controllable Latent space of Unprompted Embeddings for
Diversity Management in Text-to-Image Synthesis
Keunwoo Parka
, Jihye Chaea
, Joong Ho Ahnb
and Jihoon Kweona,∗
aDepartments of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil,
Songpa-gu, Seoul, 05505, Korea
bDepartment of Otorhinolaryngology-Head and Neck Surgery, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro
43-gil, Songpa-gu, Seoul, 05505, Korea
A R T I C L E I N F O
Keywords:
Deep learning
Text-to-image synthesis
Medical image generation
Diffusion
Style embedding
Data augmentation
A B S T R A C T
Text-to-image synthesis models require the ability to generate diverse images while maintaining
stability. To overcome this challenge, a number of methods have been proposed, including the
collection of prompt-image datasets and the integration of additional data modalities during
training. CLUE: Controllable Latent space of Unprompted Embeddings for
Diversity Management in Text-to-Image Synthesis
Keunwoo Parka
, Jihye Chaea
, Joong Ho Ahnb
and Jihoon Kweona,∗
aDepartments of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil,
Songpa-gu, Seoul, 05505, Korea
bDepartment of Otorhinolaryngology-Head and Neck Surgery, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro
43-gil, Songpa-gu, Seoul, 05505, Korea
A R T I C L E I N F O
Keywords:
Deep learning
Text-to-image synthesis
Medical image generation
Diffusion
Style embedding
Data augmentation
A B S T R A C T
Text-to-image synthesis models require the ability to generate diverse images while maintaining
stability. To overcome this challenge, a number of methods have been proposed, including the
collection of prompt-image datasets and the integration of additional data modalities during
training.Where the paper says it
Where the paper says it
[19] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. p. 2 In addition to the common noise prediction and clean sample parameterization, diffusion models may be trained
to predict a velocity term 𝑣𝑡
, which combines the clean signal and noise in a single target [19]. This v-prediction
parameterization has been shown to yield improved numerical stability for few step sampling scenarios. Park et al. : Preprint submitted to Elsevier Page 1 of 24
arXiv:2511.Where the paper says it
Where the paper says it
[20] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell et al. Learning transferable visual models from natural language supervision. p. 3 Stable Diffusion exemplifies this
approach by integrating powerful text encoders with efficient latent-space diffusion, achieving photorealistic synthesis
from natural language prompts with high fidelity and diversity. Method
We introduce CLUE (Controllable Latent of Unprompted Embedding), a novel architecture that enhances Stable
Diffusion’s controllability by learning prompt disentangled visual representations. This study adhered to the Declaration of
Helsinki and was approved with a waiver of informed consent by the Institutional Review Board of Seoul Asan Medical
Center (IRB No. 2021-0837).Where the paper says it
Where the paper says it
[21] Ross Wightman. Pytorch image models. p. 4 Our
implementation is based on Python 3. Core model training and inference leverage PyTorch 2. This domain adapted model better captures endoscopic image characteristics compared
to generic pre-trained models. For each variant, we randomly sampled 1,500 images from each diagnostic category,
normal, OME, and COM, yielding 4,500 images per variant.Where the paper says it
Where the paper says it
[22] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. p. 3 CLUE: Controllable Latent space of Unprompted Embeddings
Substituting the conversion formulas:
𝑥𝑡−1 =
√
̄𝛼𝑡−1 (√
̄𝛼𝑡𝑥𝑡 −
√
1 − ̄𝛼𝑡
̂𝑣
)
+
√
1 − ̄𝛼𝑡−1 (√
1 − ̄𝛼𝑡𝑥𝑡 +
√
̄𝛼𝑡
̂𝑣
)
(7)
This reverse process enables sampling by iteratively denoising from 𝑥𝑇 ∼ (0, 𝐼) to 𝑥0
. Conditional diffusion
models extend this framework by incorporating additional information to guide the reverse process through the
conditioning variable 𝑐 in 𝑣𝜃
(𝑥𝑡
, 𝑡, 𝑐). Park et al. : Preprint submitted to Elsevier Page 1 of 24
arXiv:2511. 12) [21] for vision backbones and utility routines. Dataset
Asan Medical Center (AMC) endoscopic dataset is retrospectively collected from patients who visited the
Department of Otolaryngology at Seoul Asan Medical Center between January 2018 and December 2020.Where the paper says it
Where the paper says it
Where the paper says it
[23] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. p. 4 dev0) for diffusion-based components and employ the timm
library (0. 12) [21] for vision backbones and utility routines. dev0) for diffusion-based components and employ the timm
library (0. 12) [21] for vision backbones and utility routines.Where the paper says it
Where the paper says it
[24] Karl Pearson. Liii. on lines and planes of closest fit to systems of points in space. p. 0 CLUE: Controllable Latent space of Unprompted Embeddings for
Diversity Management in Text-to-Image Synthesis
Keunwoo Parka
, Jihye Chaea
, Joong Ho Ahnb
and Jihoon Kweona,∗
aDepartments of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro 43-gil,
Songpa-gu, Seoul, 05505, Korea
bDepartment of Otorhinolaryngology-Head and Neck Surgery, University of Ulsan College of Medicine, Asan Medical Center, 88, Olympic-ro
43-gil, Songpa-gu, Seoul, 05505, Korea
A R T I C L E I N F O
Keywords:
Deep learning
Text-to-image synthesis
Medical image generation
Diffusion
Style embedding
Data augmentation
A B S T R A C T
Text-to-image synthesis models require the ability to generate diverse images while maintaining
stability. To overcome this challenge, a number of methods have been proposed, including the
collection of prompt-image datasets and the integration of additional data modalities during
training.Where the paper says it
[25] Harold Hotelling. Analysis of a complex of statistical variables into principal components. p. 7 Examples of synthetic datasets are shown in Appendix C. Feature Space Analysis
Principal Component Analysis (PCA) [24, 25] was performed using features extracted from an InceptionV3 model
fine-tuned on Dataset 1A.Where the paper says it
[26] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. p. 0 81) and
improved recall to 70. A classifier trained on synthetic-only data at 1000% scale
achieved an 𝐹1
score of 83. This v-prediction
parameterization has been shown to yield improved numerical stability for few step sampling scenarios. 𝑥𝑡 =
√
̄𝛼𝑡𝑥0 +
√
1 − ̄𝛼𝑡
𝜖, 𝑣𝑡 =
√
̄𝛼𝑡
𝜖 −
√
1 − ̄𝛼𝑡𝑥0
, ̄𝛼𝑡 =
∏𝑡
𝑖=1
(1 − 𝛽𝑖
) (4)
The relationships between these parameterizations enable flexible conversion:
̂𝑣 = 𝑣𝜃
(𝑥𝑡
, 𝑡, 𝑐), ̂𝑥𝑡
0
=
√
̄𝛼𝑡𝑥𝑡 −
√
1 − ̄𝛼𝑡
̂𝑣, ̂𝜖𝑡 =
√
1 − ̄𝛼𝑡𝑥𝑡 +
√
̄𝛼𝑡
̂𝑣 (5)
𝑣𝜃
is the neural network predicting velocity, 𝑐 is the conditioning information, and ̂𝑥
𝑡
0
, ̂𝜖𝑡
are the estimated clean
image and noise at timestep 𝑡, respectively.Where the paper says it
Where the paper says it
[27] Thomas Cover and Peter Hart. Nearest neighbor pattern classification. p. 5 Preprocessing
Each raw RGB endoscopic frame is first converted to grayscale and binarized with a fixed intensity threshold of
26 to isolate the bright circular field of view, after which the largest contour is extracted and subjected to a Euclidean
distance transform, its maximum distance point defining the center and radius of the maximal inscribed circle. A binary
mask of this circle is then applied to the original color image to remove all pixels outside the endoscopic aperture, the
masked region is cropped via its axis aligned bounding rectangle, and the result is resized to 512 × 512 pixels using
bilinear interpolation with nearest neighbor resizing for the mask. This v-prediction
parameterization has been shown to yield improved numerical stability for few step sampling scenarios. 𝑥𝑡 =
√
̄𝛼𝑡𝑥0 +
√
1 − ̄𝛼𝑡
𝜖, 𝑣𝑡 =
√
̄𝛼𝑡
𝜖 −
√
1 − ̄𝛼𝑡𝑥0
, ̄𝛼𝑡 =
∏𝑡
𝑖=1
(1 − 𝛽𝑖
) (4)
The relationships between these parameterizations enable flexible conversion:
̂𝑣 = 𝑣𝜃
(𝑥𝑡
, 𝑡, 𝑐), ̂𝑥𝑡
0
=
√
̄𝛼𝑡𝑥𝑡 −
√
1 − ̄𝛼𝑡
̂𝑣, ̂𝜖𝑡 =
√
1 − ̄𝛼𝑡𝑥𝑡 +
√
̄𝛼𝑡
̂𝑣 (5)
𝑣𝜃
is the neural network predicting velocity, 𝑐 is the conditioning information, and ̂𝑥
𝑡
0
, ̂𝜖𝑡
are the estimated clean
image and noise at timestep 𝑡, respectively.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Style embedding injection enables disentanglement of implicit features by injecting style embeddings into a new second attention layer of the U-Net architecture, allowing for precise control over image characteristics not captured in prompts.
-
The double cross-attention mechanism modulates latent features first by a cross-attention layer attending to the prompt embedding and then by a second cross-attention layer attending to the sampled style latent, enabling both semantic alignment through text and stylistic control through the learned style representation.
-
The training objective combines a denoising loss between predicted velocity and true velocity with a KL regularization loss between the approximate posterior of the style encoder and a standard Gaussian prior, ensuring both high fidelity generation and adherence to a controllable latent space.
-
Increasing style variance sigma from 0.0 to 0.5 expands the feature distributions in the PCA space, successfully introducing controlled diversity that better approximates the spread and coverage of real endoscopic image features without compromising diagnostic integrity.
-
The framework can generate diverse yet stable images from limited datasets by achieving a substantial improvement in generation quality, reducing FID from 46.81 to 9.30 on otitis media data and improving recall to 70.29%.
-
CLUE-generated synthetic data can serve as an effective data augmentation method for domainspecific applications, maintaining high classification performance even when training exclusively on synthetic data at 1000% scale, achieving an F1 score of 83.21% in one configuration.
-
CLUE demonstrates superior cross-institution generalization capability; models trained exclusively on CLUE-generated synthetic data at 1000% scale achieved a F1 score of 76.77%, compared to the vanilla model’s 60.61%.
Abstract
Text-to-image synthesis models require the ability to generate diverse images while maintaining stability. To overcome this challenge, a number of methods have been proposed, including the collection of prompt-image datasets and the integration of additional data modalities during training. Although these methods have shown promising results in general domains, they face limitations when applied to specialized fields such as medicine, where only limited types and insufficient amounts of data are available. We present CLUE (Controllable Latent space of Unprompted Embeddings), a generative model framework that achieves diverse generation while maintaining stability through fixed-format prompts without requiring any additional data. Based on the Stable Diffusion architecture, CLUE employs a Style Encoder that processes images and prompts to generate style embeddings, which are subsequently fed into a new second attention layer of the U-Net architecture. Through Kullback-Leibler divergence, the latent space achieves continuous representation of image features within Gaussian regions, independent of prompts. Performance was assessed on otitis media dataset. CLUE reduced FID to 9.30 (vs. 46.81) and improved recall to 70.29% (vs. 49.60%). A classifier trained on synthetic-only data at 1000% scale achieved an F1 score of 83.21% (vs. 73.83%). Combining synthetic data with equal amounts of real data achieved an F1 score of 94.76%, higher than when using only real data. On an external dataset, synthetic-only training achieved an F1 score of 76.77% (vs. 60.61%) at 1000% scale. The combined approach achieved an F1 score of 85.78%, higher than when using only the internal dataset. These results demonstrate that CLUE enables diverse yet stable image generation from limited datasets and serves as an effective data augmentation method for domain-specific applications.
Sources
- A very preliminary analysis of DALL-E 2
- IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models
- Progressive Distillation for Fast Sampling of Diffusion Models
- Denoising Diffusion Implicit Models
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs