Mitigating Long-Tail Bias via Prompt-Controlled Diffusion Augmentation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Mitigating Long-Tail Bias via Prompt-Controlled Diffusion Augmentation".
Paper summary: Tom: So we're looking at this paper now, "Mitigating Long-Tail Bias via Prompt-Controlled Diffusion Augmentation." Basically, it tackles a big problem in satellite imagery segmentation where some classes are huge and others are tiny, and those tiny classes get totally ignored by the AI.
Jane: Right. The main issue is that dominant classes take up most of the pixels, which means the model learns to focus almost entirely on them and misses rare ones entirely, especially when you have different types of data coming from different sources, like urban versus rural areas <ref:2602.04749#pg2>. So what does this paper claim is the fix?
Lu: They propose a prompt-controlled diffusion augmentation framework. Instead of just trying to fix the loss function by doing things like class re-weighting or focal loss, which they say mostly just stabilize things without actually exposing the model to those rare classes <ref:2602.04749#pg2>.
Tom: It claims this framework generates new paired label image samples with explicit control over what the semantic composition looks like and what domain it is in. I need to know if that's just making up data or something more substantial.
Jane: Well, they have two stages for this process, starting with Stage A where they use a discrete diffusion model, called D3PM, to create semantic layouts <ref:2602.04749#pg3>. This stage is conditioned on a desired class ratio target and the domain label you want to simulate <ref:2602.04749#pg3>.
Meng: Creating synthetic layouts sounds good, but how do you make sure those generated images actually look like real remote sensing imagery, not just abstract maps? What's the catch there?
Lu: Stage B takes those generated layouts and uses a fine-tuned Stable Diffusion network to turn them into photorealistic satellite images <ref:2602.04749#pg4>. Crucially, this stage makes sure they keep the spatial structure and the domain appearance consistent <ref:2602.04749#pg4>.
Tom: So Stage A controls the semantic makeup based on those ratios, and Stage B handles the visual realism while keeping that domain context intact. What's driving that control in Stage A?
Paper summary: Jane: They define a conditioning embedding where the final input combines a ratio embedding, let's call it er, with a domain embedding ed, and they use a learnable scalar alpha to mix them together like this: e equals er plus alpha ed <ref:2602.04749#pg3>. That alpha is what controls how much the domain influences the semantic layout generation.
Meng: Okay, so they are using diffusion models for layout generation, which is interesting because it gives them that diversity they need for augmentation <ref:2602.04749#pg8>. But what about their training objective? How do you train this to actually achieve those class ratios?
Lu: For Stage A, the training involves minimizing a negative variational lower bound loss with an auxiliary denoising cross-entropy term for stability, and adding a masked ratio constraint to get the stage A loss function <ref:2602.04749#pg4>. They use this two-weight ratio-matching loss where one term enforces the requested ratios directly, and there's a smaller, weighted term that softly regularizes the classes they didn't explicitly target <ref:2602.04749#pg3>.
Tom: That sounds like they are punishing the model when it doesn't match the desired class ratios in the generated layouts. So, what are they testing this on? What were the results like when they mixed these synthetic pairs into their real training set?
Jane: They use a controlled sampling protocol to mix these synthetic samples with their existing real training set to train state-of-the-art segmentation models <ref:2602.04749#pg4>. And the results, as shown in Table one they say that adding these ratio-controlled synthetic pairs consistently improves segmentation across different backbones <ref:2602.04749#pg2>.
Meng: That's a bit vague. How much of an improvement are we talking about? Is it just a few percentage points across the board, or is it huge for the classes that were previously neglected?
Lu: The paper says the largest gains come specifically on minority and mid-tail classes rather than just focusing on improving the head classes <ref:2602.04749#pg2>. They found in-domain mean intersection over union increases for all models, but particularly strong improvements are seen in agriculture, road, and water segmentation <ref:2602.04749#pg2>.
Paper summary: Tom: So it's not just making the dominant classes look better; it’s actually helping those smaller classes get segmented correctly. That's a significant finding for practical applications. What about when they look at domain generalization, like moving from one domain to another?
Jane: They also found that this synthesis helps with transfer in both directions between domains <ref:2602.04749#pg4>. This is important because their dataset, LoveDA, has a specific Urban and Rural split where the statistical differences between those two areas are quite substantial <ref:2602.04749#pg2>.
Meng: From an engineering standpoint, if we're deploying this at scale, what’s the overhead? Are we talking about generating massive amounts of data just to fine-tune a model?
Lu: The method is designed to be complementary; it synthesizes additional labeled data instead of only reshaping the loss function <ref:2602.04749#pg2>. They argue that diffusion models provide strong fidelity and diversity for this purpose <ref:2602.04749#pg8>.
Tom: So, to wrap up on what we’ve heard about "Mitigating Long-Tail Bias via Prompt-Controlled Diffusion Augmentation," the authors are focusing on adding the right data in the right proportions, not just throwing more random data at the problem <ref:2602.04749#pg2>.
Jane: Exactly. The framework is a way to generate paired label image samples with explicit control over what's in them and which domain they belong to, which helps targeted enrichment of underrepresented classes <ref:2602.04749#pg2>.
Lu: It’s about using the diffusion models to create this structured, controllable data augmentation pipeline that addresses both within-domain imbalance and cross-domain shifts simultaneously <ref:2602.04749#pg3>.
Meng: So, for deployment, the real question is how robust this augmentation step is when you move to a completely new domain not seen in the training set, like moving from Urban to Rural data entirely <ref:2602.04749#pg4>.
Tom: That's the next big test. But for now, what we have here is a method that uses prompt control within a diffusion pipeline to synthesize targeted data for segmentation tasks facing long-tail class imbalance in remote sensing imagery <ref:2602.04749#pg2>.
Jane: And it points toward using generative augmentation not just as a way to add volume, but as a precise tool for balancing the learning signal, especially when dealing with complex cross-domain shifts like those seen in datasets such as LoveDA <ref:2602.04749#pg4>.
Conclusion: Tom: So we’ve looked at how they use these diffusion models to generate synthetic data that balances class ratios and domain shifts in remote sensing imagery <ref:2602.04749#pg3>.
Jane: Right, the core idea is that instead of just throwing more random pictures at a model, you create the right kinds of pictures—the ones where the rare classes are properly represented—in controlled ways <ref:2602.04749#pg2>.
Lu: The authors are using a two-stage process. First, they use a discrete diffusion model to plan out the semantic layout based on what ratio and domain you want <ref:2602.04749#pg3>. Then, a second diffusion network draws the actual picture from that layout while making sure it looks real and matches the domain <ref:2602.04749#pg4>.
Meng: I’m looking at this framework, and my immediate thought is how much computational overhead that two-stage generation adds to a production pipeline <ref:2602.04749#pg3>.
Tom: That's fair, Meng. But the goal isn't just adding cost; it’s about getting a better model trained on data that actually helps it recognize those small classes <ref:2602.04749#pg2>. It’s about precision, not just volume.
Jane: And what this means for us is that if you have a dataset where some things are super common and others are almost invisible, this isn't just a data problem anymore <ref:2602.04749#pg2>. It’s about engineering the training signal so the AI learns to see everything equally <ref:2602.04749#pg3>.
Lu: They show that this approach actually helps when you move data between different real-world settings, like switching from urban satellite views to rural ones <ref:2602.04749#pg4>. It suggests that controlling the visual style during augmentation is just as important as controlling the class balance itself <ref:2602.04749#pg3>.
Tom: So, in short, they’re using prompt control within a diffusion pipeline to generate targeted data to fix class imbalance and domain shifts in remote sensing segmentation <ref:2602.04749#pg2>.
Jane: Exactly. The authors are proving that adding the right data in the right proportions is far more effective than just adding more of whatever you already have <ref:2602.04749#pg2>.
Meng: That makes sense for improving model performance, but we still need to see how stable this goes when we try to deploy it on massive, messy datasets where the underlying domain shift is really unpredictable <ref:2602.04749#pg4>.
Lu: That’s the next frontier. We’ve shown that controlling the composition helps with in-domain performance, but generalizing this augmentation to completely unseen domains is definitely where we need to focus our attention <ref:2602.04749#pg3>.
Buddhi Wijenayake, Nichula Wasalathilake, Parakrama Ekanayake, Roshan Godaliyadda, Vishal M. Patel, Vijitha Herath
University of Peradeniya · Johns Hopkins University
cs.CV
Submitted: 2026-02-04
Updated: 2026-04-26
Comments: Accepted to Publication at 2026 IEEE International Geoscience and Remote Sensing Symposium (IGARSS)
DOI: 10.1109/IGARSS60577.2026.11730595
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 89/100
The gist: Long-tailed class imbalance remains a fundamental obstacle in semantic segmentation of high-resolution remotesensing imagery…
Key concepts
- Long-tailed Class Imbalance
- This occurs when a few dominant classes take up most of the pixels in an image, leaving rare classes with very few examples. This biases learning toward the frequent classes and makes it hard for models to accurately identify and segment the minority classes.
- Prompt-Controlled Diffusion Augmentation
- This is a two-step process using diffusion models. The first step creates a semantic map based on desired class ratios and domain, acting as a 'prompt' for the second step. The second step then generates high-resolution images that look realistic while adhering to the specified semantic structure.
- Domain-Aware Synthesis
- The framework explicitly controls both the semantic composition (class ratios) and the domain (e.g., Urban vs. Rural). This ensures that synthetic data is not just random noise but specifically tailored to fill gaps in underrepresented classes within a particular geographical setting.
Terminology
Summary
Long-tailed class imbalance remains a fundamental obstacle in semantic segmentation of high-resolution remotesensing imagery, where dominant classes shape learned representations and rare classes are systematically under-segmented <ref:2602. p. 004749#pg2> This challenge becomes more acute in cross-domain settings such as LoveDA, which exhibits an explicit Urban/Rural split with substantial appearance differences and inconsistent classfrequency statistics across domains <ref:2602. p. 004749#pg4> We propose a promptcontrolled diffusion augmentation framework that generates paired label-image samples with explicit control over semantic composition and domain, enabling targeted enrichment of underrepresented classes rather than indiscriminate dataset expansion <ref:2602. p. 004749#pg2>
The Problem Context
Semantic segmentation of high-resolution remote sensing imagery supports key geospatial applications such as urban mapping, land-cover monitoring, and environmental assessment <ref:2602.04749#pg2> Performance in realistic settings is often limited by severe pixel-level class imbalance where a few dominant categories occupy most pixels, while minority classes appear sparsely, creating a long-tailed training signal that biases learning toward frequent classes and degrades rareclass recognition <ref:2602. p. 004749#pg2> This challenge is amplified in LoveDA which is explicitly organized into Urban and Rural domains with distinct scene structure, appearance, and inconsistent class distributions across domains <ref:2602. p. 004749#pg2>
The Proposed Framework
The proposed architecture is a two-stage, domain-aware generative augmentation pipeline that synthesizes paired label–image samples with controllable semantic composition <ref:2602.04749#pg2> In the first Stage (Stage A), a ratio and domain-conditioned discrete diffusion model (D3PM) generates semantic layouts that match desired class-ratio targets within a specified domain <ref:2602. p. 104749#pg3> In the second stage (Stage B), a fine-tuned Stable Diffusion network translates these layouts into photorealistic remotesensing images while preserving spatial structure and domain appearance <ref:2602. p. 104749#pg4>
Training Signal Construction
Stage A utilizes a D3PM to generate semantic layouts conditioned on a ratio target and a domain label, where the conditioning embedding is defined as e = er + α ed <ref:2602.04749#pg4> The training objective for Stage A involves minimizing the negative variational lower bound LVLB with an auxiliary denoising cross-entropy term LCE for stabilization, and adding the masked ratio constraint to obtain the stage A loss function LA = LVLB + 0. p. 25LCE + Lratio <ref:2602. p. 204749#pg4> Stage B employs a latent diffusion model guided by ControlNet features from the layout, with FiLM gated residual injection and a domain/ratio prompt for domain-consistent appearance <ref:2602. p. 104749#pg4>
Impact of Synthetic Data
The resulting synthetic corpus is mixed with the real training set using a controlled sampling protocol to train the state-of-the-art segmentation models <ref:2602.04749#pg4> Table I shows that adding ratio-controlled synthetic pairs consistently improves segmentation across backbones, with the largest gains concentrated on minority and mid-tail classes rather than only the head classes <ref:2602. p. 304749#pg2> Specifically, in-domain mIoU increases for all models, with particularly strong improvements in agriculture, road, and water <ref:2602. p. 304749#pg2> In domain generalization, synthesis has also improved transfer in both directions <ref:2602. p. 304749#pg4>
Conclusion
We presented a prompt-controlled diffusion augmentation framework for semantic segmentation under long-tailed class imbalance and domain shift in remote-sensing imagery <ref:2602.04749#pg2> The key value of the proposed framework is therefore not simply to add more data, but to add the right data in the right proportions <ref:2602. p. 304749#pg2> Future work will extend this idea to more datasets, tasks, and stronger forms of controllable generative augmentation <ref:2602. p. 304749#pg2>
REFERENCES
[1] Q. Yuan, H. Shen, T. Li, Z. Li, S. Li, Y. Jiang, H. Xu, W. Tan, Q. Yang, J. Wang, J Gao and L Zhang “Deep learning in environmental remote sensing: Achievements and challenges” Remote Sensing of Environment vol 241 p 111716 2020 <ref:2602. p. 404749#pg4>
[2] L Ma, Y Liu, X Zhang, Y Ye, G Yin and B A Johnson “Deep learning in remote sensing applications: A meta-analysis and review” ISPRS Journal of Photogrammetry and Remote Sensing vol 152 pp 166–177 2019 <ref:2602.04749#pg4>
[3] J Wang, Z Zheng, A Ma, X Lu and Y Zhong “Loveda: A remote sensing land-cover dataset for domain adaptive semantic segmentation” 2022 [Online] Available: https://arxiv.org/abs/2110. p. 408733 <ref:2602. p. 004749#pg4>
[4] Y Cui, M Jia, T.-Y Lin, Y Song and S Belongie “Classbalanced loss based on effective number of samples” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2019 pp 9260–9269 <ref:2602. p. 404749#pg4>
[5] T.-Y Lin, P Goyal, R Girshick, K He and P Dollar “Focal loss for dense object detection” in 2017 IEEE International Conference on Computer Vision (ICCV) 2017 pp 2999–3007 <ref:2602. p. 404749#pg4>
[6] A Shrivastava, A Gupta and R Girshick “Training regionbased object detectors with online hard example mining” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016 pp 761–769 <ref:2602.04749#pg4>
[7] E D Cubuk, B Zoph, D Mane, V Vasudevan and Q V Le “Autoaugment: Learning augmentation strategies from data” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2019 <ref:2602.04749#pg4>
[8] J Ho, A Jain and P Abbeel “Denoising diffusion probabilistic models” arXiv preprint arXiv:2006.11239 2020 Available: https://arxiv. p. 4org/abs/2006. p. 411239 <ref:2602. p. 004749#pg4>
[9] R Rombach, A Blattmann, D Lorenz, P Esser and B Ommer “High-resolution image synthesis with latent diffusion models” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2022 pp 10 674–10 685 <ref:2602.04749#pg4>
[10] A Sebaq and M ElHelw “Rsdiff: Remote sensing image generation from text using diffusion model” Neural Computing and Applications vol 36 no 36 pp 23 103–23 111 2024 Available: http://dx.doi.org/10.1007/s00521-024-10363-3 <ref:2602. p. 404749#pg4>
[11] O Baghirli, H Askarov, I Ibrahimli, I Bakhishov and N Nabiyev “Satdm: Synthesizing realistic satellite image with semantic layout conditioning using diffusion models” arXiv preprint arXiv:2309.16812 2023 Available: https://arxiv. p. 4org/abs/2309. p. 416812 <ref:2602. p. 004749#pg4>
[12] B Kim, M Bae and J.-G Lee “Sample-efficient multi-round generative data augmentation for long-tail instance segmentation” in Advances in Neural Information Processing Systems (NeurIPS) 2025 <ref:2602. p. 404749#pg4>
[13] J Niemeijer, J Ehrhardt, H Handels and H Uzunova “Uncertainty-aware controlnet: Bridging domain gaps with synthetic image generation” arXiv preprint arXiv:2510.11346 2025 Available: https://arxiv. p. 4org/abs/2510. p. 411346 <ref:2602. p. 004749#pg4>
[14] R Yu, S Liu, X Yang and X Wang “Distribution shift inversion for out-of-distribution prediction” arXiv preprint arXiv:2306.08328 2023 Available: https://arxiv. p. 4org/abs/2306. p. 4 p. 408328 <ref:2602. p. 004749#pg4>
[15] Y Qin, H Zheng, J Yao, M Zhou and Y Zhang “Classbalancing diffusion models” arXiv preprint arXiv:2305.00562 2023 Available: https://arxiv. p. 4org/abs/2305. p. 400562 <ref:2602. p. 004749#pg4>
[16] J Austin, D D Johnson, J Ho, D Tarlow and R van den Berg “Structured denoising diffusion models in discrete state-spaces” CoRR vol abs/2107.03006 2021 <ref:2602. p. 404749#pg4>
[17] L Zhang, A Rao and M Agrawala “Adding conditional control to text-to-image diffusion models” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) 2023 pp 10 674–10 685 <ref:2602.04749#pg4>
[18] E Perez, F Strub, H de Vries, V Dumoulin and A Courville “Film: Visual reasoning with a general conditioning layer” in Proceedings of the AAAI Conference on Artificial Intelligence 2018 <ref:2602.04749#pg4>
[19] A Radford, J W Kim, C Hallacy, A Ramesh, G Goh, S Agarwal, G Sastry, A Askell P Mishkin J Clark G Krueger and I Sutskever “Learning transferable visual models from natural language supervision” arXiv preprint arXiv:2103.00020 2021 Available: https://arxiv. p. 4org/abs/2103. p. 400020 <ref:2602. p. 004749#pg4>
[21] O Ronneberger, P Fischer and T Brox “U-net: Convolutional networks for biomedical image segmentation” in Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2015 pp 234–241 <ref:2602.04749#pg4>
[22] H Zhao, J Shi, X Qi, X Wang and J Jia “Pyramid scene parsing network” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017 pp 6230–6239 <ref:2602.04749#pg4>
[23] A Ma, J Wang, Y Zhong and Z Zheng “Factseg: Foreground activation-driven small object semantic segmentation in largescale remote sensing imagery” IEEE Transactions on Geoscience and Remote Sensing vol 60 pp 1–16 2022 <ref:2602.04749#pg4>
[24] J Wang, K Sun, T Cheng, B Jiang, C Deng, Y Zhao, D Liu, Y Mu M Tan X Wang W Liu and B Xiao “Deep highresolution representation learning for visual recognition” IEEE Transactions on Pattern Analysis and Machine Intelligence vol 43 no 10 pp 3349–3364 2021 <ref:2602.04749#pg4>
[25] K Yamazaki, T Hanyu, M Tran, A de Luis, R McCann, H Liao, C Rainwater M Adkins J Cothren and N Le “Aerialformer: Multi-resolution transformer for aerial image segmentation” 2023 Available: https://arxiv.org/abs/2306.06842 <ref:2602. p. 0
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved AI systems can generate paired label-image samples with explicit control over semantic composition and domain, enabling targeted enrichment of underrepresented classes rather than indiscriminate dataset expansion. This is achieved through a promptcontrolled diffusion augmentation framework that conditions synthesis on domain and class-ratio targets.
-
The system can synthesize layouts that satisfy class-ratio targets while preserving realistic spatial co-occurrence using a domain-aware, masked, ratio-conditioned discrete diffusion model (D3PM). This stage generates semantic layouts conditioned on both a ratio target and a domain label, addressing within-domain imbalance and cross-domain shift simultaneously.
-
The system can render photorealistic, domainconsistent images from these layouts using a ControlNetguided diffusion model. This second stage translates the generated semantic layout into high-fidelity remote sensing images while preserving spatial structure and domain appearance via FiLM gated residual injection guided by a domain/ratio prompt.
-
Downstream segmentation models can achieve consistent gains across backbones, especially for minority and mid-tail classes, even under cross-domain shift conditions. The synthetic corpus is mixed with real data using a controlled sampling protocol to train models like U-Net, PSPNet, FactSeg, HRNet, and AerialFormer on the LoveDA dataset.
-
The framework moves beyond standard remedies by adding
the right data in the right proportions,
specifically targeting minority classes and domain imbalances rather than relying solely on class re-weighting or resampling that preserve underlying pixel-frequency statistics.
Sources
- LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation
- Denoising Diffusion Probabilistic Models
- SatDM: Synthesizing Realistic Satellite Image with Semantic Layout Conditioning using Diffusion Models
- Uncertainty-Aware ControlNet: Bridging Domain Gaps with Synthetic Image Generation
- Distribution Shift Inversion for Out-of-Distribution Prediction
- Class-Balancing Diffusion Models
- Learning Transferable Visual Models From Natural Language Supervision
- AerialFormer: Multi-resolution Transformer for Aerial Image Segmentation
Related papers
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs
- From Formulas to Protocols: A Compositional View of Binary Target Segmentation Metrics
- Traffic Sign Recognition in Autonomous Driving: Dataset, Benchmark, and Field Experiment
- Efficient All-in-One Weather Restoration using Spectral Harmonization