COLLAR: Cascaded Object-Level Latent Refinement for High-Fidelity Conditional Generation

arXiv:2606.00954 · cs.CV · Submitted 2026-05-31 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "COLLAR: Cascaded Object-Level Latent Refinement for High-Fidelity Conditional Generation".

Jane: Here's the gist of the paper. Achieving high-fidelity object-level control in Diffusion Transformers remains a significant challenge despite the introduction of structural priors like depth and Canny maps Current object-level conditional generation methods frequently suffer from visual artifacts and struggle to maintain precise control over objects within small localized regions To address these limitations, we propose Cascaded Object-Level Latent Refinement (COLLAR), a training-free framework that progressively optimizes object-level features via the Field-of-View (FoV) expansion Extensive experiments on the COCO-MIG and COCO-POS benchmarks demonstrate that our approach consistently outperforms state-of-the art methods across semantic alignment, image quality, and spatial fidelity Proposed Framework Components COLLAR is a training-free method that introduces two core modules: (a) Cross-Scale Semantic Alignment (CSSA) module and (b) Cyclic Feature Injection (CFI) module.

Tom: The CSSA module uses attention-based manipulation to dynamically integrate bounding-box-restricted local features into corresponding extended Field-of-View branches.

Jane: This design strictly preserves local semantic alignment while significantly enhancing the spatial contextual awareness of features.

Tom: The CFI module establishes a reciprocal background feedback mechanism It extracts global background features to extended-FoV branches and feeds object-level features into the global branch.

Paper summary: Tom: So, to wrap up on "COLLAR: Cascaded Object-Level Latent Refinement for High-Fidelity Conditional Generation," this training-free framework introduces the Cross-Scale Semantic Alignment and Cyclic Feature Injection modules to achieve high fidelity object control under global structural constraints.

Jane: Essentially, they're tackling the problem where existing methods fail at precise object control by dynamically adjusting how local features blend into the global generation process, aiming to improve attribute alignment and spatial fidelity.

Lu: The authors show that this works across benchmarks like COCO-MIG and COCO-POS, consistently beating baselines in quality metrics while also showing good inference efficiency.

Meng: It seems like a solid addition for production because it brings high quality control without requiring the heavy training overhead of traditional methods.

Tom: The main implication is that we can get much finer, instance-level control in AI image generation, which opens doors for things like creative design and synthetic dataset construction.

Jane: But we have to remember that as this capability grows, it also means we could be able to produce very convincing misleading content or sophisticated deepfakes.

Lu: And just like with any foundation model approach, the paper inherits all the societal biases and stereotypes that were already baked into the pre-training data.

Meng: So, while it’s a great technical step for quality, we need to keep an eye on those ethical considerations as these tools become more powerful.

Conclusion: Tom: So we're wrapping up on COLLAR, which stands for Cascaded Object-Level Latent Refinement for High-Fidelity Conditional Generation, and you know what that means in simple terms is that this method lets AI generate images with much tighter control over specific objects inside the picture.

Jane: Yeah, it’s about taking those messy details where models usually get blurry or miss things and refining them step by step using structural information like depth maps and Canny edges to keep everything looking sharp.

Lu: The core idea is a training-free approach that uses two main parts, Cross-Scale Semantic Alignment and Cyclic Feature Injection, to make sure the object details align perfectly with the global scene structure.

Meng: It’s training-free which is good because it doesn't require massive fine-tuning sessions just to get better control over things you already have.

Tom: The authors show they did this across COCO benchmarks, and they are consistently beating the current state of the art in terms of image quality and spatial accuracy.

Jane: That’s what we saw, but I want to focus on what this actually means for us right now. It moves us closer to creating AI that can reliably place specific items exactly where we want them in a generated scene.

Lu: The architecture uses attention-based fusion between local and extended field of view branches, which lets the object details pull in the necessary context from the wider image while staying locked onto their own boundaries.

Tom: And they even found that this approach is surprisingly fast for inference, actually matching some of the latency numbers we see on other regional models.

Jane: But they’re not done yet, because there are still challenges with how well these methods handle conflicting instructions between what you ask for textually and what the scene's geometry actually allows.

Lu: That’s a valid point, and they actually acknowledge that by mentioning their next step is using a semantic-structural-aware feature scaling strategy to make the synthesis more robust.

Meng: For deployment, if it keeps its inference speed while delivering this level of spatial fidelity, that’s exactly what we need to talk about for real applications.

Tom: Exactly. So while COLLAR tackles object control directly, we still have to figure out how to make sure the underlying foundation model isn't bringing old biases along for the ride.

Jane: That’s a huge ethical piece that stays with us, because as we get better at making AI draw things precisely, we also need to be careful about what those drawings are actually representing.

Xinlong Zhang, Jia Wei, Xiaoyu Zhang, Teng Zhou, Chengyu Lin, Yongchuan Tang

Zhejiang University

cs.CV

Submitted: 2026-05-31

Updated: 2026-11-14

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 99/100

The gist: Achieving high-fidelity object-level control in Diffusion Transformers remains a significant challenge despite the introduction of structural priors like depth and Canny maps Current object-level…

Key concepts

Cross-Scale Semantic Alignment (CSSA)
This module enhances semantics by injecting localized object features into extended Field-of-View branches using attention. It precisely maps local features to their corresponding spatial regions in the extended branch, ensuring that local semantic details are correctly integrated while boosting spatial awareness.
Cyclic Feature Injection (CFI)
This module establishes reciprocal feedback by extracting background features from the global branch and injecting them into extended-FoV branches. Object features are then fed back into the global branch via weighted summation, creating a central hub for smooth blending of object details into the overall generation process.
Extended Field-of-View (extended-FoV)
This concept refers to branches in the diffusion model that are expanded to handle object-level features. These branches allow localized, high-detail information from objects to be smoothly incorporated into the broader, global generation process while strictly preserving core semantics.
Early-Stopping Strategy
A dynamic termination mechanism used during inference. It monitors the cosine similarity between high-frequency components of local and global features. When this similarity drops below a threshold, it stops injecting object features, ensuring that precise local geometry is established before further refinement.

Terminology

Summary

Achieving high-fidelity object-level control in Diffusion Transformers remains a significant challenge despite the introduction of structural priors like depth and Canny maps Current object-level conditional generation methods frequently suffer from visual artifacts and struggle to maintain precise control over objects within small localized regions To address these limitations, we propose Cascaded Object-Level Latent Refinement (COLLAR), a training-free framework that progressively optimizes object-level features via the Field-of-View (FoV) expansion Extensive experiments on the COCO-MIG and COCO-POS benchmarks demonstrate that our approach consistently outperforms state-of-the art methods across semantic alignment, image quality, and spatial fidelity

Proposed Framework Components

COLLAR is a training-free method that introduces two core modules: (a) Cross-Scale Semantic Alignment (CSSA) module and (b) Cyclic Feature Injection (CFI) module. The CSSA module uses attention-based manipulation to dynamically integrate bounding-box-restricted local features into corresponding extended Field-of-View branches. p. 1 This design strictly preserves local semantic alignment while significantly enhancing the spatial contextual awareness of features. p. 1 The CFI module establishes a reciprocal background feedback mechanism It extracts global background features to extended-FoV branches and feeds object-level features into the global branch. p. 1 Serving as a central hub for feature optimization, extended-FoV branches allow object-level features to smoothly blend into the global generation process while strictly retaining core semantics. p. 1

Mechanism of Feature Alignment

The Cross-Scale Semantic Alignment (CSSA) module enhances semantics by injecting object features from the local branch into the extended-FoV branch using an attention-based feature fusion mechanism. Specifically, it extracts Qi,L, Ki,L, and Vi,L from the local branch and feeds them into the DiT layer of the corresponding extended-FoV branch through concatenation with extendedbranch features. p. 4 The position embeddings for features from the local branch are directly assigned the exact spatial subset of indices corresponding to their shared overlapping region, which empowers the extended branch to precisely localize and attend to the local priors. p. 4

Reciprocal Feedback and Dynamic Regulation

The Cyclic Feature Injection (CFI) module facilitates reciprocal feature feedback by first extracting background features from the global branch that spatially align with the extended region of the extended-FoV branch. These extracted features are then injected into the extended-FoV branch to smooth object semantics. p. 4 Object-level features are then precisely injected into the corresponding regions of the global branch via a weighted summation with the feature set from extended-FoV branches. p. 5 An object-level early-stopping strategy is employed, where the cosine similarity between high-frequency components of local and global features dynamically determines termination based on a threshold σ. p. 5

Experimental Validation and Efficiency

Extensive experiments on the COCO-MIG and COCO-POS benchmarks demonstrate that our approach consistently outperforms existing baselines across multiple metrics. Quantitative comparisons show that under the Canny modality, our method exhibits leading performance, yielding the highest spatial fidelity (e. p. 6g., an OSR of 50. p. 693 and a 72.73 for Ours). p. 7 The inference efficiency analysis shows that COLLAR achieves a latency on par with Regional-FLUX while consistently surpassing it across all remaining metrics. p. 8 Ablation studies confirm that the full framework, which replaces the naive fusion with the Cyclic Feature Injection (CFI) module, successfully reconciles conflicts and achieves the highest performance across all primary spatial metrics while maintaining a highly competitive FID of 16. p. 83.

Conclusion and Future Directions

We present COLLAR, a training-free, plug-and-play framework for precise instance-level control in spatially-conditioned image generation. The framework introduces the Cross-Scale Semantic Alignment (CSSA) module and the Cyclic Feature Injection (CFI) module. p. 0 However, COLLAR has two main limitations: first, the extended-FoV branch incurs additional computational overhead compared to simple noise-blending methods Second, resolving local conflicts between textual semantics and spatial geometry remains challenging. p. 9 The authors aim to address this through a semantic-structural-aware feature-scaling strategy for more robust synthesis. p. 9

Societal Implications

Our proposed framework enables precise, spatially-conditioned image generation with multiinstance control, benefiting applications like creative design and synthetic dataset construction However, such capabilities could be exploited to produce highly convincing, misleading content or sophisticated deepfakes Additionally, as our approach builds upon existing generative foundation models, it inevitably inherits the societal biases and stereotypes embedded in their pre-training data.

Inference Process Summary

Algorithm 1 illustrates the complete inference process of our method. The process involves iteratively processing local latents, injecting global background features into the extended-FoV latent, and then fusing these with object-level features into the global branch. p. 5 A crucial step is the structural monitoring and early-stopping strategy based on the cosine similarity between high-frequency components to dynamically terminate injection when local geometry is established. p. 5 The final global update synthesizes the image using all harmonized object representations. p. 4

Ablation Study Insights

The ablation study on the FoV Expansion Factor γ indicates that γ = 0.5 provides an optimal balance between global coherence and local semantic alignment. p. 5 p. 5 Increasing the maximum guidance steps S from 5 to 10 significantly boosts the mAP (71. p. 50 → 72.3) and ISR (42. p. 6 p. 61 → 44.1). The optimal configuration for achieving high-fidelity generation alongside precise instance-level control is identified as K = 57 and λ = 0. p. 0 p. 05. The most significant performance leap is observed upon introducing the normalized fusion weight λ = 0.5, which elevates ISR to 45. p. 6 p. 661 and mAP to a peak of 73. p. 7 p. 793.

Integration Versatility

Table 7 demonstrates the broad compatibility and effectiveness of our proposed method when integrated with the state-of-the-art conditional generation frameworks OminiControl and EasyControl. When applied to OminiControl, our method yields a substantial increase in ISR (from 18. p. 8 p. 813 to 26.02) and mAP (from 44. p. 3 p. 325 to 56.63). For EasyControl, integrating our strategy increases the OSR from 46. p. 8 p. 828 to 63.66 and nearly doubles the ISR to 34. p. 6 p. 677. Furthermore, Table 8 shows that our method significantly outperforms both the vanilla SD3+ControlNet baseline and DreamRenderer in precise spatial control on the SD3 architecture. p. 8 p. 8

Qualitative Results

Figure 3 illustrates the qualitative comparison between our COLLAR and several state-of-the-art baselines under various spatial conditions. Visualization results demonstrate that COLLAR effectively preserves local structural integrity while maintaining precise attribute alignment The higher visual quality of small objects further validates our model’s robust control capability. p. 7

User Study Feedback

Figure 4(a) shows that users preferred our method for spatial fidelity and image quality in the user study. The ablation study on model designs confirms that our method can achieve spatial fidelity and image quality. p. 8 The optimal configuration identified is K = 57 and λ = 0.5.

Inference Efficiency

Table 4 summarizes the inference latency and peak VRAM, showing that COLLAR achieves a latency on par with Regional-FLUX while consistently surpassing it across all remaining metrics. NoiseCollage incurs the highest overhead among the baselines. p. 8 p. 8 p. 8 p. 8 The computational efficiency comparison shows that FLUX and 3DIS achieve the fastest inference, as the former is a base model without additional control and the latter relies solely on attention-mask manipulation. p. 8 p. 8 NoiseCollage incurs the highest overhead among the baselines. The comparison hinges on inference latency. p. 8 p. 8 p. 8

Societal Impacts

Our proposed framework enables precise, spatially-conditioned image generation with multiinstance control, benefiting applications like creative design and synthetic dataset construction However, such capabilities could be exploited to produce highly convincing, misleading content or sophisticated deepfakes. Additionally, as our approach builds upon existing generative foundation models, it inevitably inherits the societal biases and stereotypes embedded in their pre-training data.

References

[1] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, and 1 others. 2022. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers. p. 9 arXiv preprint arXiv:2211. p. 901324

[3] Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, Jie Lei, Tengyu Ma, Baishan Guo, Arpit Kalla

[4] Anthony Chen, Jianjin Xu, Wenzhao Zheng, Gaole Dai

[5] Hongyu Chen, Yiqi Gao, Min Zhou

[6] Junsong Chen, Jincheng Yu, Chongjian Ge

[7] Xi Chen, Lianghua Huang, Yu Liu

[8] Yiyun Chen and Weikai Yang. 2025. Refadgen: High-fidelity advertising image generation. p. 9 arXiv preprint arXiv:2508. p. 911695

[9] Zhennan Chen, Yajie Li, Haofan Wang

[10] Bo Cheng, Yuhang Ma, Liebucha Wu

[11] Patrick Esser, Sumith Kulal, Andreas Blattmann

[12] Yuwei Guo, Ceyuan Yang, Anyi Rao

[13] Runze He, Bo Cheng, Yuhang Ma

[14] Jonathan Ho, Ajay Jain, Pieter Abbeel

[15] Edward J Hu, Yelong Shen, Phillip Wallis

[16] Minghui Hu, Jianbin Zheng, Daqing Liu

[17] Yuval Kirstain, Adam Polyak, Uriel Singer

[18] Phillip Y. Lee, Taehoon Yoon

[19] Seung Hyun Lee, Jijun Jiang, Yiran Xu

[20] Danfeng Li, Hui Zhang, Sheng Wang

[21] Ming Li, Taojiannan Yang, Huafeng Kuang

[22] Yuheng Li, Haotian Liu, Qingyang Wu

[23] Tsung-Yi Lin, Michael Maire, Serge Belongie

[24] Xiang Liu, Junjun Jiang, Wei Han

[25] Yuhang Ma, Xiaoshi Wu, Keqiang Sun

[26] Chong Mou, Xintao Wang, Liangbin Xie

[27] William Peebles and Saining Xie

[28] Dustin Podell, Zion English, Kyle Lacey

[29] Can Qin, Shu Zhang, Ning Yu

[30] Colin Raffel, Noam Shazeer, Adam Roberts

[31] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh

[32] Scott Reed, Zeynep Akata, Xinchen Yan

[33] Robin Rombach, Andreas Blattmann, Dominik Lorenz

[34] Chitwan Saharia, William Chan, Saurabh Saxena

[35] Takahiro Shirakawa and Seiichi Uchida

[36] Shuyuan Tu, Qi Dai, Zhi-Qi Cheng

[37] Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla

[38] Qiang Xiang, Shuang Sun, Binglei Li

[39] Jiayu Xiao, Henglei Lv, Liang Li

[40] Shitao Xiao, Yueze Wang, Junjie Zhou

[41] Denis Zavadski, Johann-Friedrich Feiden

[42] Hui Zhang, Dexiang Hong, Yitong Wang

[43] Lvmin Zhang, Anyi Rao

[44] Xinlong Zhang, Zejian Li, Wei Li

[45] Yuxuan Zhang, Yirui Yuan, Yiren Song

[46] Shihao Zhao, Dongdong Chen, Yen-Chun Chen

[47] Guangcong Zheng, Xianpan Zhou, Xuewei Li

[48] Dewei Zhou, Mingwei Li, Zongxin Yang

[49] Dewei Zhou, You Li, Fan Ma

[50] Dewei Zhou, Ji Xie, Zongxin Yang

Ablation Study Details

Table 5 shows that as γ increases from 0.2 to 1.0, we observe a consistent reduction in FID (16. p. 62 → 15.3). The results indicate that γ = 0. p. 75 provides an optimal balance between global coherence and local semantic alignment. Effect of Guidance Step Control (σ and S) reveals that increasing the maximum guidance steps S from 5 to 10 significantly boosts the mAP (71. p. 50 → 72.3) and ISR (42.1 → 44.1). The optimal configuration for achieving high-fidelity generation alongside precise instance-level control is identified as K = 57 and λ = 0.5. The most significant performance leap is observed upon introducing the normalized fusion weight λ = 0.5, which elevates ISR to 45.61 and mAP to a peak of 73.93.

Integration with Other Methods

Table 7 demonstrates the broad compatibility and effectiveness of our proposed method when integrated with the state-of-the-art conditional generation frameworks OminiControl and EasyControl. When applied to OminiControl, our method yields a substantial increase in ISR (from 18.13 to 26.02) and mAP (from 44.25 to 56.63). For EasyControl, integrating our strategy increases the OSR from 46.28 to 63.66 and nearly doubles the ISR to 34.77. Table 8 shows that our method significantly outperforms both the vanilla SD3+ControlNet baseline and DreamRenderer in precise spatial control on the SD3 architecture.

Table 4 summarizes the inference latency and peak VRAM, showing that COLLAR achieves a latency on par with Regional-FLUX while consistently surpassing it across all remaining metrics. NoiseCollage incurs the highest overhead among the baselines. The comparison hinges on inference latency. The computational efficiency comparison shows that FLUX and 3DIS achieve the fastest inference, as the former is a base model without additional control and the latter relies solely on attention-mask manipulation. NoiseCollage incurs the highest overhead among the baselines. The comparison hinges on inference latency.

Our proposed framework enables precise, spatially-conditioned image generation with multiinstance control, benefiting applications like creative design and synthetic dataset construction However, such capabilities could be exploited to produce highly convincing, misleading content or sophisticated deepfakes Additionally, as our approach builds upon existing generative foundation models, it inevitably inherits the societal biases and stereotypes embedded in their pre-training data.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Cross-Scale Semantic Alignment (CSSA) module can be used to dynamically integrate bounding-box-restricted local features into corresponding extended Field-of-View branches to preserve local semantic alignment while significantly enhancing the spatial contextual awareness of features.

  2. Cyclic Feature Injection (CFI) module enables a reciprocal background feedback mechanism which allows for the seamless integration of instances without compromising fidelity by extracting background features from the global branch and feeding object-level features into it.

  3. The object-level earlystopping strategy, based on the cosine similarity between high-pass filtered local and global latent features, ensures that high-frequency signals accurately represent the emergence of local edges and contours, which reduces inference costs while ensuring final images remain natural.

  4. The framework can achieve superior performance on new or existing conditional generation models by demonstrating compatibility with OminiControl and EasyControl, leading to substantial increases in metrics like ISR and mAP when integrated.

  5. The method achieves state-of-the-art object-level controllability on SD3 architectures, successfully bridging the gap where native models catastrophically fails in multi-instance spatial adherence.

Sources

Related papers