Steering Visual Generation in Unified Multimodal Models with Understanding Supervision
cs.CV, cs.AI
Submitted: 2026-05-07
Updated: 2026-11-16
Comments: Accepted to NeurIPS 2026. Project page: https://lzy-tony.github.io/uno
Project page: https://lzy-tony.github.io/uno
License: http://creativecommons.org/licenses/by/4.0/
The gist: Unified multimodal models are envisioned to bridge the gap between understanding and generation.
Terminology
Abstract
Unified multimodal models are envisioned to bridge the gap between understanding and generation. Yet, to achieve competitive performance, state-of-the-art models adopt largely decoupled understanding and generation components. This design, while effective for individual tasks, weakens the connection required for mutual enhancement, leaving the potential synergy empirically uncertain. We propose to explicitly restore this synergy by introducing Understanding-Oriented Post-Training (UNO), a lightweight framework that treats understanding not only as a distinct task, but also a direct supervisory signal to steer generative representations. By incorporating objectives that encode semantic abstraction (captioning) and structural details (visual regression), we enable effective gradient flow from understanding to generation. Extensive experiments on image generation and editing demonstrate that understanding can serve as an effective catalyst for generation.
Sources
- GPT-4 Technical Report
- Qwen2.5-VL Technical Report
- BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset
- Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
- Thinking with Generated Images
- EditMGT: Unleashing Potentials of Masked Generative Transformers in Image Editing
- Emerging Properties in Unified Multimodal Pretraining
- SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation
- VQ-VA World: Towards High-Quality Visual Question-Visual Answering
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment
- GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluation
- NoHumansRequired: Autonomous High-Quality Image Editing Triplet Mining
- FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space
- REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformers
- OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generation
- Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models
- Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generation
- UniWorld-V1: High-Resolution Semantic Encoders for Unified Visual Understanding and Generation
- Step1X-Edit: A Practical Framework for General Image Editing
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs