A Lightweight Velocity Correction Framework for Improving Instruction Following in Few-Step Text-to-Image Generation
cs.CV
Submitted: 2026-10-09
Updated: 2026-10-09
Code: https://github.com/modelscope/DiffSynth-Studio
License: http://creativecommons.org/licenses/by/4.0/
The gist: Few-step text-to-image generators produce high-quality images with few network forward passes, yet struggle with complex prompts involving multiple subjects, interactions, spatial relations,
Abstract
Few-step text-to-image generators produce high-quality images with few network forward passes, yet struggle with complex prompts involving multiple subjects, interactions, spatial relations, counting, and attribute binding. Can these errors be reduced without changing the generator's parameters or increasing its forward-pass count? We propose a lightweight correction framework for a frozen 6B-parameter generator. The generator predicts velocities that determine how its intermediate image representation is updated; a corrector with fewer than 20M parameters learns residual adjustments at selected sampling steps. Because earlier corrections change the states encountered at later steps, our On-Trajectory Flow-Bridge Supervision derives residual targets at the states actually reached during corrected generation. A single corrector is shared across selected steps and runs in parallel with the generator. Only the corrector is trained, requiring neither reward optimization nor re-distillation. To our knowledge, this is the first framework for lightweight velocity-residual adaptation of fully frozen few-step text-to-image generators that preserves their original sampling schedule and forward-pass count. On our target complex-scene distribution, the framework achieves a +17.60-percentage-point NetWin over frozen Z-Image-Turbo in pairwise instruction-following evaluation, with only 26.9 ms (0.84%) additional end-to-end latency per image and no increase in backbone evaluations.
Sources
- Safe Few-Step Generation via Velocity Editing
- Diffusion-RPO: Aligning Diffusion Models through Relative Preference Optimization
- Distribution Matching Distillation Meets Reinforcement Learning
- D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Models
- Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluation
- SDXL-Lightning: Progressive Adversarial Diffusion Distillation
- Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference
- Steering Rectified Flow Models in the Vector Field for Controlled Image Generation
- Aligning Text-to-Image Diffusion Models with Reward Backpropagation
- Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis
- Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer
- Trajectory Consistency Distillation: Improved Latent Consistency Distillation by Semi-Linear Consistency Function with Trajectory Mapping
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models