DUET: A Diversity-Quality Duet of Distillation Experts for Two-Step Video Generation

Diffusion models have enabled high-quality video generation in recent years, but the high cost of iterative sampling hinders their practical deployment. Few-step distillation alleviates this cost, yet exposes a quality–diversity trade-off between its two dominant paradigms: trajectory-level distillation (e.g., sCM) favors diversity, whereas distribution-level distillation (e.g., DMD) favors quality. Targeting extreme two-step video generation, we introduce DUET, which reconciles the two paradigms through a noise-level duet of experts: an sCM expert takes the high-noise step to lay out diverse structure, and a DMD expert takes the low-noise step to refine appearance detail. Since the two experts are trained independently with their native objectives, DUET sidesteps the optimization difficulties of loss-level combinations and delivers quality and diversity jointly rather than trading one for the other. We further identify the relay interface and the high-noise stage as the remaining bottlenecks, and address them with RL-guided expert adaptation, yielding DUET+. With the Wan2.1-T2V1.3B backbone, DUET lifts the two-step quality of sCM close to the level of DMD while retaining nearly all of its structural diversity—about twice that of DMD—and DUET+ further improves overall quality while preserving this diversity advantage. Together, these results establish noise-level expert specialization as a simple, effective paradigm for reconciling diversity and quality in two-step video generation.

Improvements to AI systems:

1. Noise-Level Expert Delegation: Implement a two-stage generation pipeline where different models (or model heads) specialize in distinct noise ranges—one trained for high-noise (structural/diversity) and another for low-noise (detail/quality). This avoids loss-level combination conflicts and allows independent optimization per stage.

2. RL-Guided Expert Adaptation at Relay Interface: Add a reinforcement learning (RL) fine-tuning step specifically at the transition point between the two experts, optimizing the latent representation handed off from the high-noise to the low-noise stage. This reduces information loss and improves coherence without sacrificing diversity.

3. Decoupled Training Objectives: Train each expert with its native objective (e.g., trajectory-level for diversity, distribution-level for quality) rather than mixing losses. This prevents gradient interference and yields better convergence, making the system more stable and scalable.

4. Adaptive Noise Scheduling for Two-Step Sampling: Use a learned or RL-optimized noise split (e.g., 50% high-noise, 50% low-noise) that dynamically adjusts based on content complexity, improving output fidelity for both simple and complex scenes.

What the improved AI system can do:

• Generate high-quality videos in just two sampling steps, matching the visual fidelity of slower multi-step diffusion models.

• Maintain structural diversity (e.g., varied scene compositions, object placements, camera angles) that is typically lost in fast distillation methods.

• Produce sharp, detailed textures and fine-grained motion in low-noise stages, while preserving global coherence from the high-noise stage.

• Adaptively handle diverse prompts—from simple static scenes to complex dynamic actions—without quality degradation.

• Be trained more efficiently and stably, reducing the need for complex loss balancing or adversarial tuning.

• Serve as a drop-in replacement for existing two-step video generators, enabling real-time or near-real-time applications like interactive content creation, video editing, and simulation.

Diffusion models have enabled high-quality video generation in recent years, but the high cost of iterative sampling hinders their practical deployment. Few-step distillation alleviates this cost, yet exposes a quality--diversity trade-off between its two dominant paradigms: trajectory-level distillation (e.g., sCM) favors diversity, whereas distribution-level distillation (e.g., DMD) favors quality. Targeting extreme two-step video generation, we introduce DUET, which reconciles the two paradigms through a noise-level duet of experts: an sCM expert takes the high-noise step to lay out diverse structure, and a DMD expert takes the low-noise step to refine appearance detail. Since the two experts are trained independently with their native objectives, DUET sidesteps the optimization difficulties of loss-level combinations and delivers quality and diversity jointly rather than trading one for the other. We further identify the relay interface and the high-noise stage as the remaining bottlenecks, and address them with RL-guided expert adaptation, yielding DUET+. With the Wan2.1-T2V-1.3B backbone, DUET lifts the two-step quality of sCM close to the level of DMD while retaining nearly all of its structural diversity---about twice that of DMD---and DUET+ further improves overall quality while preserving this diversity advantage. Together, these results establish noise-level expert specialization as a simple, effective paradigm for reconciling diversity and quality in two-step video generation.