Less is More: Data-Efficient Adaptation for Controllable Text-to-Video Generation
cs.CV, cs.AI
Submitted: 2025-11-21
Updated: 2026-04-08
Journal ref: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14811-14821 (2026)
Code: https://github.com/csh-apprentice/Less_Is_More
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Fine-tuning large-scale text-to-video diffusion models to add new generative controls, such as those over physical camera parameters (e.g., shutter speed or aperture)…
Terminology
Abstract
Fine-tuning large-scale text-to-video diffusion models to add new generative controls, such as those over physical camera parameters (e.g., shutter speed or aperture), typically requires vast, high-fidelity datasets that are difficult to acquire. In this work, we propose a data-efficient fine-tuning strategy that learns these controls from sparse, low-quality synthetic data. We show that not only does fine-tuning on such simple data enable the desired controls, it actually yields superior results to models fine-tuned on photorealistic "real" data. Beyond demonstrating these results, we provide a framework that justifies this phenomenon both intuitively and quantitatively.
Sources
- ReCamMaster: Camera-Controlled Generative Rendering from A Single Video
- Wan-Animate: Unified Character Animation and Replacement with Holistic Replication
- DiffEdit: Diffusion-based semantic image editing with mask guidance
- TokenFlow: Consistent Diffusion Features for Consistent Video Editing
- AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning
- CameraCtrl: Enabling Camera Control for Text-to-Video Generation
- VACE: All-in-One Video Creation and Editing
- HunyuanVideo: A Systematic Framework For Large Video Generative Models
- Decoupled Weight Decay Regularization
- Movie Gen: A Cast of Media Foundation Models
- Towards Accurate Generative Models of Video: A New Metric & Challenges
- Wan: Open and Advanced Large-Scale Video Generative Models
- Depth Any Video with Scalable Synthetic Data
- IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models
- ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis
- Learning Video Representations without Natural Videos
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs