Learning to Interact and Direct: Co-Evolving Video Director Agents
cs.AI
Submitted: 2026-10-30
Updated: 2026-10-30
Code: https://github.com/HITsz-TMG/VideoClaw
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Video director agents orchestrate multistage production to realize users' creative visions.
Terminology
Abstract
Video director agents orchestrate multistage production to realize users' creative visions. Emerging research on their self-evolution advances screenplay planning effectively. However, the mutual reinforcement between directing and user interaction remains largely overlooked. Improved interaction clarifies user intent and informs workflow refinement, while more reliable execution shifts collaboration from troubleshooting to creative decisions. To this end, we introduce LIT (Learning to Interact and DirecT), a framework comprises a director agent, a simulated user, and a meta engineer. Simulated user assigns creative tasks to probe the director's weaknesses, while the meta engineer uses the task traces to revise the director's interaction and production strategy. Although this approach can address discovered interaction and production issues, it still relies on passive attribution by the meta agent. This limits its ability to adapt to broader user interaction patterns and production techniques. To solve this problem, we equip LIT with a co-evolution paradigm. The meta agent proactively learns external knowledge, thereby improving its ability to discover problems and propose solutions. The meta agent, simulated user, and director together form an evolutionary closed loop, collaboratively enhances the system's directing and interaction capabilities. Across the Test and OOD Test splits, LIT improves production and Interaction scores by 19.1% and 32.0%, respectively, relative to the initial pipeline. Controlled ablations reveal the correlation between improvements in production and interaction, where the co-evolution mechanism further encourage directing skill and interaction mode exploration.
Sources
- Exploration by Random Network Distillation
- ViMax: Agentic Video Generation
- VBench: Comprehensive Benchmark Suite for Video Generative Models
- ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Models
- DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
- FIFO-Diffusion: Generating Infinite Videos from Text without Training
- VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planning
- VideoStudio: Generating Consistent-Content and Multi-Scene Videos
- Curiosity-driven Exploration by Self-supervised Prediction
- Bridging Creative Intent and Visual Quality: Creator-Driven Recurrent Video Generation with Agentic Feedback Loops
- Toolformer: Language Models Can Teach Themselves to Use Tools
- AniMaker: Multi-Agent Animated Storytelling with MCTS-Driven Clip Generation
- Reflexion: Language Agents with Verbal Reinforcement Learning
- MovieChat: From Dense Token to Sparse Memory for Long Video Understanding
- Co-Director: Agentic Generative Video Storytelling
- Voyager: An Open-Ended Embodied Agent with Large Language Models
- Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
- FilmBench: A Film-Grade Benchmark for Cinematic Video Generation
- AnimAgents: Coordinating Multi-Stage Animation Pre-Production with Human-Multi-Agent Collaboration
- VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generation
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents