VidUEU-Agent: A Data-Curation Agent for Multimodal Understanding, Editing, and Unified Tasks
cs.CV
Submitted: 2026-11-03
Updated: 2026-11-03
Code: https://github.com/LRuca/VidUEU-Agent
License: http://creativecommons.org/licenses/by/4.0/
The gist: The advancement of multimodal learning is heavily constrained by the scarcity and high annotation cost of high-quality instruction data.
Abstract
The advancement of multimodal learning is heavily constrained by the scarcity and high annotation cost of high-quality instruction data. While raw videos offer an abundant and dynamic source of visual knowledge, transforming them into precise instruction data remains highly challenging. To bridge this gap, we present VidUEU-Agent, an automated pipeline that synthesizes instruction data from raw videos for multimodal understanding, editing, and unified tasks. VidUEU-Agent follows a four-stage pipeline: it performs global perception to filter low-quality shots, mines keyframes and multimodal signals, constructs metadata and routes the metadata to synthesize instruction samples. More importantly, our agent framework can autonomously iterate the data distribution based on downstream training results, thereby optimizing data quality, balance, and training effectiveness. Scaling this pipeline, we construct VidUEU-100K, a large-scale dataset of 100K training samples spanning multiple domains and tasks, providing a robust foundation for developing comprehensive multimodal capabilities. Extensive experiments demonstrate that fine-tuning baseline models on VidUEU-100K yields significant and consistent performance gains across diverse representative benchmarks, validating the effectiveness of our proposed agent framework and dataset.
Sources
- Qwen3-VL Technical Report
- Qwen2.5-VL Technical Report
- InstructPix2Pix: Learning to Follow Image Editing Instructions
- BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset
- ShareGPT4V: Improving Large Multi-Modal Models with Better Captions
- Are We on the Right Way for Evaluating Large Vision-Language Models?
- Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
- WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation
- PaddleOCR 3.0 Technical Report
- DeepSeek-V3 Technical Report
- Emerging Properties in Unified Multimodal Pretraining
- ToolScope: An Agentic Framework for Vision-Guided and Long-Horizon Tool Use
- Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage
- The Llama 3 Herd of Models
- Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
- Text-VQA Aug: Pipelined Harnessing of Large Multimodal Models for Automated Synthesis
- MMCTAgent: Multi-modal Critical Thinking Agent Framework for Complex Visual Reasoning
- M$^3$IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning
- LAION-SG: An Enhanced Large-Scale Dataset for Training Complex Image-Text Models with Structural Annotations
- A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs