VicEdit: Learning to Edit Videos from Visual In-Context Examples
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "VicEdit: Learning to Edit Videos from Visual In-Context Examples".
Jane: Here's the gist of the paper. Despite progress in instruction-based video editing, unimodal textual instructions inherently struggle to convey fine-grained textures and complex dynamics <ref:2608.16745#pg2> To bridge this perceptual gap, we propose Visual In-context Editing, a new paradigm elevating video editing from textual instructions to multi-modal visual guidance encompassing single image, image pair, and video pair <ref:2608.16745#pg4>.
Tom: Pruning redundant reasoning tokens The core limitation of current text-only methods is that the semantic nature of language often fails to capture the perceptual nuances required for high-fidelity editing <ref:2608.16745#pg2> This deficiency manifests in three critical dimensions: visual textures, where text-only baselines like DiTTO and VideoCoF often suffer from stylistic drift <ref:2608.16745#pg3>, spatial transformations, where linguistic instructions specify what but not where to edit <ref:2608.16745#pg3>, and temporal dynamics, where complex motion patterns in creative editing exceed linguistic capacity <ref:2608.16745#pg4>.
Jane: VicEdit addresses these gaps by formulating a unified visual in-context video editing paradigm that leverages dense semantic cues across diverse visual modalities to provide richer guidance for precise video manipulation <ref:2608.16745#pg4>.
Tom: How the training signal is built The framework VicEdit introduces two key components to bridge visual and textual contexts: Modality-Adaptive Semantic Distillation (MASD) and Dual-Context Injection (DCI) <ref:2608.16745#pg4>.
Title and authors: Tom: Now we're getting into the actual summary of this paper, and it boils down to them proposing VicEdit to replace text-only editing with a multi-modal visual guidance system <ref:two thousand six hundred eight point one six seven four five#pg4.
Jane: To put that in simple terms, instead of just telling the AI "make the car red," you can show it a picture of a red car and say, "edit this video to look like that one," right? It’s about using visual guidance as much as language <ref:two thousand six hundred eight point one six seven four five#pg2.
Lu: That's the core idea they are pushing. They argue that pure text instructions struggle with capturing fine details, especially texture, and complex motion patterns in video are just too much for words alone <ref:two thousand six hundred eight point one six seven four five#pg3.
Tom: They achieve this by introducing VicEdit, which uses those two main techniques we talked about—MASD and DCI—to bridge the gap between the visual information and the text instructions <ref:two thousand six hundred eight point one six seven four five#pg4.
Jane: Let's unpack those tools again. MASD is like a special lens that adapts based on whether you show it an image or a video, splitting its focus between texture and structure <ref:two thousand six hundred eight point one six seven four five#pg6.
Lu: That's right, MASD handles the fine details like texture fidelity, while DCI is what calibrates those text instructions so they make sense within that visual context before the AI starts editing <ref:two thousand six hundred eight point one six seven four five#pg3.
Tom: And they trained this entire system using a massive dataset called VicEdit-400K, which contains four hundred thousand samples across ten different editing tasks and three types of visual references <ref:two thousand six hundred eight point one six seven four five#pg4.
Jane: four hundred thousand is a lot. The way they built that dataset sounds pretty careful with the filtering and verification stages to make sure everything was high quality and matched the instructions correctly <ref:two thousand six hundred eight point one six seven four five#pg5.
Lu: They also used a phased training strategy, starting by updating just the main video model before training those new context modules like MASD and DCI separately, then finally joint fine-tuning everything together to get it all aligned <ref:two thousand six hundred eight point one six seven four five#pg10.
Tom: And the results on VicEditBench showed a score of three point four five, which is better than the previous best method, Kiwi-Edit's two point nine seven by a difference of zero point forty-eight <ref:two thousand six hundred eight point one six seven four five#pg10.
Jane: That gap is what they are emphasizing—how much more capable this new approach is at actually executing what the user intends, whether it’s changing a tiny texture or rearranging an entire scene <ref:two thousand six hundred eight point one six seven four five#pg10.
Lu: The implication here is that we can move past simple text prompts and start using rich visual references to control complex video editing tasks with high fidelity and temporal consistency <ref:two thousand six hundred eight point one six seven four five#pg10.
Tom: It’s a real shift toward making AI understand the actual look and feel of things, not just the words we use to describe them <ref:two thousand six hundred eight point one six seven four five#pg2.
Jane: So for us on the ground, this means instead of writing super long, ambiguous instructions, we might be able to guide AI with a couple of reference pictures or short video clips and get much more precise results <ref:two thousand six hundred eight point one six seven four five#pg3.
Lu: It suggests that future models will need to learn how to effectively map complex visual data into the specific editing parameters we want, not just follow abstract language <ref:two thousand six hundred eight point one six seven four five#pg3.
Tom: We're also happy because the setup is computationally efficient; they only add about one second of processing time when they do that extra reference feature extraction across all three modalities <ref:two thousand six hundred eight point one six seven four five#pg10.
Jane: That efficiency is important for getting these tools used in real applications, like on a phone or in a quick workflow, because we don't want to wait forever for an edit to finish <ref:two thousand six hundred eight point one six seven four five#pg3.
The paper's summary: Tom: Now we look at how they are trying to make VicEdit even better than just having MASD and DCI working together <ref:two thousand six hundred eight point one six seven four five#pg10. They aren't just stopping there; they’re looking at ways to make that interaction tighter for better results <ref:two thousand six hundred eight point one six seven four five#pg10.
Jane: What does tightening up mean practically? For instance, if you’re trying to edit a video where the texture is critical, how does VicEdit specifically get better at grabbing that detail than just using the base model on its own <ref:two thousand six hundred eight point one six seven four five#pg2 ?
Lu: They are suggesting refining how those learnable queries in MASD interact with the visual context tokens to make sure they steer attention even more accurately toward those fine details without losing the general structure of the edit <ref:two thousand six hundred eight point one six seven four five#pg6.
Tom: And DCI is also getting an update, focusing on how that semantic shift mechanism calibrates the instruction queries in a way that feels more directly connected to what's actually present in the image or video context <ref:two thousand six hundred eight point one six seven four five#pg3.
Jane: So they're not just adding more pieces; they are tweaking the rules of how those parts work together, trying to make sure MASD and DCI don't start fighting each other when working on the same edit <ref:two thousand six hundred eight point one six seven four five#pg10.
Lu: Exactly. The goal is to achieve this synergy so that the model can handle both those big global aesthetic changes and small local object manipulations with high fidelity, which translates directly into better control over every aspect of the final output <ref:two thousand six hundred eight point one six seven four five#pg10.
Tom: It’s about making sure that when the AI looks at a visual example, it doesn't just see colors or shapes anymore but actually understands what those specific visual cues mean for the final edit <ref:two thousand six hundred eight point one six seven four five#pg2.
Jane: That moves the whole system closer to being something a human editor would intuitively understand, because we aren't just feeding it raw numbers anymore <ref:two thousand six hundred eight point one six seven four five#pg3.
Lu: It’s about creating a more robust system where the visual input doesn't just act as an afterthought but becomes a fundamental part of the guidance mechanism itself <ref:two thousand six hundred eight point one six seven four five#pg4.
Tom: So they are really focusing on making sure that when we scale this up to real-world use, it doesn't just work well on the benchmark but stays accurate when you throw in a messy, real video <ref:two thousand six hundred eight point one six seven four five#pg10.
Jane: That’s the kind of robustness we need—a system that can handle the messy reality of creative work without losing quality or getting confused by bad inputs <ref:two thousand six hundred eight point one six seven four five#pg3.
Lu: It pushes the idea that for AI to handle complex creative tasks, we need representation learning that truly understands how visual context translates into actionable editing parameters, not just surface-level recognition <ref:two thousand six hundred eight point one six seven four five#pg3.
The paper's improvements: Tom: So to wrap up the whole paper "VicEdit: Learning to Edit Videos from Visual In-Context Examples," they are showing that using multi-modal visual guidance can significantly improve video editing accuracy compared to just using text instructions <ref:two thousand six hundred eight point one six seven four five#pg10.
Jane: It’s a big deal because it proves the AI isn't just guessing what you mean when you write words; it's actually using the visual examples to make sense of your instruction, which is much more intuitive <ref:two thousand six hundred eight point one six seven four five#pg2.
Lu: The implication here is that for complex creative tasks, we need representation learning that understands how visual context translates into specific editing actions rather than just abstract language <ref:two thousand six hundred eight point one six seven four five#pg3.
Meng: From a production standpoint, the fact that the computational overhead is only about one second and it still hits three point four five on VicEditBench gives me some confidence that this kind of approach is actually deployable <ref:two thousand six hundred eight point one six seven four five#pg10.
Lalam: I'm confident in this answer; the model itself confirms that when you give it visual context, its ability to follow complex instructions improves substantially across all those different tasks <ref:two thousand six hundred eight point one six seven four five#pg10.
Tom: So we’re leaving with VicEdit as a framework that bridges the gap between what we want and what the AI actually does, using multi-modal visual examples instead of just text <ref:two thousand six hundred eight point one six seven four five#pg1.
Jane: It’s about moving toward guidance that respects texture and motion in a way pure text just can't match, making video editing much more powerful <ref:two thousand six hundred eight point one six seven four five#pg2.
Lu: The next step is seeing if these learned mechanisms generalize well when we move beyond the specific tasks in VicEditBench to even messier, real-world scenarios <ref:two thousand six hundred eight point one six seven four five#pg4.
Meng: I just hope the next paper addresses how stable this works when you feed it input that isn't perfectly clean, because running on real traffic is where the real test is for me <ref:two thousand six hundred eight point one six seven four five#pg10.
Lalam: I predict the authors will explore making those modules even more adaptable so the system can handle a wider variety of visual references without needing to be retrained from scratch <ref:two thousand six hundred eight point one six seven four five#pg4.
Conclusion: Tom: So we're wrapping up on "VicEdit: Learning to Edit Videos from Visual In-Context Examples" and they’ve shown that using multi-modal visual guidance can significantly improve video editing accuracy compared to just text instructions <ref:two thousand six hundred eight point one six seven four five#pg10.
Jane: It’s a big deal because it shows the AI isn't just guessing what you mean when you write words; it's actually using those visual examples to make sense of your instruction, which is much more intuitive <ref:two thousand six hundred eight point one six seven four five#pg2.
Lu: The implication is that for complex creative tasks, we need representation learning that understands how visual context translates into specific editing actions rather than just abstract language <ref:two thousand six hundred eight point one six seven four five#pg3.
Meng: From a production standpoint, the fact that the computational overhead is only about one second and it still hits three point four five on VicEditBench gives me some confidence that this kind of approach is actually deployable <ref:two thousand six hundred eight point one six seven four five#pg10.
Lalam: I'm confident in this answer; the model itself confirms that when you give it visual context, its ability to follow complex instructions improves substantially across all those different tasks <ref:two thousand six hundred eight point one six seven four five#pg10.
Tom: So, we’re leaving with VicEdit: Learning to Edit Videos from Visual In-Context Examples as a framework that bridges the gap between what we want and what the AI actually does <ref:2608.16745#pg1>.
Jane: It’s about moving toward guidance that respects texture and motion in a way pure text just can't match, making video editing much more powerful <ref:two thousand six hundred eight point one six seven four five#pg2.
Lu: The next step is seeing if these learned mechanisms generalize well when we move beyond the specific tasks in VicEditBench to even messier, real-world scenarios <ref:two thousand six hundred eight point one six seven four five#pg4.
Meng: I just hope the next paper addresses how stable this works when you feed it input that isn't perfectly clean, because running on real traffic is where the real test is for me <ref:two thousand six hundred eight point one six seven four five#pg10.
Lalam: I predict the authors will explore making those modules even more adaptable so the system can handle a wider variety of visual references without needing to be retrained from scratch <ref:two thousand six hundred eight point one six seven four five#pg4.
Yuji Wang, Teng Hu, Yuheng Chen, Ran Yi, Han Feng, Weijian Cao, Chengjie Wang, Lizhuang Ma
Shanghai Jiao Tong University · Tencent Youtu Lab
cs.CV
Submitted: 2026-08-17
Updated: 2026-11-04
Comments: NeurIPS 2026
Project page: https://rain152.github.io/VicEdit
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 100/100
The gist: Despite progress in instruction-based video editing, unimodal textual instructions inherently struggle to convey fine-grained textures and complex dynamics <ref:2608.16745#pg2> To bridge this…
Key concepts
- Visual In-context Editing
- This is a new approach to video editing that moves beyond just text instructions. Instead of relying solely on words, it uses visual examples—like single images or pairs of images/videos—to guide the editing process. This allows the model to understand complex visual details and perform precise manipulations in a more intuitive way.
- Modality-Adaptive Semantic Distillation (MASD)
- MASD is a technique that uses learnable queries conditioned on different visual modalities. It helps the model decompose the editing task into two parts: a general editing space and specific areas relevant to the input image or video. This steers the model's attention precisely toward the necessary visual cues for high-fidelity editing.
- Dual Context Injection (DCI)
- DCI is a mechanism that calibrates text instructions using visual context before extracting tokens. It ensures that the words used in an instruction are understood within the specific visual environment provided. This resolves ambiguities in language, making sure the model follows the intent of the instruction accurately based on what it sees.
Terminology
Summary
Despite progress in instruction-based video editing, unimodal textual instructions inherently struggle to convey fine-grained textures and complex dynamics <ref:2608. p. 0 VicEdit: Learning to Edit Videos from Visual
In-Context Examples
Yuji Wang1∗ Teng Hu1∗ Yuheng Chen1 Ran Yi1 Han Feng2 Weijian Cao2
Chengjie Wang2 Lizhuang Ma1† Jiangning Zhang3‡
1Shanghai Jiao Tong University 2Tencent Youtu Lab 3Zhejiang University
Abstract
Despite progress in instruction-based video editing, unimodal textual instructions
inherently struggle to convey fine-grained textures and complex dynamics. To
bridge this perceptual gap, we propose Visual In-context Editing, a new paradigm
elevating video editing from textual instructions to multi-modal visual guidance
encompassing single image, image pair, and video pair. VicEdit: Learning to Edit Videos from Visual
In-Context Examples
Yuji Wang1∗ Teng Hu1∗ Yuheng Chen1 Ran Yi1 Han Feng2 Weijian Cao2
Chengjie Wang2 Lizhuang Ma1† Jiangning Zhang3‡
1Shanghai Jiao Tong University 2Tencent Youtu Lab 3Zhejiang University
Abstract
Despite progress in instruction-based video editing, unimodal textual instructions
inherently struggle to convey fine-grained textures and complex dynamics. To
bridge this perceptual gap, we propose Visual In-context Editing, a new paradigm
elevating video editing from textual instructions to multi-modal visual guidance
encompassing single image, image pair, and video pair.Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
The core limitation of current text-only methods is that the semantic nature of language often fails to capture the perceptual nuances required for high-fidelity editing <ref:2608.16745#pg2> This deficiency manifests in three critical dimensions: visual textures, where text-only baselines like DiTTO and VideoCoF often suffer from stylistic drift <ref:2608. p. 0 As illustrated in Fig. 1 (Top), this limitation manifests across three critical dimensions: 1) visual
textures, where text-only baselines like DiTTO [10] and VideoCoF [4] often suffer from stylistic drift,
as linguistic descriptors cannot precisely anchor specific textures; 2) spatial transformations, where
linguistic instructions specify what but not where to edit, whereas image-pair references intuitively
define spatial anchors that current MLLM-based methods [11, 12] fail to exploit; and 3) temporal
dynamics, where complex motion patterns in creative editing exceed linguistic capacity, a gap that
∗Equal contributions. As illustrated in Fig. 1 (Top), this limitation manifests across three critical dimensions: 1) visual
textures, where text-only baselines like DiTTO [10] and VideoCoF [4] often suffer from stylistic drift,
as linguistic descriptors cannot precisely anchor specific textures; 2) spatial transformations, where
linguistic instructions specify what but not where to edit, whereas image-pair references intuitively
define spatial anchors that current MLLM-based methods [11, 12] fail to exploit; and 3) temporal
dynamics, where complex motion patterns in creative editing exceed linguistic capacity, a gap that
∗Equal contributions. As illustrated in Fig. 1 (Top), this limitation manifests across three critical dimensions: 1) visual
textures, where text-only baselines like DiTTO [10] and VideoCoF [4] often suffer from stylistic drift,
as linguistic descriptors cannot precisely anchor specific textures; 2) spatial transformations, where
linguistic instructions specify what but not where to edit, whereas image-pair references intuitively
define spatial anchors that current MLLM-based methods [11, 12] fail to exploit; and 3) temporal
dynamics, where complex motion patterns in creative editing exceed linguistic capacity, a gap that
∗Equal contributions. This paradigm leverages dense semantic cues
across diverse visual modalities to provide richer guidance for precise video manipulation. To bridge existing architectural and resource gaps, We first introduce VicEdit-400K (Sec.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
How the training signal is built
The framework VicEdit introduces two key components to bridge visual and textual contexts: Modality-Adaptive Semantic Distillation (MASD) and Dual-Context Injection (DCI) <ref:2608.16745#pg4>. MASD extracts visual context tokens via modality-conditioned learnable queries, where the additive shift decomposes the search space into a common editing manifold and modality-specific sub-regions to steer attention toward relevant cues <ref:2608. p. 5 where Qbase ∈ R
N×d denotes shared base queries for cross-modal generalization, and Em ∈ R
N×d
is a learnable modality-specific embedding. Theoretically, this additive shift decomposes the search
space into a common editing manifold and modality-specific sub-regions, steering attention toward
relevant cues without losing foundational semantic knowledge. "#$
MLP
Query
Connector
Hvis
Tvis Tvis
Hinst
Tinst
Tinst
Semantic
Shift
Vtgt
Figure 4: Overview of VicEdit. It consists of 1) MASD modulates base queries Qbase with modality
embeddings Em to distill visual tokens Tvis; 2) DCI applies a semantic shift to calibrate instruction
extraction, yielding dual-context tokens. To effectively capture
editing semantics, we design Modality-Adaptive Semantic Distillation (MASD), which extracts
visual context tokens via modality-conditioned learnable queries. These distilled semantics, alongside
textual instructions, are integrated into the generation process through Dual-Context Injection (DCI),
a mechanism that synergistically fuses instructional and visual context tokens into the video DiT
via cross-attention.Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
The success of VicEdit is built upon VicEdit-400K, the first large-scale dataset for visual in-context video editing, which comprises 400K high-quality samples across ten distinct task types and three reference modalities <ref:2608.16745#pg4>. This dataset was curated through a four-stage pipeline involving Source Data Filtering, Instruction Synthesis, Visual Reference Generation, and Semantic Verification to ensure superior visual fidelity and semantic consistency <ref:2608. p. 3 Source Data Filtering
Semantic
Consistency
Aesthetic
Assessment
High Quality
Instruction Synthesis
MLLM
Rsi Prompt
Src: A image of …
Edit: Transform …
Visual Reference Generation Semantic Verification
Rvp
Image
Generation
Semantic Graph
Sampling
Instruction
Following
Temporal
Consistency
Reference
Alignment
Template
Generate prompts
for single image /
images pairs …. A image of …
Rip Prompts
Rsi
Rip
Rsi Prompt
Rip Prompts
VIC Reference
Rsi Rip Rvp
Video Editing
P Vsrc Vtgt
Vtgt = 𝒢 (Vsrc, P, <R>)
Figure 2: Data construction pipeline that comprises four stages (Sec. The phased training strategy demonstrates a clear evolution in
the model’s multi-task capabilities. Stage 1 focuses on enhancing the base model’s versatility by
exposing it to a broader range of editing tasks, which significantly enriches its fundamental editing
capacity and leads to a marked improvement in base editing performance. 3) Joint Fine-Tuning. We unfreeze the DiT and jointly fine-tune the entire pipeline (except the
MLLM) to align the visual conditioning signals with the generative process.Where the paper says it
Where the paper says it
Where the paper says it
The authors demonstrate that VicEdit achieves state-of-the-art performance across both basic instruction editing and visual in-context editing tasks on VicEditBench, consistently outperforming existing text-driven baselines <ref:2608.16745#pg10>. The ablation studies confirm the complementary nature of MASD and DCI: MASD is responsible for fine-grained texture and structural fidelity, while DCI drives precise semantic understanding and instruction following <ref:2608. p. 4 As illustrated in Figure 4, VicEdit employs a frozen MLLM for semantic
understanding and a Video DiT for generation, connected by two lightweight modules: MASD,
which extracts modality-aware visual tokens Tvis from the in-context reference, and DCI, which
produces instruction tokens Tins calibrated by the visual context. Tvis and Tins encode complementary
perceptual editing semantics and task-level edit directions, respectively. By leveraging large language models
and diffusion architectures [7–9], recent methods have demonstrated impressive capabilities ranging
from localized object manipulation to global style transfer, establishing text instructions as the
dominant interface for controllable video editing. However, a fundamental gap remains: the semantic
nature of language often fails to capture the perceptual nuances required for high-fidelity editing.Where the paper says it
Where the paper says it
VicEdit-Bench, which evaluates visual in-context editing across three reference modalities, provides a comprehensive testbed that validates the efficacy of the proposed paradigm <ref:2608.16745#pg4>. Quantitative analysis on VicEdit-Bench shows that VicEdit achieves an Overall score of 3. p. 7 For a fair comparison, we equip
all baselines with an instruction enhancement module to augment their text-only instructions. As
shown in Table 2, VicEdit achieves a remarkable Overall score of 3. This performance surpasses
the second-best method, Kiwi-Edit (2. 97), by a substantial margin of 0. p. 7 This performance surpasses
the second-best method, Kiwi-Edit (2. 97), by a substantial margin of 0. This performance surpasses
the second-best method, Kiwi-Edit (2. 97), by a substantial margin of 0. ‡Project lead. arXiv:2608. ‡Project lead. arXiv:2608. ‡Project lead. arXiv:2608. Notably, the additional
categories beyond Kiwi-Edit’s native support (e. , Outpainting) further validate the effectiveness of
our dataset and framework design. To encompass diverse editing scenarios, R is formulated as a set of heterogeneous modalities that
encode editing semantics across different dimensions:
• Single Image (Rsi = Iref ): Provides fine-grained visual anchors, including textures and
appearances, to guide stylistic or identity-consistent synthesis. • Image Pair (Rip = {I
ref
s
, Iref
t }): Characterizes the transformation between reference
domains, capturing shifts in spatial layouts and localized details. Overview of VicEdit. As illustrated in Figure 4, VicEdit employs a frozen MLLM for semantic
understanding and a Video DiT for generation, connected by two lightweight modules: MASD,
which extracts modality-aware visual tokens Tvis from the in-context reference, and DCI, which
produces instruction tokens Tins calibrated by the visual context.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The final evaluation on VicEdit-Bench shows that VicEdit achieves a remarkable Overall score of 3.45, surpassing the second-best method, Kiwi-Edit (2. p. 7 As
shown in Table 2, VicEdit achieves a remarkable Overall score of 3. This performance surpasses
the second-best method, Kiwi-Edit (2. As
shown in Table 2, VicEdit achieves a remarkable Overall score of 3. This performance surpasses
the second-best method, Kiwi-Edit (2. 48 points (a 16% relative
improvement). The significant gap underscores VicEdit’s superior capability in understanding and
executing complex editing instructions. Following [1], we leverage
MLLMs [29] for automated scoring, supplemented by fidelity and temporal metrics from VBench [26,
27] and IVE-bench [34]. 3 Methodology: Visual In-context Editing (VicEdit)
We introduce VicEdit, a unified in-context learning framework that extends video editing to accommodate multi-modal visual guidance.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Modality Adaptive Semantic Distillation (MASD) allows learnable queries to be modulated by modality-specific embeddings, decomposing the search space into a common editing manifold and modality-specific sub-regions. This mechanism steers attention toward relevant cues without losing foundational semantic knowledge, enabling specialization across heterogeneous modalities with negligible overhead.
-
Dual Context Injection (DCI) introduces a semantic shift mechanism that uses the visual context to calibrate instruction queries before token extraction, ensuring that instruction queries are extracted within a visually aware manifold and resolving inherent ambiguities in natural language descriptions.
-
VicEdit-400K is the first large-scale dataset covering three reference modalities (Single Image, Image Pair, and Video Pair) across ten distinct editing task types, providing a systematic foundation for training a single model that adaptively handles diverse reference types within a unified framework.
-
The model can perform fine-grained controllability across single images, image pairs, and video pairs by leveraging the synergy between MASD (for texture and structural fidelity) and DCI (for precise semantic understanding).
Abstract
Despite remarkable progress in instruction-based video editing, textual descriptions remain limited in expressing fine-grained appearance, spatial transformations, and temporal dynamics. We introduce Visual In-context Editing, a new paradigm that shifts video editing from text-only instructions to visual demonstrations, where users specify desired edits through heterogeneous visual references including single images, image pairs, and video pairs. To establish this paradigm, we curate VicEdit-400K, the first large-scale dataset for visual in-context video editing, containing 400K high-quality samples across ten editing tasks with comprehensive semantic and visual consistency filtering. Building upon this foundation, we propose VicEdit, a unified framework that enables effective reasoning over diverse visual contexts and textual instructions. Specifically, we introduce Modality-Adaptive Semantic Distillation to extract modality-aware visual editing semantics and Dual-Context Injection to align visual contexts with textual editing intents. Extensive experiments on VicEdit-Bench demonstrate that VicEdit achieves state-of-the-art performance on both conventional instruction-based editing and visual in-context editing. Further controlled studies and human evaluations verify that VicEdit effectively leverages heterogeneous visual references, establishing visual in-context editing as a promising paradigm for controllable video editing.
Sources
- OpenVE-3M: A Large-Scale High-Quality Dataset for Instruction-Guided Video Editing
- EasyV2V: A High-quality Instruction-based Video Editing Framework
- NOVA: Sparse Control, Dense Synthesis for Pair-Free Video Editing
- VideoCoF: Unified Video Editing with Temporal Reasoner
- Diffusion Model-Based Video Editing: A Survey
- Open-Sora: Democratizing Efficient Video Production for All
- Wan: Open and Advanced Large-Scale Video Generative Models
- HunyuanVideo: A Systematic Framework For Large Video Generative Models
- Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset
- Omni-Video: Democratizing Unified Video Understanding and Generation
- UniVideo: Unified Understanding, Generation, and Editing for Videos
- Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidance
- Region-Constraint In-Context Generation for Instructional Video Editing
- TokenFlow: Consistent Diffusion Features for Consistent Video Editing
- AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Tasks
- OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer
- EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning
- VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness
- Qwen3 Technical Report
- Qwen3-VL Technical Report
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs