PRIMEdit: Probability Redistribution for Instance-Aware Multi-Object Video Editing With a New Benchmark Dataset

The gist The PRIMEdit framework enables faithful and disentangled edits in single- and multi-instance levels, as well as an applicability to more fine-grained partial instance level without the need for additional training.

Overview of Challenges

Recent zero-shot video editing techniques primarily focus on global or single-object edits, which can lead to unintended changes in other parts of the video. When multiple objects require localized edits, existing methods face challenges such as unfaithful editing, editing leakage, and lack of suitable evaluation datasets and metrics. Local VE aims to address this problem by accurately manipulating specific objects in videos. The four critical challenges persisting in multi-instance VE are (i) Attention leakage, (ii) Unfaithful editing, (iii) Lack of fast and intuitive multi-instance VE methods, and (iv) Lack of evaluation dataset and metrics.

PRIMEdit Framework

PRIMEdit is a zero-shot framework that introduces two key modules to achieve faithful edits and reduce attention leakage. The framework modifies each instance based on its corresponding edit caption, ensuring that the region outside the masks remains unedited. It falls under the category of inversion-based VE where initial latents are generated using a VAE encoder and then inverted using DDIM Inversion to yield a sequence of inverted latents.

Instance-centric Probability Redistribution (IPR)

To enhance editing faithfulness and increase the likelihood of objects appearing within their masks, PRIMEdit introduces Instance-centric Probability Redistribution (IPR) in the cross-attention layers. IPR is inspired by attention modulation and operates exclusively within the cross-attention layers to avoid significant computational overhead. The manipulation involves splitting the attention map into sets inside and outside the instance mask, and then redistributing probabilities for tokens S, T, and E based on experimental observations. Specifically, it zeros out T and E tokens’ probabilities for outside regions to prevent edits outside the mask, while redistributing attention probability values from S to T and E tokens to increase editing faithfulness.

Disentangled Multi-instance Sampling (DMS)

The Disentangled Multi-instance Sampling (DMS) module is designed to disentangle the multi-instance VE process and minimize attention leakage. DMS comprises two sampling strategies: Series Noise Sampling (SNS) and Parallel Noise Sampling (PNS) preceded by latent fusion and re-inversion. In SNS, each instance is independently edited using its target caption and mask, while the background noise is derived from inverted latents. The instance latents are then harmonized through Parallel Noise Sampling (PNS) after latent fusion and re-inversion to obtain the final fused latent.

Evaluation and Metrics

The paper introduces a new evaluation benchmark called MIVE Dataset, consisting of 200 videos with varying numbers and sizes of instances accompanied by instance-level captions and masks. A novel metric called the Cross-Instance Accuracy (CIA) Score is proposed to quantify attention leakage in multi-instance editing tasks. The CIA Score is calculated as the mean of diagonal elements of an n x n similarity matrix, where each row sets the highest similarity score to 1 and all others to 0, indicating that each cropped instance aligns best with its caption. The evaluation metrics include Global Temporal Consistency (GTC), Global Textual Faithfulness (GTF), Frame Accuracy (FA), Local Temporal Consistency (LTC), Local Textual Faithfulness (LTF), Instance Accuracy (IA), and Cross-Instance Accuracy (CIA).

Experimental Results

Experiments on the MIVE dataset demonstrate that PRIMEdit significantly outperforms recent state-of-the-art methods in terms of editing faithfulness, accuracy, and leakage prevention. Quantitative comparison shows PRIMEdit achieves the best scores in key multi-instance VE metrics (LTC, LTF, IA, and CIA) demonstrating its superior ability to maintain temporal consistency, fine-grained textual alignment, and instance-aware modifications. The user study supports PRIMEdit’s robustness and effectiveness with participants favoring our method.

Efficiency and Limitations

PRIMEdit is both faster and more memory-efficient for multi-instance video editing compared to VideoGrain [65]. However, the model struggles with reflection consistency due to mask limitations, covering only the instance, not its reflection. Future works can build on approaches like [16] to ensure consistency for both the instance and its reflection. The final model strikes a balanced trade-off between LTC, faithfulness, leakage, and background preservation.

Conclusion

In this work, we introduce PRIMEdit, a novel multi-instance video editing framework featuring Instancecentric Probability Redistribution (IPR) and Disentangled Multi-instance Sampling (DMS). Our method achieves faithful and disentangled edits with minimal attention leakage, outperforming existing SOTA methods in both qualitative and quantitative analyses on our newly proposed MIVE dataset. We further propose a new Cross-Instance Accuracy (CIA) Score to quantify attention leakage. Our user study supports PRIMEdit’s robustness and effectiveness with participants favoring our method.

1. Instance-centric Probability Redistribution (IPR) in cross-attention layers will ensure precise localization by manipulating attention probabilities, as described by In our IPR, we split Ai,j of an instance into two sets depending on the mask m: outside the instance AO,j = 0 and inside the instance AI,j = 1. This prevents artifacts like oversaturation by avoiding modification of padding tokens while strategically redistributing attention probabilities from S to T and E tokens to improve editing faithfulness.

2. Disentangled Multi-instance Sampling (DMS) will significantly reduce attention leakage by decoupling the sampling process into independent instance edits and harmonizing them later. This is achieved through Series Noise Sampling (SNS), which we independently edit each instance i using its target caption ci and its mask mi, followed by latent fusion and re-inversion to obtain the final fused latent, ensuring that the background is replaced with the inverted latent noise at each step to ensure it remains unchanged.

3. The Cross-Instance Accuracy (CIA) Score will be implemented as a novel metric to quantify attention leakage across instances, defined as the mean of diagonal elements, which addresses the limitation that existing metrics cannot account for cases where the text prompt for one instance leaks to another instance. This allows researchers to specifically evaluate leakage, as seen in the quantitative comparison showing that PRIMEdit achieves better Leakage Scores than SOTA methods.

4. The MIVE Dataset construction pipeline will be utilized to generate high-quality, diverse training data by employing LLaVA [35] to generate captions and use Llama 3 [17] to summarize the captions, followed by manual refinement and target caption generation using Llama 3 for retexturing or swapping instances, ensuring the dataset provides 200 videos with varying numbers of objects per video and the number of instances per object class.

5. The training procedure will involve a hybrid sampling strategy combining Series Noise Sampling (SNS) and Parallel Noise Sampling (PNS) to mitigate temporal inconsistency. This involves fusing independent instance latents via z˙1:N t = ΣM i=1(zˆ 1:N t i · mi) + ˜z 1:N t · mB, followed by re-inversion using DDIM for l steps after the latent fusion to obtain z¨1:N t+l.

Recent Artificial Intelligence (AI)-based video editing has enabled users to edit videos through simple text prompts, significantly simplifying the editing process. However, recent zero-shot video editing techniques focus on global or single-object edits, which can lead to unintended changes in other parts of the video. When editing multiple objects only within their localized regions, existing methods face challenges, such as unfaithful editing, editing leakage, and lack of suitable evaluation datasets and metrics. To overcome these limitations, we propose P robability R edistribution for I nstance-aware M ulti-object Video Edit ing (PRIMEdit). PRIMEdit is a zero-shot framework that introduces two key modules: (i) Instance-centric Probability Redistribution (IPR) to ensure precise localization and faithful editing and (ii) Disentangled Multi-instance Sampling (DMS) to prevent editing leakage. Additionally, we present the MIVE dataset for multi-instance video editing, including its subsets categorized by instance counts and sizes for controlled analysis across different levels of scene complexity. We also introduce the Cross-Instance Accuracy (CIA) Score that is firstly proposed to evaluate the editing leakage in multi-instance video editing tasks, which exhibits strong correlation with human evaluation as measured by Spearman's ρ= 0.86. Our extensive qualitative, quantitative, and user study evaluations demonstrate that PRIMEdit significantly outperforms recent state-of-the-art methods in terms of editing faithfulness, accuracy, and leakage prevention, setting a new benchmark for multi-instance video editing.