A Model Merging Approach for Continual MLLM Unlearning
Yuhang Wang, Linlin Zhang, Haoxuan Ji, Xianmin Ye, Zhenxing Niu, Haichang Gao
cs.LG, cs.AI
Submitted: 2026-08-05
Comments: 17 pages, 5 figures
License: http://creativecommons.org/licenses/by/4.0/
The gist: Multimodal large language model (MLLM) unlearning methods have been proposed to remove private, sensitive, or proprietary information from well-trained models.
Terminology
Abstract
Multimodal large language model (MLLM) unlearning methods have been proposed to remove private, sensitive, or proprietary information from well-trained models. However, most existing MLLM unlearning methods are designed for one-shot requests and fail to adequately address continual scenarios, as repeatedly applying one-shot operations leads to cumulative utility degradation, unlearning rebound, and retention drift. We introduce Merging for Continual Unlearning (MCU), an approach that dynamically merges multiple one-shot unlearning adapters into a unified adapter upon receiving each new unlearning request.Through a leave-one-out merging analysis, we reveal that these unlearning adapters exhibit strong cross-task dependencies. Such dependencies have two contrasting effects: they can facilitate cross-task unlearning transferability, but they can also introduce severe interference that degrades unlearning effectiveness and compromises retained knowledge. To address this challenge, MCU projects the adapters into a shared representation space, preserves their dominant directions, suppresses over-concentrated coordinates, and reconfigures cross-task dependencies to mitigate interference while enhancing transferability. Experiments on ICU-Bench and MLLMU-Bench demonstrate that MCU achieves superior unlearning effectiveness while preserving both retained knowledge and general multimodal utility.
Sources
- MU-Bench: A Multitask Multimodal Benchmark for Machine Unlearning
- Editing Models with Task Arithmetic
- PULSE: Practical Evaluation Scenarios for Large Multimodal Model Unlearning
- MLUBench: A Benchmark for Lifelong Unlearning Evaluation in MLLMs
- Digger: Detecting Copyright Content Mis-usage in Large Language Model Training
- CATA: Continual Machine Unlearning via Conflict-Averse Task Arithmetic
- TOFU: A Task of Fictitious Unlearning for LLMs
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- ICU-Bench:Benchmarking Continual Unlearning in Multimodal Large Language Models
- Robust MLLM Unlearning via Visual Knowledge Distillation
- Null Space Constrained Contrastive Visual Forgetting for MLLM Unlearning
- PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for Multimodal Large Language Models
- What Matters for Model Merging at Scale?
- Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks