Validity-Gated Evaluation of Multimodal Machine Unlearning
cs.CV, cs.LG
Submitted: 2026-05-04
Updated: 2026-11-01
Comments: 13 Pages , 3 figures, TNNLS
Code: https://github.com/neurips26/UnifiedUnl
License: http://creativecommons.org/licenses/by/4.0/
The gist: Machine unlearning aims to remove the influence of training data without retraining a model from scratch.
Terminology
Abstract
Machine unlearning aims to remove the influence of training data without retraining a model from scratch. Evaluation of vision-language models relies on forget accuracy (FA), retain accuracy (RA), membership inference, and similarity to a retain-only reference, but evaluation pipelines are rarely tested. We propose a validity-gated unlearning audit (VGUA) with three checks: the forget set must be memorized without duplicate leakage into the retain set, the scorer must measure correctness, and the training code must implement the intended procedure. Applying these checks reveals that random MMBench splits leak duplicate questions, a lenient scorer hides memorization, and two coding errors remove the signal while training completes. After correcting these problems, we evaluate 234 unlearned models using six methods, three learning rates, and three backbones. Each model is compared with a retain-only reference trained using low-rank adaptation (LoRA). The results depend on update strength. Among models with RA at or above 0.70, lower FA generally indicates closer agreement with the retrained FA-RA operating point. Aggressive updates overshoot this point, collapse RA, and reverse method rankings. Relative to the original model, 78% of healthy models move closer to the retrained model's scores, whereas 94% move farther from it in activation space. Membership-inference advantage does not reliably track retraining alignment. Under option-shuffled training, unlearned models answer 25% of reordered forgotten ScienceQA questions, compared with 6% for the retain-only reference. These results show that multimodal unlearning conclusions depend on pipeline validity, update strength, and the comparison criterion.
Sources
- TOFU: A Task of Fictitious Unlearning for LLMs
- MUSE: Machine Unlearning Six-Way Evaluation for Language Models
- Unlearning Sensitive Information in Multimodal LLMs: Benchmark and Attack-Defense Evaluation
- OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs