SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearning
summary
The gist
SALMUBench introduces a novel benchmark to evaluate machine unlearning methods in CLIP models under the “Right to be Forgotten” scenario by measuring association-level forgetting and collateral…
In short
SALMUBench introduces a new benchmark to test how well machine unlearning methods remove sensitive information from CLIP models. It measures both forgetting accuracy and collateral damage using structured datasets. The evaluation reveals three failure modes: ineffective, over-generalized, or catastrophically damaging methods, helping researchers diagnose unlearning weaknesses.
Key concepts
- SALMU Dataset
- A synthetic dataset created for evaluating text-image association unlearning. It is built through five stages including anchor seeding from faces, identity-preserving image generation, PII assignment, and LLM caption paraphrasing to force methods to target semantic associations.
- Forgetting Efficacy
- This pillar of evaluation measures how successfully a method removes sensitive associations from the 'forget set.' Metrics like RetFail and Association Strength quantify this removal. It determines if the intended information has actually been erased from the model's knowledge.
- Utility Impact
- This pillar measures collateral damage by assessing how much useful, general knowledge is lost during unlearning. Metrics such as General Knowledge (GenKnow) and Inter-Identity Similarity quantify whether erasing sensitive data negatively affects the model's overall performance on non-sensitive tasks.
Terminology used across episodes
This episode discusses
- SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearning · Paper Radio
- Data Filtering Networks
The paper
SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearning · Read on arXiv
Computer Vision Center at Universitat Politecnica de Catalunya · Universitat Autonoma de Barcelona
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearning".
Jane: Here's the gist of the paper. SALMUBench introduces a novel benchmark to evaluate machine unlearning methods in CLIP models under the “Right to be Forgotten” scenario by measuring association-level forgetting and collateral damage The core contribution is a novel evaluation protocol built on structured holdout sets (holdout identity and holdout association) which enables nuanced diagnosis of unlearning failures, distinguishing between ineffective, catastrophically damaging, and over-generalizing methods Benchmark Construction The foundation of the benchmark is the SALMU dataset, a synthetic corpus designed for a comprehensive and reproducible evaluation of text-image association-level machine unlearning Its construction follows five sequential stages: (1) anchor seeding from a face dataset; (2) identity-preserving image generation; (3) CLIP-based filtering and demographic curation; (4) Personally Identifiable Information (PII) assignment; and (5) LLM caption paraphrasing The pipeline proceeds through the following five stages: (1) anchor seeding from a face dataset, (2) identity-preserving image generation, (3) CLIP-based filtering and demographic curation, (4) Personally Identifiable Information (PII) assignment, and (5) LLM caption paraphrasing Stage 5 involves paraphrasing a base sentence with a sensitive attribute using gemma3:12b
26: following one of five linguistic directives to force unlearning methods to target semantic associations rather than surface patterns Model Training Setup To support a realistic and robust evaluation of machine unlearning, two ViT/B-16 CLIP models are trained from scratch: the Clean model and the Compromised model The Clean model is trained exclusively on the retain set, while the Compromised model is trained on the union of the retain and sensitive sets Both models adopt a ViT/B-16 architecture and are trained from scratch for 32 epochs with approximately 400 million samples over 32 epochs This controlled setup ensures that any differences in their behavior are attributable solely to the presence of the sensitive data defined in the sensitive set Evaluation Protocol Metrics The SALMUBench framework includes a standardized protocol structured into two fundamental pillars: Forgetting Efficacy and Utility Impact Pillar 1, Forgetting Efficacy, quantifies the removal of sensitive associations from the forget set, with key metrics including RetFail (Retrieval Failure), Association Strength (AssocStr), Association Coherence Score (ACS), Identity Zero-Shot Classification (IdZSC), and Core Association Robustness (CoreAssoc) Pillar 2, Utility Impact, measures the preservation of useful knowledge and quantifies collateral damage using metrics such as General Knowledge (GenKnow), Inter-Identity Similarity (InterIdSim), Intra-Identity Similarity (IntraIdSim), Visual Identity Integrity (VisIdInt), and Fragile Knowledge Similarity (FragSim) Failure Mode Taxonomy The paper categorizes unlearning methods into three distinct failure modes based on their performance across efficacy and utility metrics Type 1, Catastrophically Damaging Methods, are characterized by significant drops in general knowledge (GenKnow), such as with Shuffled Captions and Direct Similarity Minimization Type 2, Over-generalized Forgetting, occurs when methods erase more than intended by pushing ‘AssocStr’ below the Clean model’s baseline of 0.142 Type 3, Ineffective Forgetting, represents the most straightforward failure where the method is simply not able to erase the target information Key Findings on Method Performance The analysis reveals that utility-efficient deletion is feasible, but current methods exhibit distinct failure modes: they either fail to forget effectively or overgeneralize by erasing more than intended The best performers (VLUnlearn, DELETE and CLIPErase) precisely reach the target efficacy metric (RetFail=0.001) with negligible impact on general utility (GenKnow ≥ 0.632) However, DELETE and CLIPErase still incur noticeable collateral damage to holdout associations (InterIdSim and IntraIdSim ≤ 0.
Paper summary: Tom: So looking at the abstract of "SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearning", they set up a scenario where you have CLIP models that are contrastively trained on this synthetic data, and they compare a model polluted with sensitive data against one that isn't.
Jane: The thesis is that current evaluations fail to diagnose fine-grained, association-level forgetting in these embedding-based foundation models like CLIP, so they built this benchmark to bridge that gap for the "Right to be Forgotten" scenario.
Lu: They introduce a novel evaluation protocol using structured holdout sets—holdout identity and holdout association—to precisely measure unlearning efficacy and collateral damage when you try to remove sensitive information from these models.
Meng: I’m interested in how they isolate the effects because right now, it feels like most unlearning tests just show a big drop in overall performance without telling you if that drop is good or bad for the rest of the model.
Lalam: They claim this setup allows them to distinguish between ineffective, catastrophically damaging, and over-generalizing methods based on how they perform across both efficacy and utility metrics.
Tom: So it’s a structured way to test if a method is just not working at all, if it’s totally destroying useful knowledge in the process, or if it's just forgetting more than you intended.
Conclusion: Jane: So, looking at the whole SALMUBench framework, the authors are trying to move beyond just checking if a model remembers something to actually measuring how well it can selectively forget specific personal associations within those models.
Tom: The implication is that for any system using these multimodal models, you need this kind of detailed evaluation before you consider deploying an unlearning method because the risk of collateral damage is much higher than we previously thought.
Lu: It’s a controlled setup where they pre-train two models from scratch, a Compromised one polluted with sensitive data and a Clean one without it, which gives them that precise control over what the model learned.
Meng: From an engineering side, it means if we want to build tools to help companies erase private data from their deployed CLIP systems, we need these kinds of metrics—like checking general knowledge preservation—to make sure our tools don't accidentally ruin the whole system.
Lalam: The paper suggests that while utility-efficient deletion is possible, current methods often fall into those failure modes they defined: either failing to forget effectively or overgeneralizing by erasing more than intended.
Tom: So the authors are pointing out that while we can delete data, we still have to worry about whether that deletion process harms the model’s ability to do other useful things.
More episodes
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI