SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearning

arXiv:2603.26316 · cs.CV, cs.LG · Submitted 2026-03-27 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearning".

Jane: Here's the gist of the paper. SALMUBench introduces a novel benchmark to evaluate machine unlearning methods in CLIP models under the “Right to be Forgotten” scenario by measuring association-level forgetting and collateral damage The core contribution is a novel evaluation protocol built on structured holdout sets (holdout identity and holdout association) which enables nuanced diagnosis of unlearning failures, distinguishing between ineffective, catastrophically damaging, and over-generalizing methods Benchmark Construction The foundation of the benchmark is the SALMU dataset, a synthetic corpus designed for a comprehensive and reproducible evaluation of text-image association-level machine unlearning Its construction follows five sequential stages: (1) anchor seeding from a face dataset; (2) identity-preserving image generation; (3) CLIP-based filtering and demographic curation; (4) Personally Identifiable Information (PII) assignment; and (5) LLM caption paraphrasing The pipeline proceeds through the following five stages: (1) anchor seeding from a face dataset, (2) identity-preserving image generation, (3) CLIP-based filtering and demographic curation, (4) Personally Identifiable Information (PII) assignment, and (5) LLM caption paraphrasing Stage 5 involves paraphrasing a base sentence with a sensitive attribute using gemma3:12b

26: following one of five linguistic directives to force unlearning methods to target semantic associations rather than surface patterns Model Training Setup To support a realistic and robust evaluation of machine unlearning, two ViT/B-16 CLIP models are trained from scratch: the Clean model and the Compromised model The Clean model is trained exclusively on the retain set, while the Compromised model is trained on the union of the retain and sensitive sets Both models adopt a ViT/B-16 architecture and are trained from scratch for 32 epochs with approximately 400 million samples over 32 epochs This controlled setup ensures that any differences in their behavior are attributable solely to the presence of the sensitive data defined in the sensitive set Evaluation Protocol Metrics The SALMUBench framework includes a standardized protocol structured into two fundamental pillars: Forgetting Efficacy and Utility Impact Pillar 1, Forgetting Efficacy, quantifies the removal of sensitive associations from the forget set, with key metrics including RetFail (Retrieval Failure), Association Strength (AssocStr), Association Coherence Score (ACS), Identity Zero-Shot Classification (IdZSC), and Core Association Robustness (CoreAssoc) Pillar 2, Utility Impact, measures the preservation of useful knowledge and quantifies collateral damage using metrics such as General Knowledge (GenKnow), Inter-Identity Similarity (InterIdSim), Intra-Identity Similarity (IntraIdSim), Visual Identity Integrity (VisIdInt), and Fragile Knowledge Similarity (FragSim) Failure Mode Taxonomy The paper categorizes unlearning methods into three distinct failure modes based on their performance across efficacy and utility metrics Type 1, Catastrophically Damaging Methods, are characterized by significant drops in general knowledge (GenKnow), such as with Shuffled Captions and Direct Similarity Minimization Type 2, Over-generalized Forgetting, occurs when methods erase more than intended by pushing ‘AssocStr’ below the Clean model’s baseline of 0.142 Type 3, Ineffective Forgetting, represents the most straightforward failure where the method is simply not able to erase the target information Key Findings on Method Performance The analysis reveals that utility-efficient deletion is feasible, but current methods exhibit distinct failure modes: they either fail to forget effectively or overgeneralize by erasing more than intended The best performers (VLUnlearn, DELETE and CLIPErase) precisely reach the target efficacy metric (RetFail=0.001) with negligible impact on general utility (GenKnow ≥ 0.632) However, DELETE and CLIPErase still incur noticeable collateral damage to holdout associations (InterIdSim and IntraIdSim ≤ 0.

Paper summary: Tom: So looking at the abstract of "SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearning", they set up a scenario where you have CLIP models that are contrastively trained on this synthetic data, and they compare a model polluted with sensitive data against one that isn't.

Jane: The thesis is that current evaluations fail to diagnose fine-grained, association-level forgetting in these embedding-based foundation models like CLIP, so they built this benchmark to bridge that gap for the "Right to be Forgotten" scenario.

Lu: They introduce a novel evaluation protocol using structured holdout sets—holdout identity and holdout association—to precisely measure unlearning efficacy and collateral damage when you try to remove sensitive information from these models.

Meng: I’m interested in how they isolate the effects because right now, it feels like most unlearning tests just show a big drop in overall performance without telling you if that drop is good or bad for the rest of the model.

Lalam: They claim this setup allows them to distinguish between ineffective, catastrophically damaging, and over-generalizing methods based on how they perform across both efficacy and utility metrics.

Tom: So it’s a structured way to test if a method is just not working at all, if it’s totally destroying useful knowledge in the process, or if it's just forgetting more than you intended.

Conclusion: Jane: So, looking at the whole SALMUBench framework, the authors are trying to move beyond just checking if a model remembers something to actually measuring how well it can selectively forget specific personal associations within those models.

Tom: The implication is that for any system using these multimodal models, you need this kind of detailed evaluation before you consider deploying an unlearning method because the risk of collateral damage is much higher than we previously thought.

Lu: It’s a controlled setup where they pre-train two models from scratch, a Compromised one polluted with sensitive data and a Clean one without it, which gives them that precise control over what the model learned.

Meng: From an engineering side, it means if we want to build tools to help companies erase private data from their deployed CLIP systems, we need these kinds of metrics—like checking general knowledge preservation—to make sure our tools don't accidentally ruin the whole system.

Lalam: The paper suggests that while utility-efficient deletion is possible, current methods often fall into those failure modes they defined: either failing to forget effectively or overgeneralizing by erasing more than intended.

Tom: So the authors are pointing out that while we can delete data, we still have to worry about whether that deletion process harms the model’s ability to do other useful things.

Computer Vision Center at Universitat Politecnica de Catalunya · Universitat Autonoma de Barcelona

cs.CV, cs.LG

Submitted: 2026-03-27

Updated: 2026-03-27

Comments: Accepted to CVPR 2026. Project page: http://cvc-mmu.github.io/salmubench

Journal ref: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 39351-39360

Code: https://github.com/joke2k/faker

Project page: http://cvc-mmu.github.io/salmubench

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 89/100

The gist: SALMUBench introduces a novel benchmark to evaluate machine unlearning methods in CLIP models under the “Right to be Forgotten” scenario by measuring association-level forgetting and collateral…

Key concepts

SALMU Dataset
A synthetic dataset created for evaluating text-image association unlearning. It is built through five stages including anchor seeding from faces, identity-preserving image generation, PII assignment, and LLM caption paraphrasing to force methods to target semantic associations.
Forgetting Efficacy
This pillar of evaluation measures how successfully a method removes sensitive associations from the 'forget set.' Metrics like RetFail and Association Strength quantify this removal. It determines if the intended information has actually been erased from the model's knowledge.
Utility Impact
This pillar measures collateral damage by assessing how much useful, general knowledge is lost during unlearning. Metrics such as General Knowledge (GenKnow) and Inter-Identity Similarity quantify whether erasing sensitive data negatively affects the model's overall performance on non-sensitive tasks.

Terminology

Summary

SALMUBench introduces a novel benchmark to evaluate machine unlearning methods in CLIP models under the “Right to be Forgotten” scenario by measuring association-level forgetting and collateral damage The core contribution is a novel evaluation protocol built on structured holdout sets (holdout identity and holdout association) which enables nuanced diagnosis of unlearning failures, distinguishing between ineffective, catastrophically damaging, and over-generalizing methods

Benchmark Construction

The foundation of the benchmark is the SALMU dataset, a synthetic corpus designed for a comprehensive and reproducible evaluation of text-image association-level machine unlearning Its construction follows five sequential stages: (1) anchor seeding from a face dataset; (2) identity-preserving image generation; (3) CLIP-based filtering and demographic curation; (4) Personally Identifiable Information (PII) assignment; and (5) LLM caption paraphrasing The pipeline proceeds through the following five stages: (1) anchor seeding from a face dataset, (2) identity-preserving image generation, (3) CLIP-based filtering and demographic curation, (4) Personally Identifiable Information (PII) assignment, and (5) LLM caption paraphrasing Stage 5 involves paraphrasing a base sentence with a sensitive attribute using gemma3:12b [26] following one of five linguistic directives to force unlearning methods to target semantic associations rather than surface patterns

Model Training Setup

To support a realistic and robust evaluation of machine unlearning, two ViT/B-16 CLIP models are trained from scratch: the Clean model and the Compromised model The Clean model is trained exclusively on the retain set, while the Compromised model is trained on the union of the retain and sensitive sets Both models adopt a ViT/B-16 architecture and are trained from scratch for 32 epochs with approximately 400 million samples over 32 epochs This controlled setup ensures that any differences in their behavior are attributable solely to the presence of the sensitive data defined in the sensitive set

Evaluation Protocol Metrics

The SALMUBench framework includes a standardized protocol structured into two fundamental pillars: Forgetting Efficacy and Utility Impact Pillar 1, Forgetting Efficacy, quantifies the removal of sensitive associations from the forget set, with key metrics including RetFail (Retrieval Failure), Association Strength (AssocStr), Association Coherence Score (ACS), Identity Zero-Shot Classification (IdZSC), and Core Association Robustness (CoreAssoc) Pillar 2, Utility Impact, measures the preservation of useful knowledge and quantifies collateral damage using metrics such as General Knowledge (GenKnow), Inter-Identity Similarity (InterIdSim), Intra-Identity Similarity (IntraIdSim), Visual Identity Integrity (VisIdInt), and Fragile Knowledge Similarity (FragSim)

Failure Mode Taxonomy

The paper categorizes unlearning methods into three distinct failure modes based on their performance across efficacy and utility metrics Type 1, Catastrophically Damaging Methods, are characterized by significant drops in general knowledge (GenKnow), such as with Shuffled Captions and Direct Similarity Minimization Type 2, Over-generalized Forgetting, occurs when methods erase more than intended by pushing ‘AssocStr’ below the Clean model’s baseline of 0.142 Type 3, Ineffective Forgetting, represents the most straightforward failure where the method is simply not able to erase the target information

Key Findings on Method Performance

The analysis reveals that utility-efficient deletion is feasible, but current methods exhibit distinct failure modes: they either fail to forget effectively or overgeneralize by erasing more than intended The best performers (VLUnlearn, DELETE and CLIPErase) precisely reach the target efficacy metric (RetFail=0.001) with negligible impact on general utility (GenKnow ≥ 0. p. 7632) However, DELETE and CLIPErase still incur noticeable collateral damage to holdout associations (InterIdSim and IntraIdSim ≤ 0. p. 7

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Implement a novel evaluation protocol built on structured holdout sets (holdout identity and holdout association) to precisely measure unlearning efficacy and collateral damage, allowing for nuanced diagnosis of unlearning failures, distinguishing between ineffective, catastrophically damaging, and over-generalizing methods.

  2. Develop methods that achieve utility-efficient deletion by ensuring the model’s predictions become identical to those of a model retrained from scratch without the sensitive data while preserving its original utility scores.

  3. Design models that can selectively forget specific pieces of information about a person (association-level unlearning) without inadvertently erasing other related facts for the same person, as quantified by measuring Intra-Identity Similarity.

  4. Create unlearning systems capable of maintaining high general knowledge (GenKnow) while successfully removing sensitive associations, as demonstrated by methods like VLUnlearn and DELETE achieving a Target Score close to the Clean model for efficacy metrics.

  5. Engineer training pipelines that utilize synthetic data construction stages, including anchor seeding, identity-preserving image generation, CLIP-based filtering and curation, PII assignment via procedural generation (names, locations, contact details), and LLM caption paraphrasing to simulate complex association patterns.

  6. Develop unlearning algorithms that can operate effectively within the embedding space of contrastively trained foundation models like CLIP by adapting existing unimodal or non-contrastive frameworks using techniques such as Direct Similarity Minimization or Negative Gradient.

  7. Implement a standardized statistical analysis framework to rigorously interpret benchmark results, aiming for an Unlearned model to become statistically indistinguishable from its ideal target—the Clean model for efficacy metrics and the original Compromised model for utility metrics—to confirm perfect state convergence.

Sources

Related papers