Spectral Saliency for Machine Unlearning
cs.LG, cs.AI
Submitted: 2026-08-16
Updated: 2026-09-26
Code: https://github.com/mseitzer/pytorch-fid
License: http://creativecommons.org/licenses/by/4.0/
The gist: Machine unlearning (MU) aims to remove the influence of specific training data while preserving model utility.
Terminology
Abstract
Machine unlearning (MU) aims to remove the influence of specific training data while preserving model utility. As the name suggests, MU can be viewed as the inverse of learning, using gradient-based updates to reduce the influence of a forget-set by counteracting the previously learned behavior. Recently, Muon, a gradient descent variant, has been introduced. Muon applies spectral magnitude normalization to encourage exploration of rare directions and demonstrates promising performance. Inspired by Muon, we adopt the spectral view for unlearning and propose Spectral Saliency Unlearning (SSU). SSU thresholds weak singular components and updates only those directions supported by a confident unlearning signal. We further provide theoretical justification for this thresholding approach from the perspective of the forgetting-retention trade-off. Experiments across image classifiers, diffusion models, and LLMs demonstrate SSU's effectiveness.
Sources
- Evaluating Machine Unlearning via Epistemic Uncertainty
- Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent
- Muon Optimizes Under Spectral Norm Constraints
- TOFU: A Task of Fictitious Unlearning for LLMs
- Llama 2: Open Foundation and Fine-Tuned Chat Models
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks