RADNPO: Reference-free Adaptive Negative Preference Optimization for LLM Unlearning
cs.CR
Submitted: 2026-09-28
Updated: 2026-09-28
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Large language models (LLMs) can memorize sensitive, private, or copyrighted content during pre-training, making machine unlearning necessary for removing targeted knowledge.
Terminology
Abstract
Large language models (LLMs) can memorize sensitive, private, or copyrighted content during pre-training, making machine unlearning necessary for removing targeted knowledge. Recent preference optimization (PO)-based unlearning methods improve stability over gradient ascent (GA)-based methods by introducing alignment-style objectives, which effectively suppress the probability of forget targets. However, target suppression alone does not sufficiently constrain the next-token distribution after unlearning. Existing methods provide limited control over how suppressed probability mass is redistributed and insufficiently adapt forgetting strength to target confidence and distributional concentration. Even after target suppression, probability mass may remain concentrated on a few non-target tokens, potentially producing repetitive or uninformative outputs. To address these limitations, we propose Reference-free ADaptive Negative Preference Optimization (RADNPO), which explicitly guides next-token probability redistribution. Specifically, RADNPO contrasts each forget target with alternative tokens favored by the current next-token distribution and adaptively modulates token-level forgetting strength using target confidence and next-token concentration. Experiments on TOFU and MUSE demonstrate that RADNPO achieves a better trade-off between forgetting quality and model utility than current baselines.
Sources
- OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics
- KTO: Model Alignment as Prospect Theoretic Optimization
- Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning
- Federated Unlearning: How to Efficiently Erase a Client in FL?
- The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning
- TOFU: A Task of Fictitious Unlearning for LLMs
- Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks
- In-Context Unlearning: Language Models as Few Shot Unlearners
- Proximal Policy Optimization Algorithms
- MUSE: Machine Unlearning Six-Way Evaluation for Language Models
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond
- Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning
- CALIBURN: Self-Calibrated LLM Unlearning Alignment
- Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
Related papers
- SoK: AI-Augmented Binary Reversing
- Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
- Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
- Efficient Fuzzy PSI under One-Sided Assumptions
- Sealing the Audit-Runtime Gap for LLM Skills
- Token Composition: A Graph Based on EVM Logs