Crossing the Margin Cliff: Toward Relearn-Robust LLM Unlearning via Margin Calibration
The episode discusses a paper detailing how current LLM unlearning methods fail due to a structural limitation called the margin cliff. To overcome this plateau where gradients vanish, hosts examine 'Margin Calibration' (MC, or Margin Calibration), a new technique designed to achieve robust forgetting while maintaining model utility.