Shortcuts in the Tail: Debiasing via Post-Hoc Spectral Compression of Fine-Tuning Updates
The episode discusses the paper "Shortcuts in the Tail," which presents a method for debiasing Large Language Models (LLMs). The authors propose using post-hoc spectral compression, specifically truncating the tail of SVD decomposition. This provides a scalable way to mitigate systemic bias without requiring full retraining, allowing researchers to maintain high accuracy while targeting and removing shortcut behaviors.