Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights

Name: Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights
Uploaded: 2026-10-15T02:42:48.510407+00:00
Duration: 465 s
Description: Spectral weight decay is a method to reduce neural network weight rank by applying additive shrinkage based on nuclear norm regularization instead of standard squared Frobenius penalties. This technique lowers effective rank, enabling efficient low-rank compression and improving robustness against label noise across various models like LLaMA and BERT.

Spectral weight decay is a method to reduce neural network weight rank by applying additive shrinkage based on nuclear norm regularization instead of standard squared Frobenius penalties. This technique lowers effective rank, enabling efficient low-rank compression and improving robustness against label noise across various models like LLaMA and BERT.

Open the full episode