Optimizer choice matters for the emergence of Neural Collapse
The episode discusses the paper "Optimizer choice matters for the emergence of Neural Collapse," which was authored by researchers from the University of Basel. The hosts review extensive testing that demonstrates that Neural Collapse is not a universal constant. They conclude that how weight decay is applied—specifically comparing methods like Adam and AdamW—is critical to determining if a model collapses.