Dropout Universality: Scaling Laws and Optimal Scheduling at the Edge-of-Chaos
The episode discusses a paper titled "Dropout Universality: Scaling Laws and Optimal Scheduling at the Edge-of-Chaos." The hosts explain how treating dropout as a perturbation near critical signal propagation allows for optimal scheduling. They find that front-loaded dropout schedules can reduce test loss by 18 to 35 percent in MLPs and Vision Transformers with a fixed training budget.