A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classification
The study investigates late-time dynamics in online classification using a teacher-student model with softmax loss. It finds that learning scales as 1/3 because only examples near decision boundaries contribute to learning, forming a boundary layer. Learning rate schedules can improve this scaling towards 1/2 by managing residual noise.