From One-Pass SGD to Data Reuse: Mini-Batch Scaling Laws in Sketched Linear Regression
This episode discusses a paper detailing scaling laws for mini-batch Stochastic Gradient Descent (SGD) in linear regression. The authors developed exact formulas showing how batch size influences error components. Key findings include specific design rules for choosing batch size and proving that sampling without replacement is more efficient than with replacement.