Why Does Adaptive Batching Help LLM Pretraining? A Perspective from Unbounded Variance
cs.LG, math.OC
Submitted: 2026-10-01
Updated: 2026-10-01
Sources
- Towards Weaker Variance Assumptions for Stochastic Optimization
- Lower Bounds and Proximally Anchored SGD for Non-Convex Minimization Under Unbounded Variance
- Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Rates
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- An Empirical Model of Large-Batch Training
- 2 OLMo 2 Furious
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks