Optimizer-dependent training dynamics converge to the same one-third optimal data scaling
cs.LG
Submitted: 2026-09-29
Updated: 2026-09-29
Sources
- Chinchilla Scaling: A replication attempt
- Scaling Optimal LR Across Token Horizons
- A Dynamical Model of Neural Scaling Laws
- Understanding Optimization in Deep Learning with Central Flows
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Deep Learning Scaling is Predictable, Empirically
- Risk and parameter convergence of logistic regression
- Scaling Laws for Neural Language Models
- A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classification
- Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be
- Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining
- FOCUS: First Order Concentrated Updating Scheme
- Inverse Depth Scaling From Most Layers Being Similar
- Universal One-third Time Scaling in Learning Peaked Distributions
- Neural Thermodynamic Laws for Large Language Model Training
- Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
- A Solvable Model of Neural Scaling Laws
- Stochastic Gradient Descent as Approximate Bayesian Inference
- An Empirical Model of Large-Batch Training
- On the Convergence of Adam and Beyond
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks