How Does Local Landscape Geometry Evolve in Language Model Pre-Training?
cs.LG, cs.AI
Submitted: 2026-09-30
Updated: 2026-09-30
Code: https://github.com/karpathy/nanoGPT
Sources
- Unveiling the Basin-Like Loss Landscape in Large Language Models
- Adaptive Gradient Methods at the Edge of Stability
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Training Compute-Optimal Large Language Models
- MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
- Three Factors Influencing Minima in SGD
- Scaling Laws for Neural Language Models
- Adam: A Method for Stochastic Optimization
- Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methods
- AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods
- MiniMax-01: Scaling Foundation Models with Lightning Attention
- DeepSeek-V3 Technical Report
- An Empirical Model of Large-Batch Training
- Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training
- AdaBatchGrad: Combining Adaptive Batch Size and Adaptive Step Size
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theory
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective
- OPT: Open Pre-trained Transformer Language Models
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks