Accelerating LLM Pretraining with Zeroth-Order Curvature on Idle Serving Compute
cs.LG, cs.AI
Submitted: 2026-11-02
Updated: 2026-11-16
Project page: http://skylion007.github.io/OpenWebTextCorpus
Terminology
Sources
- Scalable Second Order Optimization for Deep Learning
- Gradients without Backpropagation
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- DiLoCo: Distributed Low-Communication Training of Language Models
- M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity
- Training Compute-Optimal Large Language Models
- Scaling Laws for Neural Language Models
- $100K or 100 Days: Trade-offs when Pre-Training with Academic Resources
- Adam: A Method for Stochastic Optimization
- Valve: Production Online-Offline Inference Colocation with Jointly-Bounded Preemption Latency and Rate
- Muon is Scalable for LLM Training
- Decoupled Weight Decay Regularization
- Scaling Forward Gradient With Local Losses
- OPT: Open Pre-trained Transformer Language Models
- Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis