Greedy Local Learning for Language Model Pretraining: Gaps and Objective Design
cs.LG, cs.AI
Submitted: 2026-10-04
Updated: 2026-10-04
Comments: Accepted at the NeurIPS 2026 Workshop on Collaborative, Open, and Decentralized Training of Foundation Models (CODEC-FM). 13 pages, 3 figures, 8 tables
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Sources
- Backward Feature Correction: How Deep Learning Performs Deep (Hierarchical) Learning
- DeepSeek-V3 Technical Report
- DiLoCo: Distributed Low-Communication Training of Language Models
- DiPaCo: Distributed Path Composition
- OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training
- INTELLECT-1 Technical Report
- Parallel Training of Deep Networks with Local Updates
- DeMo: Decoupled Momentum Optimization
- Rethinking Local Learning: A Cheaper and Faster Recipe for LLM Post-Training
- LLaMA: Open and Efficient Foundation Language Models
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Representation Learning with Contrastive Predictive Coding
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks