SOLO: Pretraining Billion-Parameter Language Models with Shared-Output Local Learning
cs.LG, cs.DC
Submitted: 2026-09-28
Updated: 2026-09-28
Sources
- Deep Learning without Weight Transport
- Gradients without Backpropagation
- Eliciting Latent Predictions from Transformers with the Tuned Lens
- Training Deep Nets with Sublinear Memory Cost
- Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
- DiLoCo: Distributed Low-Communication Training of Language Models
- Depth-Adaptive Transformer
- LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
- Interlocking Backpropagation: Improving depthwise model-parallelism
- Faster Multi-GPU Training with PPLL: A Pipeline Parallelism Framework Leveraging Local Learning
- The Forward-Forward Algorithm: Some Preliminary Investigations
- Fix your classifier: the marginal value of training the last weight layer
- Decoupled Parallel Backpropagation with Convergence Guarantee
- Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similarity
- Parallel Training of Deep Networks with Local Updates
- Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures
- Putting An End to End-to-End: Gradient-Isolated Learning of Representations
- Pointer Sentinel Mixture Models
- Deep supervised learning using local errors
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks