STAPLE: Static Sparse Upcycling of Pretrained Checkpoints via Embedding Modules
cs.LG
Submitted: 2026-10-08
Updated: 2026-10-08
Code: https://github.com/Infini-AI-Lab/STAPLE
Project page: https://infini-ai-lab.github.io/STAPLE
Sources
- Program Synthesis with Large Language Models
- Memory Layers at Scale
- Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Training Verifiers to Solve Math Word Problems
- MeKi: Memory-based Expert Knowledge Injection for Efficient LLM Scaling
- Improving Sparse Memory Finetuning
- Upcycling Large Language Models into Mixture of Experts
- Measuring Mathematical Problem Solving With the MATH Dataset
- Training Compute-Optimal Large Language Models
- Scaling Laws for Neural Language Models
- Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints
- Large Memory Layers with Product Keys
- Fast Inference from Transformers via Speculative Decoding
- DataComp-LM: In search of the next generation of training sets for language models
- Continual Learning via Sparse Memory Finetuning
- Scaling Embeddings Outperforms Scaling Experts in Language Models
- Dirichlet-Prior Shaping: Guiding Expert Specialization in Upcycled MoEs
- Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization
- 2 OLMo 2 Furious
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks