STAPLE: Static Sparse Upcycling of Pretrained Checkpoints via Embedding Modules
STAPLE introduces a method to integrate static sparse memory into pretrained models during upcycling, avoiding costly full pretraining. It modifies Mixture-of-Experts strategies by adding token-indexed embeddings as a residual pathway alongside existing computation. This approach allows models to learn complementary, token-specific knowledge efficiently while preserving the original pretrained knowledge.