Sonar-MoE: In-Backward Expert Updates for Memory-Efficient Mixture-of-Experts Training
cs.LG, cs.AI, cs.CL, cs.PF
Submitted: 2026-11-02
Updated: 2026-11-05
Code: https://github.com/ibm-granite/granite-3.0-language-models
Terminology
Sources
- Training Deep Nets with Sublinear Memory Cost
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
- DeepSeek-V3 Technical Report
- 8-bit Optimizers via Block-wise Quantization
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations
- Measuring Massive Multitask Language Understanding
- LoRA: Low-Rank Adaptation of Large Language Models
- Tutel: Adaptive Mixture-of-Experts at Scale
- Mixtral of Experts
- FORGE: Fused On-Register Gradient Elimination for Memory-Efficient LLM Training
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- Memory Efficient Mixed-Precision Optimizers
- PyTorch Distributed: Experiences on Accelerating Data Parallel Training
- Sparse Backpropagation for MoE Training
- RevFFN: Memory-Efficient Full-Parameter Fine-Tuning of Mixture-of-Experts LLMs with Reversible Blocks
- BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models
- AdaLomo: Low-memory Optimization with Adaptive Learning Rate
- OLMoE: Open Mixture-of-Experts Language Models
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis