MatMoE: Matryoshka Mixture-of-Experts with Dynamic Mixed-Precision Quantization for Efficient Inference
cs.LG, cs.AI
Submitted: 2026-11-01
Updated: 2026-11-01
Comments: Accepeted by ICCAD 2026
Code: https://github.com/ggml-org/llama.cpp
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Program Synthesis with Large Language Models
- Evaluating Large Language Models Trained on Code
- Training Verifiers to Solve Math Word Problems
- DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
- MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- EAQuant: Enhancing Post-Training Quantization for MoE Models via Expert-Aware Optimization
- MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance
- MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensators
- Mixtral of Experts
- SqueezeLLM: Dense-and-Sparse Quantization
- DP-LLM: Runtime Model Adaptation with Dynamic Layer-wise Precision Assignment
- QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Experts
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- DeepSeek-V3 Technical Report
- A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications
- OLMoE: Open Mixture-of-Experts Language Models
- Matryoshka Quantization
- Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs
- OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis