Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading
MoE-SpeQ overcomes I/O bottlenecks in Mixture-of-Experts models by using a small, quantized draft model to predict future expert needs and proactively prefetch them. This co-design of speculative execution and offloading significantly reduces generation time, achieving up to 2.34x speedup over existing methods while reducing memory footprint by 43% on commodity hardware.