Ostinato: Fast CPU-GPU MoE Inference for Local Agents via Resident Experts in Every Forward Pass
cs.DC, cs.LG
Submitted: 2026-11-08
Updated: 2026-11-08
Code: https://github.com/ikawrakow/ik_llama
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs
- DeepSeek-V3 Technical Report
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference
- MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache
- Qwen3 Technical Report
- PDScope: Breaking the Prefill-Decode Seesaw for Memory-Bound MoE Inference
- HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference
- TraceLab: Characterizing Coding Agent Workloads for LLM Serving
Related papers
- Programmable Probabilistic Computer with 1,000,000 p-bits
- BatchGen: An Architecture for Scalable and Efficient Batch Inference
- Seamless Reconfiguration for DAG-BFT (Technical Report)
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- TIRx: A Unified Tile Primitive ML Compiler Across Heterogeneous Backends