Cascadia: A Control-Plane-Free Alternative to Hyperconverged AI Infrastructure
cs.DC, cs.AI
Submitted: 2026-09-30
Updated: 2026-09-30
Comments: 26 pages
Code: https://github.com/labscommunity/pipeline-sharded-inference-paper
Project page: https://ibm.github.io/storage-fusion/watsonx/overview
License: http://creativecommons.org/licenses/by/4.0/
The gist: We present Cascadia, a system for serving large language models on fleets of commodity Intel AIPCs using their CPU, integrated-GPU, and NPU resources.
Abstract
We present Cascadia, a system for serving large language models on fleets of commodity Intel AIPCs using their CPU, integrated-GPU, and NPU resources. Every node embeds ingress, scheduling, and execution; inference requests require no dedicated routing control plane. Nodes join a libp2p QUIC mesh using CA-issued ed25519 admission certificates, gossip signed capabilities, exchange live load over direct peer streams, and route OpenAI-compatible requests to eligible peers. An operator-run certificate authority handles admission and fleet management outside the inference path. Three serving modes share one interface: whole-model execution on one node, load-balanced replicas, and pipeline-sharded chains using the compilation and speculative decoding mechanism of our companion paper. Optional KV-cache mobility reuses compatible conversation prefixes after a routing move, with cold recomputation on a miss. Signed response receipts and hash-chained logs support provenance and audit. A three-node Phi-3.5-mini NPU testbed delivered 3.10x the response throughput of its one-node configuration under ten concurrent requests; a separate four-node deployment recorded 4.06x the throughput of direct single-node serving. Paired latency observations, runtime measurements, and internal functional checks characterize the tested configurations. We compare Cascadia with IBM, Nutanix, VMware, and HPE platforms on deployment footprint, hardware requirements, scheduling, scaling, licensing, and trust, using vendor documentation. The paper repository provides benchmark scripts, curated measurements, and a claim-to-evidence map.
Sources
- Petals: Collaborative Inference and Fine-tuning of Large Models
- CHESS: Optimizing LLM Inference via Channel-Wise Thresholding and Selective Sparsification
- Efficient Memory Management for Large Language Model Serving with PagedAttention
- TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices
- Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters
- Model-Distributed Inference for Large Language Models at the Edge
- Splitwise: Efficient generative LLM inference using phase splitting
- Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
- PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU
- SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment
- Parallax: Efficient LLM Inference Service over Decentralized Environment
- Faster MoE LLM Inference for Extremely Large Models
- TD-Pipe: Temporally-Disaggregated Pipeline Parallelism Architecture for High-Throughput LLM Inference
- DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
Related papers
- iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
- SAMM: Sharded Automated Market Maker
- InferScale: GPU-Native KV Injection for Personalized LLM Serving
- Vigil: Accountable Liveness against Selective Silence
- Steelhead: Interleaving Partially Synchronous and Asynchronous Commit Rules on a Shared DAG
- Democratizing MoE inference on commodity GPUs with CoMoE