TrimMoE A communication aware and adaptive depth framework for distributed edge inference
Ning Li, Shuting Bai, Xin Yuan, Wenchao Xu, Song Guo, Haijun Zhang
cs.NI, cs.CL
Submitted: 2026-08-12
Updated: 2026-08-13
Comments: 17 pages, 11 figures
License: http://creativecommons.org/licenses/by/4.0/
The gist: Serving Mixture-of-Experts (MoE) large language models across distributed edge servers is bottlenecked by the cross-server expert transmission.
Terminology
Abstract
Serving Mixture-of-Experts (MoE) large language models across distributed edge servers is bottlenecked by the cross-server expert transmission. The existing approaches mainly focus on how to reach a remote expert faster. However, in this paper, we instead consider whether a given layer, and the layers after it, need to be executed at all. To this end, a communication-aware adaptive-depth framework is proposed in this paper, termed TrimMoE, which couples layer skipping and confidence-based early exit with substitute execution and server-expert selection under a unified quality budget. Specifically, in the offline stage, TrimMoE freezes the backbone, trains the lightweight per-layer exit heads, calibrates the per-layer importance thresholds, and allocates the expert replicas by a skip/exit-aware redundancy benefit. In the online stage, a transition-aware look-ahead anticipates the token movement, so that the depth reduction targets the costliest transmissions, and besides, two feedback rules adapt the delay-quality weights and the exit threshold. Moreover, we prove that the substitution-and-skipping proxy degradation never exceeds the configured budget, and that the early exit is admitted only under a calibrated confidence gate. On a heterogeneous 10-server testbed with Switch-Base-8E, Qwen-MoE-A2.7B, and Mixtral-8x7B, TrimMoE reduces the average latency by up to 62.8%, lowers the cross-server traffic and the remote-execution ratio, and sustains high throughput under load, while keeping the task-quality degradation within a 2% bound.
Sources
- A Survey on Efficient Inference for Large Language Models
- Mixtral of Experts
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Mixture-of-Depths: Dynamically allocating compute in transformer-based language models
- PROBE: Co-Balancing Computation and Communication in MoE Inference via Real-Time Predictive Prefetching
- OrderMoE: An expert similarity driven distributed edge MoE inference
Related papers
- HiFiNet: Hierarchical Fault Identification in Wireless Sensor Networks via Edge-Based Classification and Graph Aggregation
- PACC: Protocol-Aware Cross-Layer Compression for Compact Network Traffic Representation
- TraceCodec: A Compiler-Backed Neural Codec for Network Traffic Traces
- Embodied AI in 6G Networks: From Intelligent Connectivity to Physical Intelligence
- Lightweight GenAI for Network Traffic Generation: Fidelity, Augmentation, and Classification
- NVIDIA Spectrum-X: Hardware-Accelerated Multiplane Networking for Large-Scale AI Training