Load Balancing with Evolving Workloads in Large-Scale Language Model Serving
The research proposes a novel load-balancing principle called BFIO for LLM serving systems that use synchronization barriers. It addresses wasted computational power by optimizing worker assignments based on a short lookahead prediction of future workload imbalance at each step. The method provides strong theoretical guarantees, proving it significantly reduces long-run imbalance and offers substantial energy savings compared to simpler scheduling methods like First-Come, First-Served.