Astrolabe: Balancing Load in LLM Serving with Randomized Prediction-Guided Scheduling
The episode discusses the paper "Astrolabe: Balancing Load in LLM Serving with Randomized Prediction-Guided Scheduling" by Wei Da and Evangelia Kalyvianaki. The hosts explain how Astrolabe uses prediction models to estimate response lengths, simulates latency on candidate servers, and uses a randomized selection strategy to send requests, leading to significant improvements in speed and stability over traditional load balancing methods.