Mind the Delay: Latency-aware Query Routing for Heterogeneous LLM pools
The paper introduces a latency-aware routing framework that optimizes query assignment to different Large Language Models by jointly considering accuracy, cost, and time-to-first-token (TTFT) latency. It uses a Serving Framework Simulation (SFS) estimator to predict TTFT based on the query's workload at each model instance. This allows the router to select the best model that balances these three factors, significantly improving response utility.