The Routing Plateau: Understanding the Accuracy Limits of LLM Routers
Many LLM routing methods hit a performance ceiling far below an ideal oracle router because they learn broad, global patterns instead of specific instance-level correctness signals. The study found this 'routing plateau' is caused by a bottleneck where routers fail to predict which model will be correct for a given query. Scaling data and fine-tuning can close about 14% of this gap.