When LLM Routers Overpay: The Cost of Strong-Model Preference
cs.AI
Submitted: 2026-02-03
Updated: 2026-11-05
Comments: Accepted to NeurIPS 2026
Code: https://github.com/AIGNLAI/EquiRouter
License: http://creativecommons.org/licenses/by/4.0/
The gist: LLM routing aims to reduce inference costs by assigning easy queries to less expensive models and reserving stronger models for cases in which they are needed.
Terminology
Abstract
LLM routing aims to reduce inference costs by assigning easy queries to less expensive models and reserving stronger models for cases in which they are needed. However, we find that existing routers often violate this principle under loose cost budgets: as the budget increases, they increasingly select the strongest and most expensive model even when less expensive feasible candidates provide comparable or identical utility. We refer to this behavior as strong-model over-selection under loose budgets. Our analysis shows that ambiguity alone does not explain this phenomenon. Substantial overpayment can arise when differently priced models offer equivalent or near-equivalent utility and the router retains a persistent model-level preference for costly models; budget relaxation exposes this preference by making premium candidates feasible. Motivated by this diagnosis, we propose EquiRouter, a decision-aligned router that learns query-dependent model rankings, preserves meaningful utility differences, and explicitly prefers lower-cost models among equivalent candidates through a Cost-Aware Ranking Objective and a lightweight Query--Model Interaction Representation. Experiments across diverse text, multimodal, continuous-utility, and large-model-pool settings show that EquiRouter consistently improves cost efficiency while maintaining competitive peak and whole-curve performance. Code is available at https://github.com/LAMDA-Model-Reuse/EquiRouter.
Sources
- What is the Role of Small Models in the LLM Era: A Survey
- Tryage: Real-time, intelligent Routing of User Prompts to Large Language Models
- RouterBench: A Benchmark for Multi-LLM Routing System
- MMR-Bench: A Comprehensive Benchmark for Multimodal LLM Routing
- Routoo: Learning to Route to Large Language Models Effectively
- Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systems
- MixLLM: Dynamic Routing in Mixed Large Language Models
- Leveraging Uncertainty Estimation for Efficient LLM Routing
- Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
- Eagle: Efficient Training-Free Router for Multi-LLM Inference
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents