Dynamic Mixed-Precision Routing for Efficient Multi-step LLM Interaction
cs.AI
Submitted: 2026-02-02
Updated: 2026-11-10
Comments: 17 pages
Code: https://github.com/vllm-project/llm-compressor
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language models (LLMs) achieve strong performance in long-horizon agentic tasks through multi-step interaction and reasoning at test time.
Terminology
Abstract
Large language models (LLMs) achieve strong performance in long-horizon agentic tasks through multi-step interaction and reasoning at test time. However, repeatedly invoking these models incurs substantial inference cost over an episode. Post-training quantization (PTQ) reduces this cost, but using a fixed low precision throughout an episode can sharply lower task success. We observe that interaction steps differ sharply in their sensitivity to quantization: given the same context, decoding a step at low precision turns a success into a failure at only about 1% of the steps, which we call critical steps. Based on this observation, we propose Dynamic Mixed-Precision Routing (DMR), a framework that adaptively selects between high-precision and low-precision versions of the same LLM at each decision step. Leveraging an entire-space factorization to jointly predict a step's divergence and its effect on the outcome, DMR learns to recognize these critical steps by training only a lightweight router on the model's hidden state. By reusing a shared high-precision KV cache and the hidden states, DMR avoids redundant context prefill and a separate text encoder, enabling precision switching with less than 1 ms of overhead per step and adding less than 0.11% to step latency for routing. This mixed-precision design also enables nested quantization, reducing the memory footprint by 39% on average compared with FP16. Across three long-horizon agentic benchmarks with nested quantization, DMR achieves an average 2.44x decode speedup over FP16 at comparable task success rates, without fine-tuning the underlying model. On ALFWorld with GPTQ, DMR achieves a 1.71x decode speedup with 86.9% success, compared with 87.2% for FP16.
Sources
- Universal Model Routing for Efficient LLM Inference
- WebResearcher: Unleashing unbounded reasoning capability in Long-Horizon Agents
- DeepAgent: A General Reasoning Agent with Scalable Toolsets
- CATP-LLM: Empowering Large Language Models for Cost-Aware Tool Planning
- Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learning
- Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents