SteerDistill: Quantizing LLM Agents via Closed-Loop Teacher Steering
cs.AI, cs.LG
Submitted: 2026-11-10
Updated: 2026-11-10
Comments: Preprint
Code: https://github.com/air-embodied-brain/SteerDistill
Project page: https://air-embodied-brain.github.io/SteerDistill
License: http://creativecommons.org/licenses/by/4.0/
The gist: Low-bit quantization reduces the inference cost of LLM agents but can compromise task success.
Terminology
Abstract
Low-bit quantization reduces the inference cost of LLM agents but can compromise task success. Examining failed rollouts, we find that quantized agents overlook task requirements or environmental conditions, producing environment transitions that diverge from successful full-precision references. Correcting these behavioral divergences remains difficult: offline data may miss the student's failure contexts, while its own rollouts may provide few correct responses in those contexts. We introduce SteerDistill, a quantization-aware distillation method in which a full-precision teacher guides the student to revise failed responses and iteratively refines its guidance using feedback from student execution. This closed-loop teacher steering produces locally verified behavioral corrections. We internalize these corrections by training the student without guidance to match the steered student's predictions at corrected turns and the full-precision teacher's at other turns, enabling unassisted inference. Across five agentic benchmarks and three models, SteerDistill exceeds the strongest quantized baseline by 1.67--2.18 points in average score under W4A16 and stays within one point of BF16. On τ squared-Bench, it outperforms QeRL by 1.84 percentage points in success rate using 67% fewer training GPU hours and produces shorter successful trajectories than the quantized baselines.
Sources
- ReAct: Synergizing Reasoning and Acting in Language Models
- ScaleWoB: Guiding GUI Agents with Coding Agents via Large-Scale Environmental Synthesis
- Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edge
- EfficientQAT: Efficient Quantization-Aware Training for Large Language Models
- ReQAT: Achieving Full-Precision Reasoning Accuracy with 4-bit Floating-Point Quantization-Aware Training
- Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery
- QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs
- OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
- QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
- SpinQuant: LLM quantization with learned rotations
- TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents
- On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents
- EDGE-OPD: Internalizing Privileged Context with Evidence Guided On-Policy Distillation
- EDGE: Experience-Distillation for Guided Exploration in Agentic Reinforcement Learning
- Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillation
- From Scoring to Acting: Outcome-Verified Comparative Self-Distillation for LLM Agents
- On-Policy Context Distillation for Language Models
- Sample-Efficient Learning from Agent Experience
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents