Beyond Token Savings: A Systematic Study of Context Compression in LLM Agents
cs.LG, cs.AI
Submitted: 2026-09-26
Updated: 2026-09-26
Code: https://github.com/google-gemini/gemini-cli
License: http://creativecommons.org/licenses/by/4.0/
The gist: As LLM agents tackle longer tasks, they increasingly compress growing histories of reasoning, actions, and tool outputs.
Terminology
Abstract
As LLM agents tackle longer tasks, they increasingly compress growing histories of reasoning, actions, and tool outputs. Compression can reduce token use, but it also changes the information available for later decisions. Existing agentic harnesses bundle decisions about what to compress, when to compress, and how much to remove into fixed policies. A systematic characterization is needed to disentangle these decisions and reveal how each affects task success and execution cost. We systematically vary these decisions across three open-weight models on SWE-bench Verified and Terminal-Bench 1.0. Across nearly 35,000 agent runs, we measure task success, token use, end-to-end latency, and estimated cost. We find that fewer tokens need not mean faster or cheaper execution: on Terminal-Bench with Qwen, policies using roughly one-third as many tokens can take 20-80% longer than the uncompressed agent. Policies with similar overall success can solve different tasks, while the same policy can perform quite differently across models. Our results motivate evaluating compression by its effects on agent execution and tailoring policies to the task, model, and workload.
Sources
- CoACT: Action-Preserving Observation Compression for Coding Agents
- Context Length Alone Hurts LLM Performance Despite Perfect Retrieval
- ACON: Optimizing Context Compression for Long-horizon LLM Agents
- Characterizing How Complex Agentic AI Systems Handle General Tasks: A Trace-Based Simulation Study
- The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspective
- Rethinking AI Cloud Infrastructure for Agentic Serving Systems with the Aries Experimentation Framework
- The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management
- Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale
- LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
- Towards Reliable Context Compression for Long-Horizon Agents: An Empirical Study of Execution Instability
- Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
- Towards Understanding, Analyzing, and Optimizing Agentic AI Execution: A CPU-Centric Perspective
- Preble: Efficient Distributed Prompt Scheduling for LLM Serving
- How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions
- TokenPilot: Cache-Efficient Context Management for LLM Agents
- Architectural Implications of Agentic AI Workflows
- KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving
- Agentic AI Workload Characteristics
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks