Lost in Context Compaction: When LLM Systems Drop Side Constraints

arXiv:2608.11242 · cs.CL, cs.AI · Submitted 2026-07-31 · Read on arXiv

cs.CL, cs.AI

Submitted: 2026-07-31

Updated: 2026-10-11

Code: https://github.com/ZhiqiEliWang/compaction-integrity

License: http://creativecommons.org/licenses/by/4.0/

The gist: When the context window is under pressure, LLM systems compact prior context to continue ongoing tasks.

Terminology

Abstract

When the context window is under pressure, LLM systems compact prior context to continue ongoing tasks. We identify a class of user-issued instructions, Session Constraints (SCs), such as "do not delete any emails until I confirm," that are meant to constrain LLM's behavior for the remainder of a session but are silently dropped during compaction. To quantify this loss, we introduce COMPINT, an evaluation suite that evaluates compactors across three long-context scenarios: multi-turn chat, agentic trajectory, and long-horizon research. Current compactors retain only 17% of injected SCs on average, and most perform worse than running the same task without compaction. Retention varies sharply with compactor, prompt, context length, SC phrasing, and injection location, showing that the loss is systematic rather than tied to any single setting. We propose an SC-aware extractor that runs alongside the compactor as a plug-and-play module, achieving over 90% retention across all three scenarios without modifying the compactor or LLM. The COMPINT evaluation suite and accompanying implementation are available at https://github.com/ZhiqiEliWang/compaction-integrity.

Sources

Related papers