Context Language Models
cs.AI, cs.CL, cs.LG
Submitted: 2026-09-29
Updated: 2026-09-29
Code: https://github.com/facebookresearch/context-language-models
Project page: https://autocompact.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: We introduce Context Language Models (CLMs), language models that natively manage their own context.
Terminology
Abstract
We introduce Context Language Models (CLMs), language models that natively manage their own context. We implement this by treating the context as a file and allowing the model to make unrestricted updates to this file. This allows the model to learn what is most important to maintain in context, and naturally extends to multi-agent systems where multiple agent contexts coexist as files. Building CLMs zero-shot with existing models outperforms SOTA context management strategies across a variety of tasks: 11.4% higher accuracy with 21.5% fewer FLOPs on BrowseComp-Plus, 5% higher scores with 59% fewer FLOPs on 12-hour EdgeBench, and 65% greater improvement with the same compute on a 24-hour multi-repository agent-swarm task. Moreover, by shifting context management from external harness control to intrinsic model behavior, CLMs naturally enable both in-context and parametric learning of context-management strategies. We show that CLMs can be steered with natural-language instructions evolved through a standard skill-optimization loop, improving held-out accuracy by up to 35.9 points on a context-management task while reducing compute. We also introduce an online reinforcement learning method for CLMs, improving Qwen3.5-9B performance on BrowseComp-Plus by 47.6% while using 12% fewer FLOPs. Finally, we co-design Suffix Cache Reuse for CLM serving, further reducing server-side compute by 35% relative to standard SGLang at matched performance.
Sources
- BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent
- MEMENTO: Teaching LLMs to Manage Their Own Context
- Meta-Harness: End-to-End Optimization of Model Harnesses
- Self-Compacting Language Model Agents
- ACM: Agentic Context Management for Long Horizon Tasks
- CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents
- Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
- AlphaEvolve: A coding agent for scientific and algorithmic discovery
- MemGPT: Towards LLMs as Operating Systems
- Think Dense, Not Long: Dynamic Decoupled Conditional Advantage for Efficient Reasoning
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Scaling Long-Horizon LLM Agent via Context-Folding
- AutoMem: Automated Learning of Memory as a Cognitive Skill
- ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarization
- SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
- Meta Context Engineering via Agentic Skill Evolution
- AgentFold: Long-Horizon Web Agents with Proactive Context Management
- Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agents
- Recursive Language Models
- Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems