MemoryWalker: Stop Training Agents on Contexts They Never Saw
cs.LG, cs.CL
Submitted: 2026-09-01
Updated: 2026-09-01
Comments: Your Memory-Compressing Harness Makes Training and Inference Inconsistent
Code: https://github.com/jiangxinke/Agentic-RAG-R1
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Production agent harnesses such as Claude Code and Qwen-Agent compress context during rollout, but training under compression creates a conditioning problem: every eviction branches the effective
Terminology
Abstract
Production agent harnesses such as Claude Code and Qwen-Agent compress context during rollout, but training under compression creates a conditioning problem: every eviction branches the effective history, so the learning object is a tree rather than a sequence. Existing linearizations either retain the rightmost path, causing time-travel leakage, or replay a depth-first traversal, causing train-inference mismatch. We introduce two exact, gradient-equivalent corrections: LogitTree, a segmented K-forward traversal, and a packed 4D attention mask. LogitTree requires K+1 backward passes; the 4D mask requires a custom kernel and white-box eviction records. We also propose SDCC (Self-Distillation for Conditioning Consistency), a single-backward-pass variational relaxation. At each eviction, it minimizes forward KL between the compressed student and a stop-gradient teacher on the reconstructed pre-eviction prefix. A residual per-junction KL of epsilon KL gives an O(sqrt(epsilon KL)) bound on the train-deployment total-variation gap. SDCC also applies to black-box harnesses. On seven web-search benchmarks with TC-RAG, AgentFold, MemexRL, Claude Code, and OpenCode, naive training inflates the train-rollout log-probability gap, especially on eviction-heavy batches. The exact methods stay at the no-compression floor, and SDCC substantially closes the gap, with lower logit drift and higher rollout rewards.
Sources
- Reweighted Wake-Sleep
- CIG: Measuring Conversational Information Gain in Deliberative Dialogues with Semantic Memory Dynamics
- REDSearcher: A Scalable and Cost-Efficient Framework for Long-Horizon Search Agents
- Understanding the planning of LLM agents: A survey
- Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
- Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation
- Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
- Lost in the Middle: How Language Models Use Long Contexts
- MemGPT: Towards LLMs as Operating Systems
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- Proximal Policy Optimization Algorithms
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Deep Research: A Systematic Survey
- AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities
- Memex(RL): Scaling Long-Horizon LLM Agents via Indexed Experience Memory
- WideSearch: Benchmarking Agentic Broad Info-Seeking
- A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challenges
- Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning
- AgentFold: Long-Horizon Web Agents with Proactive Context Management
- StackPlanner: A Centralized Hierarchical Multi-Agent System with Task-Experience Memory Management
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks