APEX-EM: Non-Parametric Online Learning for Autonomous Agents via Structured Procedural-Episodic Experience Replay
cs.CL, cs.AI, cs.IR
Submitted: 2026-03-31
Updated: 2026-08-31
Comments: 22 pages, 6 figures, Accepted to EMNLP 2026
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: LLM agents rerun full reasoning for every task, even one they solved moments earlier.
Terminology
Abstract
LLM agents rerun full reasoning for every task, even one they solved moments earlier. We introduce APEX-EM, a non-parametric experience memory that stores complete procedural-episodic traces in a typed Procedural Knowledge Graph (PKG) and retrieves them through three channels: semantic search, structural-signature matching over abstract operation sequences, and graph traversal. A Plan-Retrieve-Generate-Iterate-Ingest (PRGII) workflow produces, quality-gates, and commits experiences, indexing both successes and failures so the agent learns what to reuse and what to avoid. No weights change during deployment. We evaluate on five benchmarks: BigCodeBench, KGQAGen-10k, HLE, Lifelong Agent Bench, and ALFWorld. Because prior work uses different backbones, we base our claims on same-backbone comparisons that hold model capability fixed. On held-out BigCodeBench transfer with a shared GPT-4o backbone, APEX-EM gains +7.6,pp over the no-memory baseline, 3.3 times MemRL's +2.3,pp under the identical setup. On Lifelong Agent Bench with a shared GPT-4o-mini backbone, it gains +1.4,pp (OS) and +1.0,pp (DB) cumulative success. On KGQAGen-10k, frozen memory transfers to a blind 1, 079-question test split at 73.7% versus 42.0% with no memory, approaching an oracle handed the ground-truth subgraph (84.9%). Across three Opus scales the memory gain stays at +27 to +32,pp, so it adds to model capability rather than substituting for it. Component analysis shows no single mechanism dominates: teacher feedback is negligible for code but adds +10.3,pp on structured queries, structural signatures give 3.3 times the transfer of semantic-only retrieval, and within-epoch iteration recovers most of the gain when rich feedback is unavailable. These results argue for modular memory composed per domain.
Sources
- Learning to Continually Learn via Meta-learning Agentic Memory Designs
- Hindsight Experience Replay
- Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory
- Uncertainty quantification for stationary and time-dependent PDEs subject to Gevrey regular random domain deformations
- MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agents
- AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
- MemGPT: Towards LLMs as Operating Systems
- Humanity's Last Exam
- Zep: A Temporal Knowledge Graph Architecture for Agent Memory
- SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning
- MIRIX: Multi-Agent Memory System for LLM-Based Agents
- A-MEM: Agentic Memory for LLM Agents
- Diagnosing and Addressing Pitfalls in KG-RAG Datasets: Toward More Reliable Benchmarking
- The doubly librating Plutinos
- Unbiased surveys of dust-enshrouded galaxies using ALMA
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering