M: Every Task Deserves Its Own Memory Harness
summary
The gist
Large Language Model (LLM) agents rely on a memory harness to reuse past experience across diverse tasks because different tasks require different storage and retrieval behaviors.
In short
LLM agents need different memory systems for different tasks, but existing ones are fixed. MSTAR automatically discovers task-optimized memory programs by treating them as executable Python code with Schema, Logic, and Instruction components. This evolutionary search finds unique memory designs for conversation, planning, and reasoning tasks.
Key concepts
- Memory Harness
- A system that allows a Large Language Model agent to store, retrieve, organize, and use past experiences. The paper argues that a single harness is often inadequate because different tasks require fundamentally different ways of storing and retrieving information.
- MSTAR
- An evolutionary method that automatically discovers the best memory harnesses. It represents the memory as executable Python code (Schema, Logic, Instruction) and uses reflective agents and population-based search to iteratively improve these programs for specific tasks.
- Memory Program Search Space
- The entire design of the memory harness is treated as a program written in Python. This space includes defining data formats (Schema), backend operations (Logic), and interaction methods (Instruction). MSTAR searches this space to find the optimal combination for any given task.
- Task-Specific Optimization
- The results show that the best memory programs are structurally different for every domain, such as conversation versus embodied planning. This means every task benefits from its own custom memory harness discovered through the search process.
Terminology used across episodes
This episode discusses
- M: Every Task Deserves Its Own Memory Harness · Paper Radio
- GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning
- PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning
- HealthBench: Evaluating Large Language Models Towards Improved Human Health
- FLEX: Continuous Agent Evolution via Forward Learning from Experience
- Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory
- A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence
- AlphaEvolve: A coding agent for scientific and algorithmic discovery
- ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory
- ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
- Evaluating Memory Structure in LLM Agents · Paper Radio
- Learning to Continually Learn via Meta-learning Agentic Memory Designs
- MemEvolve: Meta-Evolution of Agent Memory Systems
- MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agents · Paper Radio
The paper
M: Every Task Deserves Its Own Memory Harness · Read on arXiv
Wenbo Pan, Shujie Liu, Xiangyang Zhou, Shiwei Zhang, Wanlu Shi
City University of Hong Kong · Microsoft
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "M: Every Task Deserves Its Own Memory Harness".
Jane: Here's the gist of the paper. Large Language Model (LLM) agents rely on a memory harness to reuse past experience across diverse tasks because different tasks require different storage and retrieval behaviors.
Tom: The paper introduces MSTAR, an evolutionary method that automatically discovers task-optimized memory harnesses through reflective code evolution.
Jane: MSTAR Methodology MSTAR represents a memory harness as an executable Python program with a fixed interface consisting of Schema, Logic, and Instruction components that are jointly optimized.
Tom: This approach allows the system to search over the memory harness itself with two mechanisms: a reflector agent that debugs failures and generates improvements, and a population-based program search strategy.
Paper summary: Tom: So we're wrapping up this look at MSTAR, which is essentially this method that lets an AI agent find its own specific memory program for a given task instead of using one fixed design everywhere.
Lu: The paper, "M: Every Task Deserves Its Own Memory Harness," shows that this evolutionary search finds programs that are structurally different for conversation versus embodied planning versus reasoning tasks.
Meng: I'm looking at the conclusion now, and it really boils down to how we should think about memory design in general.
Tom: They argue that instead of picking a single universal memory setup, every task deserves its own tailored memory harness <M: Every Task Deserves Its Own Memory Harness>.
Jane: That’s the big picture—it suggests that when you build an agent, you can't just assume one structure will fit everything; you need to be ready to evolve the memory harness for each new job.
Lu: The authors make a strong case for this by showing that this discovery process makes memory design more transparent because it’s represented as executable code.
Meng: It moves memory from being just a configuration file you pick, to something dynamically discovered by the AI itself.
Tom: That shift means the system is actually learning how to build its own specialized toolset for every different kind of work it gets assigned.
Jane: It’s a big idea for how we approach building these complex agents in the future.
Lu: And what's really interesting is that they also laid out some clear risks, like the possibility that this discovery could lead to very specific rules encoded in the memory, which limits its ability to transfer to other tasks.
Tom: So it’s a trade-off: you get super tailored performance for one job, but you have to be cautious about generalizability across different kinds of jobs.
Jane: And that's exactly what MSTAR is doing by providing this concrete way to discover those necessary structures on the fly.
Lu: We'll talk more later about how this impacts the broader idea of making memory design more transparent through executable code and how researchers can use these evolved programs to understand agent behavior better.
Conclusion: Tom: So we're wrapping up this look at MSTAR, which is this method that lets an AI agent find its own specific memory program for a given task instead of using one fixed design everywhere.
Jane: Right, it’s about letting the system discover the best way to store and retrieve information based on what the job actually requires.
Lu: The paper, "M: Every Task Deserves Its Own Memory Harness," shows that this evolutionary search finds programs that are structurally different for conversation versus embodied planning versus reasoning tasks twenty-two twenty-three.
Meng: I'm looking at the conclusion now, and it really boils down to how we should think about memory design in general.
Tom: They argue that instead of picking a single universal memory setup, every task deserves its own tailored memory harness.
Jane: That’s the big picture—it suggests that when you build an agent, you can't just assume one structure will fit everything; you need to be ready to evolve the memory harness for each new job.
Lu: The authors make a strong case for this by showing that this discovery process makes memory design more transparent because it’s represented as executable code.
Meng: It moves memory from being just a configuration file you pick, to something dynamically discovered by the AI itself.
Tom: That shift means the system is actually learning how to build its own specialized toolset for every different kind of work it gets assigned.
Jane: It’s a big idea for how we approach building these complex agents in the future.
Lu: And what's really interesting is that they also laid out some clear risks, like the possibility that this discovery could lead to very specific rules encoded in the memory, which limits its ability to transfer to other tasks.
Tom: So it’s a trade-off: you get better task-specific performance for one job but you have to deal with that computational overhead during the search phase.
Jane: And they also flagged a risk where the discovered programs can drift toward very specific rules that might make them really good for one task but terrible when you try to apply them to something new.
Lu: That’s exactly what they pointed out, so it’s not a perfect transfer mechanism yet; it's discovery rather than guaranteed generalization.
Meng: So the authors are suggesting that memory design should be treated as a search problem rather than just picking a configuration file.
Tom: It shifts the focus from designing one perfect memory system to having an AI agent actively evolve its own memory structure for every new job it gets.
Jane: And that’s what MSTAR is doing, trying to make that discovery process work reliably across different types of tasks.
Lu: We’ll talk more later about how this impacts the broader idea of making memory design more transparent through executable code and how researchers can use these evolved programs to understand agent behavior better.
More episodes
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines
- 2601.15318-On the closest balanced game