Learning by Doing: Autonomously Compiling and Curating Agent Skills from Verified Traces
cs.AI, cs.CL
Submitted: 2026-11-01
Updated: 2026-11-01
Comments: 25 pages, 9 figures, 14 tables
Code: https://github.com/agentskills/agentskills
License: http://creativecommons.org/licenses/by/4.0/
The gist: Memory systems for production LLM agents mostly store semantic memory (facts about users and domains) or episodic memory (earlier interactions).
Terminology
Abstract
Memory systems for production LLM agents mostly store semantic memory (facts about users and domains) or episodic memory (earlier interactions). Research systems such as Trace2Skill, WikiSkill, and SkillRevise learn procedures from agent runs, package them as Skills, and evaluate them on benchmarks. We describe Zep Agent Skills, a production service that gives deployed agents this procedural memory. The service uses only verified attempts as evidence, cites the supporting events for each field of a Skill, and governs the admission and distribution of each Skill version. The service records each task attempt as a Trace with ordered events, an outcome, and a verification strength. A compiler with separate Reflector and Curator roles turns verified Traces into versioned Skills. Scans and admission gates control which candidates become active, and verified use outcomes change the search rank of each Skill. The Agent Skills format is an open standard that Anthropic originated and donated to the Agentic AI Foundation, so any host of the format can load a compiled Skill. Agents retrieve Skills through lexical search with progressive disclosure, over an API or MCP. We evaluate Zep Agent Skills on τ cubed-bench banking, AppWorld, and EnterpriseOps-Gym. Skills compiled from the Traces of a teacher model increase the pass rate of three weaker executor models on the same tasks, at every reasoning effort, by 4.5 to 18.8 points. Most of the gain is on the tasks that the teacher solved. On ALFWorld, Skills compiled from the runs of a small model give a gain similar to or larger than the gain that prior work reports for continuous fine-tuning, with no training.
Sources
- Voyager: An Open-Ended Embodied Agent with Large Language Models
- Agent Workflow Memory
- ExpeL: LLM Agents Are Experiential Learners
- Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models
- Cognitive Architectures for Language Agents
- MemGPT: Towards LLMs as Operating Systems
- Zep: A Temporal Knowledge Graph Architecture for Agent Memory
- Reflexion: Language Agents with Verbal Reinforcement Learning
- Memp: Exploring Agent Procedural Memory
- LEGOMem: Modular Procedural Memory for Multi-agent LLM Systems for Workflow Automation
- Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents
- AutoSkill: Experience-Driven Lifelong Learning via Skill Self-Evolution
- Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills
- Dynamic Agent Skills: A Lifecycle Survey and Taxonomy of Evolving Skill Libraries
- Anything2Skill: Compiling External Knowledge into Reusable Skills for Agents
- SkillRevise: Improving LLM-Authored Agent Skills via Trace-Conditioned Skill Revision
- SkillLift: Learning Dense Rubrics from Sparse Oracles for Efficient Skill Evolution
- WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution
- Procedural Graphs: Self-Evolving Execution Structures for LLM Agents
- Self-Evolving Agents through Privileged Retrospective Distillation
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents