CompSkill: Compiling Execution Experience for Skill Retrieval
cs.CL
Submitted: 2026-10-31
Updated: 2026-10-31
Comments: 33 pages. Code: https://github.com/satoshinji2992/CompSkill
Code: https://github.com/satoshinji2992/CompSkill
License: http://creativecommons.org/licenses/by/4.0/
The gist: LLM agents increasingly rely on skill documents, yet most retrievers rank them by description alone and ignore what earlier executions revealed, even though a relevant skill can hurt execution.
Terminology
Abstract
LLM agents increasingly rely on skill documents, yet most retrievers rank them by description alone and ignore what earlier executions revealed, even though a relevant skill can hurt execution. We present CompSkill, which indexes skills with learned 512-d keys and compiles matched same-task comparisons into skill offsets and capped key residuals (KeyOpt), so experience changes which document the executor receives, never its text. Under a logistic outcome model with unknown task difficulty, matched comparisons cancel that difficulty and only discordant ones carry information; on support executions, predictions improve with each discordant task. An experience hypergraph keeps each comparison with its executor and document versions, recalls missed skills and grounds a cited review. Compact keys raise SkillRet NDCG@10 from 64.0 to 70.9 at one eighth of the dimension. With one document, CompSkill completes 256 and 260 of 332 ToolQA tasks with GPT-5.6-luna and DeepSeek-V4.1-Flash, respectively, more than every baseline, and stays on par with the strongest baselines on AppWorld and ALFWorld. On 332 newly generated ToolQA tasks, it completes 256 with GPT-5.6-luna, significantly more than BM25, SkillRet-8B, Tool2Vec and RankGPT; over all 844 ToolQA tasks, it completes 24 more than the strongest baseline. Selection matters beyond relevance: pairing the right guide with a same-database sibling instead of another database's guide costs 8 of 96 tasks. Code, models and support data: https://github.com/satoshinji2992/CompSkill
Sources
- Don't Offer What Can't Be Done: Deterministic Executability Gating for LLM Skill Selection at Scale
- A Systematic Survey of Agentic Skills: Architecture, Lifecycle, and Security
- The Router Within: Eliciting Native Skill Routing from a Frozen LLM
- Task Decomposition-Guided Reranking for Adaptive Agent Skill Retrieval
- From Local to Global: A Graph RAG Approach to Query-Focused Summarization
- SE-GoS: Self-Evolving Graph-of-Skills for Skill Library at Scale
- SelfEvoSkill: Paired Execution Audits for Skill Revision without External Outcome Supervision
- Compositional Skill Routing for LLM Agents: Decompose, Retrieve, and Compose
- Field-Aware Agent Skill Retrieval
- From Relevance to Execution Utility: Reward-Aware Dynamic Execution Gating for Skill-Based LLM Agents
- SkillCommit: Evolving Agent Skills through Behaviorally Validated Scope Expansion
- SkillReason: Reasoning-Enhanced Agent Skill Retrieval for Implicit User Requests
- When Retrieval Partially Conflicts with Skills: Understanding Procedural Hallucination in Language Models
- SoK: Agentic Skills -- Beyond Tool Use in LLM Agents
- SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agents
- Comparative Approaches to Agent Retrieval over Large Skill Libraries
- Organizing, Orchestrating, and Benchmarking Agent Skills at Ecosystem Scale
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data