SkillSentry: Reliable Skill Execution for LLM Agents via Runtime Assurance

SKILLSENTRY: Reliable Skill Execution for LLM Agents via Runtime Assurance

Abstract

LLM agents are increasingly equipped with skills to perform complex tasks through multi-step reasoning and tool use. Although skills provide reusable procedural knowledge, agents may still execute them unreliably. Even when an agent has demonstrated the capability to complete tasks under the guidance of a skill, it may fail to do so consistently across similar tasks or repeated runs due to deviations from the skill procedure or incorrect execution of individual steps. Such instability limits the practical reliability of LLM agents. To address this problem, we propose SKILLSENTRY, a skill-oriented runtime assurance framework built upon a new domain-specific language (DSL) for representing runtime guidance for skill execution. SKILLSENTRY initializes the runtime guidance by combining a skill specification extracted from the corresponding skill document with execution experience mined from historical successful and failed traces. It then wraps around the agent execution loop to monitor and guide skill execution under the current guidance, while iteratively refining the guidance using newly collected traces. We evaluate SKILLSENTRY on 15 skills across two LLM agents, each paired with two backbone models, i.e., Claude Code with Claude-Haiku-4.5 and Claude-Opus-4.6, and Codex with GPT-5.2 and GPT-5.4. Our results show that SKILLSENTRY improves the task success rate of LLM agents by 24.1% across skills, on average, while exhibiting lower variability across repeated runs.

Introduction

Large language models (LLMs) have shown strong capabilities in understanding user instructions, generating code, and solving complex reasoning questions. Building on these capabilities, LLM agents further extend LLMs from passive text generation to autonomous task execution, i.e., planning intermediate steps, invoking external tools, observing execution feedback, and refining their actions. As user tasks become more complex and diverse, skills have emerged as an important abstraction for packaging reusable task-solving knowledge. A skill usually describes how an agent should complete a class of tasks through predefined steps, including high-level procedures, tool-use patterns, and task-specific constraints. By providing such procedural guidance, skills help LLM agents reuse validated task-solving strategies, constrain the execution space, and improve their capability to complete complex tasks.

However, equipping an LLM agent with a skill does not guarantee reliable execution. Even when the LLM agent has demonstrated the capability to complete a task under the guidance of a skill before, it may fail to reproduce the correct execution consistently in later runs. Such failures do not necessarily indicate a lack of fundamental model capability for task-solving. Rather, they often arise from the generative nature of LLM agent execution, where each action is produced from an evolving context of prior reasoning, tool feedback, and intermediate observations. Thus, small variations during agent execution may lead to deviations from the skill procedure or incorrect execution of individual steps, resulting in task failures.

Existing approaches improve the reliability of LLM agent execution from different perspectives. Some studies focus on acquiring, generating, or selecting skills for better reusability. Some automatic prompt optimization and reflection-based approaches refine prompts, instructions, in-context memories, or skills based on execution feedback. A few studies improve the underlying model capability through supervised fine-tuning, reinforcement learning, or post-training on agent execution traces. These approaches mainly improve LLM agents by updating external artifacts or the underlying model parameters, thereby enhancing the task-solving capability before agent deployment. In contrast, some studies focus on runtime guardrails and enforcement systems, constraining agent behaviors during execution according to user-specified rules, safety policies, or domain constraints. They are effective for preventing unsafe or policy-violating actions, but a skill-based agent may still fail without violating explicit safety policies. Although prior approaches can improve LLM agent capabilities or constrain unsafe runtime behavior, it remains unclear how to improve the runtime reliability of skill execution when an agent already has the basic capability to complete the task but may still deviate from the skill procedure or execute individual steps incorrectly.

To bridge this gap, we propose SKILLSENTRY, a skill-oriented runtime assurance framework for improving the reliability of LLM agent execution. SKILLSENTRY does not aim to teach an LLM agent fundamentally new skills or compensate for tasks beyond the capability of the underlying model. Instead, it targets cases where an agent has demonstrated the capability to complete tasks under the guidance of a skill but may still execute the skill unreliably across similar tasks or repeated runs.

SKILLSENTRY is built upon a new domain-specific language (DSL) for representing runtime guidance for skill execution, and consists of an initialization stage and a self-evolving stage. In the initialization stage, it extracts a skill specification from each skill document, capturing the expected steps, step dependencies, constraints, and completion requirements. It also mines execution experience from historical successful and failed traces, including validated action patterns, failure-associated action patterns, and step-level suggestions and warnings. By combining the skill specification with the mined experience, SKILLSENTRY constructs the initial runtime guidance.

In the self-evolving stage, the agent executes new task queries under current runtime guidance through the step-aware runtime assurance mechanism. SKILLSENTRY monitors agent execution, delivers step-level suggestions and warnings to improve the execution of individual skill steps, intervenes upon detecting procedural deviations or failure-associated action patterns, and checks the completion of required steps before accepting the final output. The resulting traces are used to refine the execution experience, enabling SKILLSENTRY to iteratively improve the runtime guidance as new execution traces are collected.

We evaluate SKILLSENTRY on 15 skills across two widely used LLM agents, each paired with two backbone models, i.e., Claude Code with Claude-Haiku-4.5 and Claude-Opus-4.6, and Codex with GPT-5.2 and GPT-5.4. Our evaluation shows that SKILLSENTRY improves the average task success rate by 24.1% across skills, reduces the standard deviation (SD) of task success rates across repeated runs by 41.1%, and introduces little runtime overhead. Further self-evolving analysis shows that SKILLSENTRY can iteratively refine the runtime guidance for reliable skill execution. Our ablation study confirms the contribution of failure-associated action patterns, suggestions, and warnings in the runtime guidance to the final effectiveness. Step-level guidance delivery is more effective than placing all guidance in the system prompt, increasing the task success rate by 5.8% and reducing the SD by 26.8%. Finally, our generalization study shows that the constructed runtime guidance can transfer across different backbone models within the same agent to some extent.

The main contributions of this work are as follows:

• We design a domain-specific language for representing skill-oriented runtime guidance that combines a skill specification extracted from the skill document with execution experience mined from successful and failed agent traces.

• We propose SKILLSENTRY, a skill-oriented runtime assurance framework that wraps around the agent execution loop, monitors and guides skill execution, and iteratively refines its runtime guidance using newly collected execution traces.

• We implement a prototype of SKILLSENTRY, and conduct experiments to demonstrate its effectiveness and efficiency.

Background and Motivation

Background. An LLM agent completes tasks through iterative interactions with external environments. To guide multi-step planning and execution, skills have been introduced as reusable procedural knowledge for completing a class of tasks. A skill usually contains a name, a description specifying when it should be used, and a procedure describing the expected execution steps. An LLM agent executes a skill through the agent runtime. The runtime maintains the interaction context, presents the skill and available tools to the model, receives model-generated actions, executes tool calls, and returns observations to the model for the next decision. Therefore, the abstract procedure described in a skill is realized as a concrete runtime trace comprising multiple iterations of action planning, tool invocation, and execution feedback, followed by a final output. Modern agent systems commonly expose hooks as extension points in the runtime loop. Hooks can be triggered at different lifecycle points, such as before an action is executed, after execution feedback is returned, or before the agent terminates. These hooks allow an external module to observe the next action and provide additional information to the agent during execution. In this work, we use hooks to monitor and guide LLM agents' skill execution at runtime.

Motivation. Although skills provide procedural knowledge for task completion, runtime decisions depend on iterative reasoning and execution feedback, making the execution process sensitive to small variations in intermediate contexts. Thus, an agent may understand the overall skill but still deviate from the skill procedure or execute individual steps incorrectly. The paper illustrates this problem with two cases of Claude Code under the guidance of the skill for macroeconomic timeseries detrending. In Case-1, the task query asks the agent to compute the correlation between the cyclical components of personal consumption expenditure (PCE) and private fixed investment (PFI) from 1973 to 2024. The skill procedure requires the agent to explicitly harmonize mixed-frequency inputs before real-term conversion, log transformation, HP filtering, and correlation analysis. The agent had previously completed the task query successfully, demonstrating its capability to complete such tasks with the skill. However, later repeated executions of the same task query failed occasionally because it skipped the step of frequency harmonization, deviating from the skill procedure. In Case-2, the query was semantically rewritten without changing the intended task or expected output. After this benign rewrite, Codex failed to complete the task successfully because it used an incorrect HP filtering parameter. The skill explicitly specifies λ = 1600 for quarterly data, but the agent applied λ = 100, which is intended for annual data. This parameter mismatch changed the extracted cyclical components and eventually led to an incorrect correlation result. These cases show that skill-related tasks may still fail even when the agent has previously completed similar tasks successfully under the guidance of corresponding skills. These observations motivate SKILLSENTRY, which wraps around the LLM agent execution loop, monitors the agent execution progress against the skill procedure, and intervenes when necessary to guide the agent towards reliable skill execution.

Methodology

The key idea of SKILLSENTRY is to represent the expected skill procedure and historical execution experience as structured runtime guidance, use this runtime guidance to monitor and guide agent skill execution, and iteratively refine it using newly collected execution traces. SKILLSENTRY consists of an initialization stage and a self-evolving stage. In the initialization stage, it first extracts a skill specification from the corresponding skill document using an LLM-based parser. The extracted specification captures the expected skill steps, their dependencies, step-level constraints, and the steps required for task completion. SKILLSENTRY then uses historical successful and failed execution traces to mine initial execution experience. Successful traces provide validated action patterns and step-level suggestions, whereas failed traces reveal failure-associated action patterns and corresponding warnings. By combining the extracted specification with the mined experience, SKILLSENTRY constructs the initial runtime guidance for the skill.

In the self-evolving stage, the agent executes new task queries under the current runtime guidance through the step-aware runtime assurance mechanism. SKILLSENTRY wraps around the agent execution loop through runtime hooks, and inspects the actions planned by the agent through a procedure checker and a termination checker. The procedure checker monitors whether the execution follows the steps embedded in the runtime guidance. When the execution reaches a specific step, it delivers the corresponding step-level suggestions and warnings to help the agent execute the step correctly, and hints the agent to re-plan when procedural deviations or failure-associated action patterns are detected. The termination checker verifies that all required skill steps have been completed before accepting the final output, and prompts the agent to continue planning otherwise. The resulting new traces are collected for later execution experience mining to refine the runtime guidance. The updated guidance is used in the next round of runtime assurance, forming an iterative optimization loop that progressively improves the runtime guidance for reliable skill execution.

DSL for Skill-Oriented Runtime Guidance. Although skill documents provide procedural knowledge described in natural language, they do not explicitly expose the information needed for runtime procedure checking. Moreover, agent execution traces are typically unstructured. The key challenge is to transform these unstructured data into structured representations that can be consumed by the runtime assurance mechanism. To this end, SKILLSENTRY introduces a domain-specific language (DSL) for skill-oriented runtime guidance. The DSL combines the skill specification extracted from the skill document and execution experience mined from successful and failed agent execution traces. A runtime guidance instance is associated with one skill and consists of a set of procedure steps and termination requirements. Each step is marked by a unique identifier and a short description. The depends on field records prerequisite steps, determining the dependencies between steps, while the constraints field records explicit requirements on step execution. Two types of constraints are considered: a tool constraint specifies a tool required by the step, and a parameter constraint specifies a required parameter value or condition. The termination field lists the required steps that must be completed before the final output can be accepted. Each step further contains a set of logical actions. A step may involve multiple logical actions, and each logical action may have multiple alternative action patterns. Each pattern specifies an observable tool call and an optional regular expression in the command match field that matches commands realizing the action. Matching any action pattern indicates that the corresponding logical action has been performed, whereas a step is considered completed only after all of its logical actions have been matched during runtime execution. The on enter field stores suggestions and warnings for step execution, which are delivered to the agent to support more reliable step execution when the execution reaches the corresponding step. Each step also contains a set of failure patterns, which record failure-associated action patterns together with reasons explaining their potential risks.

Skill Specification Extraction. Given the DSL, SKILLSENTRY first extracts the skill specification from the corresponding skill document. A skill document is written in natural language. Therefore, SKILLSENTRY uses an LLM-based parser with GPT-5.4 to transform each skill document into the specification fields defined by the DSL. Specifically, given a skill document, the parser extracts fo…

Based on the paper, here are the specific improvements I can make to AI systems and what the improved system can do:

1. Implement a runtime assurance layer with a DSL for skill execution monitoring — I can build a structured representation of skill procedures (steps, dependencies, constraints, termination requirements) that wraps around the agent execution loop, enabling real-time verification of whether actions align with the expected skill procedure.

2. Add a procedure checker with finite-state machine (FSM) tracking — I can construct an FSM from skill steps and dependencies to track execution progress, detect deviations (e.g., skipping required steps like frequency harmonization), and intervene with targeted hints for re-planning when the agent attempts out-of-order actions.

3. Integrate failure-pattern detection with advisory blocking — I can mine historical failed traces to identify action patterns associated with failures (e.g., using λ=100 instead of λ=1600 for HP filtering), temporarily deny such actions, and return failure reasons to the agent for reconsideration, while allowing the action if the agent insists.

4. Deliver step-level contextual guidance at runtime — Instead of placing all guidance in a system prompt, I can inject step-specific suggestions and warnings only when the agent enters the corresponding step, reducing information overload and improving adherence (5.8% higher success rate, 26.8% lower variability vs. system-prompt delivery).

5. Implement a termination checker for required-step verification — Before accepting final output, I can verify all required skill steps are completed, and if any are missing, deny termination and prompt the agent to continue planning.

6. Enable self-evolving guidance through iterative trace mining — I can continuously refine runtime guidance by mining new successful/failed traces, updating action patterns, suggestions, and warnings, with diagnosis of fields as too restrictive (FP), too weak (FN), missing coverage (MC), or OK, leading to progressive improvement (58.2% success rate increase after 10 iterations).

7. Add cross-model guidance transfer — I can transfer evolved runtime guidance across backbone models within the same agent, retaining 94.2% of native performance, enabling faster deployment when switching models.

8. Incorporate fail-open exception handling — I can ensure the assurance layer never disrupts the agent runtime by defaulting to allow actions/termination on internal exceptions, recording errors for later diagnosis.

• Execute skills with high reliability: Achieve 24.1% higher task success rates on average across diverse skills (e.g., 210.8% improvement on excitation-signal-design, 131.6% on weighted-gdp-calc), with 41.1% lower variability across repeated runs.

• Prevent procedural deviations: Detect and correct out-of-order or skipped steps in real-time, such as forcing frequency harmonization before real-term conversion in macroeconomic timeseries detrending.

• Recover from known failure modes: Recognize and warn against historically unsuccessful actions (e.g., incorrect HP filtering parameters) before they cause task failure.

• Maintain efficiency: Add only 7.8% more inference turns and 8.7% more token cost, with negligible computational overhead (0.8% of total execution time).

• Adapt and improve over time: Continuously learn from new execution traces, stabilizing performance while reducing variance as more data is collected.

• Transfer across models: Apply learned runtime guidance to different backbone models within the same agent, preserving most of the reliability gains without retraining.

• Operate safely: Never block valid actions permanently, allow auxiliary exploration, and degrade gracefully on internal errors, ensuring the agent remains functional even if the assurance layer fails.

LLM agents are increasingly equipped with skills to perform complex tasks through multi-step reasoning and tool use. Although skills provide reusable procedural knowledge, agents may still execute them unreliably. Even when an agent has demonstrated the capability to complete tasks under the guidance of a skill, it may fail to do so consistently across similar tasks or repeated runs due to deviations from the skill procedure or incorrect execution of individual steps. Such instability limits the practical reliability of LLM agents. To address this problem, we propose SkillSentry, a skill-oriented runtime assurance framework built upon a new domain-specific language (DSL) for representing runtime guidance for skill execution. SkillSentry initializes the runtime guidance by combining a skill specification extracted from the corresponding skill document with execution experience mined from historical successful and failed traces. It then wraps around the agent execution loop to monitor and guide skill execution under the current guidance, while iteratively refining the guidance using newly collected traces. We evaluate SkillSentry on 15 skills across two LLM agents, each paired with two backbone models, i.e., Claude Code with Claude-Haiku-4.5 and Claude-Opus-4.6, and Codex with GPT-5.2 and GPT-5.4. Our results show that SkillSentry improves the task success rate of LLM agents by 24.1% across skills, on average, while exhibiting lower variability across repeated runs.