Agent Skill Framework: Perspectives on the Potential of Small to Medium Language Models in Industrial Environments

The gist The effectiveness of Agent Skills for small to medium language models in industrial environments remains underexplored, revealing a tradeoff between GPU cost and agent performance

Agent Skills Paradigm

Agent skills are an effective framework for agent-centric context engineering, acting as a static cheat-sheet that enables Large Language Models to focus on salient information through progressive context management This approach optimizes both the length and quality of context windows while improving LLMs’ use of tools, documents, and external knowledge resources Unlike Retrieval Augmented Generation (RAG) systems which rigidly encode textual or visual data into vectorspace databases, the Skills design directly leverages the In-Context Learning (ICL) capabilities of LLMs and emergent reasoning to dynamically select the most pertinent information and contextual knowledge

Model Performance Across Scales

The study systematically investigated the Skill paradigm in resource-constrained industrial settings, finding that very small models struggle with reliable skill selection, while models around 30B–80B benefit substantially Specifically, Tiny Models Fail at Skill Routing Across these three datasets where extremely small models such as Gemma-3-4B-it and Gemma-3-270M-it still struggle to retrieve the appropriate skill However, for most medium-sized to large models (>=30B), retrieval errors decrease substantially, and the error mass shifts downstream toward reasoning boundaries and instruction/format compliance

Skill Selection Strategies

To evaluate skill-selection performance, three distinct CE strategies were tested:

1. Direct Instruction (DI) uses a minimal prompt to mimic raw user input

2. Full-Skill Instruction (FSI) provides a fixed context containing the entire temporary Skill repository, requiring the model to identify the correct Skill among multiple specifications

3. Agent Skill Instruction (ASI) loads Skill information on demand, where the model determines whether additional Skill details are needed, retrieves the relevant Skill, and generates an answer conditioned on that information

Evaluation Datasets and Metrics

The evaluation utilized three datasets: IMDB for binary sentiment classification, FiNER for XBRL tagging requiring strong domain knowledge in finance, and InsurBench for a real-world insurance claims classification task To account for practical efficiency, metrics included Cls ACC (Classification Accuracy), Cls F1 (F1 score), Skill ACC (skill-selection accuracy), Avg GT (average processing time per task in minutes), and Avg VRAM Time (GPU memory, defined as GB·min) which represents the time cost per task

Key Findings on Model Robustness

The results indicate a practical deployment threshold at approximately 12B–30B parameters, below which skill routing breaks down even under minimal interference For instance, Gemma-3-4B-it achieves only a 0.78 success rate on InsurBench Furthermore, the Qwen3-80B Thinking model exhibited severely uncontrolled resource consumption, with VRAM·Time reaching 94.802 GB·min on IMDB, 94.503 GB·min on FiNER, and 181.003 GB·min on InsurBench — representing 22×, 18×, and 34× the cost of Instruct under the same method

Optimization Insights

For lightweight SLMs in Agent Skills deployments, chat history benefits are largest for very small models (e.g. Gemma-3-4b-it and Gemma3-270m-it) Moreover, replacing the keyword Skill with its synonyms showed minimal impact on performance, while Expertise consistently outperformed Skill across metrics, suggesting it as a potentially superior alternative The study concludes that the 30B to 80B scale is well suited for agent-skill pipelines that has acceptable accuracy with budget constraints

The paper provides actionable guidance on model selection, skill keyword configuration, and chat-history management for organizations with resource-constrained, security-sensitive industrial environments.

1. A system can reliably select appropriate skills for complex tasks by implementing an Agent Skill Instruction (ASI) strategy, as this method showed high skill-selection accuracy, with ASI 0.950 on IMDB and 0.997 on FiNER, significantly outperforming Direct Instruction (DI).

2. For small language models (SLMs) below 4B parameters, the system can be optimized by employing a deterministic message-trimming policy for conversational history to manage context-window constraints while minimizing VRAM time usage, as this is mainly recommended for lightweight SLMs in Agent Skills deployments.

3. The architecture can be tuned to leverage more descriptive skill keywords; specifically, replacing Skill with synonyms like Expertise or Knowhow demonstrates substantial improvements in performance and GPU memory efficiency on InsurBench.

4. To improve reasoning robustness, the system should adopt a model scale between 30B and 80B parameters to achieve acceptable accuracy with budget constraints, as this range is well suited for agent-skill pipelines that has acceptable accuracy with budget constraints.

Skills are getting increasingly used in recent agentic frameworks, performing robustly with proprietary LLMs. Yet, their effectiveness for small and medium-sized open-source models (270M--80B) remains underexplored, especially under industrial, resource-constrained conditions. As such, we present a large-scale empirical study evaluating the skill paradigm's feasibility across two open-source classification tasks: one insurance-claim task based on proprietary data, and a set of Python programming problems. We find that very small models frequently fail to retrieve or select correct skills, substantially limiting their usefulness in the evaluated skill-augmented settings, while larger 30B--80B models gain substantial benefits from skill augmentation. Furthermore, we provide a deployment-oriented GPU profile, reporting VRAM time occupancy (GB times min) across models under our evaluation settings---a deployment-oriented metric that may help characterize GPU-memory costs, where small and medium models are typically preferred for VRAM control. These results characterize how the benefits of skill augmentation vary with model scale, task complexity, and GPU budget, and provide deployment-oriented observations.