Long SKILL Compliance as Logical Reasoning: Closure-Grounded Detection with Scaling-Guided On-Policy Distillation
Shuaitao Zhao, Feng Ni, Lichao Ma, Jiaye Lin, Fei Han, Yang Wei, Lu Pan
cs.AI
Submitted: 2026-08-08
Updated: 2026-08-11
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: The increasing complexity of enterprise business scenarios has promoted the widespread adoption of long SKILL documents in agent systems, posing new challenges for compliance detection: large models
Terminology
Abstract
The increasing complexity of enterprise business scenarios has promoted the widespread adoption of long SKILL documents in agent systems, posing new challenges for compliance detection: large models incur substantial inference costs, while small models may fail to maintain detection accuracy. To address this gap, we propose SkillCDG, a graph-based framework for long SKILL compliance detection. SkillCDG represents complex business policies as a two-layer constraint dependency graph, where the upper layer indexes SKILL descriptions for scenario routing and the lower layer captures dependencies among atomic constraints within each SKILL. During inference, two-level retrieval followed by dependency closure supports compliance judgment and source traceability. We comprehensively evaluate the framework on three enterprise datasets and two controlled public benchmark variants. Experimental results demonstrate that SkillCDG outperforms baseline methods by up to 12.8 percentage points in detection F1 score, while reducing token consumption by a maximum 64.3%. Moreover, we further investigate the inherent relationships among policy-graph complexity, model scale, and detection performance. Comparative experiments conducted on four checkpoints from a single model family validate a concise and effective scaling trend: end-to-end detection correctness exhibits a complexity-differentiated scaling pattern, and the complexity metric derived from the constraint dependency graph can effectively quantify instance difficulty and the performance improvement potential of models. Leveraging this insightful scaling trend, we conduct adaptive training sample selection and adopt on-policy distillation to efficiently enhance the compliance detection capability of small-scale models.
Sources
- Beyond IVR: Benchmarking Customer Support LLM Agents for Business-Adherence
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- From Local to Global: A Graph RAG Approach to Query-Focused Summarization
- LightRAG: Simple and Fast Retrieval-Augmented Generation
- SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?
- Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following
- Distilling the Knowledge in a Neural Network
- LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
- Scaling Laws for Neural Language Models
- SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks
- KGGen: Extracting Knowledge Graphs from Plain Text with Language Models
- Predicting Task Performance with Context-aware Scaling Laws
- LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
- Generalizing Verifiable Instruction Following
- AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios
- Skill Retrieval Augmentation for Agentic AI
- SkillTester: Benchmarking Utility and Security of Agent Skills
- SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment
- SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems