Specification-based Security Evaluation of Real-World Agents Tool-Calling under Long-Horizon Context Tasks
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: I'm Nadia.
Elias: And I'm Elias, here with Priya.
Nadia: We're looking at the paper: "Specification-based Security Evaluation of Real-World Agents Tool-Calling under Long-Horizon Context Tasks".
Elias: Here's the gist of the paper. The paper introduces RAS-Eval, a comprehensive security benchmark designed to evaluate Large Language Model agents in real-world environments by supporting JSON, LangGraph, and MCP tool formats The results show that attacks reduced agent task completion rates (TCR) by 36.78% on average and achieved an 85.65% success rate in academic settings Benchmark Construction The dataset is structured into four distinct components: test cases, attack tasks, toolkits, and scenarios The test cases and attack tasks are serialized using JavaScript Object Notation (JSON) format for seamless integration The toolkit encompasses a diverse set of resources including scripts designed to support the LangGraph paradigm, Python scripts tailored for the MCP, and JSON objects To evaluate LLM agents in both real-world and simulated environments, two distinct execution modes were designed for the tools Real Execution involves collecting real-world APIs and MCP-compliant tools from open-source repositories on GitHub Simulated Execution implements simulated tool environments operating on a shared in-memory dictionary object that persists throughout an agent’s task execution Risk Categorization and Failure Taxonomy The risks of all attack tasks and test cases are classified and mapped to the CWE categories The 11 vulnerability types include CWE-77 Command Injection, CWE-89 SQL Injection, and CWE-434 Unrestricted Upload of File with Dangerous Type among others A hierarchical classification system defines six atomic failure modes for granular diagnosis These atomic modes are F1 (Partial Tool Omission), F2 (Sequential Violation), F3 (Null Execution), F4 (Stack Overflow), F5 (Extraneous Invocation), and F6 (Runtime Execution Fault) Compound failures can yield up to 32 different reasons for failure, enabling precise root cause analysis Experimental Validation and Scaling Laws The evaluation utilized eight representative Large Language Models across all test cases, with attack tasks conducted exclusively on the GLM4-Flash model Performance metrics such as Task Completion Rate (TCR), Task Incompletion Rate (TIR), and Task Fail Rate (TFR) were used to evaluate model performance The empirical validation showed that scaling laws hold for security capabilities, with larger models outperforming smaller counterparts For instance, the average attack success rate was 85.65% in academic settings Effectiveness of Attack Tasks The effectiveness of the benchmark's attack tasks was validated by comparing agent performance before and after attacks across various scenarios Table 8 shows that post-attack, average performance scores drop significantly across scenarios, with an average attack success rate of 73.44% This confirms the effectiveness of the benchmark’s attack tasks in evaluating model vulnerabilities The distribution of reasons for task failure shows a difference before and after the attack, indicating vulnerability exposure Discriminative Power and Consistency The benchmark's discriminative power was validated by calculating the Kappa coefficient κ between human and model annotations The average Kappa coefficient of all models is 0.6499, indicating relatively good agreement among the evaluated models Furthermore, the fitting results for the scaling law demonstrated a strong fit with an R2 of 0.9051 This suggests that approximately 90.51% of the variance in performance is explained by the model size In this work, RAS-Eval is proposed as a novel LLM agent security evaluation dataset for dynamic, open, and real-world environments It supports JSON, LangGraph, and MCP tool formats while evaluating agents powered by 7 mainstream LLMs across 7 scenarios The findings offer new ways to design more robust LLM agents REFERENCES
1: Mahyar Abbasian, Iman Azimi, Amir M Rahmani, and Ramesh Jain.
Nadia: Conversational health agents: A personalized llm-powered agent framework.
Elias: arXiv preprint arXiv:2310.02374, 2023.
2: Junkai Li, Yunghwei Lai, Weitao Li, Jingyi Ren, Meng Zhang, Xinhui Kang, Siyu Wang, Peng Li, Ya-Qin Zhang, Weizhi Ma, et al.
Paper summary: Nadia: So to get into this paper "Specification-based Security Evaluation of Real-World Agents Tool-Calling under Long-Horizon Context Tasks," the main thing is that they are addressing the lack of a standard way to evaluate agent security when they are actually using tools in dynamic settings >
Elias: They introduce RAS-Eval, which is this comprehensive security benchmark. It supports JSON, LangGraph, and Model Context Protocol formats for tool execution >
Priya: The thesis of the paper is that you need a benchmark that covers both simulated and real-world tool execution for LLM agents to be truly robust >
Nadia: Right. And they claim this benchmark reveals significant vulnerabilities, showing that attacks can reduce agent task completion rates by thirty-six point seventy-eight percent on average in academic settings > <ref:2506.15253#pg1>
Elias: They also show an eighty-five point sixty five percent success rate for those attacks in academic settings, which is the main result they highlight about the vulnerability exposure >
Priya: It matters because it gives us a structured way to see these vulnerabilities mapped to eleven Common Weakness Enumeration categories and defines six atomic failure modes for diagnosis >
Nadia: That structure helps researchers actually pinpoint where the agent is failing, which is much better than just getting a vague failure report >
Elias: And they also show that scaling laws hold for security capabilities. This means bigger models tend to be more secure when evaluating these agent tool-calling scenarios, according to the paper "Specification-based Security Evaluation of Real-World Agents Tool-Calling under Long-Horizon Context Tasks"
Priya: So, in short, it’s about creating a benchmark that tests real agent behavior against specific attack types and shows that model size affects security performance >
Nadia: It really matters because it gives us a tool to design more robust agents for critical domains like healthcare or finance without just guessing at where the weak points are >
Elias: So, they’re providing a comprehensive dataset that allows people to test these agents in diverse, real-world scenarios and understand the security landscape of agent tool usage >
Conclusion: Nadia: Thinking about the title, "Specification-based Security Evaluation of Real-World Agents Tool-Calling under Long-Horizon Context Tasks," it really boils down to testing agents when they are actually using tools over long periods >
Elias: That’s right. It’s not just a quick check; it involves checking tool calling, context length, and the environment they're operating in >
Priya: What this implies is that for any agent deployment, you need to consider the full lifecycle of its tasks and how those tools interact over time >
Nadia: And the authors are providing this RAS-Eval benchmark as a way to make that consideration concrete for practitioners, giving them something tangible to work with >
Elias: It’s about moving from theoretical concerns about agent security to practical evaluation, using things like JSON and MCP formats that are actually used in development today >
Priya: So the implication is that we need standardized testing protocols for these agents so we can build trust in them when they're making decisions based on external tools >
Nadia: Exactly. It gives us a common language to discuss vulnerabilities, which helps everyone—the developers and the security researchers—understand each other better >
Elias: So this paper is about establishing a formal way to benchmark the security of LLM agents in these specific, long-horizon contexts, providing a foundation for future research on agent safety >
Zhejiang University
cs.CR, cs.AI
Submitted: 2025-06-18
Updated: 2026-11-11
Comments: 22 pages, 5 figures
Code: https://github.com/lanzer-tree/RAS-Eval
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 91/100
The gist: The paper introduces RAS-Eval, a comprehensive security benchmark designed to evaluate Large Language Model agents in real-world environments by supporting JSON, LangGraph…
Key concepts
- RAS-Eval
- A comprehensive security benchmark designed to evaluate LLM agents in dynamic, open, and real-world environments. It supports various tool formats like JSON, LangGraph, and MCP to test agent robustness against attacks.
- Tool Formats
- The benchmark supports three distinct tool formats: JSON for general serialization of test cases and tasks; LangGraph for defining agent workflows; and MCP (Model Context Protocol) scripts tailored for Python-based tools. This allows evaluation across different real-world agent architectures.
- Failure Taxonomy
- A hierarchical classification system that defines six atomic failure modes: F1 through F6. These modes, such as Partial Tool Omission or Null Execution, allow for granular diagnosis of why an agent failed during a security test, helping pinpoint the exact root cause of vulnerabilities.
Terminology
Summary
The paper introduces RAS-Eval, a comprehensive security benchmark designed to evaluate Large Language Model agents in real-world environments by supporting JSON, LangGraph, and MCP tool formats The results show that attacks reduced agent task completion rates (TCR) by 36. p. 178% on average and achieved an 85. p. 065% success rate in academic settings
Benchmark Construction
The dataset is structured into four distinct components: test cases, attack tasks, toolkits, and scenarios The test cases and attack tasks are serialized using JavaScript Object Notation (JSON) format for seamless integration The toolkit encompasses a diverse set of resources including scripts designed to support the LangGraph paradigm, Python scripts tailored for the MCP, and JSON objects To evaluate LLM agents in both real-world and simulated environments, two distinct execution modes were designed for the tools Real Execution involves collecting real-world APIs and MCP-compliant tools from open-source repositories on GitHub Simulated Execution implements simulated tool environments operating on a shared in-memory dictionary object that persists throughout an agent’s task execution
Risk Categorization and Failure Taxonomy
The risks of all attack tasks and test cases are classified and mapped to the CWE categories The 11 vulnerability types include CWE-77 Command Injection, CWE-89 SQL Injection, and CWE-434 Unrestricted Upload of File with Dangerous Type among others A hierarchical classification system defines six atomic failure modes for granular diagnosis These atomic modes are F1 (Partial Tool Omission), F2 (Sequential Violation), F3 (Null Execution), F4 (Stack Overflow), F5 (Extraneous Invocation), and F6 (Runtime Execution Fault) Compound failures can yield up to 32 different reasons for failure, enabling precise root cause analysis
Experimental Validation and Scaling Laws
The evaluation utilized eight representative Large Language Models across all test cases, with attack tasks conducted exclusively on the GLM4-Flash model Performance metrics such as Task Completion Rate (TCR), Task Incompletion Rate (TIR), and Task Fail Rate (TFR) were used to evaluate model performance The empirical validation showed that scaling laws hold for security capabilities, with larger models outperforming smaller counterparts For instance, the average attack success rate was 85.65% in academic settings
Effectiveness of Attack Tasks
The effectiveness of the benchmark's attack tasks was validated by comparing agent performance before and after attacks across various scenarios Table 8 shows that post-attack, average performance scores drop significantly across scenarios, with an average attack success rate of 73.44% This confirms the effectiveness of the benchmark’s attack tasks in evaluating model vulnerabilities The distribution of reasons for task failure shows a difference before and after the attack, indicating vulnerability exposure
Discriminative Power and Consistency
The benchmark's discriminative power was validated by calculating the Kappa coefficient κ between human and model annotations The average Kappa coefficient of all models is 0.6499, indicating relatively good agreement among the evaluated models Furthermore, the fitting results for the scaling law demonstrated a strong fit with an R2 of 0. p. 89051 This suggests that approximately 90. p. 951% of the variance in performance is explained by the model size
Conclusion
In this work, RAS-Eval is proposed as a novel LLM agent security evaluation dataset for dynamic, open, and real-world environments It supports JSON, LangGraph, and MCP tool formats while evaluating agents powered by 7 mainstream LLMs across 7 scenarios The findings offer new ways to design more robust LLM agents
REFERENCES
[1] Mahyar Abbasian, Iman Azimi, Amir M Rahmani, and Ramesh Jain. Conversational health agents: A personalized llm-powered agent framework. p. 1 arXiv preprint arXiv:2310. p. 002374, 2023.
[2] Junkai Li, Yunghwei Lai, Weitao Li, Jingyi Ren, Meng Zhang, Xinhui Kang, Siyu Wang, Peng Li, Ya-Qin Zhang, Weizhi Ma, et al. Agent hospital: A simulacrum of hospital with evolvable medical agents. p. 0 arXiv preprint arXiv:2405. p. 002957, 2024.
[3] Wenqi Shi, Ran Xu, Yuchen Zhuang, Yue Yu, Jieyu Zhang, Hang Wu, Yuanda Zhu, Joyce Ho, Carl Yang, and May D Wang. Ehragent: Code empowers large language models for few-shot complex tabular reasoning on electronic health records. p. 7 arXiv preprint arXiv:2401. p. 007128, 2024.
[4] Yangyang Yu, Haohang Li, Zhi Chen, Yuechen Jiang, Yang Li, Denghui Zhang, Rong Liu, Jordan W Suchow, and Khaldoun Khashanah. Finmem: A performance-enhanced llm trading agent with layered memory and character design. p. 0 In Proceedings of the AAAI Symposium Series, volume 3, pages 595–597, 2024. p. 8
[5] Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang, Jiaqi Bai, Linzheng Chai, Zhao Yan, Qian-Wen Zhang, Di Yin, Xing Sun et al. Mac-sql: A multi-agent collaborative framework for text-to-sql. p. 1 arXiv preprint arXiv:2312. p. 011242.
[6] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. p. 0 React: Synergizing reasoning and acting in language models, 2023. p. 7 URL https://arxiv. p. 0org/abs/2210.03629.
[7] Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, and Minlie Huang. p. 0 Agentsafetybench: Evaluating the safety of llm agents. p. 0 arXiv preprint arXiv:2412. p. 014470.
[8] Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao, Zhenting Wang, Chenlu Zhan, Hongwei Wang, and Yongfeng Zhang. p. 0 Agent security bench (ASB): Formalizing and benchmarking attacks and defenses in LLM-based agents. p. 1 In The Thirteenth International Conference on Learning Representations, 2025. p. 0
[9] Fábio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models. p. 6 arXiv preprint arXiv:2211. p. 009527.
[10] Xiangrui Cai, Haidong Xu, Sihan Xu, Ying Zhang, et al. Badprompt: Backdoor attacks on continuous prompts. p. 0 Advances in Neural Information Processing Systems, 35:37068–37080, 2022. p. 4
[11] Anthropic. p. 0 Introduction - model context protocol, 04 2025. p. 0
[12] Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois et al. p. 0 Identifying the risks of lm agents with an lm-emulated sandbox. p. 0 In The Twelfth International Conference on Learning Representations, 2024.
[13] Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner et al. Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents. p. 0 In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. p. 6
[14] Huihao Jing, Haoran Li, Wenbin Hu, Qi Hu, Heli Xu et al. Mcip: Protecting mcp safety via model contextual integrity protocol. p. 0 arXiv preprint arXiv:2505. p. 014590.
[15] Vineeth Sai Narajala and Idan Habler. Enterprise-grade security for the model context protocol (mcp): Frameworks and mitigation strategies. p. 0 arXiv preprint arXiv:2504. p. 008623.
[16] Kai Mei, Xi Zhu, Wujiang Xu, Wenyue Hua et al. Aios: Llm agent operating system. p. 0 arXiv:2403. p. 016971.
[17] Balaji Rama, Kai Mei, and Yongfeng Zhang. Cerebrum (aios sdk): A platform for agent development, deployment, distribution, and discovery. p. 0 In 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics, 2025. p. 0
[18] Hanjun Luo, Shenyu Dai, Chiming Ni et al. Agentauditor: Human-level safety and security evaluation for llm agents. p. 0 arXiv preprint arXiv:2506. p. 000641.
[19] Steve Christey, J Kenderdine, J Mazella, and B Miles. Common weakness enumeration. p. 0 Mitre Corporation, 2013.
[20] Jinze Bai, Shuai Bai, Yunfei Chu et al. Qwen technical report. p. 6 arXiv preprint arXiv:2309. p. 016609.
[21] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet et al. Llama: Open and efficient foundation language models. p. 6 arXiv preprint arXiv:2302. p. 013971.
[22] Team GLM, Aohan Zeng, Bin Xu et al. p. 8 Chatglm: A family of large language models from glm-130b to glm-4 all tools, 2024. p. 8
[23] DeepSeek-AI. p. 6 Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. p. 6
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Implement RAS-Eval as a comprehensive security benchmark supporting both simulated and real-world tool execution across JSON, LangGraph, and MCP formats to address the
absence of standardized evaluation benchmarks for these agents in dynamic environments.
-
Utilize the 3802 attack tasks mapped to 11 Common Weakness Enumeration CWE categories to provide comprehensive security coverage, exposing vulnerabilities such as
CWE-77 Command Injection
andCWE-89 SQL Injection.
-
Develop a novel failure mode taxonomy comprising six atomic failure modes (F1 through F6) enabling granular diagnosis of agent failures, allowing for precise root cause analysis of security failures.
-
Integrate the empirically validated scaling laws for security capabilities, confirming that
larger models outperform smaller counterparts
when evaluated using RAS-Eval across diverse scenarios. -
Enhance agent performance by using the unified performance score calculated as
scorei = ncorrect / (ncorrect + nwrong + nlack)
to objectively measure task success and differentiate model competencies. -
Improve attack detection capabilities by leveraging the Attack Success Rate (ASR) metric, defined as whether
the output r of the tool satisfies the attack target Sattack,
to quantify how effectively adversarial inputs lead to desired outputs.
Sources
- Conversational Health Agents: A Personalized LLM-Powered Agent Framework
- Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agents
- EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Records
- MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL
- Agent-SafetyBench: Evaluating the Safety of LLM Agents
- Ignore Previous Prompt: Attack Techniques For Language Models
- MCIP: Protecting MCP Safety via Model Contextual Integrity Protocol
- Enterprise-Grade Security for the Model Context Protocol (MCP): Frameworks and Mitigation Strategies
- AIOS: LLM Agent Operating System
- AgentAuditor: Human-Level Safety and Security Evaluation for LLM Agents
- Qwen Technical Report
- LLaMA: Open and Efficient Foundation Language Models
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data