Breaking MCP with Function Hijacking Attacks: Novel Threats for Function Calling and Agentic Models
cs.CR, cs.AI, cs.CL
Submitted: 2026-04-22
Updated: 2026-08-31
Comments: Accepted to TMLR (08/26)
Code: https://github.com/GraySwanAI/nanoGCG
License: http://creativecommons.org/licenses/by/4.0/
The gist: The growth of agentic AI has drawn significant attention to function calling Large Language Models (LLMs), which are designed to extend the capabilities of AI-powered system by invoking external
Terminology
Abstract
The growth of agentic AI has drawn significant attention to function calling Large Language Models (LLMs), which are designed to extend the capabilities of AI-powered system by invoking external functions. Injection and jailbreaking attacks have been extensively explored to showcase the vulnerabilities of LLMs to user prompt manipulation. The expanded capabilities of agentic models introduce further vulnerabilities via their function calling interface. Recent work in LLM security showed that function calling can be abused, leading to data tampering and theft, causing disruptive behavior such as endless loops, or causing LLMs to produce harmful content in the style of jailbreaking attacks. This paper introduces a novel function hijacking attack (FHA) that manipulates the tool selection process of agentic models to force the invocation of an attacker-chosen function. While existing attacks focus on semantic preference of the model for function-calling tasks, we show that FHA is largely agnostic to the context semantics and remains effective across domains and function sets. We demonstrate that FHA generalizes to unseen queries and payload perturbations under a fixed target model, reaching 62.5% to 81.9% ASR on held-out queries across 4 function-calling LLMs (instructed and reasoning models), evaluated on the Berkeley Function Calling Leaderboard (BFCL). We further evaluate the cross-model transferability of FHA, showing that FHA can be transferred to other model sizes and families (11.2-27.6% ASR). Our findings further demonstrate the need for strong guardrails and modules for agentic systems.
Sources
- Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks
- AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
- Small Language Models are the Future of Agentic AI
- AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
- Query-Based Adversarial Prompt Generation
- The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies
- Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions
- Mistral 7B
- Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Calling
- AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
- Improving Small-Scale Large Language Models Function Calling for Reasoning Tasks
- GPT-4 Technical Report
- Gorilla: Large Language Model Connected with Massive APIs
- Qwen3 Technical Report
- Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAI
- Toolformer: Language Models Can Teach Themselves to Use Tools
- LLaMA: Open and Efficient Foundation Language Models
- Chain-of-Tools: Utilizing Massive Unseen Tools in the CoT Reasoning of Frozen Language Models
- The Dark Side of Function Calling: Pathways to Jailbreaking Large Language Models
- ReAct: Synergizing Reasoning and Acting in Language Models
Related papers
- SoK: AI-Augmented Binary Reversing
- Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
- Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
- Estimating the Social Cost of Corporate Data Breaches
- Efficient Fuzzy PSI under One-Sided Assumptions
- Sealing the Audit-Runtime Gap for LLM Skills