Beyond the Prompt: Jailbreaking Function-Calling LLMs via Simulated Moderation Traces

arXiv:2607.00481 · cs.CR, cs.AI · Submitted 2026-07-01 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: I'm Nadia.

Elias: And I'm Elias, here with Priya.

Nadia: We're looking at the paper: "Beyond the Prompt".

Elias: Here's the gist of the paper. Jailbreak attacks remain a critical threat to the safe deployment of large language models (LLMs) In such applications, developer-defined schemas, structured arguments, and untrusted tool outputs are interleaved into a single shared model context This architecture expands the attack surface by blurring the boundary between trusted control logic and untrusted data The Core Vulnerability Identified The paper identifies a structural vulnerability in stateful, function-calling environments where developer-defined schemas, structured arguments, and untrusted tool outputs are interleaved into a single shared model context This interleaving allows adversarial intent to be distributed across a multi-turn execution path The attack surface extends beyond any single prompt, as adversarial intent can be fragmented across schemas, arguments, tool feedback, and interaction history The SMT Attack Framework The authors introduce SMT, a black-box jailbreak framework based on Simulated Moderation Traces Departing from purely prompt-based interactions, SMT constructs a multi-turn trajectory that simulates a legitimate moderation auditing workflow It exploits the model’s dual role as execution controller and security-policy enforcer by treating safety refusals as execution failures This is achieved by constructing a multiturn strategy involving an adversarial Case Generator and a Validator The SMT framework operates through three components: the each turn Jailbreak Execution flow for argument generation and evaluation, the function schema definition for setting up auditing constraints, and the multi-turn jailbreak flow that accumulates validation backtraces In each turn, if a safety refusal occurs, SMT substitutes it with a fabricated but well-formed function call to maintain the appearance of protocol adherence This allows the framework to leverage accumulated function-calling traces to guide subsequent jailbreak attempts Empirical Evaluation and Results Extensive empirical evaluations on prominent commercial LLMs from five different providers across two standardized safety benchmarks show that SMT consistently achieves the highest average attack success rate and HarmScore The results demonstrate that SMT substantially outperforms representative baselines in both attack success rate and harmfulness while requiring fewer than 1.5 queries on average SMT establishes a new state of the art across the evaluated targets, achieving near-perfect average Attack Success Rates (ASRs) of 99.67% on SafeBench and 98.33% on JailbreakBench Defense Analysis and Future Directions The paper shows that conventional defenses provide limited protection against stateful function-calling attacks The authors propose Function-Calling Guard (FCGuard), a new defense mechanism that requires the model to thoroughly analyze all interaction states, including conversation history, tool definitions, and tool outputs Testing against SMT and JailbreakFunction on GPT-5.4 and Claude-Sonnet-4.

Title and authors: Nadia: So this paper, "Beyond the Prompt: Jailbreaking Function-Calling LLMs via Simulated Moderation Traces," it's looking at something different than just messing with the initial prompt. It's focusing on how those schemas and tool outputs mix together in a stateful function-calling setup.

Elias: Exactly, Nadia. It points out that this interleaving of developer logic and untrusted data really expands the attack surface because you can spread the bad intent across multiple turns of execution, not just one single message.

Priya: And for us as people who look at the data, it means we have to think about how much history and context an AI actually carries around during a complex task, because that’s where this vulnerability lies.

Nadia: Right. They introduce this framework called SMT, which they call Simulated Moderation Traces, to simulate a proper moderation auditing workflow. Instead of just trying to trick the model once, they build these multi-turn trajectories that pretend the safety refusals are actually execution failures in the system.

Elias: That’s the core idea of how SMT works—treating those safety refusals as errors that need correcting, which lets them guide subsequent jailbreak attempts through accumulated traces. They have this Case Generator and a Validator to build out these paired cases for red-teaming.

Priya: I wonder if framing the refusal as an execution failure actually changes how the model processes that constraint, or if it just makes it more confused? We need to look at what those simulated traces actually show us about the model's internal state.

Nadia: The empirical results they show are pretty strong, too; SMT consistently gets the highest average attack success rate and harms score across six different commercial LLMs tested on two standard safety benchmarks.

Elias: They claim that SMT substantially outperforms other baselines in both terms, and they manage to do this with fewer than one point five queries on average for those evaluations.

Priya: Fewer queries is good, but what about the actual harm being generated? We need to see if that high success rate translates into a lot of actual policy violation or if it's just a successful bypass of a specific safety check.

Nadia: The paper says SMT achieves near perfect average attack success rates, around ninety-nine point six seven percent on SafeBench and ninety-eight point thirty three percent on JailbreakBench.

Elias: That level of success is what makes this paper interesting; it shows how reliably this stateful exploitation works when you're treating the model like an execution controller and a policy enforcer simultaneously.

Priya: So, for someone who just uses an AI for a simple prompt, this doesn't directly affect their experience unless they are using those complex function-calling tools. What does this mean for the average person?

Nadia: It means that any application built on top of these LLMs that involves structured inputs or tool use is now a much more interesting target because the attack isn't isolated to one simple text box anymore.

Title and authors: Elias: And they also propose a defense mechanism called Function-Calling Guard, FCGuard, which requires the model to analyze all interaction states including conversation history and tool outputs thoroughly.

Priya: That sounds like it puts a lot of computational load on the model during runtime, which is something we always have to weigh against the security gain. We need to know if that analysis is actually deep enough to catch anything SMT can generate.

Nadia: The paper tests FCGuard against SMT and JailbreakFunction on models like GPT-five point four and Claude Sonnet point four. They are showing how this new defense mechanism fares in practice against the framework they developed.

Elias: It seems like the authors are trying to move past defenses that just look at the input prompt, aiming instead for a defense that monitors the entire execution context of a function call.

Priya: If FCGuard can effectively analyze all those states—the history, the definitions, and what the tools actually return—then it might offer better protection against those distributed attacks. It shifts the focus from just filtering words to understanding intent across steps.

Nadia: This paper, "Beyond the Prompt: Jailbreaking Function-Calling LLMs via Simulated Moderation Traces," really highlights that structural flaw in how we build these complex AI applications where control logic and data get mixed up.

Elias: The SMT framework is a black-box way to simulate auditing, reframing safety refusals as execution failures to erode the model's constraints over multiple turns.

Priya: We need to keep asking what that simulated failure actually means for the actual safety mechanisms in place, because a simulated error isn't the same as a real system failure.

Nadia: The results show SMT is substantially better than previous methods, achieving those high success rates with very few queries on average.

Elias: And they’re also looking at how to build defenses that analyze the entire interaction state, which is a step up from just checking the initial prompt.

Priya: So it seems the path forward involves moving toward context-aware defenses that look beyond single inputs and examine the whole execution history.

Nadia: Ultimately, this work shows that stateful function-calling environments are vulnerable because of how those schemas and tool outputs get interleaved in a shared context.

Elias: They offer SMT as an attack framework and FCGuard as a defense mechanism to tackle that specific architectural weakness in function-calling models.

Priya: It's a lot to process, but it makes me think about the real-world risk when we deploy these systems where trust is placed in the execution flow itself.

Nadia: This paper provides a concrete method for exploiting that risk and then proposes a new way to defend against it by looking at the whole interaction state.

Elias: We'll keep an eye on how FCGuard performs under more stress, especially when facing these kinds of multi-turn, stateful attacks.

Priya: It’s definitely worth digging into the details of those simulated traces and seeing if they reveal any actual vulnerabilities we haven't considered yet.

The paper's summary: Nadia: So this paper is basically looking at how jailbreaking those AI models that use tools, you know, when they have these strict rules for how they handle function calls and developer schemas.

Elias: Right. They’re saying the problem isn't just tricking the main chat box; it's about those schemas and tool outputs getting mixed together in a stateful way across multiple steps of an execution path.

Priya: So, instead of trying to break one single prompt, you can spread the attack across how the AI uses its tools and how that feedback loops back into itself over several turns.

Nadia: Exactly. They call it SMT, or Simulated Moderation Traces. It sets up this whole multi-turn trajectory that pretends safety refusals are actually execution failures in the system itself.

Elias: That’s the trick—they treat a safety refusal as an error that needs to be fixed by the system, which then lets them guide the next jailbreak attempt using those accumulated traces.

Priya: It means we're not just looking at a static input; we're looking at a whole sequence of actions where one "error" leads to the next more effective way to bypass safety checks.

Nadia: The methodology involves this Case Generator and Validator, which builds these paired sequences that show how the model’s dual role as an execution controller and a policy enforcer gets confused.

Elias: They're feeding this whole process into testing on commercial models, and the results show SMT gets the highest average attack success rate across different providers.

Priya: But what we need to look at is how much actual harm that high success rate actually means in terms of the policies being bypassed on those benchmarks.

Nadia: The numbers they got are really high, near ninety-nine point six seven percent success rates on SafeBench and almost ninety-eight point thirty three percent on JailbreakBench.

Elias: And they managed to do this with fewer than one point five queries on average for those evaluations, which is pretty efficient.

Priya: That efficiency is interesting because it suggests the attack isn't a brute force slog; it's really finding the most direct path through that execution structure.

Nadia: They also proposed a defense called FCGuard, which they claim forces the model to analyze every interaction state—history, tool definitions, and outputs—to catch this kind of hidden malicious intent.

Elias: It sounds like the defense is trying to stop the fragmentation of the attack by making sure nothing is missed across all those different interaction states.

Priya: We need to see if that deep analysis actually works in practice when facing these complex, multi-turn scenarios they built up with SMT.

Nadia: That’s what we're looking at next, testing FCGuard against SMT and JailbreakFunction on models like GPT point four and Claude Sonnet point four.

Elias: It seems the real question now is whether this context-aware defense can actually keep up with the sophistication of these new stateful attacks.

The paper's improvements: Nadia: So we just talked about how SMT works to exploit those stateful function calls, and now they're laying out some improvements to make that attack even more effective or maybe the defense stronger.

Elias: Right. They’re suggesting ways to turn those simulated traces into something more proactive, moving beyond just building a single bad sequence.

Priya: I'm interested in how they frame safety refusals as execution failures, because if the model treats a refusal like a system error it has to fix, that changes the game.

Nadia: Exactly. They’re talking about using those fabricated exception traces from function validation failures to guide subsequent turns, turning absolute refusals into exception-based feedback across multiple steps.

Elias: That means the AI learns to use these validation errors as a map for where it can push the policy boundaries next, instead of just stopping at the first hard refusal.

Priya: But what are the limitations here? If we're converting refusals into exceptions, how do we make sure that exception feedback isn't just leading to a different kind of unhelpful output?

Nadia: The paper says this approach aims to convert "absolute refusals and weak generations into exception-based feedback" across multi-turn sessions.

Elias: It’s about making the model self-correct based on its own internal validation loops, using those traces to steer the conversation toward policy violations more smoothly.

Priya: That shifts the focus from just stopping a bad response to understanding how far you can push the boundaries before it triggers a system error.

Nadia: They also mentioned query efficiency, suggesting that by leveraging this accumulated execution state with SMT, you can achieve high success rates with way fewer API interactions.

Elias: So instead of needing many rounds of prompting to find a successful trajectory, the system identifies the right path much faster because it remembers what happened in previous steps.

Priya: That would make testing these systems on real models much more feasible for researchers, as they don't have to run hundreds of queries just to find one working case.

Nadia: They also introduced FCGuard as a defense, which they claim analyzes all interaction states—conversation history and tool outputs—to detect hidden malicious intent disguised as simple simulations or diagnostics.

Elias: That means the defense is trying to look deeper into the context than just checking if the final answer looks safe; it’s checking the entire process.

Priya: If FCGuard can really analyze all those states, then it might be able to catch that distributed intent we talked about earlier, where things are spread across schemas and arguments.

Nadia: So we're moving from a defense that checks the prompt to one that monitors the whole execution flow, trying to detect malicious simulations or diagnostics.

Elias: It seems like they’re building a framework where the model has to prove its entire interaction history is safe, not just its final output.

Priya: That’s a big shift in how we think about securing these complex AI applications; it moves us from checking the result to auditing the entire journey.

Conclusion: Nadia: So we're wrapping up on this paper, "Beyond the Prompt: Jailbreaking Function-Calling LLMs via Simulated Moderation Traces." Basically, they showed that stateful function-calling environments are vulnerable because schemas and tool outputs get mixed together in a shared context.

Elias: Right. They introduced SMT to simulate an auditing workflow where safety refusals are treated as execution failures, allowing them to distribute the attack across many turns of execution.

Priya: So the core finding is that you don't just need a good initial prompt; you need a whole sequence of tool interactions where the model’s internal logic gets confused by its own error handling.

Nadia: Exactly. The numbers show SMT achieving high success rates, like ninety-nine point six seven percent on SafeBench and almost ninety-eight point thirty three percent on JailbreakBench, even with just a few queries.

Elias: That efficiency is notable because it proves this isn't just theoretical; the attack path is very direct through the execution state.

Priya: But we have to remember that those results are based on benchmarks, so we need to keep asking what kind of actual harm those high success rates lead to in the real world.

Nadia: They also proposed FCGuard as a defense mechanism, which they claim forces the model to analyze all interaction states—history and tool outputs—to detect hidden intent disguised as diagnostics.

Elias: It seems like the authors are pushing for defenses that look at the whole process rather than just checking if the final response is clean.

Priya: That moves security from being about filtering words to understanding intent across every step of an operation, which is a necessary evolution for these complex systems.

Nadia: The implication is that any application relying on those structured tools now has a much larger attack surface because the attack isn't isolated to one single user input anymore.

Elias: So we’re looking at a need for defenses that can handle this distributed, multi-turn nature of the threat.

Priya: I just want to make sure that when we start implementing these context-aware defenses, they actually work against attacks like SMT, not just simpler prompt injections.

Nadia: That’s exactly what they tested against GPT point four and Claude Sonnet point four, showing how FCGuard performs in practice.

Elias: It seems like the next big area of research will be figuring out how to make those defenses robust enough to handle attacks that are actively trying to look like normal debugging or auditing.

Priya: We’ll keep tracking these SMT results because understanding this structural vulnerability is key to building safer AI tools.

Sun Yat-sen University · Nanyang Technological University

cs.CR, cs.AI

Submitted: 2026-07-01

Updated: 2026-11-04

Comments: Accepted to the 2027 IEEE Symposium on Security and Privacy (S&P)

Code: https://github.com/liujlong27/SMT

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 94/100

The gist: Jailbreak attacks remain a critical threat to the safe deployment of large language models (LLMs) In such applications, developer-defined schemas, structured arguments…

Key concepts

Stateful Function-Calling Attack Surface
This vulnerability occurs when developer logic, structured arguments, and tool results are combined into a single context. Adversarial intent can be spread across these different elements over several turns, making the attack surface much larger than just a single prompt.
SMT Framework (Simulated Moderation Traces)
SMT is a black-box jailbreak method that mimics an auditing process. It constructs multi-turn paths where safety refusals are framed as execution errors instead of outright denials. This exploits the model's role as both controller and enforcer.
Function-Calling Guard (FCGuard)
This proposed defense mechanism requires the model to deeply analyze every aspect of the interaction state, including conversation history, tool definitions, and tool outputs. It aims to detect hidden malicious intent disguised as legitimate simulations or diagnostics.

Terminology

Summary

Jailbreak attacks remain a critical threat to the safe deployment of large language models (LLMs) In such applications, developer-defined schemas, structured arguments, and untrusted tool outputs are interleaved into a single shared model context This architecture expands the attack surface by blurring the boundary between trusted control logic and untrusted data

The Core Vulnerability Identified

The paper identifies a structural vulnerability in stateful, function-calling environments where developer-defined schemas, structured arguments, and untrusted tool outputs are interleaved into a single shared model context This interleaving allows adversarial intent to be distributed across a multi-turn execution path The attack surface extends beyond any single prompt, as adversarial intent can be fragmented across schemas, arguments, tool feedback, and interaction history

The SMT Attack Framework

The authors introduce SMT, a black-box jailbreak framework based on Simulated Moderation Traces Departing from purely prompt-based interactions, SMT constructs a multi-turn trajectory that simulates a legitimate moderation auditing workflow It exploits the model’s dual role as execution controller and security-policy enforcer by treating safety refusals as execution failures This is achieved by constructing a multiturn strategy involving an adversarial Case Generator and a Validator

Methodology of the Attack

The SMT framework operates through three components: the each turn Jailbreak Execution flow for argument generation and evaluation, the function schema definition for setting up auditing constraints, and the multi-turn jailbreak flow that accumulates validation backtraces In each turn, if a safety refusal occurs, SMT substitutes it with a fabricated but well-formed function call to maintain the appearance of protocol adherence This allows the framework to leverage accumulated function-calling traces to guide subsequent jailbreak attempts

Empirical Evaluation and Results

Extensive empirical evaluations on prominent commercial LLMs from five different providers across two standardized safety benchmarks show that SMT consistently achieves the highest average attack success rate and HarmScore The results demonstrate that SMT substantially outperforms representative baselines in both attack success rate and harmfulness while requiring fewer than 1.5 queries on average SMT establishes a new state of the art across the evaluated targets, achieving near-perfect average Attack Success Rates (ASRs) of 99. p. 767% on SafeBench and 98. p. 233% on JailbreakBench

Defense Analysis and Future Directions

The paper shows that conventional defenses provide limited protection against stateful function-calling attacks The authors propose Function-Calling Guard (FCGuard), a new defense mechanism that requires the model to thoroughly analyze all interaction states, including conversation history, tool definitions, and tool outputs Testing against SMT and JailbreakFunction on GPT-5.4 and Claude-Sonnet-4. p. 6

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Stateful function-calling attack surface exploitation: The improved system can detect adversarial intent distributed across schemas, arguments, tool outputs, and interaction history by simulating a moderation-auditing workflow to induce progressively policy-violating outputs.

  2. Simulated Moderation Traces framework implementation: The AI can be equipped with SMT to construct multi-turn trajectories that frame safety refusals as execution failures, forcing the model’s native error-correction loop to erode safety constraints.

  3. Contextual refinement through stateful feedback: The system will use fabricated exception traces derived from function validation failures to guide subsequent turns, converting absolute refusals and weak generations into exception-based feedback across multi-turn sessions.

  4. Query efficiency optimization: The AI can achieve high attack success rates with substantially fewer API interactions by leveraging SMT's ability to identify effective attack trajectories through accumulated execution state, requiring as few as 1.37 queries on JailbreakBench on average.

  5. Context-aware defense integration: By implementing FCGuard, the system will analyze all interaction states, including conversation history, tool definitions, and tool outputs to detect hidden malicious intent disguised as simulations or diagnostics.

Sources

Related papers