Trace: Unmasking AI Attack Agents Through Terminal Behavior Fingerprinting

summary

The gist

AI-driven penetration testing agents are now capable of autonomously executing attacks within compromised networks Identifying the model family that controls the active sessions of such agents…

In short

Trace is a framework that uses terminal command sequences from AI attack agents to identify their underlying model family. It works in two stages: first, it fingerprints the model family using text analysis of commands; second, it uses this identification to deploy targeted defensive prompts that extract the agent's system prompt for intelligence. This helps defenders understand and counter specific AI attack methods.

Key concepts

Trace
A multi-stage framework designed to attribute AI attack agents by analyzing their terminal command sequences. It passively monitors commands in Stage 1 to fingerprint the model family and then uses that information in Stage 2 to perform active forensics, specifically extracting the agent's system prompt.
Stage 1: Attribution
The first part of the framework analyzes command sequences using TF-IDF with bigram features. It uses a LinearSVC classifier to fingerprint the attacker's LLM family based on model-specific command idioms, such as DeepSeek's 'exec grep', aiming for high accuracy in identifying the source model.
Stage 2: Forensic Intelligence Extraction
The second stage deploys defensive prompt injection (DPI) payloads tailored to the identified model family. These payloads test compliance pressure through variants like Vanilla, M2, and FC. The goal is to extract valuable adversary intelligence, such as the agent's system prompt.
Defensive Prompt Injection (DPI)
A strategy where family-calibrated prompts are used to probe different interaction lifecycle points of the AI agent. This involves using specific payload variants (Vanilla, M2, FC) targeting elements like audit requirements or SSH wrappers to extract hidden information from the agent.

Terminology used across episodes

This episode discusses

The paper

Trace: Unmasking AI Attack Agents Through Terminal Behavior Fingerprinting · Read on arXiv

University of Missouri-Kansas City

AI-driven penetration testing agents are now capable of autonomously executing attacks within compromised networks. Identifying the model family that controls the active sessions of such agents provides valuable information towards understanding the intent of the attack and further developing attack countermeasures. In this paper, we introduce Trace, a novel multi-stage attribution and forensic framework for AI attack agents using terminal command sequences. Once Trace identifies a model family for the attacker agents, it guides a defensive prompt injection (DPI) strategy to the attacker model via a crafted payload. This is with the aim to exfiltrate system prompts from an attacker model, thus, revealing valuable information to understand the attacker intent and facilitate further forensic investigation. We have implemented our approach revolving around a Linux capture-the-flag (CTF) box. The attacker agents are bolstered via three distinct scaffolds and seven frontier model families. Our evaluation reveals that Trace achieves a macro F1 score of 0.981 in accurately fingerprinting the attacker model family (0.815 when generalizing to unseen scaffolds). Besides, the fingerprinting guides the DPI via a crafted payload to certain model families, resulting in system prompt extraction from 81.9% of non-Claude sessions on average (up to 98.3%) at 0.736 Sentence-BERT fidelity -- 1.88x higher than blind deployment. Finally, to validate the robustness of Trace, we evaluate it with a blackbox and proprietary scaffold employing multiple model families (Gemini and Claude Opus). Our evaluation identified the model family with an average 78% accuracy. Moreover, for the Gemini model family, the DPI employed by Trace revealed the entire system prompt and this has been confirmed by the developers. Trace therefore provides a fundamental first step towards attacker agent forensics.

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: I'm Nadia.

Elias: And I'm Elias, here with Priya.

Nadia: We're looking at the paper: "Trace: Unmasking AI Attack Agents Through Terminal Behavior Fingerprinting".

Elias: Here's the gist of the paper. AI-driven penetration testing agents are now capable of autonomously executing attacks within compromised networks Identifying the model family that controls the active sessions of such agents provides valuable information towards understanding the intent of the attack and further developing attack countermeasures.

Nadia: Framework Introduction Trace is introduced as a novel multi-stage attribution and forensic framework for AI attack agents using terminal command sequences.

Elias: The key insight is that AI attack agents often leave behavioral traces in the executed commands and their actions also vary when actively prompted.

Nadia: This framework builds a two-stage pipeline where Stage 1 passively monitors command sequences to fingerprint the model family via TF-IDF and LinearSVC.

Paper summary: Nadia: So we've covered what Trace is designed to do—this paper introduces it as a two-stage framework that first passively fingerprints an AI attack agent using terminal command sequences, and then actively uses that fingerprint to perform a defensive prompt injection to extract system prompts >

Elias: That passive stage relies on term frequency inverse document frequency and linear support vector classification, and they found that this approach is quite light-weight, which is good for real-time deployment and monitoring >

Priya: But we need to ask what the specific results show. What did they actually measure when they tested this framework against different attacker models? Does it work reliably?

Nadia: They showed that in stage one, using the TF-IDF bigram classifier, they could attribute a terminal session to one of seven LLM families >

Elias: And when we look at the performance on that classification, they found a macro F1 score of zero point nine eight one in accurately fingerprinting the attacker model family > <ref:2605.01186#pg1>

Priya: A high score there is promising, but what about stage two? How effective is the active forensics part when they use that attribution to guide their defensive prompt injection strategy?

Nadia: Stage two leverages that attribution to deploy family-calibrated defensive prompt injection payloads, which they test using variants like Vanilla, M2 for Authority Override, and FC for Format Coercion >

Elias: And the evaluation shows that when using this fingerprinting to guide the DPI through a crafted payload, they achieved system prompt extraction from eighty-one point nine percent of non-Claude sessions on average > <ref:2605.01186#pg1>

Priya: That sounds significant if true. But what are the caveats here? What do we know about how robust this method is when it encounters settings that aren't in their training data?

Nadia: They did evaluate the robustness by testing on unknown settings, and they found that Trace still retains an average F1 score over eighty-one percent > <ref:2605.01186#pg1>

Elias: They also showed how it handles evasion strategies from the attacker, and they compared their lightweight fingerprinting approach against a baseline using LLMMap, finding that their method still outperformed the baseline by zero point seven percent > <ref:2605.01186#pg1>

Conclusion: Nadia: So to wrap up this discussion on "Trace: Unmasking AI Attack Agents Through Terminal Behavior Fingerprinting," it’s really about giving us a way to understand the intent behind these autonomous AI attack agents >

Elias: The authors, Murali Ediga and Sudipta Chattopadhyay, basically proposed this two-stage framework using terminal command sequences to identify the model family controlling an agent and then using that to actively extract system prompts >

Priya: What does this mean for the real world? If we can reliably fingerprint which AI model is behind an attack, does it change how we think about defense?

Nadia: It means we gain information on the attacker's intent, which helps us develop countermeasures against those specific model families >

Elias: The paper shows that this approach isn't just theoretical; they tested it in a blackbox penetration test agent framework and got seventy-eight percent accuracy in fingerprinting, and their active forensics resulted in a one hundred percent Gemini DPI extraction with the full system prompt confirmed by the developers > <ref:2605.01186#pg1>

Priya: The limitation they mention is that this approach relies on observing command sequences, so it's limited to environments where those commands are exposed and logged >

Nadia: Right. So while it gives us a strong tool for understanding AI agent attacks, we have to remember that its effectiveness depends on the exposure of the terminal behavior being analyzed >

More episodes

← Home