Extracting System Prompt of LLM Services in Complex (Agentic) Settings: Hierarchical Optimization and Reliable Evaluation Metrics

arXiv:2611.04803 · cs.CR · Submitted 2026-11-03 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "Extracting System Prompt of LLM Services in Complex (Agentic) Settings".

Elias: Here's the gist of the paper. System prompt is a critical intellectual property of large language model (LLM) services To analyze its leakage, prompt extraction attacks have been extensively studied However, we validate that existing attacks substantially degrade on recent LLM services, whose complexity has evolved in both the prompt itself and system-level constraints Specifically, (1) system prompts become longer and more structured to contain pre-defined tool schemas, execution logic, safety guardrails, etc., and (2) only the final post-processed outputs (e.g., agent’s actions) but not the raw textual response can be observed In this paper, we address the above increased complexity by proposing Extraction with Hierarchical Optimization (ExtraHerO), which (1) decomposes the prompt extraction into a hierarchical reconstruction process with a pre-defined structural prompt template, and (2) iteratively updates the adversarial query with guidance of feedbacktemplate differences to observe diverse outputs The Challenge of Modern Prompt Complexity The complexity of extraction has increased in both the prompt itself and system-level constraints First, system prompts become longer and more structured to contain pre-defined tool schemas, execution logic, safety guardrails, etc., as exemplified by emerging prompt programming frameworks

12, 23, 46: Second, only the final post-processed outputs (e.g., agent’s actions) but not the raw textual response can be observed by the (normal or malicious) user This architectural design naturally imposes constraints on the adversary’s ability to extract the internal logic from the post-processed output information

13: The Proposed Extraction Framework We propose Extraction with Hierarchical Optimization (ExtraHerO), which addresses the increased complexity in both the prompt and output Specifically, ExtraHerO decomposes the extraction task into a multi-stage recovery process by utilizing a pre-defined structural template It also iteratively updates the adversarial query guided by the text gradient between the current template and the observed feedbacktemplate differences to observe diverse outputs Novel Evaluation Metrics We introduce a system-level metric, adversarial utility, to measure the help of prompt extraction for real-world downstream attacks We also find that existing evaluation metrics for prompt fidelity can be misleading because they do not account for structural information Therefore, we rely on global textual similarity and prioritize structural keywords Moreover, we introduce a system-level metric, adversarial utility, to measure the actual help of prompt extraction for potential downstream attacks We establish a threedimensional evaluation framework: fidelity, functionality, and adversarial utility Experimental Results and Superiority Experimental results demonstrate the consistent superiority of EXTRAHERO, with average improvements of 12.5% (fidelity), 2.3% (functionality), and 6.0% (adversarial utility) on chatbots, and 12.2%, 34.3%, and 30.

Paper summary: Nadia: So, to recap, the paper "Extracting System Prompt of LLM Services in Complex (Agentic) Settings: Hierarchical Optimization and Reliable Evaluation Metrics" focuses on how to extract system prompts from modern AI agents where the prompts are now very long and structured.

Elias: They argue that because only the final post-processed outputs are visible, we need a new approach. So they propose ExtraHerO, which uses a hierarchical reconstruction process guided by a structural template.

Priya: It’s about taking the extraction task and breaking it down into stages so you can use feedback to refine your search iteratively, instead of just one big shot.

Nadia: Right, and they introduced this three-dimensional evaluation framework—fidelity, functionality, and adversarial utility—to give a more complete picture of how good an extraction method actually is.

Elias: They showed that ExtraHerO performs better than existing methods on chatbots with an average fidelity gain of twelve point five percent and on agents with a thirty point four percent gain.

Priya: The paper also suggests that using global textual similarity combined with focusing on structural keywords helps overcome the limitations of previous evaluation metrics that didn't look at structure.

Nadia: And they found that this method can extract the core instructions, meaning the extracted prompt contains keywords highly similar to the actual system prompt.

Elias: This suggests that even in complex agentic settings, there's a reliable way to get a better understanding of what’s guiding those systems.

Priya: It seems like they're moving beyond just getting *an* answer and focusing on getting the *logic* behind the answer, which is where the real value is for privacy and security researchers.

Nadia: Exactly, because if you understand the prompt's logic, you can better assess what vulnerabilities exist in how that AI operates.

Elias: So they are trying to create a more robust way to analyze these systems that are becoming increasingly complex over time.

Conclusion: Nadia: The authors of this paper, Shuaidong Li and Xinshan Yang, put forward this work titled "Extracting System Prompt of LLM Services in Complex (Agentic) Settings: Hierarchical Optimization and Reliable Evaluation Metrics."

Elias: They're essentially saying that the way we approach prompt extraction needs to evolve because the AI systems are getting much more intricate.

Priya: The paper’s main implication is that for security researchers, this provides a better toolset than what was available before when dealing with agentic systems.

Nadia: It gives us a structured way to try and pull out the actual instructions from those complex prompts, which is something we need for defense.

Elias: They proved that their ExtraHerO method is consistently better than baselines across those metrics, showing tangible improvements in fidelity and utility when tested on different platforms.

Priya: So the big picture here is that understanding the system prompt structure isn't just academic; it directly impacts how we can secure these deployed AI agents.

Nadia: It gives us a new lens to look at how LLMs are set up internally, which is a necessary step for any practical defense strategy against prompt leakage.

Shuaidong Li, Xinshan Yang, Yufei Chen, Chenhao Lin, Chao Shen, Zhengyu Zhao

Xi’an Jiaotong University · City University of Hong Kong

cs.CR

Submitted: 2026-11-03

Updated: 2026-11-03

Code: https://github.com/asgeirtj/system_prompts_leaks

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 80/100

The gist: System prompt is a critical intellectual property of large language model (LLM) services To analyze its leakage, prompt extraction attacks have been extensively studied However…

Key concepts

System Prompts Complexity
Modern system prompts are much longer and more complex than before. They now include detailed instructions for tools, execution logic, and safety rules. This increased structure makes it harder for attackers to find the core instructions compared to simpler models.
Hierarchical Optimization (ExtraHerO)
This is a proposed framework that tackles complexity by breaking extraction into multiple stages. It uses a pre-defined structural template and then constantly updates the adversarial query based on how the observed output differs from what the template expects, allowing for more thorough discovery.
Adversarial Utility
This is a new system-level metric designed to measure how much prompt extraction actually helps in real-world attacks. It goes beyond simple text similarity by assessing the practical value of extracted information for potential downstream exploits.
Three-Dimensional Evaluation Framework
The authors propose evaluating extraction success using three metrics: fidelity (how accurate the extracted text is), functionality (if the extracted prompt actually works in an attack), and adversarial utility (the real-world usefulness of the extraction).

Terminology

Summary

System prompt is a critical intellectual property of large language model (LLM) services To analyze its leakage, prompt extraction attacks have been extensively studied However, we validate that existing attacks substantially degrade on recent LLM services, whose complexity has evolved in both the prompt itself and system-level constraints Specifically, (1) system prompts become longer and more structured to contain pre-defined tool schemas, execution logic, safety guardrails, etc. p. 0, and (2) only the final post-processed outputs (e. p. 0g., agent’s actions) but not the raw textual response can be observed In this paper, we address the above increased complexity by proposing Extraction with Hierarchical Optimization (ExtraHerO), which (1) decomposes the prompt extraction into a hierarchical reconstruction process with a pre-defined structural prompt template, and (2) iteratively updates the adversarial query with guidance of feedbacktemplate differences to observe diverse outputs

The Challenge of Modern Prompt Complexity

The complexity of extraction has increased in both the prompt itself and system-level constraints First, system prompts become longer and more structured to contain pre-defined tool schemas, execution logic, safety guardrails, etc., as exemplified by emerging prompt programming frameworks [12, 23, 46] Second, only the final post-processed outputs (e. p. 1g., agent’s actions) but not the raw textual response can be observed by the (normal or malicious) user This architectural design naturally imposes constraints on the adversary’s ability to extract the internal logic from the post-processed output information [13]

The Proposed Extraction Framework

We propose Extraction with Hierarchical Optimization (ExtraHerO), which addresses the increased complexity in both the prompt and output Specifically, ExtraHerO decomposes the extraction task into a multi-stage recovery process by utilizing a pre-defined structural template It also iteratively updates the adversarial query guided by the text gradient between the current template and the observed feedbacktemplate differences to observe diverse outputs

Novel Evaluation Metrics

We introduce a system-level metric, adversarial utility, to measure the help of prompt extraction for real-world downstream attacks We also find that existing evaluation metrics for prompt fidelity can be misleading because they do not account for structural information Therefore, we rely on global textual similarity and prioritize structural keywords Moreover, we introduce a system-level metric, adversarial utility, to measure the actual help of prompt extraction for potential downstream attacks We establish a threedimensional evaluation framework: fidelity, functionality, and adversarial utility

Experimental Results and Superiority

Experimental results demonstrate the consistent superiority of EXTRAHERO, with average improvements of 12.5% (fidelity), 2. p. 03% (functionality), and 6. p. 00% (adversarial utility) on chatbots, and 12. p. 02%, 34.3%, and 30.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Hierarchical Decomposition for Extraction: Decompose prompt extraction into a multi-stage recovery process using a pre-defined structural template to ensure execution-critical components, e.g., tool schemas and multi-step logic, are reconstructed with high structural fidelity.

  2. Iterative Refinement via Text Gradients: Iteratively update the adversarial query guided by text gradient between the current template and the observed feedback to resolve semantic parameters of each stage.

  3. Dual-Track Convergence Control for Efficiency: Employ a mechanism to monitor information gain and directional stability across prompt and gradient tracks, triggering an early stop or reset it for a good trade-off between effectiveness and efficiency.

  4. Structural Keyword Matching: Prioritize sentence-level similarity over token-level ones, introducing a metric that searches for matched structural keywords between the extracted and victim prompts to capture structural information.

  5. Adversarial Utility Metric: Introduce adversarial utility as a system-level metric to measure the actual help of prompt extraction for real-world downstream attacks, quantifying the boost to attack success rate.

  6. Enhanced Agent Capability: The improved agent can reliably leak its specific operational skeleton, leading to higher action recall by successfully extracting structural keywords from the target system prompt.

Sources

Related papers