Extracting System Prompt of LLM Services in Complex (Agentic) Settings: Hierarchical Optimization and Reliable Evaluation Metrics
summary
The gist
System prompt is a critical intellectual property of large language model (LLM) services To analyze its leakage, prompt extraction attacks have been extensively studied However…
In short
Existing methods for extracting LLM system prompts fail against modern, complex AI services because prompts are now long and structured with tools and rules, and users only see final actions, not raw text. This work proposes ExtraHerO, a method that breaks extraction into stages using a structural template and iteratively refines the attack query based on feedback differences to find hidden prompt information.
Key concepts
- System Prompts Complexity
- Modern system prompts are much longer and more complex than before. They now include detailed instructions for tools, execution logic, and safety rules. This increased structure makes it harder for attackers to find the core instructions compared to simpler models.
- Hierarchical Optimization (ExtraHerO)
- This is a proposed framework that tackles complexity by breaking extraction into multiple stages. It uses a pre-defined structural template and then constantly updates the adversarial query based on how the observed output differs from what the template expects, allowing for more thorough discovery.
- Adversarial Utility
- This is a new system-level metric designed to measure how much prompt extraction actually helps in real-world attacks. It goes beyond simple text similarity by assessing the practical value of extracted information for potential downstream exploits.
- Three-Dimensional Evaluation Framework
- The authors propose evaluating extraction success using three metrics: fidelity (how accurate the extracted text is), functionality (if the extracted prompt actually works in an attack), and adversarial utility (the real-world usefulness of the extraction).
Terminology used across episodes
This episode discusses
- Extracting System Prompt of LLM Services in Complex (Agentic) Settings: Hierarchical Optimization and Reliable Evaluation Metrics · Paper Radio
- Jailbreaking Black Box Large Language Models in Twenty Queries
- Reporting LLM Prompting in Automated Software Engineering: A Guideline Based on Current Practices and Expectations
- Why Are My Prompts Leaked? Unraveling Prompt Extraction Threats in Customized Large Language Models
- LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked
- UI-TARS: Pioneering Automated GUI Interaction with Native Agents
- SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks
- TextGrad: Automatic "Differentiation" via Text
- Effective Prompt Extraction from Language Models
- Universal and Transferable Adversarial Attacks on Aligned Language Models
The paper
Extracting System Prompt of LLM Services in Complex (Agentic) Settings: Hierarchical Optimization and Reliable Evaluation Metrics · Read on arXiv
Shuaidong Li, Xinshan Yang, Yufei Chen, Chenhao Lin, Chao Shen, Zhengyu Zhao
Xi’an Jiaotong University · City University of Hong Kong
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "Extracting System Prompt of LLM Services in Complex (Agentic) Settings".
Elias: Here's the gist of the paper. System prompt is a critical intellectual property of large language model (LLM) services To analyze its leakage, prompt extraction attacks have been extensively studied However, we validate that existing attacks substantially degrade on recent LLM services, whose complexity has evolved in both the prompt itself and system-level constraints Specifically, (1) system prompts become longer and more structured to contain pre-defined tool schemas, execution logic, safety guardrails, etc., and (2) only the final post-processed outputs (e.g., agent’s actions) but not the raw textual response can be observed In this paper, we address the above increased complexity by proposing Extraction with Hierarchical Optimization (ExtraHerO), which (1) decomposes the prompt extraction into a hierarchical reconstruction process with a pre-defined structural prompt template, and (2) iteratively updates the adversarial query with guidance of feedbacktemplate differences to observe diverse outputs The Challenge of Modern Prompt Complexity The complexity of extraction has increased in both the prompt itself and system-level constraints First, system prompts become longer and more structured to contain pre-defined tool schemas, execution logic, safety guardrails, etc., as exemplified by emerging prompt programming frameworks
12, 23, 46: Second, only the final post-processed outputs (e.g., agent’s actions) but not the raw textual response can be observed by the (normal or malicious) user This architectural design naturally imposes constraints on the adversary’s ability to extract the internal logic from the post-processed output information
13: The Proposed Extraction Framework We propose Extraction with Hierarchical Optimization (ExtraHerO), which addresses the increased complexity in both the prompt and output Specifically, ExtraHerO decomposes the extraction task into a multi-stage recovery process by utilizing a pre-defined structural template It also iteratively updates the adversarial query guided by the text gradient between the current template and the observed feedbacktemplate differences to observe diverse outputs Novel Evaluation Metrics We introduce a system-level metric, adversarial utility, to measure the help of prompt extraction for real-world downstream attacks We also find that existing evaluation metrics for prompt fidelity can be misleading because they do not account for structural information Therefore, we rely on global textual similarity and prioritize structural keywords Moreover, we introduce a system-level metric, adversarial utility, to measure the actual help of prompt extraction for potential downstream attacks We establish a threedimensional evaluation framework: fidelity, functionality, and adversarial utility Experimental Results and Superiority Experimental results demonstrate the consistent superiority of EXTRAHERO, with average improvements of 12.5% (fidelity), 2.3% (functionality), and 6.0% (adversarial utility) on chatbots, and 12.2%, 34.3%, and 30.
Paper summary: Nadia: So, to recap, the paper "Extracting System Prompt of LLM Services in Complex (Agentic) Settings: Hierarchical Optimization and Reliable Evaluation Metrics" focuses on how to extract system prompts from modern AI agents where the prompts are now very long and structured.
Elias: They argue that because only the final post-processed outputs are visible, we need a new approach. So they propose ExtraHerO, which uses a hierarchical reconstruction process guided by a structural template.
Priya: It’s about taking the extraction task and breaking it down into stages so you can use feedback to refine your search iteratively, instead of just one big shot.
Nadia: Right, and they introduced this three-dimensional evaluation framework—fidelity, functionality, and adversarial utility—to give a more complete picture of how good an extraction method actually is.
Elias: They showed that ExtraHerO performs better than existing methods on chatbots with an average fidelity gain of twelve point five percent and on agents with a thirty point four percent gain.
Priya: The paper also suggests that using global textual similarity combined with focusing on structural keywords helps overcome the limitations of previous evaluation metrics that didn't look at structure.
Nadia: And they found that this method can extract the core instructions, meaning the extracted prompt contains keywords highly similar to the actual system prompt.
Elias: This suggests that even in complex agentic settings, there's a reliable way to get a better understanding of what’s guiding those systems.
Priya: It seems like they're moving beyond just getting *an* answer and focusing on getting the *logic* behind the answer, which is where the real value is for privacy and security researchers.
Nadia: Exactly, because if you understand the prompt's logic, you can better assess what vulnerabilities exist in how that AI operates.
Elias: So they are trying to create a more robust way to analyze these systems that are becoming increasingly complex over time.
Conclusion: Nadia: The authors of this paper, Shuaidong Li and Xinshan Yang, put forward this work titled "Extracting System Prompt of LLM Services in Complex (Agentic) Settings: Hierarchical Optimization and Reliable Evaluation Metrics."
Elias: They're essentially saying that the way we approach prompt extraction needs to evolve because the AI systems are getting much more intricate.
Priya: The paper’s main implication is that for security researchers, this provides a better toolset than what was available before when dealing with agentic systems.
Nadia: It gives us a structured way to try and pull out the actual instructions from those complex prompts, which is something we need for defense.
Elias: They proved that their ExtraHerO method is consistently better than baselines across those metrics, showing tangible improvements in fidelity and utility when tested on different platforms.
Priya: So the big picture here is that understanding the system prompt structure isn't just academic; it directly impacts how we can secure these deployed AI agents.
Nadia: It gives us a new lens to look at how LLMs are set up internally, which is a necessary step for any practical defense strategy against prompt leakage.
More episodes
- 2608.27496-ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection
- Daily Summary for 2026-11-19
- 2607.07085-Is Randomness Necessary for Adaptive Data Analysis?
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2609.10601-Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows
- 2610.38786-Adversarial Process Disturbances at a Deployable Alarm Budget: Detector Evaluation on a Simulator and a Real Plant
- 2511.08660-Binary and Multiclass Cyberattack Classification on GeNIS Dataset
- 2604.27302-Static Attribution of Android Residential Proxy Malware Using Graph Kernels
- Daily Summary for 2026-11-18