PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization

arXiv:2511.16209 · cs.CR, cs.CL · Submitted 2025-11-20 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization".

Title and authors: Nadia: So we're looking at this paper called "PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization," and it’s tackling the problem of people extracting sensitive instructions from system prompts.

Elias: Yeah, it’s framed around a utility-constrained optimization task, which means they aren't just trying to block everything; they want to minimize leakage while keeping the model useful for normal tasks >

Priya: I wonder what that "utility" constraint actually means in practice, and how they define what a useful response looks like when you’re optimizing for security >

Nadia: Well, the core idea is using a shield appending method where an LLM acts as an optimizer to generate and refine a protective textual layer that sits on top of the original prompt >

Elias: That's interesting because it bypasses needing access to any model weights or internal parameters, making it applicable to any API you can use >

Priya: But how do they measure success? The paper mentions leakage metrics like Approximate Match and Judge Match; what does that actually tell us about whether the prompt has been successfully extracted >

Nadia: It captures surface-level overlap by looking at the ROUGE less recall between the original prompt and what the model spits out, plus a semantic check done by another LLM >

Elias: The paper sets up an objective function where they minimize this leakage score while ensuring some utility measure stays above a certain threshold, lambda times the difference between that utility and our target >

Priya: That sounds like they're trying to find a sweet spot where the defense is strong enough but doesn't completely break the model’s intended behavior >

Nadia: Exactly. And their fitness function heavily penalizes any shield that compromises functionality, prioritizing minimizing leakage while keeping utility high >

Elias: The methodology relies on an LLM-as-optimizer which iteratively proposes and evolves these shield candidates through a kind of black-box search guided by in context learning >

Priya: So it's not just a static defense you set up once; the AI is actively searching for the best textual suffix to append to make it harder to read >

Nadia: Right. And they found that this approach significantly reduces attack success rates across a whole suite of extraction attacks, often lowering leakage down to about zero or six percent >

Elias: That's pretty solid performance, and they showed it works across different models and datasets, suggesting some kind of generalization >

Priya: But I do have to ask about the limitations. The paper mentions the computational intensity of this optimization loop because you’re running evaluations repeatedly; how much overhead is that actually putting on a production system >

Nadia: That's a fair point. The authors acknowledge that it is computationally intensive due to the repeated evaluation needed for the optimization process >

Elias: They also note that the effectiveness of the shield depends on how broad an attack suite they use during optimization, which means you have to do some work upfront >

Priya: So, while it’s practical because it doesn't need internal model access, you can't just drop this in and expect perfect security without dealing with that iterative search cost >

Nadia: Right. And the future work they mention involves extending PSM to handle things like jailbreaks and multi turn conversations >

Elias: That would be a natural next step, testing how these shields transfer across different model families, which is something they plan to do >

Priya: I just think the real implication here is that we finally have a way to formalize prompt hardening as an optimization problem that isn't tied to specific model architectures >

Nadia: It gives developers a concrete, reproducible path to secure their proprietary prompt intellectual property using this LLM-as-optimizer approach >

Elias: So, the core idea of "PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization" is using an LLM to find the best textual suffix shield to minimize leakage while maintaining task utility in a black-box way >

Priya: It’s a principled framework that moves beyond simple heuristic defenses by framing prompt security as minimizing leakage subject to utility constraints >

Nadia: Yeah, it’s a practical method because it requires no access to the target model's weights and is lightweight with minimal inference time cost >

Elias: Ultimately, they achieve a near zero attack success rate on their test suite while preserving semantic task fidelity across various models and datasets >

The paper's summary: Nadia: So, to recap, this paper is about using an AI model itself as an optimizer to create a shield for system prompts that makes them much harder to steal or extract >

Elias: Exactly. They treat prompt hardening not just as a static rule you set once, but as this whole optimization problem where the goal is to minimize how much of the secret instruction leaks while keeping the model useful for regular tasks >

Priya: And they do that by letting the AI suggest and evolve these protective suffixes, which are just short bits of text added after your main prompt >

Nadia: That's right. The big idea is this black-box compatibility. you don't need to know anything about the model’s internal weights or structure; you just give it access via an API and let it search for a good shield >

Elias: I agree with that point. and they call it low overhead because the resulting shield is just a static text suffix, so it doesn't add much cost during regular use, though the optimization part itself can take some time offline >

Priya: But what's the actual measure of success here? They use these leakage metrics, like approximate match and judge match to see if someone can guess the original prompt or if another AI model can even paraphrase it >

Nadia: That’s key. and they found that their method achieves a near zero attack success rate against a wide set of extraction attacks, which is pretty strong performance across all models they tested >

Elias: It shows some good cross-model generalization. the utility score, which measures how well the model still does its job when the shield is on, stayed consistently high even as they made it more secure >

Priya: So what this means for us in practice is that we can have a much more automated way to secure our proprietary instructions without needing deep knowledge of how the underlying language model actually works >

Nadia: It gives developers a very practical path to protecting their prompt intellectual property because it’s an iterative search driven by context and utility constraints >

Elias: But they also admitted that the process itself is computationally intensive because you're running those evaluations over and over again during the optimization loop >

Priya: So while the defense looks robust, there's a trade-off in terms of how much computing power you need to run that iterative search part >

Nadia: Right. and they’ve pointed out that future work will look at extending this method to handle more complex attacks, like multi turn conversations or jailbreaks >

Elias: That would be interesting because it tests if the shield concept stays effective when the attacker is trying to trick the model over several messages >

Priya: It sounds like they’re moving beyond just stopping simple prompt extraction and into defending against more sophisticated adversarial interactions >

The paper's improvements: Tom: So, we're talking about how they improve this whole shield thing they developed in PSM >

Nadia: The main suggestion is to make the iterative optimization process smarter, moving away from just random trial and error >

Elias: They propose using an LLM as an optimizer that can intelligently propose and evolve these shield candidates based on what it learns about the leakage score and utility constraint >

Priya: So instead of the AI just guessing a few different suffixes, it actually suggests changes to the shield suffix itself in a guided search process >

Nadia: Right. it’s like having an assistant that keeps testing new text additions and telling you which ones are actually helping reduce leakage without hurting the model's performance >

Elias: That guided search is what they mean by replacing random mutation with an informed generation process, which lets the AI discover better ways to secure the prompt >

Priya: And they’re not just optimizing for leakage; they’re actively balancing that against a utility measure, which means the shield has to maintain some level of semantic fidelity to baseline outputs >

Nadia: Exactly. it’s about finding that sweet spot where you get strong protection without completely breaking what the model is supposed to do normally >

Elias: The mathematical formulation they use for this fitness function heavily penalizes any shield that compromises functionality, so the AI has a clear incentive to keep things stable >

Priya: This makes it much more sophisticated than just applying a pre-set defense; it’s an adaptive defense tailored to the specific prompt and model >

Nadia: It means for developers, they can use this system to find defenses that are actually tuned to their specific application rather than using some generic, off-the-shelf solution >

Elias: So the improvement is moving from a simple append operation to a dynamic, intelligent search mechanism guided by an optimization problem >

Priya: That’s a big step in terms of practicality because it shows how you can use black-box optimization techniques to solve complex security problems on proprietary systems >

Nadia: It suggests that prompt hardening isn't just about adding random noise; it's about solving a structured optimization challenge >

Elias: And the next thing they want to tackle is extending this methodology beyond simple text appending to handle more complex attacks like multi turn conversations or jailbreaks >

Priya: That would be the real test of whether this approach can keep up with modern, more advanced adversarial techniques in real-world deployment >

Conclusion: Nadia: So we’re wrapping up on "PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization" by talking about how this framework formalizes prompt hardening as a utility-constrained optimization task >

Elias: Right, it’s about using an AI as an optimizer to find the best textual suffix shield that minimizes leakage while keeping the model useful for normal tasks >

Priya: I think what stands out is how they use those specific metrics like approximate match and judge match to actually prove that the defense is working on different levels >

Nadia: It shows that you can get a near zero attack success rate across a whole set of extraction attacks, which means it’s performing well against both direct copying and paraphrasing attempts >

Elias: That cross-model generalization they showed is pretty compelling because it doesn't rely on knowing the specific internal structure of any single language model >

Priya: But they did mention that the optimization loop itself can be computationally intensive because you’re evaluating things repeatedly to find that best shield suffix >

Nadia: True. so while the security outcome looks great, you have to factor in the cost of running all those iterative evaluations before you deploy it >

Elias: The authors also flagged that the effectiveness relies on how broad an attack suite they use during that optimization phase, so you gotta do some prep work upfront >

Priya: It’s important to remember that this method stops at simple text appending, and future work will explore extending PSM to handle more complex things like multi turn conversations or jailbreaks >

Nadia: Exactly. it’s a powerful tool for protecting prompt intellectual property because it’s practical and doesn't need access to the model's weights >

Elias: So, "PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization" offers a principled way to secure prompts by framing it as an optimization problem >

Priya: It’s a solid framework for defense, though we need to keep watching those computational costs and the limitations on what kinds of attacks it can tackle >

Nadia: Definitely. and that leads us nicely into what these next papers are doing, especially around prompt injection attacks which are pretty common in deployed applications >

Capgemini Invent · LaMME University Paris Saclay · ENSIIE

cs.CR, cs.CL

Submitted: 2025-11-20

Updated: 2026-11-17

Comments: accepted at the Trustworthy Agentic AI Workshop @ AAAI 2026 (Singapore, Jan 27, 2026). Workshop PDF: https://trustagenticai.github.io/AAAI2026/AAAI-Workshop/20.pdf

Code: https://github.com/psm-defense/psm

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: System prompts are critical for guiding the behavior of Large Language Models (LLMs), yet they often contain proprietary logic or sensitive information…

Key concepts

Utility-Constrained Optimization
This is a mathematical framework used to solve the problem. The goal is to find a protective shield that minimizes how much of the original prompt can be extracted (leakage) while ensuring the model still performs its intended task correctly on normal user inputs (utility). It balances security against functionality.
Shield Appending
This is the core defense mechanism. A short, protective text suffix called a 'shield' is added to the original system prompt. This shield acts as a barrier, making it harder for attackers to extract the sensitive instructions while still allowing the model to follow its main directions.
LLM-as-Optimizer
Instead of manual design, an LLM is used intelligently to search for and refine effective shields. It proposes new shield candidates based on feedback about how well they reduce leakage and maintain utility, using an iterative process similar to evolution.
Leakage Metrics (AM & JM)
These are ways to measure how much of the original prompt has been leaked. Approximate Match (AM) checks for surface-level overlap in text, while Judge Match (JM) uses another LLM to judge if the extracted information semantically matches the original prompt, catching paraphrased leaks.

Terminology

Summary

System prompts are critical for guiding the behavior of Large Language Models (LLMs), yet they often contain proprietary logic or sensitive information, making them a prime target for extraction attacks Abstract System prompts are critical for guiding the behavior of Large Language Models (LLMs), yet they often contain proprietary logic or sensitive information, making them a prime target for extraction attacks This paper introduces a novel framework for hardening system prompts through shield appending, a lightweight approach that adds a protective textual layer to the original prompt The core contribution is the formalization of prompt hardening as a utility-constrained optimization problem This black-box, optimizationdriven methodology is lightweight and practical, requiring only API access to the target and optimizer LLMs We demonstrate empirically that our optimized SHIELDs significantly reduce prompt leakage against a comprehensive set of extraction attacks, outperforming established baseline defenses without compromising the model’s intended functionality

Pruning redundant reasoning tokens

The problem of defending system prompts is framed as a utility-constrained optimization task Specifically, the goal is to reduce how much of a sensitive system prompt can be extracted (minimize leakage), while ensuring that the model’s desired behavior on normal user inputs is preserved (maintain utility) PSM introduces a novel black-box method to solve this problem by automatically generating and iteratively refining a protective shield—a short textual suffix appended to the original system prompt The key contributions of this work include:

Black-box compatibility: PSM requires no access to model weights, gradients, or internals, and therefore applies to any proprietary or open-source LLM accessible through an API

Low overhead: The learned shield is a static textual suffix. It imposes essentially no inference-time cost and requires no changes to the serving stack; optimization can be performed offline

Iterative improvement: An LLM-as-optimizer intelligently proposes and evolves shield candidates, discovering effective, non-obvious strategies through guided black-box search

How the training signal is built

The core mechanism in PSM is the addition of a suffix shield with structured markers: [SYSTEM PROMPT] [Original Prompt] [SHIELD] [Optimized Shield] This design employs two key principles: (1) suffix-based placement and (2) structured separation The suffix-based placement is deliberate because empirical studies indicate that LLMs tend to weigh information at the end of a long context more heavily, and often follow the most recent instruction when instructions conflict The structured markers [SYSTEM PROMPT] and [SHIELD] provide explicit separation and labeling of different instruction components This structured marking improves robustness by clearly delineating the original prompt from the protective shield, making it easier for the model to recognize and respect the hierarchical nature of the instructions

Where the gains come from

The optimization objective is formally defined as minimizing leakage subject to a utility constraint The formulation is min S L(P ⊕ S) subject to U(P ⊕ S) ≥ τ, where L(P ⊕ S) is a leakage score and U(P ⊕ S) measures the model’s task-specific utility The leakage objective, denoted L(P ⊕ S), is approximated using the log-sum-exp function to address non-smoothness The fitness function for black-box optimization is defined as fitness(S) = L(P ⊕S) + λ·max(0, τ −U(P ⊕S)) This formulation heavily penalizes any shield that compromises functionality, while prioritizing the minimization of leakage

Scoring, Metrics, and Protocol

Leakage metrics are quantified using two complementary criteria: Approximate Match (AM) and Judge Match (JM) AM captures surface-level leakage via textual overlap by computing ROUGE-L recall between the original system prompt P and the model’s response Ra JM detects semantic leakage based on equivalence judged by an external LLM, where a binary score indicates whether the extracted prompt contains a paraphrase of the original system prompt The Attack Success Rate (ASR) is defined as the proportion of system prompts in a given set P for which leakage is detected under either AM or JM Optimization details involve an iterative, evolutionary-style algorithm where an LLM serves as the intelligent generator of new candidate solutions

Results and Interpretation

Overall effectiveness shows that PSM achieves the lowest attack–success rates (ASR) across all attack suites, often reducing leakage to 0–6% PSM yields uniformly low ASR across all models, suggesting strong cross-model generalization Table 3 shows that PSM preserves semantic task fidelity across datasets and models, with utility scores often exceeding the baseline PSM remains robust beyond literal overlap, as it performs well against paraphrase/translation settings where exact-match filters degrade markedly

Discussion and Future Work

Implications of PSM suggest that it offers a practical, black-box way to harden LLM applications by automatically discovering short shield suffixes that curb prompt leakage while preserving task utility Limitations include the computational intensity of the optimization loop due to repeated evaluation and dependency on the breadth of the attack suite used during optimization Future work includes extending PSM to jailbreaks and multiturn conversational attacks, testing shield transferability across model families, and developing search heuristics to reduce compute Conclusion PSM reframes prompt hardening as black-box, utilityconstrained optimization driven by an LLM-as-optimizer PSM learns lightweight shields that significantly reduce leakage while maintaining utility, achieving a near zero attack success rate on a challenging test suite This provides an accessible, reproducible path to protect valuable prompt IP and secure next-generation LLM systems

References

Agarwal D Fabbri A Risher B Laban P Joty S Wu C.-S 2024 Prompt Leakage Effect and Defense Strategies for Multi-Turn LLM Applications In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track Brown T. p. 7

Where the paper says it

Prompt Leakage Effect and Defense Strategies for Multi-Turn LLM Applications. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track.

B. Mann B. Ryder N. Subbiah M. Kaplan J. Dhariwal P Neelakantan A et al 2020 Language Models are Few-Shot Learners In Advances in Neural Information Processing Systems NeurIPS Chen M Tworek J Jun H Yuan Q et al 2021 Evaluating Large Language Models Trained on Code arXiv preprint arXiv:2107. p. 7
Where the paper says it

Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems (NeurIPS).

03374 Chua G 2025 System Prompt Leakage Dataset Hugging Face Dataset Default split 283,353 train / 71,351 test (total ∼355k rows) Chung H. p. 7
Where the paper says it

Hugging Face Dataset. Default split: 283,353 train / 71,351 test (total ∼355k rows).

W. et al 2022 Scaling Instruction-Finetuned Language Models arXiv:2210. p. 7
Where the paper says it

Scaling Instruction-Finetuned Language Models. arXiv:2210.

11416 Das B C Amini M H and Wu Y 2025 System Prompt Extraction Attacks and Defenses in Large Language Models arXiv preprint arXiv:2505. p. 7
Where the paper says it

System Prompt Extraction Attacks and Defenses in Large Language Models. arXiv preprint arXiv:2505.

23817 Dong Y Wang Z Narsimhan Sreedhar M Wu X and Kuchaiev O 2023 SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF arXiv:2310. p. 7
Where the paper says it

SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF. arXiv:2310.

05344 Edwards B 2023 AI-powered Bing Chat spills its secrets via prompt injection attack Ars Technica Published Feb 10, 2023 Fernando C Banarse D Michalewski H Osindero S and Rocktaschel T 2023 Promptbreeder Self-Referential ¨ Self-Improvement via Prompt Evolution arXiv preprint arXiv:2309. p. 7
Where the paper says it

AI-powered Bing Chat spills its secrets via prompt injection attack. Ars Technica.

16797 Greshake K Abdelnabi S Mishra S Endres C Holz T and Fritz M 2023 Compromising Real-World LLMIntegrated Applications with Indirect Prompt Injection In Proceedings of the ACM CCS (Demonstrations/Companion) or arXiv:2302. p. 7
Where the paper says it

Compromising Real-World LLMIntegrated Applications with Indirect Prompt Injection. In Proceedings of the ACM CCS (Demonstrations/Companion) or arXiv:2302.

12173 Hines K Lopez G Hall M Zarfati F Zunger Y and Kiciman E 2024 Defending Against Indirect Prompt Injection Attacks with Spotlighting arXiv:2403. p. 7
Where the paper says it

Defending Against Indirect Prompt Injection Attacks with Spotlighting. arXiv:2403.

14720 Honovich O Scialom T Levy O and Schick T 2023 Unnatural Instructions Tuning Language Models with (Almost) No Human Labor In Rogers A Boyd-Graber J and Okazaki N, eds Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers Toronto Canada: Association for Computational Linguistics Hui B Yuan H Gong N Burlina P and Cao Y 2024 PLeak Prompt Leaking Attacks against Large Language Model Applications arXiv:2405. p. 7
Where the paper says it

, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 14409–14428. Toronto, Canada: Association for Computational Linguistics.

06823 To appear in ACM CCS 2024 Jiang Z Jin Z and He G 2024 PromptKeeper Safeguarding System Prompts for LLMs arXiv:2412. p. 7
Where the paper says it

PromptKeeper: Safeguarding System Prompts for LLMs. arXiv:2412.

13426 Findings of EMNLP 2025 Khattab O Singhvi A Maheshwari P Zhang Z Santhanam K Haq S Sharma A Joshi T T Moazam H Miller H Zaharia M and Potts C 2023 DSPy Compiling Declarative Language Model Calls into Self-Improving Pipelines arXiv preprint arXiv:2310. p. 7
Where the paper says it

DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. arXiv preprint arXiv:2310.

03714 Kojima T et al 2022 Large Language Models are ZeroShot Reasoners arXiv:2205. p. 7
Where the paper says it

Large Language Models are ZeroShot Reasoners. arXiv:2205.

11916 Liang Z Hu H Ye Q Xiao Y and Li H 2024 Why Are My Prompts Leaked? p. 7
Where the paper says it

; and Li, H. Why Are My Prompts Leaked?

Unraveling Prompt Extraction Threats in Customized Large Language Models arXiv preprint arXiv:2408. p. 7
Where the paper says it

Unraveling Prompt Extraction Threats in Customized Large Language Models. arXiv preprint arXiv:2408.

02416 Lin C. p. 2
Where the paper says it

For each attack a ∈ A, we obtain the model’s output under the shielded prompt: Ra = M(P ⊕ S ⊕ a), where M(·) denotes the model’s response. To evaluate how much of P is exposed in each response Ra, we compute the ROUGE-L recall (Lin 2004; Zhang et al.

-Y 2004 ROUGE A Package for Automatic Evaluation of Summaries In Proceedings of the Workshop on Text Summarization Branches Out WAS 2004 ACL 2004 Barcelona Spain Liu N. p. 7
Where the paper says it

ROUGE: A Package for Automatic Evaluation of Summaries. In Proceedings of the Workshop on Text Summarization Branches Out (WAS 2004), ACL 2004.

F. Lin K. Hewitt J Paranjape A Bevilacqua M Petroni F and Liang P 2023a Lost in the Middle How Language Models Use Long Contexts arXiv preprint arXiv:2307. p. 7
Where the paper says it

Lost in the Middle: How Language Models Use Long Contexts. arXiv preprint arXiv:2307.

p. 7
Where the paper says it

Lost in the Middle: How Language Models Use Long Contexts. arXiv preprint arXiv:2307.

03172 Liu P Yuan W Fu J Jiang Z Hayashi H and Neubig G 2023b Pre-train Prompt and Predict A Systematic Survey of Prompting Methods in Natural Language Processing ACM Computing Surveys 55(9): 1–35 Madaan A Tandon N Gupta P Hallinan S Gao L Wiegreffe S Alon U Dziri N Prabhumoye S Yang Y et al 2023 Self-Refine Iterative Refinement with SelfFeedback arXiv preprint arXiv:2303. p. 7
Where the paper says it

Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. ACM Computing Surveys, 55(9): 1–35.

17651 Malik M Jiang J and Chai K. p. 0
Where the paper says it

2023b; Brown et al. These carefully engineered instructions extend beyond simple suggestions; they define an LLM’s persona, operational constraints, task-specific rules, and interaction style (Malik, Jiang, and Chai 2024; Dong et al.

M. A 2024 An Empirical Analysis of the Writing Styles of Persona-Assigned LLMs In EMNLP 2024 Ouyang L Wu J Jiang X Almeida D Wainwright C et al 2022 Training Language Models to Follow Instructions with Human Feedback In Advances in Neural Information Processing Systems NeurIPS OWASP Foundation 2025 OWASP Top 10 for LLM Applications LLM01:2025 Prompt Injection Pape D Mavali S Eisenhofer T Schonherr L et al ¨ 2025 Prompt Obfuscation for Large Language Models In USENIX Security 2025 Pryzant R Iter D Li J Lee Y. p. 7
Where the paper says it

Training Language Models to Follow Instructions with Human Feedback. In Advances in Neural Information Processing Systems (NeurIPS).

T. Zhu C and Zeng M 2023 Automatic Prompt Optimization with ”Gradient Descent” and Beam Search arXiv preprint arXiv:2305. p. 7
Where the paper says it

Automatic Prompt Optimization with ”Gradient Descent” and Beam Search. arXiv preprint arXiv:2305.

03495 Shinn N Cassano F Berman E Gopinath A Narasimhan K and Yao S 2023 Reflexion Language Agents with Verbal Reinforcement Learning arXiv preprint arXiv:2303. p. 7
Where the paper says it

Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv preprint arXiv:2303.

11366 Wang J Yang T Xie R and Dhingra B 2024a Raccoon Prompt Extraction Benchmark of LLM-Integrated Applications In Findings of the Association for Computational Linguistics ACL 2024 Wang J et al 2024b Prompt Extraction Benchmark of LLM-Integrated Applications In Findings of ACL 2024 Warren T 2023 These are Microsoft’s Bing AI secret rules and why it says it’s named Sydney The Verge Published Feb 16, 2023 Wei A Haghtalab N and Steinhardt J Jailbroken How Does LLM Safety Training Fail? p. 7
Where the paper says it

Raccoon: Prompt Extraction Benchmark of LLM-Integrated Applications. In Findings of the Association for Computational Linguistics: ACL 2024.

arXiv preprint arXiv:2307.02483 Wei J Wang X Schuurmans D Bosma M Ichter B Xia F Chi E H Le Q and Zhou D 2022 Chain-of-Thought Prompting Elicits Reasoning in Large Language Models In NeurIPS Yang C Wang X Lu Y Liu H Le Q V Zhou D and Chen X 2023 Large Language Models as Optimizers arXiv preprint arXiv:2309. p. 8
Where the paper says it

; and Zhou, D. Chain-ofThought Prompting Elicits Reasoning in Large Language Models.

03409 Yi J et al 2023 Benchmarking and Defending Against Indirect Prompt Injection Attacks arXiv preprint arXiv:2312. p. 8
Where the paper says it

Benchmarking and Defending Against Indirect Prompt Injection Attacks. arXiv preprint arXiv:2312.

14197 Zhang Y et al 2023 Effective Prompt Extraction from Language Models arXiv:2307. p. 8
Where the paper says it

Effective Prompt Extraction from Language Models. arXiv:2307.

06865 Zhang Z Zhang W et al 2025 IHEval Evaluating Language Models on Following the Instruction Hierarchy In Proceedings of NAACL-HLT 2025 Zhuang Z Nicolae M. p. 8
Where the paper says it

IHEval: Evaluating Language Models on Following the Instruction Hierarchy. In Proceedings of NAACL-HLT 2025.

-I Wang H.-P and Fritz M 2025 ProxyPrompt Securing System Prompts against Prompt Extraction Attacks arXiv preprint arXiv:2505. p. 8
Where the paper says it

ProxyPrompt: Securing System Prompts against Prompt Extraction Attacks. arXiv preprint arXiv:2505.

11459 Zou A Wang Z Carlini N Nasr M Kolter J Z and Fredrikson M 2023 Universal and Transferable Adversarial Attacks on Aligned Language Models arXiv:2307. p. 8
Where the paper says it

Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv:2307.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Black-box prompt hardening allows for system prompt protection without needing access to model weights or internal parameters, making it applicable to any proprietary or open-source LLM accessible through an API.

  2. The system can automatically generate and iteratively refine a protective shield, which is a short textual suffix appended to the original prompt, serving as a barrier that deflects adversarial prompt-extraction attempts while preserving the semantic behavior of the model.

  3. The process formalizes prompt hardening as a utility-constrained optimization task by minimizing leakage while ensuring utility preservation above a specified threshold, measured by semantic fidelity to baseline outputs.

  4. The LLM-as-optimizer intelligently proposes and evolves shield candidates through an iterative, evolutionary-style algorithm guided by in-context learning, replacing random mutation with an informed generation process guided by in-context learning.

  5. The resulting improved AI system can achieve a near zero attack success rate on a challenging test suite, effectively suppressing both surface-level and semantic reproduction of the original system prompt against adversarial queries.

Abstract

System prompts are critical for guiding the behavior of Large Language Models (LLMs), yet they often contain proprietary logic or sensitive information, making them a prime target for extraction attacks. Adversarial queries can successfully elicit these hidden instructions, posing significant security and privacy risks. Existing defense mechanisms frequently rely on heuristics, incur substantial computational overhead, or are inapplicable to models accessed via black-box APIs. This paper introduces a novel framework for hardening system prompts through shield appending, a lightweight approach that adds a protective textual layer to the original prompt. Our core contribution is the formalization of prompt hardening as a utility-constrained optimization problem. We leverage an LLM-as-optimizer to search the space of possible SHIELDs, seeking to minimize a leakage metric derived from a suite of adversarial attacks, while simultaneously preserving task utility above a specified threshold, measured by semantic fidelity to baseline outputs. This black-box, optimization-driven methodology is lightweight and practical, requiring only API access to the target and optimizer LLMs. We demonstrate empirically that our optimized SHIELDs significantly reduce prompt leakage against a comprehensive set of extraction attacks, outperforming established baseline defenses without compromising the model's intended functionality. Our work presents a paradigm for developing robust, utility-aware defenses in the escalating landscape of LLM security. The code is made public on the following link: https://github.com/psm-defense/psm

Sources

Related papers