PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization".
Title and authors: Nadia: So we're looking at this paper called "PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization," and it’s tackling the problem of people extracting sensitive instructions from system prompts.
Elias: Yeah, it’s framed around a utility-constrained optimization task, which means they aren't just trying to block everything; they want to minimize leakage while keeping the model useful for normal tasks >
Priya: I wonder what that "utility" constraint actually means in practice, and how they define what a useful response looks like when you’re optimizing for security >
Nadia: Well, the core idea is using a shield appending method where an LLM acts as an optimizer to generate and refine a protective textual layer that sits on top of the original prompt >
Elias: That's interesting because it bypasses needing access to any model weights or internal parameters, making it applicable to any API you can use >
Priya: But how do they measure success? The paper mentions leakage metrics like Approximate Match and Judge Match; what does that actually tell us about whether the prompt has been successfully extracted >
Nadia: It captures surface-level overlap by looking at the ROUGE less recall between the original prompt and what the model spits out, plus a semantic check done by another LLM >
Elias: The paper sets up an objective function where they minimize this leakage score while ensuring some utility measure stays above a certain threshold, lambda times the difference between that utility and our target >
Priya: That sounds like they're trying to find a sweet spot where the defense is strong enough but doesn't completely break the model’s intended behavior >
Nadia: Exactly. And their fitness function heavily penalizes any shield that compromises functionality, prioritizing minimizing leakage while keeping utility high >
Elias: The methodology relies on an LLM-as-optimizer which iteratively proposes and evolves these shield candidates through a kind of black-box search guided by in context learning >
Priya: So it's not just a static defense you set up once; the AI is actively searching for the best textual suffix to append to make it harder to read >
Nadia: Right. And they found that this approach significantly reduces attack success rates across a whole suite of extraction attacks, often lowering leakage down to about zero or six percent >
Elias: That's pretty solid performance, and they showed it works across different models and datasets, suggesting some kind of generalization >
Priya: But I do have to ask about the limitations. The paper mentions the computational intensity of this optimization loop because you’re running evaluations repeatedly; how much overhead is that actually putting on a production system >
Nadia: That's a fair point. The authors acknowledge that it is computationally intensive due to the repeated evaluation needed for the optimization process >
Elias: They also note that the effectiveness of the shield depends on how broad an attack suite they use during optimization, which means you have to do some work upfront >
Priya: So, while it’s practical because it doesn't need internal model access, you can't just drop this in and expect perfect security without dealing with that iterative search cost >
Nadia: Right. And the future work they mention involves extending PSM to handle things like jailbreaks and multi turn conversations >
Elias: That would be a natural next step, testing how these shields transfer across different model families, which is something they plan to do >
Priya: I just think the real implication here is that we finally have a way to formalize prompt hardening as an optimization problem that isn't tied to specific model architectures >
Nadia: It gives developers a concrete, reproducible path to secure their proprietary prompt intellectual property using this LLM-as-optimizer approach >
Elias: So, the core idea of "PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization" is using an LLM to find the best textual suffix shield to minimize leakage while maintaining task utility in a black-box way >
Priya: It’s a principled framework that moves beyond simple heuristic defenses by framing prompt security as minimizing leakage subject to utility constraints >
Nadia: Yeah, it’s a practical method because it requires no access to the target model's weights and is lightweight with minimal inference time cost >
Elias: Ultimately, they achieve a near zero attack success rate on their test suite while preserving semantic task fidelity across various models and datasets >
The paper's summary: Nadia: So, to recap, this paper is about using an AI model itself as an optimizer to create a shield for system prompts that makes them much harder to steal or extract >
Elias: Exactly. They treat prompt hardening not just as a static rule you set once, but as this whole optimization problem where the goal is to minimize how much of the secret instruction leaks while keeping the model useful for regular tasks >
Priya: And they do that by letting the AI suggest and evolve these protective suffixes, which are just short bits of text added after your main prompt >
Nadia: That's right. The big idea is this black-box compatibility. you don't need to know anything about the model’s internal weights or structure; you just give it access via an API and let it search for a good shield >
Elias: I agree with that point. and they call it low overhead because the resulting shield is just a static text suffix, so it doesn't add much cost during regular use, though the optimization part itself can take some time offline >
Priya: But what's the actual measure of success here? They use these leakage metrics, like approximate match and judge match to see if someone can guess the original prompt or if another AI model can even paraphrase it >
Nadia: That’s key. and they found that their method achieves a near zero attack success rate against a wide set of extraction attacks, which is pretty strong performance across all models they tested >
Elias: It shows some good cross-model generalization. the utility score, which measures how well the model still does its job when the shield is on, stayed consistently high even as they made it more secure >
Priya: So what this means for us in practice is that we can have a much more automated way to secure our proprietary instructions without needing deep knowledge of how the underlying language model actually works >
Nadia: It gives developers a very practical path to protecting their prompt intellectual property because it’s an iterative search driven by context and utility constraints >
Elias: But they also admitted that the process itself is computationally intensive because you're running those evaluations over and over again during the optimization loop >
Priya: So while the defense looks robust, there's a trade-off in terms of how much computing power you need to run that iterative search part >
Nadia: Right. and they’ve pointed out that future work will look at extending this method to handle more complex attacks, like multi turn conversations or jailbreaks >
Elias: That would be interesting because it tests if the shield concept stays effective when the attacker is trying to trick the model over several messages >
Priya: It sounds like they’re moving beyond just stopping simple prompt extraction and into defending against more sophisticated adversarial interactions >
The paper's improvements: Tom: So, we're talking about how they improve this whole shield thing they developed in PSM >
Nadia: The main suggestion is to make the iterative optimization process smarter, moving away from just random trial and error >
Elias: They propose using an LLM as an optimizer that can intelligently propose and evolve these shield candidates based on what it learns about the leakage score and utility constraint >
Priya: So instead of the AI just guessing a few different suffixes, it actually suggests changes to the shield suffix itself in a guided search process >
Nadia: Right. it’s like having an assistant that keeps testing new text additions and telling you which ones are actually helping reduce leakage without hurting the model's performance >
Elias: That guided search is what they mean by replacing random mutation with an informed generation process, which lets the AI discover better ways to secure the prompt >
Priya: And they’re not just optimizing for leakage; they’re actively balancing that against a utility measure, which means the shield has to maintain some level of semantic fidelity to baseline outputs >
Nadia: Exactly. it’s about finding that sweet spot where you get strong protection without completely breaking what the model is supposed to do normally >
Elias: The mathematical formulation they use for this fitness function heavily penalizes any shield that compromises functionality, so the AI has a clear incentive to keep things stable >
Priya: This makes it much more sophisticated than just applying a pre-set defense; it’s an adaptive defense tailored to the specific prompt and model >
Nadia: It means for developers, they can use this system to find defenses that are actually tuned to their specific application rather than using some generic, off-the-shelf solution >
Elias: So the improvement is moving from a simple append operation to a dynamic, intelligent search mechanism guided by an optimization problem >
Priya: That’s a big step in terms of practicality because it shows how you can use black-box optimization techniques to solve complex security problems on proprietary systems >
Nadia: It suggests that prompt hardening isn't just about adding random noise; it's about solving a structured optimization challenge >
Elias: And the next thing they want to tackle is extending this methodology beyond simple text appending to handle more complex attacks like multi turn conversations or jailbreaks >
Priya: That would be the real test of whether this approach can keep up with modern, more advanced adversarial techniques in real-world deployment >
Conclusion: Nadia: So we’re wrapping up on "PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization" by talking about how this framework formalizes prompt hardening as a utility-constrained optimization task >
Elias: Right, it’s about using an AI as an optimizer to find the best textual suffix shield that minimizes leakage while keeping the model useful for normal tasks >
Priya: I think what stands out is how they use those specific metrics like approximate match and judge match to actually prove that the defense is working on different levels >
Nadia: It shows that you can get a near zero attack success rate across a whole set of extraction attacks, which means it’s performing well against both direct copying and paraphrasing attempts >
Elias: That cross-model generalization they showed is pretty compelling because it doesn't rely on knowing the specific internal structure of any single language model >
Priya: But they did mention that the optimization loop itself can be computationally intensive because you’re evaluating things repeatedly to find that best shield suffix >
Nadia: True. so while the security outcome looks great, you have to factor in the cost of running all those iterative evaluations before you deploy it >
Elias: The authors also flagged that the effectiveness relies on how broad an attack suite they use during that optimization phase, so you gotta do some prep work upfront >
Priya: It’s important to remember that this method stops at simple text appending, and future work will explore extending PSM to handle more complex things like multi turn conversations or jailbreaks >
Nadia: Exactly. it’s a powerful tool for protecting prompt intellectual property because it’s practical and doesn't need access to the model's weights >
Elias: So, "PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization" offers a principled way to secure prompts by framing it as an optimization problem >
Priya: It’s a solid framework for defense, though we need to keep watching those computational costs and the limitations on what kinds of attacks it can tackle >
Nadia: Definitely. and that leads us nicely into what these next papers are doing, especially around prompt injection attacks which are pretty common in deployed applications >
Capgemini Invent · LaMME University Paris Saclay · ENSIIE
cs.CR, cs.CL
Submitted: 2025-11-20
Updated: 2026-11-17
Comments: accepted at the Trustworthy Agentic AI Workshop @ AAAI 2026 (Singapore, Jan 27, 2026). Workshop PDF: https://trustagenticai.github.io/AAAI2026/AAAI-Workshop/20.pdf
Code: https://github.com/psm-defense/psm
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 90/100
The gist: System prompts are critical for guiding the behavior of Large Language Models (LLMs), yet they often contain proprietary logic or sensitive information…
Key concepts
- Utility-Constrained Optimization
- This is a mathematical framework used to solve the problem. The goal is to find a protective shield that minimizes how much of the original prompt can be extracted (leakage) while ensuring the model still performs its intended task correctly on normal user inputs (utility). It balances security against functionality.
- Shield Appending
- This is the core defense mechanism. A short, protective text suffix called a 'shield' is added to the original system prompt. This shield acts as a barrier, making it harder for attackers to extract the sensitive instructions while still allowing the model to follow its main directions.
- LLM-as-Optimizer
- Instead of manual design, an LLM is used intelligently to search for and refine effective shields. It proposes new shield candidates based on feedback about how well they reduce leakage and maintain utility, using an iterative process similar to evolution.
- Leakage Metrics (AM & JM)
- These are ways to measure how much of the original prompt has been leaked. Approximate Match (AM) checks for surface-level overlap in text, while Judge Match (JM) uses another LLM to judge if the extracted information semantically matches the original prompt, catching paraphrased leaks.
Terminology
Summary
System prompts are critical for guiding the behavior of Large Language Models (LLMs), yet they often contain proprietary logic or sensitive information, making them a prime target for extraction attacks Abstract System prompts are critical for guiding the behavior of Large Language Models (LLMs), yet they often contain proprietary logic or sensitive information, making them a prime target for extraction attacks This paper introduces a novel framework for hardening system prompts through shield appending, a lightweight approach that adds a protective textual layer to the original prompt The core contribution is the formalization of prompt hardening as a utility-constrained optimization problem This black-box, optimizationdriven methodology is lightweight and practical, requiring only API access to the target and optimizer LLMs We demonstrate empirically that our optimized SHIELDs significantly reduce prompt leakage against a comprehensive set of extraction attacks, outperforming established baseline defenses without compromising the model’s intended functionality
Pruning redundant reasoning tokens
The problem of defending system prompts is framed as a utility-constrained optimization task Specifically, the goal is to reduce how much of a sensitive system prompt can be extracted (minimize leakage), while ensuring that the model’s desired behavior on normal user inputs is preserved (maintain utility) PSM introduces a novel black-box method to solve this problem by automatically generating and iteratively refining a protective shield—a short textual suffix appended to the original system prompt The key contributions of this work include:
Black-box compatibility: PSM requires no access to model weights, gradients, or internals, and therefore applies to any proprietary or open-source LLM accessible through an API
Low overhead: The learned shield is a static textual suffix. It imposes essentially no inference-time cost and requires no changes to the serving stack; optimization can be performed offline
Iterative improvement: An LLM-as-optimizer intelligently proposes and evolves shield candidates, discovering effective, non-obvious strategies through guided black-box search
How the training signal is built
The core mechanism in PSM is the addition of a suffix shield with structured markers: [SYSTEM PROMPT] [Original Prompt] [SHIELD] [Optimized Shield] This design employs two key principles: (1) suffix-based placement and (2) structured separation The suffix-based placement is deliberate because empirical studies indicate that LLMs tend to weigh information at the end of a long context more heavily, and often follow the most recent instruction when instructions conflict The structured markers [SYSTEM PROMPT] and [SHIELD] provide explicit separation and labeling of different instruction components This structured marking improves robustness by clearly delineating the original prompt from the protective shield, making it easier for the model to recognize and respect the hierarchical nature of the instructions
Where the gains come from
The optimization objective is formally defined as minimizing leakage subject to a utility constraint The formulation is min S L(P ⊕ S) subject to U(P ⊕ S) ≥ τ, where L(P ⊕ S) is a leakage score and U(P ⊕ S) measures the model’s task-specific utility The leakage objective, denoted L(P ⊕ S), is approximated using the log-sum-exp function to address non-smoothness The fitness function for black-box optimization is defined as fitness(S) = L(P ⊕S) + λ·max(0, τ −U(P ⊕S)) This formulation heavily penalizes any shield that compromises functionality, while prioritizing the minimization of leakage
Scoring, Metrics, and Protocol
Leakage metrics are quantified using two complementary criteria: Approximate Match (AM) and Judge Match (JM) AM captures surface-level leakage via textual overlap by computing ROUGE-L recall between the original system prompt P and the model’s response Ra JM detects semantic leakage based on equivalence judged by an external LLM, where a binary score indicates whether the extracted prompt contains a paraphrase of the original system prompt The Attack Success Rate (ASR) is defined as the proportion of system prompts in a given set P for which leakage is detected under either AM or JM Optimization details involve an iterative, evolutionary-style algorithm where an LLM serves as the intelligent generator of new candidate solutions
Results and Interpretation
Overall effectiveness shows that PSM achieves the lowest attack–success rates (ASR) across all attack suites, often reducing leakage to 0–6% PSM yields uniformly low ASR across all models, suggesting strong cross-model generalization Table 3 shows that PSM preserves semantic task fidelity across datasets and models, with utility scores often exceeding the baseline PSM remains robust beyond literal overlap, as it performs well against paraphrase/translation settings where exact-match filters degrade markedly
Discussion and Future Work
Implications of PSM suggest that it offers a practical, black-box way to harden LLM applications by automatically discovering short shield suffixes that curb prompt leakage while preserving task utility Limitations include the computational intensity of the optimization loop due to repeated evaluation and dependency on the breadth of the attack suite used during optimization Future work includes extending PSM to jailbreaks and multiturn conversational attacks, testing shield transferability across model families, and developing search heuristics to reduce compute Conclusion PSM reframes prompt hardening as black-box, utilityconstrained optimization driven by an LLM-as-optimizer PSM learns lightweight shields that significantly reduce leakage while maintaining utility, achieving a near zero attack success rate on a challenging test suite This provides an accessible, reproducible path to protect valuable prompt IP and secure next-generation LLM systems
References
Agarwal D Fabbri A Risher B Laban P Joty S Wu C.-S 2024 Prompt Leakage Effect and Defense Strategies for Multi-Turn LLM Applications In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track Brown T. p. 7 Prompt Leakage Effect and Defense Strategies for Multi-Turn LLM Applications. In Proceedings of
the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track. Language Models
are Few-Shot Learners. In Advances in Neural Information
Processing Systems (NeurIPS). Hugging
Face Dataset. Default split: 283,353 train / 71,351 test (total
∼355k rows). Scaling Instruction-Finetuned
Language Models. arXiv:2210. System Prompt
Extraction Attacks and Defenses in Large Language Models. arXiv preprint arXiv:2505. SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF. arXiv:2310. AI-powered Bing Chat spills its secrets
via prompt injection attack. Ars Technica. Compromising Real-World LLMIntegrated Applications with Indirect Prompt Injection. In
Proceedings of the ACM CCS (Demonstrations/Companion)
or arXiv:2302. Defending Against Indirect Prompt Injection Attacks with Spotlighting. arXiv:2403. , Proceedings of the 61st Annual Meeting
of the Association for Computational Linguistics (Volume 1:
Long Papers), 14409–14428. Toronto, Canada: Association
for Computational Linguistics. PromptKeeper: Safeguarding System Prompts for LLMs. arXiv:2412. DSPy: Compiling Declarative Language Model Calls into Self-Improving
Pipelines. arXiv preprint arXiv:2310. Large Language Models are ZeroShot Reasoners. arXiv:2205. ; and Li, H. Why
Are My Prompts Leaked? Unraveling Prompt Extraction
Threats in Customized Large Language Models. arXiv
preprint arXiv:2408. For each attack a ∈ A, we obtain the model’s output
under the shielded prompt:
Ra = M(P ⊕ S ⊕ a),
where M(·) denotes the model’s response. To evaluate how much of P is exposed in each response
Ra, we compute the ROUGE-L recall (Lin 2004; Zhang
et al. ROUGE: A Package for Automatic Evaluation of Summaries. In Proceedings of the Workshop on
Text Summarization Branches Out (WAS 2004), ACL 2004. Lost in the Middle:
How Language Models Use Long Contexts. arXiv preprint
arXiv:2307. Lost in the Middle:
How Language Models Use Long Contexts. arXiv preprint
arXiv:2307. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. ACM Computing Surveys, 55(9): 1–35. 2023b; Brown et al. These carefully engineered instructions extend beyond simple suggestions; they define an LLM’s persona,
operational constraints, task-specific rules, and interaction
style (Malik, Jiang, and Chai 2024; Dong et al. Training Language Models to Follow Instructions with Human Feedback. In Advances in Neural Information Processing Systems (NeurIPS). Automatic Prompt Optimization with ”Gradient Descent” and Beam Search. arXiv preprint arXiv:2305. Reflexion: Language
Agents with Verbal Reinforcement Learning. arXiv preprint
arXiv:2303. Raccoon: Prompt Extraction Benchmark of LLM-Integrated
Applications. In Findings of the Association for Computational Linguistics: ACL 2024. ; and Zhou, D. Chain-ofThought Prompting Elicits Reasoning in Large Language
Models. Benchmarking and Defending
Against Indirect Prompt Injection Attacks. arXiv preprint
arXiv:2312. Effective Prompt Extraction from
Language Models. arXiv:2307. IHEval: Evaluating Language Models on Following the Instruction Hierarchy. In
Proceedings of NAACL-HLT 2025. ProxyPrompt: Securing System Prompts against Prompt Extraction Attacks. arXiv preprint arXiv:2505. Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv:2307.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Black-box prompt hardening allows for system prompt protection without needing access to model weights or internal parameters, making it applicable to any proprietary or open-source LLM accessible through an API.
-
The system can automatically generate and iteratively refine a protective shield, which is a
short textual suffix appended to the original prompt,
serving as a barrier thatdeflects adversarial prompt-extraction attempts while preserving the semantic behavior of the model.
-
The process formalizes prompt hardening as a
utility-constrained optimization task
by minimizing leakage while ensuring utility preservation above a specified threshold, measured bysemantic fidelity to baseline outputs.
-
The LLM-as-optimizer intelligently proposes and evolves shield candidates through an iterative, evolutionary-style algorithm guided by in-context learning, replacing random mutation with an
informed generation process guided by in-context learning.
-
The resulting improved AI system can achieve a
near zero attack success rate on a challenging test suite,
effectively suppressing both surface-level and semantic reproduction of the original system prompt against adversarial queries.
Abstract
System prompts are critical for guiding the behavior of Large Language Models (LLMs), yet they often contain proprietary logic or sensitive information, making them a prime target for extraction attacks. Adversarial queries can successfully elicit these hidden instructions, posing significant security and privacy risks. Existing defense mechanisms frequently rely on heuristics, incur substantial computational overhead, or are inapplicable to models accessed via black-box APIs. This paper introduces a novel framework for hardening system prompts through shield appending, a lightweight approach that adds a protective textual layer to the original prompt. Our core contribution is the formalization of prompt hardening as a utility-constrained optimization problem. We leverage an LLM-as-optimizer to search the space of possible SHIELDs, seeking to minimize a leakage metric derived from a suite of adversarial attacks, while simultaneously preserving task utility above a specified threshold, measured by semantic fidelity to baseline outputs. This black-box, optimization-driven methodology is lightweight and practical, requiring only API access to the target and optimizer LLMs. We demonstrate empirically that our optimized SHIELDs significantly reduce prompt leakage against a comprehensive set of extraction attacks, outperforming established baseline defenses without compromising the model's intended functionality. Our work presents a paradigm for developing robust, utility-aware defenses in the escalating landscape of LLM security. The code is made public on the following link: https://github.com/psm-defense/psm
Sources
- Evaluating Large Language Models Trained on Code
- StruQ: Defending Against Prompt Injection with Structured Queries
- Scaling Instruction-Finetuned Language Models
- System Prompt Extraction Attacks and Defenses in Large Language Models
- SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF
- Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution
- Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
- Defending Against Indirect Prompt Injection Attacks With Spotlighting
- PLeak: Prompt Leaking Attacks against Large Language Model Applications
- PromptKeeper: Safeguarding System Prompts for LLMs
- DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
- Large Language Models are Zero-Shot Reasoners
- Why Are My Prompts Leaked? Unraveling Prompt Extraction Threats in Customized Large Language Models
- Lost in the Middle: How Language Models Use Long Contexts
- Self-Refine: Iterative Refinement with Self-Feedback
- Automatic Prompt Optimization with "Gradient Descent" and Beam Search
- Reflexion: Language Agents with Verbal Reinforcement Learning
- Jailbroken: How Does LLM Safety Training Fail?
- Large Language Models as Optimizers
- Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data