PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization
summary
The gist
System prompts are critical for guiding the behavior of Large Language Models (LLMs), yet they often contain proprietary logic or sensitive information…
In short
This work introduces PSM, a method to protect sensitive system prompts from extraction attacks by adding a protective textual shield. It frames prompt hardening as an optimization problem where an LLM acts as an optimizer to iteratively create the best shield. This black-box approach is lightweight, requires no model internals, and empirically reduces prompt leakage significantly while preserving the model's intended function.
Key concepts
- Utility-Constrained Optimization
- This is a mathematical framework used to solve the problem. The goal is to find a protective shield that minimizes how much of the original prompt can be extracted (leakage) while ensuring the model still performs its intended task correctly on normal user inputs (utility). It balances security against functionality.
- Shield Appending
- This is the core defense mechanism. A short, protective text suffix called a 'shield' is added to the original system prompt. This shield acts as a barrier, making it harder for attackers to extract the sensitive instructions while still allowing the model to follow its main directions.
- LLM-as-Optimizer
- Instead of manual design, an LLM is used intelligently to search for and refine effective shields. It proposes new shield candidates based on feedback about how well they reduce leakage and maintain utility, using an iterative process similar to evolution.
- Leakage Metrics (AM & JM)
- These are ways to measure how much of the original prompt has been leaked. Approximate Match (AM) checks for surface-level overlap in text, while Judge Match (JM) uses another LLM to judge if the extracted information semantically matches the original prompt, catching paraphrased leaks.
Terminology used across episodes
This episode discusses
- PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization · Paper Radio
- Evaluating Large Language Models Trained on Code
- StruQ: Defending Against Prompt Injection with Structured Queries
- Scaling Instruction-Finetuned Language Models
- System Prompt Extraction Attacks and Defenses in Large Language Models
- SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF
- Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution
- Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
- Defending Against Indirect Prompt Injection Attacks With Spotlighting
- PLeak: Prompt Leaking Attacks against Large Language Model Applications
- PromptKeeper: Safeguarding System Prompts for LLMs
- DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
- Large Language Models are Zero-Shot Reasoners
- Why Are My Prompts Leaked? Unraveling Prompt Extraction Threats in Customized Large Language Models
- Lost in the Middle: How Language Models Use Long Contexts
- Self-Refine: Iterative Refinement with Self-Feedback
- Automatic Prompt Optimization with "Gradient Descent" and Beam Search
- Reflexion: Language Agents with Verbal Reinforcement Learning
- Jailbroken: How Does LLM Safety Training Fail?
- Large Language Models as Optimizers
- Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models
The paper
PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization · Read on arXiv
Capgemini Invent · LaMME University Paris Saclay · ENSIIE
System prompts are critical for guiding the behavior of Large Language Models (LLMs), yet they often contain proprietary logic or sensitive information, making them a prime target for extraction attacks. Adversarial queries can successfully elicit these hidden instructions, posing significant security and privacy risks. Existing defense mechanisms frequently rely on heuristics, incur substantial computational overhead, or are inapplicable to models accessed via black-box APIs. This paper introduces a novel framework for hardening system prompts through shield appending, a lightweight approach that adds a protective textual layer to the original prompt. Our core contribution is the formalization of prompt hardening as a utility-constrained optimization problem. We leverage an LLM-as-optimizer to search the space of possible SHIELDs, seeking to minimize a leakage metric derived from a suite of adversarial attacks, while simultaneously preserving task utility above a specified threshold, measured by semantic fidelity to baseline outputs. This black-box, optimization-driven methodology is lightweight and practical, requiring only API access to the target and optimizer LLMs. We demonstrate empirically that our optimized SHIELDs significantly reduce prompt leakage against a comprehensive set of extraction attacks, outperforming established baseline defenses without compromising the model's intended functionality. Our work presents a paradigm for developing robust, utility-aware defenses in the escalating landscape of LLM security. The code is made public on the following link: https://github.com/psm-defense/psm
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization".
Title and authors: Nadia: So we're looking at this paper called "PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization," and it’s tackling the problem of people extracting sensitive instructions from system prompts.
Elias: Yeah, it’s framed around a utility-constrained optimization task, which means they aren't just trying to block everything; they want to minimize leakage while keeping the model useful for normal tasks >
Priya: I wonder what that "utility" constraint actually means in practice, and how they define what a useful response looks like when you’re optimizing for security >
Nadia: Well, the core idea is using a shield appending method where an LLM acts as an optimizer to generate and refine a protective textual layer that sits on top of the original prompt >
Elias: That's interesting because it bypasses needing access to any model weights or internal parameters, making it applicable to any API you can use >
Priya: But how do they measure success? The paper mentions leakage metrics like Approximate Match and Judge Match; what does that actually tell us about whether the prompt has been successfully extracted >
Nadia: It captures surface-level overlap by looking at the ROUGE less recall between the original prompt and what the model spits out, plus a semantic check done by another LLM >
Elias: The paper sets up an objective function where they minimize this leakage score while ensuring some utility measure stays above a certain threshold, lambda times the difference between that utility and our target >
Priya: That sounds like they're trying to find a sweet spot where the defense is strong enough but doesn't completely break the model’s intended behavior >
Nadia: Exactly. And their fitness function heavily penalizes any shield that compromises functionality, prioritizing minimizing leakage while keeping utility high >
Elias: The methodology relies on an LLM-as-optimizer which iteratively proposes and evolves these shield candidates through a kind of black-box search guided by in context learning >
Priya: So it's not just a static defense you set up once; the AI is actively searching for the best textual suffix to append to make it harder to read >
Nadia: Right. And they found that this approach significantly reduces attack success rates across a whole suite of extraction attacks, often lowering leakage down to about zero or six percent >
Elias: That's pretty solid performance, and they showed it works across different models and datasets, suggesting some kind of generalization >
Priya: But I do have to ask about the limitations. The paper mentions the computational intensity of this optimization loop because you’re running evaluations repeatedly; how much overhead is that actually putting on a production system >
Nadia: That's a fair point. The authors acknowledge that it is computationally intensive due to the repeated evaluation needed for the optimization process >
Elias: They also note that the effectiveness of the shield depends on how broad an attack suite they use during optimization, which means you have to do some work upfront >
Priya: So, while it’s practical because it doesn't need internal model access, you can't just drop this in and expect perfect security without dealing with that iterative search cost >
Nadia: Right. And the future work they mention involves extending PSM to handle things like jailbreaks and multi turn conversations >
Elias: That would be a natural next step, testing how these shields transfer across different model families, which is something they plan to do >
Priya: I just think the real implication here is that we finally have a way to formalize prompt hardening as an optimization problem that isn't tied to specific model architectures >
Nadia: It gives developers a concrete, reproducible path to secure their proprietary prompt intellectual property using this LLM-as-optimizer approach >
Elias: So, the core idea of "PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization" is using an LLM to find the best textual suffix shield to minimize leakage while maintaining task utility in a black-box way >
Priya: It’s a principled framework that moves beyond simple heuristic defenses by framing prompt security as minimizing leakage subject to utility constraints >
Nadia: Yeah, it’s a practical method because it requires no access to the target model's weights and is lightweight with minimal inference time cost >
Elias: Ultimately, they achieve a near zero attack success rate on their test suite while preserving semantic task fidelity across various models and datasets >
The paper's summary: Nadia: So, to recap, this paper is about using an AI model itself as an optimizer to create a shield for system prompts that makes them much harder to steal or extract >
Elias: Exactly. They treat prompt hardening not just as a static rule you set once, but as this whole optimization problem where the goal is to minimize how much of the secret instruction leaks while keeping the model useful for regular tasks >
Priya: And they do that by letting the AI suggest and evolve these protective suffixes, which are just short bits of text added after your main prompt >
Nadia: That's right. The big idea is this black-box compatibility. you don't need to know anything about the model’s internal weights or structure; you just give it access via an API and let it search for a good shield >
Elias: I agree with that point. and they call it low overhead because the resulting shield is just a static text suffix, so it doesn't add much cost during regular use, though the optimization part itself can take some time offline >
Priya: But what's the actual measure of success here? They use these leakage metrics, like approximate match and judge match to see if someone can guess the original prompt or if another AI model can even paraphrase it >
Nadia: That’s key. and they found that their method achieves a near zero attack success rate against a wide set of extraction attacks, which is pretty strong performance across all models they tested >
Elias: It shows some good cross-model generalization. the utility score, which measures how well the model still does its job when the shield is on, stayed consistently high even as they made it more secure >
Priya: So what this means for us in practice is that we can have a much more automated way to secure our proprietary instructions without needing deep knowledge of how the underlying language model actually works >
Nadia: It gives developers a very practical path to protecting their prompt intellectual property because it’s an iterative search driven by context and utility constraints >
Elias: But they also admitted that the process itself is computationally intensive because you're running those evaluations over and over again during the optimization loop >
Priya: So while the defense looks robust, there's a trade-off in terms of how much computing power you need to run that iterative search part >
Nadia: Right. and they’ve pointed out that future work will look at extending this method to handle more complex attacks, like multi turn conversations or jailbreaks >
Elias: That would be interesting because it tests if the shield concept stays effective when the attacker is trying to trick the model over several messages >
Priya: It sounds like they’re moving beyond just stopping simple prompt extraction and into defending against more sophisticated adversarial interactions >
The paper's improvements: Tom: So, we're talking about how they improve this whole shield thing they developed in PSM >
Nadia: The main suggestion is to make the iterative optimization process smarter, moving away from just random trial and error >
Elias: They propose using an LLM as an optimizer that can intelligently propose and evolve these shield candidates based on what it learns about the leakage score and utility constraint >
Priya: So instead of the AI just guessing a few different suffixes, it actually suggests changes to the shield suffix itself in a guided search process >
Nadia: Right. it’s like having an assistant that keeps testing new text additions and telling you which ones are actually helping reduce leakage without hurting the model's performance >
Elias: That guided search is what they mean by replacing random mutation with an informed generation process, which lets the AI discover better ways to secure the prompt >
Priya: And they’re not just optimizing for leakage; they’re actively balancing that against a utility measure, which means the shield has to maintain some level of semantic fidelity to baseline outputs >
Nadia: Exactly. it’s about finding that sweet spot where you get strong protection without completely breaking what the model is supposed to do normally >
Elias: The mathematical formulation they use for this fitness function heavily penalizes any shield that compromises functionality, so the AI has a clear incentive to keep things stable >
Priya: This makes it much more sophisticated than just applying a pre-set defense; it’s an adaptive defense tailored to the specific prompt and model >
Nadia: It means for developers, they can use this system to find defenses that are actually tuned to their specific application rather than using some generic, off-the-shelf solution >
Elias: So the improvement is moving from a simple append operation to a dynamic, intelligent search mechanism guided by an optimization problem >
Priya: That’s a big step in terms of practicality because it shows how you can use black-box optimization techniques to solve complex security problems on proprietary systems >
Nadia: It suggests that prompt hardening isn't just about adding random noise; it's about solving a structured optimization challenge >
Elias: And the next thing they want to tackle is extending this methodology beyond simple text appending to handle more complex attacks like multi turn conversations or jailbreaks >
Priya: That would be the real test of whether this approach can keep up with modern, more advanced adversarial techniques in real-world deployment >
Conclusion: Nadia: So we’re wrapping up on "PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization" by talking about how this framework formalizes prompt hardening as a utility-constrained optimization task >
Elias: Right, it’s about using an AI as an optimizer to find the best textual suffix shield that minimizes leakage while keeping the model useful for normal tasks >
Priya: I think what stands out is how they use those specific metrics like approximate match and judge match to actually prove that the defense is working on different levels >
Nadia: It shows that you can get a near zero attack success rate across a whole set of extraction attacks, which means it’s performing well against both direct copying and paraphrasing attempts >
Elias: That cross-model generalization they showed is pretty compelling because it doesn't rely on knowing the specific internal structure of any single language model >
Priya: But they did mention that the optimization loop itself can be computationally intensive because you’re evaluating things repeatedly to find that best shield suffix >
Nadia: True. so while the security outcome looks great, you have to factor in the cost of running all those iterative evaluations before you deploy it >
Elias: The authors also flagged that the effectiveness relies on how broad an attack suite they use during that optimization phase, so you gotta do some prep work upfront >
Priya: It’s important to remember that this method stops at simple text appending, and future work will explore extending PSM to handle more complex things like multi turn conversations or jailbreaks >
Nadia: Exactly. it’s a powerful tool for protecting prompt intellectual property because it’s practical and doesn't need access to the model's weights >
Elias: So, "PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization" offers a principled way to secure prompts by framing it as an optimization problem >
Priya: It’s a solid framework for defense, though we need to keep watching those computational costs and the limitations on what kinds of attacks it can tackle >
Nadia: Definitely. and that leads us nicely into what these next papers are doing, especially around prompt injection attacks which are pretty common in deployed applications >
More episodes
- 2608.27496-ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection
- Daily Summary for 2026-11-19
- 2607.07085-Is Randomness Necessary for Adaptive Data Analysis?
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2609.10601-Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows
- 2610.38786-Adversarial Process Disturbances at a Deployable Alarm Budget: Detector Evaluation on a Simulator and a Real Plant
- 2511.08660-Binary and Multiclass Cyberattack Classification on GeNIS Dataset
- 2604.27302-Static Attribution of Android Residential Proxy Malware Using Graph Kernels
- Daily Summary for 2026-11-18