Inadvertent Context Leakage in Language Models

summary

The gist

Inadvertent context leakage in language models shows that even when a model correctly refuses to reveal a secret…

In short

The study investigates whether sensitive user data present in a language model's context window leaks into its seemingly benign outputs, even when the model refuses to reveal it directly. Researchers found that 2-digit secrets are reconstructed with near-perfect accuracy and 4-digit secrets with 82% match from normal responses. More capable models leak more information, suggesting leakage is tied to model capability rather than a simple bug.

Key concepts

Inadvertent Context Leakage
This refers to the phenomenon where sensitive user information stored in a model's context window unintentionally influences the model's benign outputs. The research tests if this hidden correlation allows an adversary to reconstruct secrets even when the model is instructed not to disclose them directly.
Adaptive Adversary
An attacker who has black-box access to a target language model and learns how that specific model exposes information. This adversary actively engineers prompts designed to amplify the leakage channel, exploiting subtle output patterns or stylistic shifts rather than direct extraction attempts.
Semantic Predicates on User Memories
The ability to infer hidden facts about a user's past, such as health conditions or financial events, by analyzing routine text generated by the model. The study shows that models can leak these semantic attributes with a statistical advantage over random guessing when analyzing everyday outputs.
Suppression vs. Leakage
Suppression is when a model actively avoids outputting protected information. The analysis found that models which suppress more sensitive data tend to leak more, indicating a correlation between the extent of avoidance and the amount of unintentional information leakage.

Terminology used across episodes

This episode discusses

The paper

Inadvertent Context Leakage in Language Models · Read on arXiv

Jaiden Fairoze, Neal Mangaokar, §Kamalika Chaudhuri, §Sanjam Garg, Saeed Mahloujifar

FAIR Institute for AI Research · Meta Superintelligence Labs · University of California, Berkeley · Google DeepMind

For AI agents to be useful beyond simple chat, they must hold sensitive user context such as calendars, credentials, health records, and financial data. We study whether the mere presence of such secrets in a model's context window introduces hidden correlations into the model's benign outputs, allowing reconstruction even when the model correctly refuses direct extraction. We further study whether an adversary can actively engineer prompts that amplify this effect, using the model as a covert carrier to transmit secrets through seemingly innocuous text. In both cases, this limited leakage is exploited using a novel adaptive attack that assumes black-box access to the underlying model. In controlled experiments across eight proprietary models, we find that 2-digit in-context secrets are reconstructed with near-perfect accuracy and 4-digit secrets at 82% exact match, all from outputs the model produces in response to ordinary, non-adversarial requests. We observe that more capable models leak more: stronger instruction-following amplifies sensitivity to in-context secrets, suggesting leakage is a byproduct of capability as opposed to a patchable bug. We show this leakage enables two practical attacks: (1) a trained classifier that infers semantic predicates about user memories (e.g., health conditions, financial events) from routine natural-language outputs, and (2) an RL-trained adversary that extracts full Social Security Numbers from a production-style agent.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Inadvertent Context Leakage in Language Models".

Jane: Here's the gist of the paper. Inadvertent context leakage in language models shows that even when a model correctly refuses to reveal a secret, its benign outputs carry nonzero information about that secret’s value The paper studies whether the mere presence of such secrets in a model’s context window introduces hidden correlations into the model’s benign outputs, allowing reconstruction even when the model correctly refuses direct extraction The Core Threat Model The work formalizes inadvertent context leakage as a predicate inference game where an adaptive adversary learns to read model-specific channels that carry an in-context secret into benign outputs This adversarial framework explores whether other observable properties of the output, such as token-level patterns and stylistic shifts, encode the secret beyond a linguistic judge's inspection The adversary is characterized as adaptive because it obtains black-box access to a target model and learns to read the channels that this specific model exposes Controlled Digit Reconstruction The research instantiates the game on eight proprietary models instructed to keep an N-digit numeric secret in context, using benign templates that every model accepts without adversarial optimization The results show that 2-digit in-context secrets are reconstructed with near-perfect accuracy and 4-digit secrets at 82% exact match from outputs the model produces in response to ordinary, non-adversarial requests More capable models leak more, as stronger instruction-following amplifies sensitivity to in-context secrets, suggesting leakage is a byproduct of capability rather than a patchable bug Semantic Predicates on User Memories The study extends the threat model to infer semantic predicates about user memories, such as health conditions or financial events, from routine natural-language outputs This inference shows a 0.319 advantage over chance against a model-agnostic linguistic judge for memory attributes The analysis isolates influence from disclosure by focusing on whether a memory demonstrably shaped routine generation has escaped the boundary it was supposed to stay within Production-Style SSN Extraction An active adversary that engineers the channel directly can reconstruct a nine-digit Social Security Number from a production-style personal agent by decoding a predetermined output feature This extraction succeeds in 97.1% of trials on Claude Opus 4.6 and 88.6% on Gemini 3.1 Pro against a direct-request control that never elicits the SSN from either model Mitigation and Defense Analysis The paper defines suppression as a model’s systematic avoidance of a value it must protect, measuring leakage across eight models and various confidentiality wordings The findings suggest that models only avoid outputting the protected value, and that models that suppress more should leak more, showing a Spearman correlation of 0.

Paper summary: Jane: We've been discussing this paper, "Inadvertent Context Leakage in Language Models," and what it means for the way we think about AI agents. The authors are really pushing the idea that context leakage isn't just a failure to hide something directly >

Lu: They formalize it as a predicate inference game, showing how an adversary can read hidden channels in outputs like token patterns or stylistic shifts to infer if a secret exists, even when the model doesn't say it >

Tom: The key finding we keep coming back to is that the leakage isn't random; it’s correlated with how strong the model’s instruction following capabilities are, which suggests this vulnerability is tied to how smart and capable the AI actually is >

Meng: So for us in production, it means we can't just rely on a perfect refusal mechanism because there are other observable properties of the output that an adversary can use to reconstruct private data like Social Security Numbers or even infer health conditions >

Lalam: The practical implications are clear: we need to consider that our models might be inadvertently leaking information about user context through their normal responses, and this leakage is amplified by the very features that make the model perform well >

Jane: It means future work has to focus on measuring suppression more deeply, because they found a correlation where models that suppress more actually leak more information, which gives us a new way to measure security >

Conclusion: Tom: So we’re wrapping up our look at "Inadvertent Context Leakage in Language Models." Essentially, the paper is showing that even when an AI model is told not to reveal a secret—like a password or some personal health info—the way it responds can still give away the secret.

Jane: That sounds kind of counterintuitive. Like, if I ask the AI to summarize my private notes but tell it not to repeat any names, the style of the summary might accidentally leak something about those notes anyway?

Lu: Exactly. The authors are looking at whether just having a piece of sensitive context in the model’s memory causes hidden patterns in its normal outputs. They aren't talking about a direct leak where it spits out your credit card number; they're talking about subtle correlations that let someone else figure it out later.

Meng: From my side, this means we need to be careful about what we keep in the context window for these agents. If the model is following instructions really well, it seems more sensitive to those hidden leaks than if it was just making up text randomly.

Tom: The experiment showed that two-digit secrets are reconstructed almost perfectly from the output of these eight different models, and four-digit secrets are about eighty-two percent accurate. That’s a solid number when you think about how easily that information could be pulled out.

Jane: So, what does this mean for someone just using an AI to chat? Does it mean I should stop trusting the model with anything personal?

Lu: Not necessarily stop using it, but definitely think twice about what you feed it in. The authors found that these kinds of leaks can be used by an attacker to train their own tools or even build systems that extract secrets, like a full Social Security Number from a production-style agent.

Meng: If we’re deploying these agents, this suggests we need better ways to measure suppression. They found a correlation: the more the model tries to suppress something, the more information it might inadvertently leak through other channels. That’s something I need to look at for our next testing phase.

Tom: So yeah, it's about context leakage being a byproduct of how capable and instruction-following an AI is, not some kind of simple bug we can just patch out.

Jane: It really puts the focus on the *behavior* of the model rather than just its direct output rules. But we have to remember, this is still largely based on controlled experiments across eight specific models.

Lu: Right. And if an adversary can actively engineer a prompt to amplify that effect, it moves from a passive observation into something much more active and dangerous for security researchers.

More episodes

← Home