Privilege and confidentiality in generative AI workflows
summary
The gist
Generative AI systems store and process client data in three distinct ways: in the model’s parameters through training and memorisation, in the context window during a live session…
In short
Generative AI stores client data in three ways: model parameters (memorisation), context windows (live sessions), and knowledge databases (RAG). Risks differ across these modes, requiring tailored governance. Managing these risks means understanding how data persists in each layer, as the duty of confidentiality follows the data regardless of where it is stored.
Key concepts
- Model Memorisation
- This is permanent storage where training material becomes encoded into the AI's core parameters. Data can be extracted verbatim using specific prompts. Since there is no 'delete' button on a trained model, this mode presents the most enduring risk to confidentiality.
- Context Window
- This represents the AI’s short-term working memory during an active session. Risks include disclosure to providers, shared context between users, and prompt injection. A practical issue is 'context rot,' where long inputs are truncated or processed in batches.
- Retrieval-Augmented Generation (RAG)
- RAG uses a separate searchable knowledge store to provide information during generation. Risks include embedding persistence, where data stays in the vector database, and semantic caching. Governance must manage these specific persistence layers separately from standard document management systems.
Terminology used across episodes
This episode discusses
The paper
Privilege and confidentiality in generative AI workflows · Read on arXiv
University of Bristol Law School · University of Oxford
Generative AI (GenAI) systems store and process client data in three distinct ways: in the model's parameters through training and memorisation, in the context window during a live session, and in knowledge databases for retrieval-augmented generation (RAG). Each mode creates different and often counter-intuitive risks to confidentiality and legal professional privilege, and each calls for specific governance responses. Drawing on the first English and American decisions to address privilege and generative AI, UK and Munir v Secretary of State for the Home Department and United States v Heppner, on the orthodox privilege authorities against which those decisions must be read, and on recent computer science research, we explain the three modes of data storage and processing in terms accessible to practitioners and analyse the legal consequences of each. We then situate the analysis within the regulatory framework governing solicitors in England and Wales and within the ordinary principles of professional negligence, arguing that the standard of effective information governance (and with it the benchmark against which negligence and misconduct will be measured) is changing. Although we write primarily for SRA-regulated practitioners, our data-governance analysis is framed to extend to any jurisdiction in which the protection of privilege or professional secrecy depends on demonstrable confidentiality. The ultimate aim of this article is to help legal services professionals understand salient data leakage risks in GenAI systems and thereby facilitate a more responsible deployment of GenAI on client data and other sensitive material.
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "Privilege and confidentiality in generative AI workflows".
Elias: Here's the gist of the paper. Generative AI systems store and process client data in three distinct ways: in the model’s parameters through training and memorisation, in the context window during a live session, and in knowledge databases for retrieval-augmented generation (RAG) Each mode creates different and often counter-intuitive risks to confidentiality and legal professional privilege, necessitating specific governance responses.
Nadia: The Three Modes of Data Processing The paper examines three distinct ways in which data is stored and processed in generative AI systems: (1) model memorisation; (2) the context window; and (3) retrieval-augmented generation (RAG).
Elias: The headline message is twofold, stating that risks are manageable only if one understands the difference between these three modes of data processing.
Nadia: Model Memorisation Risks Memorisation is described as "the most permanent of the three modes of data processing considered here, and the most familiar".
Paper summary: Nadia: So, to recap what we just heard, we talked about the three distinct ways client data gets stored in generative AI systems: memorisation in the model parameters, the context window during a live session, and knowledge databases for retrieval augmented generation or RAG.
Elias: And they argue that these three modes create different risks to confidentiality and privilege, so understanding them is what manages those risks. The paper emphasizes that you can't manage anything if you don't know the difference between data baked into the model, data transiting a provider’s infrastructure during a session, or data persisting in a vector database.
Priya: I think it’s important to focus on how these persistence layers affect legal practice because the risks aren't all equal across those modes. Memorisation is described as the most permanent and familiar mode of processing here.
Nadia: Right, so when we talk about memorisation, it means statistical patterns from training material get encoded into the model’s parameters, which can then be extracted using prompting techniques to get verbatim content. It assumes that just because you type something in doesn't mean it’s gone once it's used for training data.
Elias: That assumption is a big problem because there isn't a delete button on a trained AI model, and you can extract that stored data with the right prompt, so the user can’t undo that storage permanently.
Priya: And then we have the context window, which is like its short-term working memory during a live session. The risks there include disclosure to the model provider, shared context between users, and prompt injection attacks.
Nadia: There's also this practical issue called the long-input problem where materials bigger than that window get truncated or processed in batches, which leads to what they call context rot. That’s a risk specific to the session itself.
Elias: And then there’s retrieval augmented generation, or RAG. This mode uses a separate searchable knowledge store to handle long inputs, but it introduces specific risks like embedding persistence and embedding inversion which can mean the embeddings reveal almost as much information as the original text itself.
Conclusion: Nadia: So, looking at the whole paper "Privilege and confidentiality in generative AI workflows," Janeček and Melham are essentially arguing that the risks are manageable if you grasp these three distinct modes of data processing.
Elias: They claim that putting client material into a consumer-grade tool whose terms allow retention or training use is a breach of confidentiality, which puts privilege at serious risk, no matter what legal route a court ultimately prefers.
Priya: What this means for us in practice is that on limited disclosure authorities, an enterprise-grade deployment with no training use and minimal retention shouldn't itself imperil privilege if you have the right documentation proving it.
Nadia: And the paper points out that the persistence layers that lawyers don't see, like fine-tuned parameters, shared workspaces, and vector stores, are where those unintuitive breaches are actually going to happen.
Elias: So the final message is that understanding this difference between memorisation, context window use, and RAG stores is becoming part of what technological competence looks like in two thousand twenty-six <ref:2607.05479#pg1>.
Priya: It’s about moving the focus from the AI tool's brand or source code to those technical data-handling terms and controls behind them.
Nadia: Exactly. The paper's big implication is that the duty of confidentiality remains the anchor, but you have to account for all these different channels through which a breach can occur.
More episodes
- 2608.27496-ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection
- Daily Summary for 2026-11-19
- 2607.07085-Is Randomness Necessary for Adaptive Data Analysis?
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2609.10601-Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows
- 2610.38786-Adversarial Process Disturbances at a Deployable Alarm Budget: Detector Evaluation on a Simulator and a Real Plant
- 2511.08660-Binary and Multiclass Cyberattack Classification on GeNIS Dataset
- 2604.27302-Static Attribution of Android Residential Proxy Malware Using Graph Kernels
- Daily Summary for 2026-11-18