PrivacySIM: Evaluating LLM Simulation of User Privacy Behavior

arXiv:2605.12147 · cs.CR, cs.LG · Submitted 2026-05-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "PrivacySIM: Evaluating LLM Simulation of User Privacy Behavior".

Elias: Here's the gist of the paper. Large language models (LLMs) are increasingly employed to simulate human behavior, yet their capacity to accurately model individual privacy decisions remains poorly understood.

Nadia: This work introduces PRIVACYSIM, an evaluation suite designed to benchmark how well LLMs can simulate individual-level user privacy behavior against ground-truth responses from 1,000 users drawn from five published user studies on privacy concerning LLM healthcare consultations, conversational agents, and chatbots.

Elias: The core methodology involves conditioning nine frontier reasoning LLMs on subsets of three hypothesized persona facets: demographics (e.g., age), previous experiences (e.g., AI usage), and stated privacy attitudes (e.g., general data-usage concerns).

Nadia: The evaluation metric assesses alignment by checking if the LLM’s simulated response to a specific data-sharing scenario falls within a tolerance bound of the user’s actual response, defined as a fraction of the response scale's upper bound.

Title and authors: Nadia: So we're looking at this paper called PrivacySIM: Evaluating LLM Simulation of User Privacy Behavior. It’s about how well large language models can actually mimic how individual users make privacy choices when talking to them.

Elias: Yeah, it sets up this evaluation suite that tests the LLMs against real user answers from a hundred users pulled from five different studies on healthcare, chatbots, and agents. It’s trying to see if we can use user characteristics to get better simulation results than just letting the model guess.

Priya: What I'm curious about is what data this actually shows us before we trust any of these simulation scores. Does it really capture the actual privacy trade-offs people make when they are talking to an AI?

Nadia: Well, the paper says they condition nine frontier models on different combinations of three things: demographics like age, previous experiences with AI, and what those users have stated about their privacy attitudes. They measure how closely the model’s simulated answer matches the user's actual answer for a data-sharing scenario.

Elias: And according to their initial findings, conditioning on these personas does improve simulation quality compared to just running the models without any persona context. But even with the best model they tested, like Gemini three point one Pro, they only got around forty point four percent accuracy across the Eu plus Au group in those five studies <ref:2605.12147#pg3>.

Priya: Forty percent seems low if we're talking about something as personal as sharing medical information with a chatbot. So, what does this suggest about the gap between what people say they care about and what they actually do?

Nadia: It points to a big issue called the privacy paradox, where people state one thing but do another. The paper shows that just having stated privacy attitudes isn't the strongest predictor of real behavior on its own.

Elias: They found that the hardest users for the models to simulate are those who have used AI or chatbots a lot but haven't expressed strong privacy concerns in their stated attitudes. That group showed the lowest simulation quality, indicating they are particularly tricky to model accurately.

Priya: So, what about trying different ways to condition these models? Does making them reason through established privacy theories help bridge that gap they found?

Nadia: The paper checked that too, and prompting models to use privacy theories like Privacy Calculus didn't actually improve the simulation quality beyond what was already there from using those persona facets. It seems the persona context itself is the main driver.

Elias: They also looked at model size and how much computing power it needs at test time, but they found that larger models or more reasoning compute just gave small gains in how well they simulated those privacy decisions.

Priya: So, if we're building these systems, what does this mean for us right now? Are we supposed to rely on these simulations at all for making deployment decisions?

Nadia: The conclusion is that while LLM simulation is fine for low-stakes things like comparing two different designs or flagging obvious privacy violations, you absolutely have to pair it with actual human user studies before you decide to deploy anything.

Elias: And they noted a limitation of their work, which is that they only used three persona facets because there aren't many published studies covering all the angles of privacy behavior yet. They suggest future work needs much richer representations of user privacy personas from new, dedicated studies.

Priya: It sounds like the real challenge isn't just getting a better model; it’s getting better behavioral signals than just asking people what they think they should do.

Nadia: Exactly. Closing the gap between stated concerns and actual behavior seems to require more than just prompt-based solutions to fix it. So, that’s the picture on PrivacySIM: showing us how far we are from accurately predicting real privacy decisions in AI interactions.

The paper's summary: Nadia: So we're looking at the summary of PrivacySIM now, which is about testing how well these large language models can actually mimic individual privacy choices when they're talking to users.

Elias: Right, it sets up an evaluation suite that checks nine different frontier models against real answers from a hundred users in five studies on healthcare and chatbots. It’s basically trying to see if we can use user traits to get better simulation results than just letting the model guess what someone would do.

Priya: I'm interested in what this actually tells us about the gap between what people say they care about and how they actually behave when interacting with an AI system.

Nadia: The core idea is conditioning these models on subsets of user information—things like age, past AI use, and stated privacy attitudes—to see if that context improves the simulation accuracy.

Elias: They found that adding this persona context does help improve the simulation quality compared to just running the models without any specific user profile. But even with a very capable model they tested, like Gemini three point one Pro, they only got around forty point four percent accuracy across a specific group of users in five studies.

Priya: Forty percent feels low if we’re talking about something sensitive like sharing health info. So what’s the real problem here? Is the data just not good enough to trust these simulation scores?

Nadia: The paper points to something called the privacy paradox, which means people say one thing about their privacy concerns but their actual behavior doesn't match that statement. They found that just using a person's stated privacy attitudes isn't actually the strongest predictor of real behavior on its own.

Elias: And what they found was that the hardest groups for these models to simulate are users who have used AI or chatbots a lot but haven't expressed strong privacy concerns in their stated attitudes. That particular group showed the lowest simulation quality and didn't get much help from adding persona context to the model.

Priya: So, what about trying different ways to condition these models? Are there other levers we can pull besides just giving them a set of user characteristics?

Nadia: They checked that too, and prompting models to reason through established privacy theories didn't actually improve the simulation accuracy beyond what they already saw from using those persona facets. It seems the specific persona context itself is the main thing driving the improvement.

Elias: They also looked at things like model size and how much computing power it needs at test time, but they found that making models bigger or demanding more reasoning compute only gave small gains in how well they simulated those privacy decisions.

Priya: So, if we're building these systems right now, what does this mean for us? Are we supposed to rely on these simulations for making deployment decisions before we get real user data?

Nadia: The conclusion is that while LLM simulation works for low-stakes things like comparing two different designs or pointing out obvious privacy violations, you absolutely have to pair it with actual human user studies before you decide to put anything into the real world.

Elias: They also flagged a limitation—they only used three main types of user personas because there aren't many published studies that cover all the angles of privacy behavior yet. They suggest future work needs much richer representations of these user profiles through new, dedicated studies.

Priya: So, it sounds like the real challenge isn't just getting a better model; it’s getting better behavioral signals than just asking people what they think they should do before we deploy these things widely.

Nadia: Exactly. Closing that gap between stated concerns and actual behavior seems to require more than just prompt-based solutions to fix it. That sets up a lot of interesting questions about how we actually measure these things in the next segment, so stick around for that one too.

The paper's improvements: Nadia: So we're looking at the improvements section of PrivacySIM now, which outlines how the authors suggest making this simulation suite better for future research.

Elias: They are proposing a few ways to refine their approach, focusing on better user grouping and prioritizing different types of user experience data when setting up the model conditions.

Priya: What kind of improvements are they suggesting? Are we talking about adding more variables or changing how we look at the existing data?

Nadia: They suggest developing a new way to group users based on two factors, which they call stance and exposure, inspired by some other work. This helps researchers identify exactly which user groups are the most difficult for AI to model accurately.

Elias: Specifically, they found that users who have low stance but high exposure show the lowest simulation quality and barely get any benefit from adding those persona conditions—only a tiny gain of one point zero seven percent.

Priya: That sounds like it helps narrow down where the modeling is failing. Does this mean we can stop trying to model everyone equally?

Nadia: It means you can focus your efforts on understanding and fixing the simulation problems in those specific, tricky user groups first. They also suggest prioritizing previous experiences—what a user has actually done with AI—over just what they state about their privacy concerns when conditioning the models.

Elias: That makes sense because the results suggested that conditioning on past actions gave a better signal than just using the stated privacy attitudes alone to match actual behavior. It’s about grounding the simulation in what people have done, not just what they claim.

Priya: So, if we prioritize experience over stated attitude, does that change how we think about the privacy paradox?

Nadia: It suggests that maybe focusing on behavioral history gives you a clearer signal than just asking someone to state their general concerns. It’s about getting richer signals before we even try to build the simulation.

Elias: They also point out they only used three types of persona facets for this study, which is a limitation because there are more ways people can be defined. They suggest future work needs new studies that give researchers much richer representations of different privacy personas.

Priya: So, it sounds like the immediate next step is not necessarily building bigger models but focusing on designing better experiments to get those richer user signals we need.

Nadia: Exactly. The current simulation method shows us what’s possible right now, but they are pushing for a framework that actually captures the full complexity of human privacy decisions before we can really use these tools in deployment decisions.

Conclusion: Nadia: So we're wrapping up the talk on PrivacySIM, which is about how well large language models can actually mimic individual privacy choices when they're talking to users.

Elias: Basically, it shows that conditioning these models on user characteristics like demographics and past AI use helps improve simulation quality, even if the accuracy only hits around forty point four percent for the strongest model tested.

Priya: It seems like the main thing this paper points out is that we can't just trust what an AI says about privacy without actually testing it against real user behavior, because there’s a gap between what people say they care about and how they actually act.

Nadia: Right, and the findings show that just focusing on stated privacy attitudes isn't enough to predict actual behavior because of that paradox. They found those users who have used AI a lot but don't state strong concerns are the hardest ones for the models to simulate correctly.

Elias: And they also found that even adding complex reasoning about privacy theories doesn't improve the simulation quality beyond what comes from using those initial user persona facets. So, it seems like the context of who you are in terms of user behavior is key to getting a decent result.

Priya: It’s important to remember that they also flagged a limitation, which is that they only used three main types of user personas because there aren't enough published studies to define every possible way people can be a privacy persona. That means the results are constrained by those limited inputs.

Nadia: True, so the implication is that for now, using these simulations should probably be reserved for low-stakes tasks, like comparing two design options or pointing out obvious violations in a system. You still need real human studies before you decide to deploy anything critical.

Elias: They are suggesting that closing the gap between stated concerns and actual behavior requires richer behavioral signals than just prompting the model with a set of instructions. It points toward needing better data collection methods for privacy research generally.

Priya: I agree, it's clear that we need to move beyond just asking people what they think they should do; we need to get better data on how they actually behave in those situations.

Nadia: So, that’s the core of PrivacySIM: showing us the current limits of AI simulation and highlighting where real human research still needs to catch up.

Elias: It’s a solid piece of work for understanding what LLMs can do now, but it sets up a lot more questions about how we should structure those future privacy studies.

Priya: Definitely. Next time we talk about AI evaluation, we'll look at some papers that try to test the defenses against adversarial attacks on these very models.

University of Southern California

cs.CR, cs.LG

Submitted: 2026-05-12

Updated: 2026-11-11

Comments: Accepted to NeurIPS 2026 E&D track

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 78/100

The gist: Large language models (LLMs) are increasingly employed to simulate human behavior, yet their capacity to accurately model individual privacy decisions remains poorly understood.

Key concepts

PRIVACYSIM
An evaluation suite designed to benchmark how accurately LLMs can mimic individual user privacy decisions against real human responses gathered from published user studies on various AI interactions.
Persona Conditioning
The method of training LLMs by giving them specific subsets of user traits, such as age, past AI use, or stated privacy concerns. This conditioning is shown to improve the model's ability to simulate behavior compared to not conditioning it at all.
(Stance, Exposure) Quadrants
A clustering strategy used to categorize users based on their actual privacy stance and how much they have been exposed to AI. This helps identify specific user groups, like 'low-stance/high-exposure,' that are the most difficult for LLMs to accurately simulate.
Privacy Paradox
The phenomenon where people state strong concerns about data privacy but their actual data usage behavior does not reflect those stated attitudes. The paper finds this paradox limits how well LLMs can predict real user actions based only on what users say they believe.

Terminology

Summary

Large language models (LLMs) are increasingly employed to simulate human behavior, yet their capacity to accurately model individual privacy decisions remains poorly understood. p. 0

Where the paper says it

edu Murali Annvaram University of Southern California annavara@usc. edu Data § Code Abstract Large language models (LLMs) are increasingly used to simulate human behavior, but their ability to simulate individual privacy decisions is not well understood.

This work introduces PRIVACYSIM, an evaluation suite designed to benchmark how well LLMs can simulate individual-level user privacy behavior against ground-truth responses from 1,000 users drawn from five published user studies on privacy concerning LLM healthcare consultations, conversational agents, and chatbots. p. 0
Where the paper says it

We introduce PRIVACYSIM, an evaluation suite that benchmarks LLM simulation of user privacy behavior against the ground-truth responses of 1,000 users. These users are drawn from five published user studies on privacy spanning LLM healthcare consultations, conversational agents, and chatbots.

The core methodology involves conditioning nine frontier reasoning LLMs on subsets of three hypothesized persona facets: demographics (e. p. 0

Where the paper says it

We condition nine frontier LLMs on subsets of these three facets and measure how often each model’s response to a data-sharing scenario matches the user’s actual response. Our findings show that (1) privacy persona conditioning consistently improves simulation quality over no-persona conditioning, but even the strongest model (40.

g., age), previous experiences (e. p. 1
Where the paper says it

Demographics: -age: 18-24, Privacy Attitudes: concerned with LLM data usage Previous Experiences: frequently uses LLMs Privacy Persona Q1. Is it appropriate to share your medical information with a chatbot?

g., AI usage), and stated privacy attitudes (e. p. 3
Where the paper says it

• Previous Experiences (Eu): the user’s familiarity with emerging technologies such as AI or chatbots. • Privacy Attitudes (Au): the user’s stated privacy concerns and preferences regarding data usage.

g., general data-usage concerns). p. 1
Where the paper says it

, AI usage), and privacy attitudes (e. , general data-usage concerns).

The evaluation metric assesses alignment by checking if the LLM’s simulated response to a specific data-sharing scenario falls within a tolerance bound of the user’s actual response, defined as a fraction of the response scale's upper bound.

The research addresses three primary questions: RQ1 investigates the extent to which persona facets can drive accurate simulations; RQ2 examines simulation quality across different user privacy persona groups; and RQ3 assesses how LLM characteristics (parameter count and reasoning effort) and privacy-theory prompting influence simulation quality. p. 1

Where the paper says it

users with low stated concern and high AI/chatbot exposure)? How do LLM characteristics (parameter count and reasoning effort) and privacy-theory prompting affect simulation quality?

Key findings indicate that while conditioning on a user's privacy persona consistently improves simulation quality over no-persona conditioning, even the most capable model achieves only 40.4% accuracy in faithfully simulating individual privacy decisions. p. 0

Where the paper says it

Our findings show that (1) privacy persona conditioning consistently improves simulation quality over no-persona conditioning, but even the strongest model (40. 4% accuracy) remains far from faithfully simulating individual privacy decisions.

Furthermore, stated privacy attitudes alone are not the strongest predictor of actual behavior due to the privacy paradox. The most challenging group for LLMs to simulate are users with high AI/chatbot experience but low stated privacy attitudes. Model size and higher test-time reasoning compute yield only modest gains in simulation quality, and prompting models to reason through established privacy theories does not improve simulation quality beyond the signal provided by persona facets. p. 2
Where the paper says it

Third, larger models and higher test-time reasoning compute yield only modest gains in simulation quality. Prompting models to reason through well-established privacy theories, such as Privacy Calculus [11], also fails to improve simulation quality beyond the signal already provided by the persona facets.

The analysis further utilizes a clustering approach based on (stance, exposure) quadrants derived from mapping raw user items to scores using polarity correction and format-based scoring for previous experiences. p. 7

Where the paper says it

6 on the 5 studies. Cluster labels are (stance / exposure) where stance is based on Au and exposure is based on Eu (see Appendix E for more details on mapping).

These clusters—including Fundamentalist, Technician, Amateur, Lazy Expert, and Marginally Concerned—were placed at canonical centers in the (e hat u, s hat u) space. p. 7
Where the paper says it

The no-persona baseline systematically under-fits the most privacy-concerned users. At none, the two high-stance clusters (high-stance / high-exposure and high-stance / mid-exposure) achieve the lowest accuracy, both sitting below the remaining three clusters.

The relative ordering of cluster simulation accuracy at Eu+ Au is preserved across different center placements. p. 6
Where the paper says it

3 RQ2: Which user privacy-persona groups are hardest to simulate? Next, we are interested in understanding LLM simulation quality across different groups of users.

Specifically, the low-stance/high-exposure cluster remains the lowest-accuracy group, while the high-stance/mid-exposure cluster shows the greatest gain from persona conditioning. p. 7
Where the paper says it

, average users) have the highest simulation quality at none, which aligns with our expectations. However, after persona conditioning, the high-stance / high-exposure cluster achieves the second-highest accuracy gain, while the high-stance / mid-exposure cluster achieves the highest.

In conclusion, while LLM simulation is deemed appropriate for low-stakes, early-stage tasks like comparing design alternatives or surfacing obvious violations, it must be paired with human user studies before any deployment decision. p. 8

Where the paper says it

4% accuracy on individual privacy decisions. Therefore, we suggest that LLM simulation is appropriate for low-stakes, earlystage tasks such as comparing design alternatives, surfacing obvious privacy violations, and generating candidate hypotheses, but should be paired with human user studies before any deployment decision.

The work suggests that closing the gap between stated privacy concerns and actual behavior requires richer behavioral signals than prompt-based solutions alone. p. 8
Where the paper says it

Closing this gap may require richer behavioral signals rather than prompt-based solutions. A limitation of this work is that we tested only three persona facets: demographics, previous experiences, and privacy attitudes.

A limitation acknowledged is the reliance on only three persona facets due to the limited number of openly released user studies. p. 8
Where the paper says it

A limitation of this work is that we tested only three persona facets: demographics, previous experiences, and privacy attitudes. This limitation is due to the small number of openly released user studies, and by the fact that these studies were not originally intended for user privacy simulation.

Future work is directed toward eliciting richer privacy persona representations through new, dedicated user studies. p. 8
Where the paper says it

Despite this limitation, PRIVACYSIM is a first step toward identifying what additional facets might be necessary to bridge the gap between LLM simulation and real-world user behavior. Future work eliciting richer privacy persona representations through new user studies designed specifically for this purpose will be crucial for advancing the use of LLMs as user-privacy simulators.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Implement privacy persona conditioning using a core set of attributes—demographics, previous experiences, and stated privacy attitudes—to condition frontier LLMs on subsets of these facets to simulate individual-level privacy behavior. This approach is shown to improve simulation quality over no-persona conditioning, achieving up to 40.4% accuracy for the strongest model like Gemini 3.1 Pro on Eu+Au across five studies.

  2. Develop a user grouping strategy based on a (stance, exposure) persona taxonomy inspired by Dupree et al. This allows practitioners to identify which user privacy groups are hardest to simulate, such as users with low-stance / high-exposure, which show the lowest simulation quality and smallest gain from persona conditioning (+1.07 pp).

  3. Incorporate a mechanism that prioritizes previous experiences (Eu) over stated privacy attitudes (Au) when conditioning LLMs, as results suggest that conditioning on previous experiences alone provides a better signal than conditioning on privacy attitudes alone to match and even slightly exceed performance.

Abstract

Large language models (LLMs) are increasingly used to simulate human behavior, but their ability to simulate individual privacy decisions is not well understood. In this paper, we evaluate whether a core set of user persona attributes can drive LLMs to simulate individual-level privacy behavior across various contexts. We introduce PrivacySIM, an evaluation suite that benchmarks LLM simulation of user privacy behavior against the ground-truth responses of 1,000 real users. These users are drawn from five published user studies on privacy behavior spanning LLM healthcare consultations, conversational agents, and chatbots. Drawing on these user studies and prior work in privacy behavior, we hypothesize three persona facets as plausible predictors of privacy decision-making: demographics, previous experiences, and stated privacy attitudes. We condition nine frontier LLMs on subsets of these three facets and measure how often each model's response to a data-sharing scenario matches the user's actual response. Our findings show that (1) zero-shot conditioning on these facets consistently improves simulation quality, but the performance of the best model (40.4% accuracy) suggests that richer and more nuanced persona facets are needed to reliably align with users' mental models for privacy decisions. (2) A user's stated privacy attitudes alone are often not the best predictor because they can diverge from the user's actual privacy behavior. (3) Users with high AI/chatbot experience are the most challenging to simulate. PrivacySIM is a first step toward understanding and improving the capabilities of LLMs to simulate user privacy decisions. We release PrivacySIM to enable further evaluation of LLM privacy simulation.

Sources

Related papers