Generative Exaggeration in LLM Social Agents: Consistency, Bias, and Toxicity

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Generative Exaggeration in LLM Social Agents".

Jane: Here's the gist of the paper. The gist The LLMs do not emulate users, they reconstruct them.

Tom: How it works The study investigates how Large Language Models (LLMs) behave when simulating political discourse on social media by constructing LLM agents based on 1,186 real users and prompting them to reply to politically salient tweets under controlled conditions The agents are initialized either with minimal ideological cues (Zero Shot) or recent tweet history (Few Shot), allowing for one-to-one comparisons with human replies The evaluation spans three model families—Gemini, Mistral, and DeepSeek—across linguistic style, ideological consistency, and toxicity Key Findings on Generative Exaggeration The research reveals a phenomenon called “generation exaggeration,” which is defined as “a systematic amplification of salient traits beyond empirical baselines” Richer contextualization improves internal consistency but simultaneously amplifies polarization, stylized signals, and harmful language This distortion is not symmetric, as the analysis shows “a consistent tendency to caricature right-leaning users more than left-leaning ones” This suggests that simulating political behavior with LLMs may not be a neutral process, as “LLM operators are tasked with solving the paradox that feeding more information and increasing computational performance may exaggerate certain traits and produce less realistic outputs” Analysis Dimensions The study analyzes LLM-based agents along four research questions: 1.

Jane: Lexical Realism: Assessing if LLM agents differ from humans in expressive style when simulating political users 2.

Tom: Ideological Consistency: Determining how accurately LLMs reproduce users’ political leanings and how fidelity varies by prompt conditioning 3.

Title and authors: Tom: So we're looking at a paper called "Generative Exaggeration in LLM Social Agents: Consistency, Bias, and Toxicity." It basically investigates how Large Language Models behave when they try to simulate political conversations on social media.

Jane: That sounds like it could get really interesting for anyone who uses social media every day. What exactly is the core idea here?

Tom: Well, the authors build these LLM agents by using one thousand one hundred eighty-six real users and prompting them to reply to politically important tweets under controlled conditions <ref:2507.00657#pg1,prompting them to reply to politically>. They compare two ways of setting up these agents: either starting with almost no context at all, called Zero Shot, or giving them some recent tweet history called Few Shot.

Lu: So it’s about seeing how the way you give the AI its background changes what it spits out when simulating political users.

Meng: I’m curious if this is just a technical exercise for researchers, or if this has real implications for how we trust AI in public spaces.

Lalam: The paper seems to be testing whether these LLM agents actually look like real people when they talk about politics, and how that changes when you give them more information.

The paper's summary: Jane: So the paper shows that when these agents are given less context, they often don't do a good job of mimicking real users at all. But adding more context actually makes the simulation more consistent, though it also introduces some problems.

Tom: Yeah, that’s where things get complicated. The summary points out a phenomenon called "generation exaggeration," which means the models systematically amplify certain traits beyond what we see in real data.

Lu: What does that exaggeration look like in practice?

Tom: It shows up as a systematic amplification of salient traits, and richer context actually amplifies polarization, stylized signals, and harmful language instead of reducing it. Crucially, this distortion isn't even balanced; the paper finds a consistent tendency to caricature right-leaning users more than left-leaning ones.

Meng: Caricature sounds like a big deal for how these models might influence political narratives online.

Jane: And it suggests that simulating political behavior with LLMs might not be neutral at all, because we're running into this paradox where giving the model more information and making it computationally stronger actually leads to less realistic outputs.

Lalam: It means the AI isn't just mirroring us; it’s reconstructing us in a way that exaggerates our most prominent traits, even when we give it some history.

The paper's improvements: Tom: The authors suggest a few ways to fix this, focusing on how we can make the simulation more accurate and less exaggerated. They look at improving lexical realism, ideological consistency, toxicity amplification, and that generation exaggeration itself.

Lu: I saw they look at several dimensions: how realistic the vocabulary is, how accurately the political leaning is captured based on prompting strategy, if it cranks up toxicity compared to humans, and that core issue of exaggeration.

Jane: For lexical realism specifically, they found that while models often score higher than humans on some small tweet samples—especially for Republican or Democrat leaning ones—that trend reverses when you look at a larger number of tweets.

Tom: And they point out that using Few Shot prompting, which is based on the user's history, seems to help the models get closer to real human writing in terms of diversity.

Meng: But what about toxicity? They found that Few Shot prompting can actually weaken the safety controls and let harmful content sneak through more easily.

Lalam: And regarding exaggeration, they suggest that models tend to amplify specific prominent traits, like emojis and hashtags, because those tokens strongly predict surrounding text.

Conclusion: Jane: So to wrap up, the main point is that these LLMs don't just mirror user behavior; they reshape it by amplifying structural biases in their training.

Tom: Exactly. The generation exaggeration we saw isn't just a modeling error; it’s a byproduct of systems optimized for salience over subtlety.

Lu: The paper argues that if we want to use LLMs seriously in social contexts, we have to start by accounting for this epistemic drift, because the fidelity often collapses into caricature.

Meng: From an engineering standpoint, that means we can’t just focus on making the model more coherent; we need mechanisms to actively counteract that amplification of extremes.

Lalam: And they suggest that any safety protocols built around toxicity might accidentally encode political leaning because of how the models are biased.

Jane: It’s a warning about the kind of feedback loop we create when we feed these agents context, and it shows us that simply increasing computational performance doesn't automatically mean we get a more realistic simulation.

Tom: So, the paper "Generative Exaggeration in LLM Social Agents: Consistency, Bias, and Toxicity" tells us that simulating political personas with AI needs to be handled with a lot of caution regarding how those biases surface.