Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthropomorphism, and Maxims

summary

Video file (mp4)

The gist

The gist The FRANZ framework introduces an automated method for characterizing Large Language Model responses across four dimensions: cultural positioning, use of generalizing language,

In short

FRANZ is an automated framework to audit how Large Language Models frame subjective answers by measuring four dimensions: cultural positioning, generalizing language, anthropomorphism cues, and adherence to conversational maxims. Using a large corpus of Reddit questions, the study found that insider framing and anthropomorphism are tightly linked across different countries.

Key concepts

Cultural Positioning
This refers to how the LLM frames its response based on its perceived cultural stance or perspective—whether it sounds like an insider or an outsider. It reveals the underlying viewpoint through which the model is communicating, which is crucial for understanding subjective queries.
Anthropomorphism Cues
These are specific linguistic markers used by the LLM to make its response sound human, such as claims about personal history or desires. These cues help determine how much of a response relies on creating a relatable, human-like persona rather than just providing information.
Generalizing Language
This concept measures whether the LLM uses language that attributes characteristics to an entire group rather than individuals. It involves checking for specific grammatical forms and attributions that apply broadly to the group being discussed.
Conversational Maxims
These are Grice's maxims—Quality, Quantity, Relation, and Manner—which govern how we communicate effectively. The framework assesses whether the LLM follows or violates these rules in its response style.

Terminology used across episodes

This episode discusses

The paper

Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthropomorphism, and Maxims · Read on arXiv

Siddhesh Pawar, Sarah Masud, Haneul Yoo, Alice Oh, Isabelle Augenstein

University of Copenhagen · KAIST

Large language models (LLMs) are being increasingly used to answer subjective, information-seeking questions, where users are sensitive to how responses are communicated, not just whether the answers are correct. Existing LLM evaluations for subjective cultural queries largely focus on factual correctness, ignoring how the response is framed. To this end, we introduce FRANZ, an automated FRAmework for respoNse characteriZation to conduct communicative audit of LLM responses along four dimensions: cultural positioning, use of generalizing language, anthropomorphic cues, and adherence to conversational maxims. To enable this evaluation, we contribute SQUARE - a corpus of 376k subjective questions sourced from 57 subreddits, and mapped to 7 countries and 19 question categories. We demonstrate FRANZ's applicability by scoring responses from three open-weight LLMs. We observe that LLMs show statistically significant differences in the frequency with which they employ each response characteristic. Unlike single-dimensional audits, FRANZ reveals that insider positioning and anthropomorphism are positively coupled, with the degree of coupling varying by country, providing a diagnostic lens for identifying framing divergences.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Not What, But How".

Jane: Here's the gist of the paper. The gist The FRANZ framework introduces an automated method for characterizing Large Language Model responses across four dimensions: cultural positioning, use of generalizing language, anthropomorphism cues, and adherence to conversational maxims.

Tom: Framework Overview FRANZ is a sociolinguistically grounded framework that operationalizes four distinct yet interrelated facets of response framing <ref:2606.02493#pg19>.

Jane: It shifts the focus from what LLM outputs encode to how they are framed <ref:2606.02493#pg19>.

Tom: The four characteristics audited are cultural positioning, use of generalizing language, anthropomorphic cues, and adherence to conversational maxims <ref:2606.02493#pg19>.

Paper summary: Tom: So, we're talking about this paper called "Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthropomorphism, and Maxims" right now. The authors are Siddhesh Pawar and Sarah Masud at the University of Copenhagen and KAIST sipa, sarah.masud

Jane: And what this paper is really about is that existing ways we test these large language models for subjective questions just focus on if the answer is factually correct, completely ignoring how it's actually framed. This framework introduces FRANZ to automate a way of auditing those responses across four specific dimensions right now.

Lu: The core idea here is moving the focus from what the AI outputs encode to how those outputs are framed, which means we're looking at cultural positioning, use of generalizing language, anthropomorphic cues, and adherence to conversational maxims <ref:2606.02493#pg1>.

Meng: So it’s about checking the communication style of the AI instead of just checking if the facts are right. But what makes this work is their data contribution called SQUARE 376k subjective questions sourced from fifty-seven subreddits and mapping those to seven countries and nineteen question categories <ref:2606.02493#pg1>.

Lalam: That corpus was built by taking a big pushshift reddit crawl, finding subreddits like r/Ask or r/AskA as a starting point, and then manually expanding that list <ref:2606.02493#pg1>.

Tom: And they used Llama-three point one-8B-it, Gemma-three-12b-it, and Ministral-3B-14B-it to score the responses to see how these four characteristics appear in the output <ref:2606.02493#pg2>.

Jane: The paper claims that LLMs show model specific signatures when you look at how they adapt knowledge production by users of insider and outsider groups <ref:2606.02493#pg3>.

Lu: They look at how persona steering surfaces variations when the AI is asked to inhabit specific demographic stances, which is something we see in other work too <ref:2606.02493#pg3>.

Meng: The framework unifies these dimensions by tracing how cultural positioning—which includes insider positioning—is connected to anthropomorphism, and how that coupling varies by country <ref:2606.02493#pg1>.

Tom: So what they found was that insider framing and anthropomorphism are positively coupled, meaning when an AI positions itself as being an insider, it’s more likely to be judged as empathetic or emotional <ref:2606.02493#pg2>.

Paper summary: Jane: That coupling isn't the same everywhere; the degree of coupling changes depending on which country you look at <ref:2606.02493#pg1>.

Lu: They also found that generalizing language appears largely orthogonal to both cultural positioning and anthropomorphism, meaning those things don't strongly influence whether a response uses group-level claims about individuals <ref:2606.02493#pg3>.

Meng: And adherence to conversational maxims, like Grice’s maxims on quality or quantity, seems to be largely orthogonal to all three of those other characteristics <ref:2606.02493#pg3>.

Lalam: So the analysis across Llama, Gemma, and Mistral showed statistically significant differences in how often they use each of these response characteristics <ref:2606.02493#pg2>.

Tom: It seems like the main implication for us right now is that we need to be careful with these couplings because if you overdo one thing, it can inflate the other one, which is a real concern for how we deploy these models <ref:2606.02493#pg1>.

Jane: That bidirectional coupling between positioning and anthropomorphism means that insider-oriented responses are likely to be judged as more affectively anthropomorphic, and vice versa <ref:2606.02493#pg1>.

Lu: That bidirectional coupling warrants caution because you can't just optimize one dimension without affecting the other <ref:2606.02493#pg1>.

Meng: From a deployment standpoint, if we want to debias these models, we can’t just target one characteristic; it looks like a multi-objective optimization challenge <ref:2606.02493#pg1>.

Tom: So the paper shows us that for subjective questions, the framing itself is what matters more than just the correct answer <ref:2606.02493#pg1>.

Jane: It moves the evaluation away from just factual correctness and toward understanding how these models communicate culturally <ref:2606.02493#pg1>.

Lu: The framework is designed to provide a diagnostic lens for identifying these framing divergences across all four dimensions <ref:2606.02493#pg1>.

Meng: The paper also flags some limitations, and one of them is that the cultural proxy they use is coarse and limited by the metadata they have available, because country-named subreddits only select a particular slice of online discourse <ref:2606.02493#pg1>.

Lalam: And another limitation they mention is that LLM judges are imperfect even after human validation on these subjective annotation tasks <ref:2606.02493#pg1>.

Paper summary: Tom: Plus, the paper notes that Grice’s maxims themselves are calibrated to Western communicative norms, so we have to consider if the observed maxim violation rates might be conflating model behavior with those cultural assumptions <ref:2606.02493#pg1>.

Jane: So what this means for someone just listening to the show is that when you hear an AI answer something subjective, you can start asking not just 'is it right?' but 'how is it positioned,' and 'is it trying to sound empathetic or generalizing?'

Lu: It gives us a way to look at those nuanced communication signals across positioning, generalization, anthropomorphism, and maxims <ref:2606.02493#pg1>.

Meng: It shows that we have model specific signatures when you analyze these framing characteristics across different AI models like Llama and Gemma <ref:2606.02493#pg2>.

Tom: Overall, this paper, "Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthropomorphism, and Maxims," gives us a specific tool to audit the communication style of these models <ref:2606.02493#pg1>.

Jane: It’s about using this framework to understand how LLM-mediated knowledge production is adapted by users of insider and outsider groups <ref:2606.02493#pg3>.

Lu: The work uses Generalized Estimating Equations, which is a statistical method to get population level estimates while accounting for the fact that responses from three different LLMs to the same post aren't independent <ref:2606.02493#pg1>.

Meng: So what this means for us is that we have a more structured way to evaluate these models beyond just looking at accuracy scores <ref:2606.02493#pg1>.

Tom: It points toward the fact that training variations shape models’ default response tendencies in ways that are complex and coupled <ref:2606.02493#pg1>.

Jane: This paper, "Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthropomorphism, and Maxims," provides a starting point for evaluations that aren't content focused benchmarks like CultureBank or BLEnD <ref:2606.02493#pg1>.

Lu: It supports examining how different communities employ these response characteristics at a granular level within the SQUARE corpus <ref:2606.02493#pg1>.

Meng: So we're seeing that the way an AI communicates is a measurable and analyzable thing, not just some vague feeling <ref:2606.02493#pg1>.

Tom: That’s what this paper is really about, using FRANZ to audit responses for cultural positioning, generalization language, anthropomorphic cues, and maxim adherence <ref:2606.02493#pg1>.

Conclusion: Tom: So this paper is called "Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthropomorphism, and Maxims" and it’s basically a way to audit how Large Language Models actually talk about stuff instead of just checking if they got the facts right.

Jane: That’s right. The authors are Siddhesh Pawar and Sarah Masud at the University of Copenhagen and KAIST, sipa, sarah.masud. They built this framework called FRANZ to look at four specific things: cultural positioning, generalizing language, anthropomorphic cues, and adherence to conversational maxims.

Lu: They use a corpus called SQUARE which has three hundred seventy-six thousand subjective questions from fifty-seven subreddits across seven countries and nineteen categories. It’s a lot of data to work with for this kind of analysis.

Meng: The core idea is that we need to look at how the AI frames its answers, not just the content itself. They set up an LLM-as-a-judge approach to do this, which means they don't have humans manually labeling everything.

Lalam: The results show that there are statistically significant differences in how often different open-weight models like Llama and Gemma use these response characteristics when answering subjective questions.

Tom: It seems the main finding is about coupling; specifically, insider positioning seems to be positively coupled with anthropomorphism, meaning when an AI sounds like it’s speaking from an insider perspective, it tends to also sound more empathetic or emotional.

Jane: That coupling isn't uniform across the board; the degree of that connection shifts depending on the country you are looking at. They found this relationship varies by cultural context.

Lu: And they also found that generalizing language seems pretty separate from both cultural positioning and anthropomorphism, meaning those things don't strongly influence whether an AI uses group-level claims about people.

Meng: The adherence to conversational maxims, like Grice’s rules on quality or quantity, doesn't seem to connect much with the other three characteristics they audited. It’s largely orthogonal.

Tom: So what this means for us is that we have a multidimensional lens now for spotting these framing divergences in AI responses instead of just looking at a single metric like accuracy.

Jane: It’s about understanding how LLMs communicate culturally, not just what they know. This framework gives us a way to diagnose *how* an AI is positioning itself in conversation.

Lu: The implication is that we can't just optimize for one trait; the tight coupling between positioning and anthropomorphism suggests that pushing one can inadvertently boost the other.

Meng: That means debiasing these models won't be a simple single-target fix; it looks like a multi-objective optimization challenge because of those couplings.

Tom: It really highlights that training variations shape default response tendencies in complex ways, and we need to be careful about the tradeoffs when we fine-tune or deploy these systems.

Jane: This work provides a starting point for evaluations that aren't just content focused benchmarks like CultureBank, but something that looks at the actual communication style.

Lu: The authors also flagged some limitations on their methodology; for example, they use a cultural proxy based on country-named subreddits, which is coarse and limited to that particular slice of online discourse.

Meng: Plus, the LLM judges used in their system are imperfect even after human validation, so we have to keep that in mind when interpreting the scores.

Tom: So for anyone listening who isn't reading the paper right now, this means moving beyond just checking if an AI is factually right to actually understanding its communicative stance.

Jane: Next time you interact with an AI on a subjective topic, try to think about whether it’s positioning itself as an insider or trying to be empathetic, and see how that connects to the overall framing.

More episodes

← Home