Flout at Your Own Risk: LLMs Struggle with Pragmatic Cooperativity Under Epistemic Asymmetry

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Flout at Your Own Risk".

Jane: Here's the gist of the paper. The gist The results show that while LLMs exhibit certain pragmatic capabilities in collaborative settings, and these can be elicited through prompting and post-training, they still face challenges in pragmatic communication with incomplete information, and that certain failure modes do correlate with floutings of Grice’s maxims that go unrecognized Collaborative Epistemic Asymmetry The paper formalizes a notion of collaborative epistemic asymmetry (CEA) to explicitly connect Gricean maxims to quantitative task outcomes This setting simulates situations where parties with different background knowledge and capabilities must bring them together to solve a problem, but no one individual knows the solution at the outset Collaborative epistemic asymmetry (CEA) is parameterized by ⟨W, U, G, d, Pi⟩ An utterance is actionable if it is (1) informative and (2) goal-oriented Lemma 1 states that an utterance u is actionable for listener L if it results in a non-trivial belief update to L’s posterior over world states and reduces L’s uncertainty about the goal G Under CEA, if utterance u is actionable, then u adheres to Grice’s cooperative maxims Experimental Setup and Methodology The investigation was performed in a setting derived from the Distributed Partial Information Puzzle (DPIP) Lego task using the CRAFT simulator This task simulates a "jigsaw" problem where Directors provide partial views of a single structure, and the Builder must place blocks according to their instructions The preference dataset D was constructed by pairing the most actionable Director utterance yw with a suboptimal utterance yl from the same turn Experiments tested three main conditions: Prompting Experiments, Tool-Calling Experiments, and Post-Training Experiments Prompting and Tool-Calling Effects The paper assessed how prompting the Builder LLM to act as a literal listener (L0) versus a pragmatic listener (L1) affects performance The "Pragmatic Builder" prompt instructs the model to identify which Director’s utterance was the most informative and think through questions relating to Gricean maxims Results show that eliciting pragmatic reasoning from the GPT-4.1 Builder with the Pragmatic Builder prompt infix results in the best overall performance with use of the exploration tool The combination of a pragmatic prompt and move exploration is identified as being both important and reliable for improving performance Post-Training Strategies To address RQ3, open-weight models acting as Directors were post-trained using SFT, DPO (Direct Preference Optimization), and IPO (Identity Preference Optimization) methods against the preference dataset D The results indicate that supervised fine-tuning may produce “safe” actionable-sounding utterances decoupled from actual task state However, DPO alignment brings performance back up when the Builder is explicitly Literal or, in particular, Pragmatic Failure Modes and Maxim Flouting The analysis of specific failure modes correlated with floutings of Grice’s maxims that go unrecognized Quality Flouting occurs if a director does not accurately describe their wall view, which is shown to correlate with improved actionable quality of utterances over SFT models Relation Flouting happens when a Director’s wall is fully completed but the “owner” gives an instruction anyway, and this strongly correlates with move failure Quantity Flouting shows that overinformativenessadversely affects actionability as Lemma 1 predicts, while flouting the maxim of quantity does not necessarily guarantee entropy reduction toward the goal The paper concludes that flouting Gricean maxims under epistemic asymmetry is associated with adverse impacts on agent-agent coordination and task performance Closed model Builder agents are better pragmatic listeners when there is not too much context to incorporate into their reasoning Open model directors can be aligned to be more optimal in their utterances but they remain fragile Future work may explore novel optimization techniques to enable Builder/Listener models to adaptively seek information from specific speakers or in certain ways Ethics Statement Due to human diversity of expression and the human tendency to behave in ways that challenge even theoretically rigorous and empirically validated AI systems, our results have bearing on human-LLM interactions and suggest that assumptions of current LLMs’ pragmatic competence in collaborative settings should be treated cautiously or even skeptically These conclusions would need to be tested in real multiparty human-AI collaborations in order to assess the precise effects of human diversity on LLM performance, as well as susceptibility to factors such as bias against non-normative modes of communication Additionally, because of the focus of our work on agent-agent interactions in collaborative tasks with measurable outcomes, “flouting” as used in our experiments focuses less on social nuance in human communication, such as irony, hyperbole, or the use metaphors to imply meaning In favor of measuring instances where LLMs break the rules outlined by the maxims of conversation without direct intent to imply additional meaning References The paper cites numerous works supporting its theoretical and empirical foundations, including those on Grice’s maxims, Rational Speech Act frameworks, and theory of mind in LLMs Specific references include Clark & Brennan (Clark & Brennan, 1991), Grice (1975), and various papers on LLM reasoning and collaboration such as Butler et al.

Tom: (2025) and Nath et al.

Jane: (2026) The work also builds upon concepts from RSA frameworks, dynamic epistemic logic, and preference alignment methods like DPO and IPO Specific citations include Goodmann & Stuhlmuller (Goodman & Stuhlmuller ¨, 2013), Filipi & Wales (Filipi & Wales, 2004), and Eisenstein et al.

Tom: (2026) The empirical data is grounded in the Distributed Partial Information Puzzle dataset by Zhu et al.

Title and authors: Tom: So we're looking at this paper called "Flout at Your Own Risk: LLMs Struggle with Pragmatic Cooperativity Under Epistemic Asymmetry". We just talked about how they formalize collaborative epistemic asymmetry to connect Grice's maxims to task success.

Jane: Right, and the authors are Hannah VanderHoeven, Abhijnan Nath, and Nikhil Krishnaswamy from the Situated Grounding and Natural Language Lab at Colorado State University. They set up a multi-party collaboration task using something called CRAFT to simulate a jigsaw problem.

Lu: The authors set up this setting by pairing the most actionable Director utterance with a suboptimal one from the same turn to build their preference dataset, which is really smart for testing these behaviors against real performance.

Meng: I'm curious if this setup—pairing good and bad utterances—is actually scalable when you have hundreds of different agents interacting simultaneously.

Lalam: The paper tests three main strategies: prompting experiments, tool-calling experiments, and post-training experiments to see how these models respond in different ways.

The paper's summary: Tom: So the main gist of this paper is that while these LLMs show some pragmatic abilities when we guide them with prompts or train them further, they still struggle when information is incomplete and they have to communicate with others.

Jane: They specifically look at how prompting and post-training can help elicit more pragmatic reasoning from models like GPT-four point one, and they found that combining a good prompt with an exploration tool helps performance a lot.

Lu: They also looked at post-training strategies, using supervised fine-tuning, direct preference optimization, and identity preference optimization against their dataset to see if they could make the Director agents more robust against listeners who aren't being perfectly cooperative.

Meng: The results suggest that supervised fine-tuning might give you utterances that *sound* actionable but aren't actually based on the current task state, which is a big red flag for deployment.

Lalam: The paper also analyzed specific failure modes and found they correlate directly with LLMs flouting Grice’s maxims in ways we don't usually expect, like quality floutings or relation floutings that lead to move failures.

The paper's improvements: Tom: So the paper points out a few specific issues they found in the current state of things, and they propose some directions for improvement based on those findings.

Jane: They suggest that quality flouting—when a director doesn't accurately describe their view—actually correlates with better actionable utterance quality when compared to models trained just with supervised fine-tuning.

Lu: They also found that while DPO alignment can boost performance when the Builder is explicitly told to be literal or pragmatic, it still has limitations because it’s aligning against a specific dataset.

Meng: The paper suggests that for directors, post-training methods like DPO or IPO could provide more robustness against listeners who are not being perfectly cooperative because they align the utterances toward examples of what actually works.

Lalam: They also point out that quantity flouting, where an utterance is just overly informative, actually hurts actionability according to their lemma one, even though violating that maxim doesn't automatically mean the model won't reach the goal faster.

Conclusion: Tom: So wrapping up this paper "Flout at Your Own Risk: LLMs Struggle with Pragmatic Cooperativity Under Epistemic Asymmetry". The main message is that flouting Gricean maxims under these conditions leads to negative impacts on how well agents coordinate and perform tasks.

Jane: They conclude that closed model Builder agents are better listeners when there isn't too much context they have to process, and open model directors can be aligned to be more optimal in their utterances if you use the right alignment techniques.

Lu: The implication for the world is that we need to think more about how these agents handle missing information and not just focus on getting them all to follow the rules perfectly.

Meng: From a deployment standpoint, it means we can't rely solely on fine-tuning for robustness; post-training alignment is a necessary step if you want those agents to be reliable in uncertain settings.

Lalam: The paper shows that we need to be careful with assumptions about what models can do pragmatically when they are operating under epistemic asymmetry, and that flouting rules without intent is something we have to measure more closely.