Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibility

arXiv:2507.12553 · cs.CL, cs.AI · Submitted 2025-07-16 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibility".

Jane: The gist: Language models have been shown to learn linear representations that discriminate between different modal categories, and these vectors reflect human categorization behavior and interpretable features.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, wrapping up this whole discussion on "Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibility." The main point is that these linear representations, the modal difference vectors, are more reliable for classifying things than just looking at the probability scores.

Jane: And what matters is that these vectors show a consistent way models learn to build their understanding of possibility and impossibility as they grow larger and more complex > Page ten >

Lu: This research moves beyond just seeing if an LM can answer a question correctly; it shows us *how* the AI structures its knowledge about the world, whether it’s coarse or fine-grained > Page two > <ref:2507.12553#pg3>

Meng: For deployment at scale, this suggests that we need to pay attention to these internal representations because they actually reflect how humans think about things like event plausibility > Page ten >

Tom: And for someone just listening to the show, the implication is that the AI isn't just spitting out text; it’s building a structured, layered model of reality based on how it processes language.

Jane: It suggests we might be able to use these vectors to test intuitive theories about what LMs think about the world and see if they match human intuition > Page two > <ref:2507.12553#pg3>

Lu: The paper suggests that LMs encode causal principles underlying the world while retaining the ability to imagine hypothetical realities > Page ten >

Meng: So, we’re looking at a way to test those theories directly using these specific mathematical features from the models themselves.

Tom: That's what this work does, identifying linear representations that discriminate between modal categories within a variety of LMs, or modal difference vectors > Page one > <ref:2507.12553#pg2>

Jane: It really gives us a new lens into how we understand modal categorization in humans by correlating projections along these vectors with human participants' ratings of interpretable features > Page ten >

Conclusion: Tom: So, we’re wrapping up our look at this paper, "Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibility." The authors are Michaelov et al., and they found that these mathematical vectors actually align with how humans categorize possibilities.

Jane: It seems the big idea is that language models aren't just guessing; they’ve built these internal structures that separate what’s probable from what’s impossible, which matches our intuition about the world.

Lu: The core finding is that these modal difference vectors are more discriminative than the standard probability estimates we usually look at when classifying sentences.

Meng: From a deployment standpoint, it means we can actually use these specific internal features to classify text better than just looking at the raw output score, which is a nice mechanism for refinement.

Tom: Exactly. And what’s interesting is how these vectors emerge as the models get bigger and more trained; they follow a specific developmental order that mirrors how we think about different kinds of impossibility.

Jane: That’s where it gets really compelling for us—when we project those vectors onto human behavioral data, they actually correlate well with what people say is "subjective event likelihood."

Lu: The paper suggests that this might be a way to test intuitive theories about what an AI learns from just reading text. It gives us a concrete feature space derived directly from the model's activations.

Meng: But we need to watch how these vectors behave when the model starts failing at scale, because if they break down, we know where our representation learning needs to improve for production systems.

Tom: Right. So what does this actually mean for the future of AI understanding? Does this help us design better models, or does it just give us a fancy way to categorize text?

Jane: It gives us a new tool—a way to peek inside the model’s head—to see if its internal logic about reality matches our own experience of what's possible.

Lu: It suggests that LMs are encoding causal principles underlying the world while still keeping the flexibility to imagine hypothetical scenarios. That’s a pretty deep claim for representation learning.

Meng: If we can use these vectors to check if an AI is reasoning about physics or just random word patterns, that could make our safety testing much more rigorous when we deploy these systems.

Tom: So, the paper shows a specific way to look at model internal states and connect them back to human-understandable concepts like imagination and physical possibility. It’s a bridge between the code and common sense.

Jane: It really does point toward testing intuitive theories about what LMs derive from raw text input, which is something we need to keep exploring as these models get more sophisticated.

Michael A. Lepori, Jennifer Hu, Ishita Dasgupta, Roma Patel, Thomas Serre, Ellie Pavlick

Brown University · Johns Hopkins University · Google DeepMind

cs.CL, cs.AI

Submitted: 2025-07-16

Updated: 2026-10-13

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: The gist: Language models have been shown to learn linear representations that discriminate between different modal categories, and these vectors reflect human categorization behavior and

Key concepts

Modal Difference Vectors
These are linear representations discovered within language models that effectively separate different modal categories, such as possible versus impossible. They are created by calculating the difference between the hidden states of two stimuli belonging to distinct categories, allowing the model to discriminate between them more accurately than standard probability scores.
Modal Categories
The paper defines four levels of possibility: Probable (commonplace and possible), Improbable (possible but rare), Impossible (violates known laws like physics), and Inconceivable (fundamentally flawed or un-evaluable). These categories help map how language models structure the concept space of events.
Contrastive Activation Addition
This is the specific technique used to create modal difference vectors. It involves taking the difference between the hidden state representations of two examples that belong to different categories, such as one representing a 'possible' scenario and another representing an 'impossible' one. This subtraction yields a vector that highlights the key differences between those concepts.
Mechanistic Interpretability
This is the field of study used to understand how internal components of language models work. The researchers use this approach to analyze LM activations, treating them as interpretable features that reveal underlying principles, such as how models distinguish between different levels of modal categories.

Terminology

Summary

The gist: Language models have been shown to learn linear representations that discriminate between different modal categories, and these vectors reflect human categorization behavior and interpretable features.

Language Model Representations Reflect Human Judgments of Event Plausibility

This work identifies linear representations, or modal difference vectors, within language models that successfully discriminate between different modal categories, which are crucial for understanding how LMs map linguistic expressions to their corresponding modal categories <ref:2507.12553#pg2> The abstract states that we identify linear representations that discriminate between modal categories within a variety of LMs, or modal difference vectors.

Modal Difference Vectors and Discrimination

The researchers find that these modal difference vectors are often more discriminative than the sentence probability itself <ref:2507.12553#pg5> They create these vectors using Contrastive Activation Addition (CAA), which involves taking the difference between hidden state representations of two stimuli belonging to different categories, such as "x+ and x- <ref:2507.12553#pg7> These vectors are used for classification by checking whether a held-out pair of stimuli x' + v¯ is larger than x' - v¯ <ref:2507.12553#pg9>. The results show that modal difference vectors outperform probability estimates and other projection-based classification baselines <ref:2507.12553#pg2> This holds true for models with at least 2B parameters, where modal difference vectors match or (sometimes drastically) outperform probability estimates at classifying stimuli by their modal category" <ref:2507.12553#pg10>.

Development of Modal Categories in Models

The study investigates how these representations develop across different model characteristics, addressing the research question RQ 2 <ref:2507.12553#pg5> They find that more coarse distinctions emerge in smaller models, in shallower layers, and earlier than more fine-grained distinctions <ref:2507.12553#pg9>. Specifically, for scaling analyses, they observe that models first distinguish the inconceivable modal category from the rest <ref:2507.12553#pg10>. The progression observed is: Models next distinguish probable and impossible events, then probable and improbable, and finally improbable and impossible <ref:2507.12553#pg10>.

Reflection of Human Categorization Behavior

The paper analyzes human behavioral categorization data to see if the LM representations reflect human intuition <ref:2507.12553#pg5> They project sentences onto three modal difference vectors—probable-improbable, improbable-impossible, and impossible-inconceivable—to create a feature space for logistic regression models <ref:2507.12553#pg9>. The results show that a feature space derived from modal difference vectors routinely outperforms the baselines when modeling human categorization behavior <ref:2507.12553#pg10>. Furthermore, they find that projections along the probable-improbable vector correlate very well with subjective event likelihood, but less well with any of the other dimensions <ref:2507.12553#pg10>.

Interpretability of Modal Difference Vectors

The final study explores whether these vectors correspond to human-understandable features, addressing RQ 4 <ref:2507.12553#pg5> They correlate projections along the three vectors with interpretable dimensions such as subjective event likelihood or imageability <ref:2507.12553#pg10>. Notably, projections along the impossible-inconceivable vector correlate selectively with features that measure whether a sentence is easy to imagine (either directly or indirectly) <ref:2507.12553#pg10>. This suggests that imagination has been shown to significantly impact adults’ estimation of event likelihood <ref:2507.12553#pg10>. The study concludes by suggesting that one might use modal difference vectors to generate hypotheses about human representations of modal categories <ref:2507.12553#pg10>.

The investigation lays the foundation for testing intuitive theories that LMs derive about the world from raw text input <ref:2507.12553#pg10>. It also points to a specific, testable hypothesis that humans distinguish between inconceivable and impossible events on the basis of imagination <ref:2507.12553#pg10>. The findings demonstrate that LMs learn to represent the difference between real life and mere fantasy to a greater extent than implied by previous research <ref:2507.12553#pg10>.

--- Page 1 ---

The gist: Language models have been shown to learn linear representations that discriminate between different modal categories, and these vectors reflect human categorization behavior and interpretable features.

How it works

  1. Modal Difference Vectors are created using Contrastive Activation Addition (CAA), which involves taking the difference between hidden state representations of two stimuli belonging to different categories, such as "x+ and x-" <ref:2507.12553#pg7>.

  2. These vectors are used for classification by checking whether a held-out pair of stimuli x' + v¯ is larger than x' - v¯ <ref:2507.12553#pg9>.

  3. The development of these vectors follows a pattern: more coarse distinctions emerge in smaller models, in shallower layers, and earlier than more fine-grained distinctions <ref:2507.12553#pg10>.

Modal Categories Studied

The paper defines four modal categories to study:

(Probable)

Scenarios that are both possible and commonplace in our world. E.g., chilling a drink using ice <ref:2507.12553#pg4>

(Improbable)

Scenarios that are possible, but not commonplace in our world. E.g., chilling a drink using snow <ref:2507.12553#pg4>

(Impossible)

Scenarios that are not possible in our world, as they violate some known law of nature (e.g., physics, biology, etc.). E.g., chilling a drink using fire <ref:2507.12553#pg4>

(Inconceivable)

Scenarios that cannot be evaluated for possibility in any possible world, due to some fundamental semantic or conceptual error. E.g., chilling a drink using yesterday <ref:2507.12553#pg4>

Key Findings

  1. Modal difference vectors are more reliable than sentence probability estimates for classification <ref:2507.12553#pg10>.

  2. The representations develop in an intuitive order: models first distinguish the inconceivable modal category from the rest, perhaps by taking advantage of distributional signals of selectional restriction violations <ref:2507.12553#pg10>.

  3. Modal difference vectors reflect human categorization distributions better than baselines, and they are required to capture fine-grained categorization behavior for the most demanding datasets <ref:2507.12553#pg10>.

  4. Projections along the probable-improbable vector correlate with human subjective event likelihood judgments <ref:2507.12553#pg10>.

  5. Projections along the impossible-inconceivable vector correlate selectively with features like imageability and places/environments <ref:2507.12553#pg10>.

Implications

The results suggest that LMs encode causal principles underlying the world while retaining the ability to imagine hypothetical realities <ref:2507.12553#pg10>. The work provides a direct means of testing intuitive theories that LMs derive about the world from raw text input <ref:2507.12553#pg10>.

The paper defines four modal categories to study:

(Probable)

Scenarios that are both possible and commonplace in our world. E.g., chilling a drink using ice <ref:2507.12553#pg4>

(Improbable)

Scenarios that are possible, but not commonplace in our world. E.g., chilling a drink using snow <ref:2507.

Improvements for AI systems

  1. The system can perform more reliable modal categorization than current methods by utilizing modal difference vectors, which are found to be more discriminative than the probability of the sentence (RQ 1). This allows for a classification accuracy improvement on generalization datasets, as shown in Figure 2, where modal difference vectors outperform probability estimates.

  2. The system can develop more robust internal representations by training it across different scales and depths, specifically by leveraging the finding that more coarse distinctions emerge in smaller models, in shallower layers, and earlier than more fine-grained distinctions (Figure 3d). This ensures that the model learns to distinguish between modal categories in a manner consistent with human cognitive development.

  3. The system can better mimic nuanced human categorization by using projections along modal difference vectors to model human participants’ categorization behavior, as these vectors reflect participants’ graded, intuitive notions of these categories better than the baselines (Section 5.2). This enables the AI to capture the graded, intuitive theories that humans use when making judgments on events.

  4. The system can generate more controllable and targeted outputs by using modal difference vectors to steer model generations, as this method results in surprisal increasing in order of "baseline < improbable < impossible < inconceivable" (Figure 8). This allows for the explicit generation of sentences expressing a desired modal category through intervention at specific layers.

Abstract

Language models (LMs) are used for a diverse range of tasks, from question answering to writing fantastical stories. In order to reliably accomplish these tasks, LMs must be able to discern the modal category of a sentence (i.e., whether it describes something that is possible, impossible, completely nonsensical, etc.). However, recent studies have called into question the ability of LMs to categorize sentences according to modality (Michaelov et al., 2025; Kauf et al., 2023). In this work, we identify linear representations that discriminate between modal categories within a variety of LMs, or modal difference vectors. Analysis of modal difference vectors reveals that LMs have access to more reliable modal categorization judgments than previously reported. Furthermore, we find that modal difference vectors emerge in a consistent order as models become more competent (i.e., through training steps, layers, and parameter count). Notably, we find that modal difference vectors identified within LM activations can be used to model fine-grained human categorization behavior. This potentially provides a novel view into how human participants distinguish between modal categories, which we explore by correlating projections along modal difference vectors with human participants' ratings of interpretable features. In summary, we derive new insights into LM modal categorization using techniques from mechanistic interpretability, with the potential to inform our understanding of modal categorization in humans.

Sources

Related papers