Epistemic Familiarity is Associated With Belief Stability in Large Language Models

summary

The gist

Large language models (LLMs) are widely used as information sources, yet small changes in semantic assumptions can destabilize their beliefs.

In short

Researchers tested how changing small semantic assumptions affects Large Language Model (LLM) beliefs using a framework called P-StaT. They found that unfamiliar synthetic statements cause greater instability than familiar fictional ones, especially in behavioral settings. Retractions often focus on ambiguous or technical concepts, suggesting that familiarity with a statement type is linked to its stability.

Key concepts

P-StaT Framework
A framework designed to evaluate belief stability by changing which statements are treated as true and measuring how an LLM's beliefs shift. It allows researchers to compare stability in both how the model processes information (representation) and how it acts on that information (behavior).
Epistemic Familiarity
This measures how much an LLM has encountered semantically related claims or entities during its pretraining. It's operationalized by comparing familiar fictional statements against unfamiliar synthetic ones, showing that familiarity with the statement type correlates with the model's belief stability.
Synthetic vs. Fictional Perturbations
Perturbations are changes made to a statement to test stability. Fictional statements come from culturally embedded corpora and are generally more familiar. Synthetic statements are generated specifically to minimize prior exposure, making them less familiar but often more destabilizing.

Terminology used across episodes

This episode discusses

The paper

Epistemic Familiarity is Associated With Belief Stability in Large Language Models · Read on arXiv

Samantha Dies, Courtney Maynard, Germans Savcisens, Tina Eliassi-Rad

Khoury College of Computer Sciences, Northeastern University · Network Science Institute, Northeastern University · Santa Fe Institute

Large language models (LLMs) are widely used as information sources, yet small changes in semantic assumptions can destabilize their beliefs. We introduce P-StaT (Perturbation Stability of Truth), a framework for evaluating belief stability under matched semantic perturbations in both representational and behavioral settings. Across 21 LLMs and three domains, we compare perturbations involving familiar Fictional statements against synthetically generated and unfamiliar Synthetic statements. Unfamiliar Synthetic perturbations generally induce greater epistemic instability than familiar Fictional perturbations, with behavioral belief retraction rates frequently exceeding 0.5 (50%). Finally, exploratory clustering analyses reveal recurring themes among retracted statements, including ambiguity, technical terminology, and obscure concepts. These results show that epistemic familiarity is systematically associated with stability under semantic reframing, suggesting that stability-based analyses can complement accuracy-based benchmarks when evaluating LLM robustness and reliability. Code and data are available at https://github.com/samanthadies/P-StaT.

DOI: 10.18653/2026.findings-emnlp.532

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Epistemic Familiarity is Associated With Belief Stability in Large Language Models".

Jane: Here's the gist of the paper. Large language models (LLMs) are widely used as information sources, yet small changes in semantic assumptions can destabilize their beliefs.

Tom: Unfamiliar Synthetic perturbations generally induce greater epistemic instability than familiar Fictional perturbations, particularly in behavioral evaluations.

Jane: Finally, exploratory clustering analyses reveal recurring themes among retracted statements, including ambiguity, technical terminology, and obscure concepts.

Tom: P-StaT Framework Introduction The paper introduces P-StaT (Perturbation Stability of Truth), a framework for evaluating belief stability under matched semantic perturbations in both representational and behavioral settings.

Paper summary: Tom: So, Jane, this paper, "Epistemic Familiarity is Associated With Belief Stability in Large Language Models," it's looking at how much an AI's beliefs can wobble when you change the background assumptions we give it.

Jane: Right. The core idea here seems to be that small shifts in what we tell the model can really destabilize its understanding, and this study introduces this framework called P-StaT to test that stability in both how the model represents things and how it behaves.

Lu: It's testing belief stability under matched semantic perturbations across twenty-one large language models and three different areas, like city locations, medical indications, or word definitions. The key thing they are comparing is using statements that we know—familiar fictional ones—against statements that are made up specifically to be unfamiliar synthetic ones.

Meng: Unfamiliar synthetic statements causing more instability than familiar fictional ones sounds like it could be a big deal for us because we’re always worried about what happens when we feed models weird data at scale.

Lalam: I see the paper uses this idea of epistemic familiarity, which is basically how likely an AI is to have seen something semantically related during its training, and they operationalize that by looking at whether statements are fictional or synthetic.

Tom: And the big finding they highlight is that these unfamiliar synthetic perturbations generally induce greater epistemic instability than those involving familiar fictional statements, especially when we look at the behavioral evaluations.

Jane: That’s a strong result because it suggests that the novelty of the information, even if synthetically generated to be new, can be more damaging to an LLM's internal beliefs than something that just sounds like a story from fiction.

Lu: And they quantified this with behavioral belief retraction rates frequently exceeding zero point five, which means fifty percent of the time it actually changed what the model believed.

Meng: Fifty percent is a lot when you’re trying to build something reliable for production, so that high rate of retraction under synthetic stress is definitely something we need to keep an eye on.

Paper summary: Lalam: And when they looked at *why* those beliefs were retracted, they found recurring themes in the statements that got flipped, like ambiguity and obscure concepts.

Tom: Ambiguity and technical jargon are interesting because those are exactly the kinds of things where a model might latch onto something it thinks is true but isn't really grounded in the data it saw.

Jane: So what they’re suggesting is that epistemic familiarity—that prior exposure to similar concepts—is systematically linked to stability when we change the semantic framing of those statements, which could help us build better robustness benchmarks for these models.

Lu: Exactly. They suggest that using stability-based analyses can really complement accuracy based benchmarks when we're trying to figure out how robust an LLM actually is.

Meng: It sounds like a way to diagnose the weaknesses without having to run massive adversarial attacks, which saves us some time and compute on our end.

Lalam: The paper basically provides this controlled setting to study how familiarity interacts with both how the model organizes its internal representations and how it behaves when we're probing it.

Tom: So what they’re saying is that if you want to evaluate an LLM’s reliability, you need to look at its stability under these kinds of semantic reframings, not just whether it gets the answer right on a test.

Jane: That makes sense because accuracy can sometimes be fooled by subtle changes in wording; this P-StaT framework seems designed to catch those subtle shifts in belief sets.

Lu: And they even looked at how the representations themselves changed across different layers of the model, finding that those retraction patterns stayed qualitatively similar across neighboring layers.

Meng: That’s reassuring for deployment because if we find a pattern that holds across different parts of the network, it suggests a more fundamental issue with how it processes information rather than just some random noise in one layer.

Lalam: It seems like they are pointing toward specific types of semantic content—ambiguous referents and rare concepts—as the most vulnerable spots for an AI's beliefs to break.

Paper summary: Tom: So, when we think about the future, this study suggests that understanding epistemic familiarity can help us design better ways to train models to be more stable when faced with unexpected inputs.

Jane: It’s about making sure that when an LLM encounters something new or ambiguous, it doesn't just guess wildly but maintains a solid foundation in its knowledge.

Lu: And for future work, they note that their operationalization of epistemic familiarity is based on proxies like cultural embeddedness and statement construction rather than something directly observable.

Meng: So the next step would be to find a more direct way to measure that actual prior exposure, instead of relying on those proxies they used in this study.

Lalam: I think that’s the natural path forward, moving from these controlled semantic perturbations toward methods that actually map onto pretraining data exposure.

Tom: It’s interesting because the authors are careful about what this framework can and cannot do; it’s for diagnosis under controlled settings, not for studying real-world adversarial attacks or conversational dynamics.

Jane: Right, so we have a tool to see where the model is shaky when we change the meaning of things, but we need more work to see how that translates into real-world safety concerns.

Lu: And they’ve shown how belief set analyses can give us a different kind of view on robustness compared to just looking at simple factuality benchmarks.

Meng: It gives us a way to probe the model's internal logic, which is valuable because it helps us pinpoint exactly where the reasoning goes astray when context shifts.

Lalam: The whole point of P-StaT is to provide that controlled comparison between representational organization and behavioral robustness under these specific semantic pressures.

Tom: So, to wrap up this paper on "Epistemic Familiarity is Associated With Belief Stability in Large Language Models," it’s showing us that how we define familiarity matters for how stable an AI's beliefs are when we mess with the meaning of what we feed it.

Jane: And the implication for us is that if you want to make these large language models more trustworthy, you need to think about their epistemic vulnerabilities by testing them against these kinds of controlled semantic reframings.

Conclusion: Tom: So we’re wrapping up this look at "Epistemic Familiarity is Associated With Belief Stability in Large Language Models."

Jane: It’s really about how much a model's beliefs can wobble when we slightly change the meaning of what we feed it.

Tom: The authors are looking at twenty-one different large language models across three areas, like medical stuff or city names.

Lu: They used this P-StaT framework to test stability both in how the AI represents things and how it behaves when we give it slightly different assumptions.

Meng: Basically, they’re asking: does knowing something is familiar help keep the AI's answer stable when you change the question a little bit?

Jane: The main finding is that unfamiliar synthetic stuff actually causes more belief instability than familiar fictional stuff, especially in how the models behave.

Tom: It seems like prior exposure to similar concepts acts as a kind of anchor for the model's truth judgments.

Lu: They found that when statements are semantically ambiguous or use technical jargon, those are the specific things where models tend to retract their beliefs most often.

Jane: That’s because those kinds of tricky statements pull at different parts of the model's internal knowledge base in different ways.

Tom: So what this means for us is that accuracy alone isn't enough; we need to look at stability too when we build these systems.

Lu: They suggest that using stability tests, like P-StaT, can actually complement traditional factuality benchmarks when you want to know if an AI is reliable.

Jane: It gives us a way to see the underlying structure of what the model believes, not just what it spits out on a simple test.

Tom: And they flagged that this study is focused on controlled semantic changes, so we need more work to see how this plays out in real-world conversations or when people try to trick the AI.

Jane: Exactly. So we've seen how familiarity affects stability under controlled conditions; now we have to figure out what happens when the inputs aren't just a carefully constructed test set.

More episodes

← Home