Shortcuts or Memorisation? Revisiting Data Corruption in Natural Language Understanding with Large Language Models
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Shortcuts or Memorisation? Revisiting Data Corruption in Natural Language Understanding with Large Language Models".
Jane: Here's the gist of the paper. Pre-trained encoders fine-tuned on natural language inference (NLI) and other language understanding tasks keep much of their accuracy after the input is perturbed, e.g.
Tom: by removing all nouns or shuffling the words The main contributions of this paper are as follows.
Jane: Robustness to Perturbation and Benchmark Differences Regardless of model size, every system that answers directly retains 36–72% of its above-chance performance without nouns on MNLI compared to only 14–29% on ANLI Only a model that reasons before answering departs from this pattern.
Tom: The paper demonstrates that the gap between MNLI and ANLI lies in the dataset, not the model Robustness against perturbation alone cannot prove language understanding or memorisation What it reveals depends on which cues remain intact even after perturbation is applied.
Paper summary: Tom: So we just talked about how these models handle losing nouns, and the paper "Shortcuts or Memorisation? Revisiting Data Corruption in Natural Language Understanding with Large Language Models" claims that even when you strip out nouns, most systems keep a lot of their accuracy on MNLI >
Jane: They found that every system that answers directly retains thirty six to seventy two percent of its above chance performance without nouns on MNLI compared to only fourteen to twenty nine percent on ANLI >
Lu: The paper also points out that the gap between those two benchmarks is in the dataset not necessarily the model itself, regardless of how large or small we make the LLM >
Meng: They also checked some contamination indicators—like exact n-gram counts in the pre-training data—and found no association with robustness >
Tom: That means they aren't seeing a simple link between how much data corruption a model sees and how robust it becomes against losing parts of the input >
Jane: Instead, they focus on items solvable solely from the hypothesis, which they found to be more robust in all three of the most robust LLMs on MNLI when compared to when function words are removed >
Lu: They built three hundred sixty eight new human-validated NLI items and found that all eight LLMs show the same shortcut pattern across these unseen items >
Tom: That means they aren't seeing memorization as the main driver here, but something else is at play in how those models work >
Jane: They compare MNLI with BioASQ, which is also robust to perturbation, but their results suggest that the structure of the data matters more than just having a large model >
Conclusion: Tom: So when we look at this paper, "Shortcuts or Memorisation? Revisiting Data Corruption in Natural Language Understanding with Large Language Models," they are suggesting that the survival of familiar NLI benchmarks isn't because the models memorized specific answers >
Jane: They argue instead that certain structural shortcuts in how the data is presented allow these models to maintain performance even when you corrupt the input, like removing all nouns >
Lu: The key evidence they offer is that for the most robust LLMs on MNLI, items that can be solved just from the initial hypothesis are more robust than items where you remove function words >
Meng: What this means for us in production is that we shouldn't just look at memorization as a failure mode when models get weird input; we should also be looking at what structural assumptions they make about the language structure itself >
Tom: And they show that these shortcut patterns are consistent across all eight models even on brand new items they haven't seen, which is pretty strong evidence for this idea >
Jane: It implies that understanding the underlying structure of natural language is a more stable thing than remembering every specific sentence in the training set >
University of Helsinki · Institute of Language and Speech Processing Athena Research Centre Athens
cs.CL
Submitted: 2026-11-12
Updated: 2026-11-19
Code: https://github.com/aarnetalman/shortcuts-or-memorisation
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 89/100
The gist: Pre-trained encoders fine-tuned on natural language inference (NLI) and other language understanding tasks keep much of their accuracy after the input is perturbed, e.g.
Key concepts
- Robustness to Perturbation
- This tests how well a model's answer holds up when parts of the input text are deliberately changed or removed, like taking out all nouns. The paper found that many models keep most of their accuracy even with these changes, indicating they rely on more than just exact memorization.
- Contamination Indicators
- These are specific metrics used to check for potential data corruption or shortcuts within the model's responses. The research found no significant link between these indicators and overall robustness, suggesting that the presence of these signs doesn't reliably predict how well a model handles noisy input.
- Hypothesis-Only Analysis
- This method involves testing if an item can be correctly classified using only the initial guess or hypothesis generated by the model, without needing further complex reasoning. The paper showed that for certain challenging items, this direct approach reveals patterns related to memorization versus shortcuts.
- MNLI vs. ANLI Gap
- This refers to the difference in performance between two specific natural language inference benchmarks. The research concluded that this performance gap is caused by the differences in the datasets used for each benchmark, not necessarily a fundamental limitation of how well a model understands language generally.
Terminology
Summary
Pre-trained encoders fine-tuned on natural language inference (NLI) and other language understanding tasks keep much of their accuracy after the input is perturbed, e. p. 0 chatzikyriakidis@athenarc. gr
Abstract
Pre-trained encoders fine-tuned on natural language inference (NLI) and other language understanding tasks keep much of their accuracy
after the input is perturbed, e. The main contributions of this paper are as follows:
• We demonstrate that the gap between MNLI
and ANLI lies in the dataset, not the model. Regardless of model size, all systems that answer directly keep much more of their performance on MNLI than on ANLI when nouns
are removed (36–72% vs.Where the paper says it
Where the paper says it
Robustness to Perturbation and Benchmark Differences
Regardless of model size, every system that answers directly retains 36–72% of its above-chance performance without nouns on MNLI compared to only 14–29% on ANLI Only a model that reasons before answering departs from this pattern. The paper demonstrates that the gap between MNLI and ANLI lies in the dataset, not the model Robustness against perturbation alone cannot prove language understanding or memorisation What it reveals depends on which cues remain intact even after perturbation is applied. p. 1 In other words, robustness against perturbation
alone cannot prove language understanding or
memorisation. What it reveals depends on which
cues remain intact even after perturbation is applied.Where the paper says it
Contamination Indicators and Shortcut Mechanisms
The paper finds no link between the three contamination indicators and robustness In all three of the most robust LLMs on MNLI, items solvable solely from the hypothesis are more robust, and this advantage is reversed when function words are removed. The strongest evidence comes from 368 new, human-validated items that none of the models can have seen All eight LLMs exhibit the same shortcut pattern in these unseen items. p. 0 Our strongest evidence comes from 368 new,
human-validated items that none of the models can have seen. All eight LLMs keep more
of their performance without nouns on items
written in MNLI’s style than on items written
without its cues.Where the paper says it
Hypothesis-Only Analysis and Fresh Items
The new items enable the most direct test: no model has seen them, but the MNLI-style items have the same type of cues as MNLI If memorisation causes the robustness on MNLI, it should disappear on Clean R no nouns R nouns+verbs Easy− Model MNLI free MNLI free MNLI free hard Table 2 shows the second pattern in all models. Within MNLI-style items, items correctly labelled by the MNLI hypothesis-only classifier show 11–28 points less loss than the remaining items when nouns are absent The new items are easier than MNLI (93–100% clean accuracy). p. 6 62–74%), as on MNLI. Within MNLI-style items, items correctly labelled
by the MNLI hypothesis-only classifier show 11–
28 points less loss than the remaining items when
nouns are absent, and for 7 of the 8 models the 95%
interval excludes zero.Where the paper says it
Conclusion on Memorisation vs. p. 7 We would
therefore argue that a benchmark score neither confirms nor refutes understanding, which is better
seen as a matter of degree, and that perturbation
experiments are most useful for finding out what a
given benchmark can and cannot distinguish. 7 Conclusion
In this paper we have revisited the effect of input
perturbations on NLI and other language understanding tasks with LLMs and fine-tuned encoders.Where the paper says it
Taken together, the results point away from memorisation and towards shortcuts as the reason why familiar NLI benchmarks survive perturbation First, the contrast between MNLI and ANLI is shared by every system that answers directly Second, none of the contamination indicators is related to robustness Third, for the open LLMs most robust on MNLI, items that can be solved from the hypothesis alone are more robust Fourth, and most directly, every LLM shows the same signature on fresh items that it cannot have seen.
Limitations and Future Directions
Our contamination checks are partial The shortcut split depends on one simple classifier; items it labels incorrectly may still contain cues that a stronger classifier would find The hosted models are analysed on the 2,000-item MNLI sample, which gives the comparison less power than for the open models Open questions remain: a larger, multiply annotated fresh set would sharpen the comparison other cues, such as lexical overlap (McCoy et al., 2019), may explain why the hosted models show no hypothesis-only effect on MNLI and it remains to be seen whether reasoning models in general escape the MNLI–ANLI contrast. p. 7 Open questions remain: a larger, multiply annotated fresh set would sharpen the comparison;
other cues, such as lexical overlap (McCoy et al. ,
2019), may explain why the hosted models show
no hypothesis-only effect on MNLI; and it remains
to be seen whether reasoning models in general
escape the MNLI–ANLI contrast. First,
we reproduce previously published encoder results
on the MNLI dataset within 1 percentage point, and
then re-examine the perturbation experiments of
Talman et al. (2021, 2022) using four open-weight
1
arXiv:2611. First,
we reproduce previously published encoder results
on the MNLI dataset within 1 percentage point, and
then re-examine the perturbation experiments of
Talman et al. (2021, 2022) using four open-weight
1
arXiv:2611.Where the paper says it
Where the paper says it
Where the paper says it
Shortcuts or Memorisation? p. 0 Shortcuts or Memorisation? Revisiting Data Corruption in Natural
Language Understanding with Large Language Models
Aarne Talman
University of Helsinki
aarne. Shortcuts or Memorisation? Revisiting Data Corruption in Natural
Language Understanding with Large Language Models
Aarne Talman
University of Helsinki
aarne. Shortcuts or Memorisation? Revisiting Data Corruption in Natural
Language Understanding with Large Language Models
Aarne Talman
University of Helsinki
aarne.Where the paper says it
Where the paper says it
Where the paper says it
Robustness to perturbation and benchmark differences. p. 5 Among all 80 comparisons, 7
have 95% confidence intervals excluding zero, with
differences in both directions and an overall average of −0. We therefore find no evidence
that exposure to the source text increases robustness.Where the paper says it
Contamination indicators and shortcut mechanisms. p. 0 Only a model that
reasons before answering departs from this pattern. We also check contamination indicators
(including exact n-gram counts in the public
pre-training data of two of the models) and
find no link between exposure and robustness.Where the paper says it
Hypothesis-only analysis and fresh items. p. 7 We recommend that robustness results on NLI benchmarks
be reported together with a hypothesis-only analysis and, where possible, on items written after the
models’ training data were collected. Open questions remain: a larger, multiply annotated fresh set would sharpen the comparison;
other cues, such as lexical overlap (McCoy et al.Where the paper says it
Conclusion on memorisation vs. p. 7 We would
therefore argue that a benchmark score neither confirms nor refutes understanding, which is better
seen as a matter of degree, and that perturbation
experiments are most useful for finding out what a
given benchmark can and cannot distinguish. 7 Conclusion
In this paper we have revisited the effect of input
perturbations on NLI and other language understanding tasks with LLMs and fine-tuned encoders.Where the paper says it
Limitations and future directions. p. 8 Limitations
Our contamination checks are partial. The n-gram
counts cover the OLMo 2 pre-training mix rather
than that of OLMo 3, sample only two 13-grams
per field, and detect only verbatim overlap; paraphrased or reformatted copies of a benchmark, for
example in code repositories, would be missed.Where the paper says it
The paper is 2611.16843. EVERY sentence must end with a reference token. p. 2 3 Models
Fine-tuned encoders. As a reference we finetune RoBERTa-base (Liu et al.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Robustness analysis should incorporate hypothesis-only classification to distinguish between memorization and shortcuts, as
robustness to perturbation on familiar NLI benchmarks reflects dataset shortcuts rather than memorisation.
This allows systems to better understand that a high score on MNLI might stem from superficial cues rather than deep language understanding of the sentence pair. -
Develop methods for assessing how different cues survive perturbations, as
robustness against perturbation alone cannot prove language understanding or memorisation. What it reveals depends on which cues remain intact even after perturbation is applied.
This enables the creation of benchmarks that specifically test the necessity of different linguistic elements (like function words) for correct inference. -
Implement a mechanism to detect answer bias in open-weight LLMs, as
Perturbed input not only removes information but also biases LLMs toward a single answer.
This allows systems to be evaluated on centred accuracy rather than raw accuracy, whichconflates two things: losing the information needed for the task, and shifting towards one answer.
-
Utilize fresh, unseen items written in MNLI style as a critical test for generalization, as
on 368 new, human-validated items that none of the models can have seen all eight LLMs exhibit the same shortcut pattern in these unseen items.
This provides a direct measure of whether performance is due to memorization or learned patterns from familiar data. -
Prioritize training on ablated data for encoders when seeking robustness, as
for encoders, training on ablated data recovers most of the loss (Talman et al., 2021, 2022).
This demonstrates that fine-tuning can mitigate the effects of data corruption by forcing the model to rely on more fundamental linguistic structures.
Abstract
Pre-trained encoders fine-tuned on natural language inference (NLI) and other language understanding tasks keep much of their accuracy after the input is perturbed, e.g. by removing all nouns or shuffling the words. Does the same hold for large language models (LLMs), and if so, is it because they have seen the benchmarks during pre-training? In this paper we revisit these perturbation experiments with eight LLMs alongside fine-tuned encoders on GLUE and the three rounds of ANLI. Regardless of model size, every system that answers directly retains 36-72% of its above-chance performance without nouns on MNLI compared to only 14-29% on ANLI. Only a model that reasons before answering departs from this pattern. We also check contamination indicators (including exact n-gram counts in the public pre-training data of two of the models) and find no link between exposure and robustness. Our strongest evidence comes from 368 new, human-validated items that none of the models can have seen. All eight LLMs keep more of their performance without nouns on items written in MNLI's style than on items written without its cues. When the perturbation instead removes the function words carrying such cues, this advantage reverses. In light of these results, we argue that robustness to perturbation on familiar NLI benchmarks reflects dataset shortcuts rather than memorisation, and tells us more about the benchmark than about the model.
Sources
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data