Shortcuts or Memorisation? Revisiting Data Corruption in Natural Language Understanding with Large Language Models

summary

The gist

Pre-trained encoders fine-tuned on natural language inference (NLI) and other language understanding tasks keep much of their accuracy after the input is perturbed, e.g.

In short

The study investigated why large language models maintain performance on natural language inference tasks even when input is intentionally corrupted, such as removing nouns or shuffling words. Findings suggest that robustness stems from shortcuts rather than memorization, as certain items are easier to solve using only the initial hypothesis. The gap between different benchmarks like MNLI and ANLI is attributed to dataset differences, not inherent model weaknesses.

Key concepts

Robustness to Perturbation
This tests how well a model's answer holds up when parts of the input text are deliberately changed or removed, like taking out all nouns. The paper found that many models keep most of their accuracy even with these changes, indicating they rely on more than just exact memorization.
Contamination Indicators
These are specific metrics used to check for potential data corruption or shortcuts within the model's responses. The research found no significant link between these indicators and overall robustness, suggesting that the presence of these signs doesn't reliably predict how well a model handles noisy input.
Hypothesis-Only Analysis
This method involves testing if an item can be correctly classified using only the initial guess or hypothesis generated by the model, without needing further complex reasoning. The paper showed that for certain challenging items, this direct approach reveals patterns related to memorization versus shortcuts.
MNLI vs. ANLI Gap
This refers to the difference in performance between two specific natural language inference benchmarks. The research concluded that this performance gap is caused by the differences in the datasets used for each benchmark, not necessarily a fundamental limitation of how well a model understands language generally.

Terminology used across episodes

This episode discusses

The paper

Shortcuts or Memorisation? Revisiting Data Corruption in Natural Language Understanding with Large Language Models · Read on arXiv

University of Helsinki · Institute of Language and Speech Processing Athena Research Centre Athens

Pre-trained encoders fine-tuned on natural language inference (NLI) and other language understanding tasks keep much of their accuracy after the input is perturbed, e.g. by removing all nouns or shuffling the words. Does the same hold for large language models (LLMs), and if so, is it because they have seen the benchmarks during pre-training? In this paper we revisit these perturbation experiments with eight LLMs alongside fine-tuned encoders on GLUE and the three rounds of ANLI. Regardless of model size, every system that answers directly retains 36-72% of its above-chance performance without nouns on MNLI compared to only 14-29% on ANLI. Only a model that reasons before answering departs from this pattern. We also check contamination indicators (including exact n-gram counts in the public pre-training data of two of the models) and find no link between exposure and robustness. Our strongest evidence comes from 368 new, human-validated items that none of the models can have seen. All eight LLMs keep more of their performance without nouns on items written in MNLI's style than on items written without its cues. When the perturbation instead removes the function words carrying such cues, this advantage reverses. In light of these results, we argue that robustness to perturbation on familiar NLI benchmarks reflects dataset shortcuts rather than memorisation, and tells us more about the benchmark than about the model.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Shortcuts or Memorisation? Revisiting Data Corruption in Natural Language Understanding with Large Language Models".

Jane: Here's the gist of the paper. Pre-trained encoders fine-tuned on natural language inference (NLI) and other language understanding tasks keep much of their accuracy after the input is perturbed, e.g.

Tom: by removing all nouns or shuffling the words The main contributions of this paper are as follows.

Jane: Robustness to Perturbation and Benchmark Differences Regardless of model size, every system that answers directly retains 36–72% of its above-chance performance without nouns on MNLI compared to only 14–29% on ANLI Only a model that reasons before answering departs from this pattern.

Tom: The paper demonstrates that the gap between MNLI and ANLI lies in the dataset, not the model Robustness against perturbation alone cannot prove language understanding or memorisation What it reveals depends on which cues remain intact even after perturbation is applied.

Paper summary: Tom: So we just talked about how these models handle losing nouns, and the paper "Shortcuts or Memorisation? Revisiting Data Corruption in Natural Language Understanding with Large Language Models" claims that even when you strip out nouns, most systems keep a lot of their accuracy on MNLI >

Jane: They found that every system that answers directly retains thirty six to seventy two percent of its above chance performance without nouns on MNLI compared to only fourteen to twenty nine percent on ANLI >

Lu: The paper also points out that the gap between those two benchmarks is in the dataset not necessarily the model itself, regardless of how large or small we make the LLM >

Meng: They also checked some contamination indicators—like exact n-gram counts in the pre-training data—and found no association with robustness >

Tom: That means they aren't seeing a simple link between how much data corruption a model sees and how robust it becomes against losing parts of the input >

Jane: Instead, they focus on items solvable solely from the hypothesis, which they found to be more robust in all three of the most robust LLMs on MNLI when compared to when function words are removed >

Lu: They built three hundred sixty eight new human-validated NLI items and found that all eight LLMs show the same shortcut pattern across these unseen items >

Tom: That means they aren't seeing memorization as the main driver here, but something else is at play in how those models work >

Jane: They compare MNLI with BioASQ, which is also robust to perturbation, but their results suggest that the structure of the data matters more than just having a large model >

Conclusion: Tom: So when we look at this paper, "Shortcuts or Memorisation? Revisiting Data Corruption in Natural Language Understanding with Large Language Models," they are suggesting that the survival of familiar NLI benchmarks isn't because the models memorized specific answers >

Jane: They argue instead that certain structural shortcuts in how the data is presented allow these models to maintain performance even when you corrupt the input, like removing all nouns >

Lu: The key evidence they offer is that for the most robust LLMs on MNLI, items that can be solved just from the initial hypothesis are more robust than items where you remove function words >

Meng: What this means for us in production is that we shouldn't just look at memorization as a failure mode when models get weird input; we should also be looking at what structural assumptions they make about the language structure itself >

Tom: And they show that these shortcut patterns are consistent across all eight models even on brand new items they haven't seen, which is pretty strong evidence for this idea >

Jane: It implies that understanding the underlying structure of natural language is a more stable thing than remembering every specific sentence in the training set >

More episodes

← Home