Cleaning or Recalling? Knowledge Dependence in LLM-based Data Cleaning [Experiment, Analysis & Benchmark]
summary
The gist
High scores on public data-cleaning benchmarks can reflect knowledge of familiar entities, making it difficult to assess how well an LLM repairs unfamiliar data The authors present CoRe…
In short
The study tested how well LLMs repair data cleaning errors when entity names change, comparing repairs against fresh records and word/syllable surrogates. Findings show that commercial models perform better with fresh data, and renaming movies significantly degrades year accuracy. It reveals knowledge dependence, where LLMs sometimes preserve incorrect facts or ignore external evidence.
Key concepts
- CoRe Protocol
- A method used to compare how well different LLMs repair data cleaning errors. It compares repairs made on original data against repairs made using word-based and syllable-based fake names to test entity familiarity and word form knowledge.
- Knowledge-Dependent Repair
- This occurs when an LLM preserves a fact that is technically wrong but familiar or assigned, rather than correcting it based on external evidence. For example, an LLM might keep a specific date even if the reference material shows it's incorrect because the fact is 'known'.
- Benchmark-Row Memory
- This refers to whether an LLM relies on the specific structure or content of a benchmark row rather than general knowledge when performing repairs. The study tested this using a screened continuation probe and found no positive evidence of this memory in the tested models.
Terminology used across episodes
This episode discusses
- Cleaning or Recalling? Knowledge Dependence in LLM-based Data Cleaning [Experiment, Analysis & Benchmark] · Paper Radio
- REIN: A Comprehensive Benchmark Framework for Data Cleaning Methods in ML Pipelines
- SEED: Domain-Specific Data Curation With Large Language Models
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- Making LLMs Work for Enterprise Data Tasks
- The Llama 3 Herd of Models · Paper Radio
- Qwen2.5 Technical Report
- TabClean: Reusable LLM-Synthesized Programs for Tabular Data Cleaning
- Benchmark Data Contamination of Large Language Models: A Survey
The paper
Cleaning or Recalling? Knowledge Dependence in LLM-based Data Cleaning [Experiment, Analysis & Benchmark] · Read on arXiv
Minzu University of China · Renmin University of China
High scores on public data-cleaning benchmarks can reflect knowledge of familiar entities, making it difficult to assess how well an LLM repairs unfamiliar data. We present CoRE (Cleaning or Recalling), a protocol comparing public benchmarks with fresh real records and paired name surrogates built from English words or syllables. The surrogates retain error positions and non-identity values while transferring the benchmark's error mechanisms. Across 21 controlled tables and eight LLMs, repair performance varies substantially with entity names on Hospital and, for commercial models, Movies. Renaming removes some information needed for repair, such as a film's identifier mapping. It also affects repairs whose candidates remain available: replacing entity names with word-based surrogates lowers DeepSeek-V4-Pro's year accuracy from 91% to 18%. On Hospital and Movies, commercial models score higher on fresh records than on word-based surrogates. A screened continuation probe finds no positive evidence of benchmark-row memory in the tested models. Familiar facts can conflict with the task. Even with explicit table-evidence instructions, commercial models rewrite assigned directors at rates of 0.000-0.404 on known films. Reference scoring can also disagree with external records. On Rayyan, an ISSN correction matching a title-matched PubMed record receives no credit, whereas 94% of TabClean's credited repairs reproduce the reference date transformation. We provide controlled tables, scoring code and materials for offline replay.
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Cleaning or Recalling? Knowledge Dependence in LLM-based Data Cleaning
Experiment, Analysis & Benchmark: ".
Jane: Here's the gist of the paper. High scores on public data-cleaning benchmarks can reflect knowledge of familiar entities, making it difficult to assess how well an LLM repairs unfamiliar data The authors present CoRe, a protocol comparing public benchmarks with fresh real records and paired name surrogates built from English words or syllables across 21 controlled tables and eight LLMs, repair performance varies substantially with entity names on Hospital and, for commercial models, Movies Renaming removes some information needed for repair, such as a film’s identifier mapping On Hospital and Movies, commercial models score higher on fresh records than on word-based surrogates A screened continuation probe finds no positive evidence of benchmark-row memory in the tested models Pruning redundant reasoning tokens The paper introduces CoRe to compare repairs on dirty/reference pairs under controlled entity replacements Each benchmark is compared with fresh real records, a word-based surrogate with fictitious English-word names, and a syllable-based surrogate with meaningless names The original and its surrogates share rows and error positions, while fresh records carry the same kinds of errors Paired substitutions test entity familiarity and word form How the training signal is built CoRe evaluates repair under four input conditions, including Fresh (F), Word-based surrogate (W), and Syllable-based surrogate (S) The three paired contrasts are O−W, W−S, and O−S These contrasts measure score changes under entity and word-form replacement Recovery depends on the target convention and the information shown to the cleaner Where the gains come from The analysis reveals that Movie identifiers lose their mappings after renaming, but year ranges retain the same candidates On Movies, DeepSeek-V4-Pro’s year accuracy falls from 91% on real films to 18% on word names and 13% on syllable names, even though the candidates are unchanged On Hospital, three commercial models lose 0.59 to 0.72 repair F1 after syllable replacement on identity columns Reference errors can also change which edits receive credit Insights into knowledge dependence The study characterizes both knowledge-dependent repair and directed rewriting of assigned values across eight LLMs A familiar fact can be the wrong value to preserve On Rayyan, 434/461 of TabClean’s credited repairs reproduce the reference date transformation, whereas an externally matched ISSN correction earns no credit The paired comparisons in CoRe retain error positions, while fresh records and a screened continuation probe examine transfer and benchmark-row memory separately Lessons and recommendations The authors suggest distinguishing missing mappings from retained candidates They recommend aligning labels, error positions and supporting records when comparing chunk-based and whole-table methods Furthermore, they advise setting source priority and testing value preservation when dealing with cross-source errors or assigned values They also recommend inspecting the edits rewarded by the reference to understand which transformations are truly correct Applying CoRe to other benchmarks Applying CoRe requires profiling errors and reference conventions, selecting a compatible fresh source, and building the two naming styles with acceptance checks The study shows that geographic codes can retain familiar meanings after names change Record which field relationships survive identity replacement when applying CoRe to other benchmarks Artifact and result replay The statistical core replays condition contrasts, conflict rates and Rayyan checks offline from Hospital/Beers tables, Movies numerical witnesses and re-scored continuation counts Authorized local validation checks historical movie inputs, replies and scoring-code hashes before re-scoring Limitations The datasets diagnose recovery mechanisms and reference transformations in entity-rich public tables, rather than estimate their prevalence across cleaning workloads Transfer to private data also depends on size, sparsity, task complexity and the availability of supporting records The study covers eight commercial and open-weight cleaners on English tables concerning mainly US entities, with decoding settings in Report §2; it does not cover Claude or Gemini The intervals describe variation across entities and replies within fixed constructions New backgrounds, regrouped prompts and repeated API calls introduce other sources of variation The intervals describe variation across entities and replies within fixed constructions Renaming Movies lowers DeepSeek-V4-Pro’s year accuracy from 91% to 18% even though the candidate years remain in the input Identifier repairs decline too, but their mappings are no longer available On Hospital, GPT repairs 74 of 95 errors with a unique restoration under the constructor’s restricted syllable inventory The paired comparisons in CoRe retain error positions, while fresh records and a screened continuation probe examine transfer and benchmark-row memory separately The probe finds no positive evidence of row memory in its tested models The assigned-value and reference analyses reveal errors that conventional repair scores can miss Some models replace assigned directors with real credits even after an explicit table-evidence instruction On Rayyan, 434/461 of TabClean’s credited repairs reproduce a reference date transformation, while an externally matched ISSN correction earns no credit For evaluation, these cases call for tests with a specified source priority and inspection of the edits counted as correct The constructions and scoring code support comparisons with other data cleaners References
Paper summary: Tom: So, thinking about the title and what we've discussed, Cleaning or Recalling? Knowledge Dependence in LLM-based Data Cleaning
Experiment, Analysis & Benchmark: points toward a necessary caution when evaluating these AI cleaning tools.
Jane: It suggests that the way we measure an AI’s ability to clean data needs to be more nuanced than just looking at the final repair percentage on a benchmark dataset.
Lu: The implication is that if you're building systems, you need to test how those systems handle changes in entity names—whether it’s swapping a real name for a word or just changing the way the entity is represented in the data.
Meng: For production, this means we can't rely on one model being universally good across all types of entities; we have to understand which parts of its knowledge are actually useful for cleaning specific tables.
Lalam: The core message is that repair performance isn't just about fixing errors; it’s about understanding the underlying knowledge dependence, and if that dependence is brittle, the repair will break when you move to new data.
Conclusion: Tom: So, we’ve looked at how these models clean data using this new CoRe protocol, and now we’re wrapping up by talking about what that whole thing actually means for us as a team.
Jane: The paper's title is "Cleaning or Recalling? Knowledge Dependence in LLM-based Data Cleaning," and it really gets to the heart of how these AI tools learn from data.
Lu: It suggests that just getting a high score on public cleaning benchmarks isn't enough because those scores might just reflect knowledge of familiar things, not actual repair skill on messy, new data.
Meng: So if we’re shipping models in production, this tells us that relying only on benchmark scores might be risky because the training signal itself can be tied to old patterns.
Lalam: Exactly. The study found that repair performance changes a lot depending on whether you compare the cleaning against fresh real records, or against fake words made up from English syllables.
Tom: It’s interesting how they showed that for some models, like DeepSeek-V4-Pro on movie data, renaming those film identifiers actually caused the year accuracy to drop way down from ninety one percent to eighteen percent.
Jane: That’s a big warning because it shows that even if you think you're just changing a name, the AI can lose critical pieces of information needed for the repair.
Lu: The researchers also found that sometimes what an AI does is actually just remembering something familiar instead of correctly cleaning the error.
Meng: That’s exactly what I worry about when we deploy these; if it's recalling a wrong fact, even with a high score, that’s a failure at scale.
Lalam: The conclusion points out that you need to distinguish between those missing mappings and just retaining candidates for the repair.
Tom: It also shows that some models can replace assigned values with real credits even when told they shouldn't, which means we have to be careful about how we instruct them.
Jane: So, the big implication is that understanding *why* an AI makes a mistake—whether it’s because it’s bad at cleaning or because its knowledge base is misleading—is now just as important as the final score.
Lu: And that opens up a whole new way to test these systems, focusing on transfer and how they handle those kinds of memory effects across different data types.
Meng: It means we need to build our own tests that specifically look for this kind of knowledge dependence instead of just running them against some pre-made benchmark.
More episodes
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines
- 2601.15318-On the closest balanced game