Cleaning or Recalling? Knowledge Dependence in LLM-based Data Cleaning [Experiment, Analysis & Benchmark]
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Cleaning or Recalling? Knowledge Dependence in LLM-based Data Cleaning
Experiment, Analysis & Benchmark: ".
Jane: Here's the gist of the paper. High scores on public data-cleaning benchmarks can reflect knowledge of familiar entities, making it difficult to assess how well an LLM repairs unfamiliar data The authors present CoRe, a protocol comparing public benchmarks with fresh real records and paired name surrogates built from English words or syllables across 21 controlled tables and eight LLMs, repair performance varies substantially with entity names on Hospital and, for commercial models, Movies Renaming removes some information needed for repair, such as a film’s identifier mapping On Hospital and Movies, commercial models score higher on fresh records than on word-based surrogates A screened continuation probe finds no positive evidence of benchmark-row memory in the tested models Pruning redundant reasoning tokens The paper introduces CoRe to compare repairs on dirty/reference pairs under controlled entity replacements Each benchmark is compared with fresh real records, a word-based surrogate with fictitious English-word names, and a syllable-based surrogate with meaningless names The original and its surrogates share rows and error positions, while fresh records carry the same kinds of errors Paired substitutions test entity familiarity and word form How the training signal is built CoRe evaluates repair under four input conditions, including Fresh (F), Word-based surrogate (W), and Syllable-based surrogate (S) The three paired contrasts are O−W, W−S, and O−S These contrasts measure score changes under entity and word-form replacement Recovery depends on the target convention and the information shown to the cleaner Where the gains come from The analysis reveals that Movie identifiers lose their mappings after renaming, but year ranges retain the same candidates On Movies, DeepSeek-V4-Pro’s year accuracy falls from 91% on real films to 18% on word names and 13% on syllable names, even though the candidates are unchanged On Hospital, three commercial models lose 0.59 to 0.72 repair F1 after syllable replacement on identity columns Reference errors can also change which edits receive credit Insights into knowledge dependence The study characterizes both knowledge-dependent repair and directed rewriting of assigned values across eight LLMs A familiar fact can be the wrong value to preserve On Rayyan, 434/461 of TabClean’s credited repairs reproduce the reference date transformation, whereas an externally matched ISSN correction earns no credit The paired comparisons in CoRe retain error positions, while fresh records and a screened continuation probe examine transfer and benchmark-row memory separately Lessons and recommendations The authors suggest distinguishing missing mappings from retained candidates They recommend aligning labels, error positions and supporting records when comparing chunk-based and whole-table methods Furthermore, they advise setting source priority and testing value preservation when dealing with cross-source errors or assigned values They also recommend inspecting the edits rewarded by the reference to understand which transformations are truly correct Applying CoRe to other benchmarks Applying CoRe requires profiling errors and reference conventions, selecting a compatible fresh source, and building the two naming styles with acceptance checks The study shows that geographic codes can retain familiar meanings after names change Record which field relationships survive identity replacement when applying CoRe to other benchmarks Artifact and result replay The statistical core replays condition contrasts, conflict rates and Rayyan checks offline from Hospital/Beers tables, Movies numerical witnesses and re-scored continuation counts Authorized local validation checks historical movie inputs, replies and scoring-code hashes before re-scoring Limitations The datasets diagnose recovery mechanisms and reference transformations in entity-rich public tables, rather than estimate their prevalence across cleaning workloads Transfer to private data also depends on size, sparsity, task complexity and the availability of supporting records The study covers eight commercial and open-weight cleaners on English tables concerning mainly US entities, with decoding settings in Report §2; it does not cover Claude or Gemini The intervals describe variation across entities and replies within fixed constructions New backgrounds, regrouped prompts and repeated API calls introduce other sources of variation The intervals describe variation across entities and replies within fixed constructions Renaming Movies lowers DeepSeek-V4-Pro’s year accuracy from 91% to 18% even though the candidate years remain in the input Identifier repairs decline too, but their mappings are no longer available On Hospital, GPT repairs 74 of 95 errors with a unique restoration under the constructor’s restricted syllable inventory The paired comparisons in CoRe retain error positions, while fresh records and a screened continuation probe examine transfer and benchmark-row memory separately The probe finds no positive evidence of row memory in its tested models The assigned-value and reference analyses reveal errors that conventional repair scores can miss Some models replace assigned directors with real credits even after an explicit table-evidence instruction On Rayyan, 434/461 of TabClean’s credited repairs reproduce a reference date transformation, while an externally matched ISSN correction earns no credit For evaluation, these cases call for tests with a specified source priority and inspection of the edits counted as correct The constructions and scoring code support comparisons with other data cleaners References
Paper summary: Tom: So, thinking about the title and what we've discussed, Cleaning or Recalling? Knowledge Dependence in LLM-based Data Cleaning
Experiment, Analysis & Benchmark: points toward a necessary caution when evaluating these AI cleaning tools.
Jane: It suggests that the way we measure an AI’s ability to clean data needs to be more nuanced than just looking at the final repair percentage on a benchmark dataset.
Lu: The implication is that if you're building systems, you need to test how those systems handle changes in entity names—whether it’s swapping a real name for a word or just changing the way the entity is represented in the data.
Meng: For production, this means we can't rely on one model being universally good across all types of entities; we have to understand which parts of its knowledge are actually useful for cleaning specific tables.
Lalam: The core message is that repair performance isn't just about fixing errors; it’s about understanding the underlying knowledge dependence, and if that dependence is brittle, the repair will break when you move to new data.
Conclusion: Tom: So, we’ve looked at how these models clean data using this new CoRe protocol, and now we’re wrapping up by talking about what that whole thing actually means for us as a team.
Jane: The paper's title is "Cleaning or Recalling? Knowledge Dependence in LLM-based Data Cleaning," and it really gets to the heart of how these AI tools learn from data.
Lu: It suggests that just getting a high score on public cleaning benchmarks isn't enough because those scores might just reflect knowledge of familiar things, not actual repair skill on messy, new data.
Meng: So if we’re shipping models in production, this tells us that relying only on benchmark scores might be risky because the training signal itself can be tied to old patterns.
Lalam: Exactly. The study found that repair performance changes a lot depending on whether you compare the cleaning against fresh real records, or against fake words made up from English syllables.
Tom: It’s interesting how they showed that for some models, like DeepSeek-V4-Pro on movie data, renaming those film identifiers actually caused the year accuracy to drop way down from ninety one percent to eighteen percent.
Jane: That’s a big warning because it shows that even if you think you're just changing a name, the AI can lose critical pieces of information needed for the repair.
Lu: The researchers also found that sometimes what an AI does is actually just remembering something familiar instead of correctly cleaning the error.
Meng: That’s exactly what I worry about when we deploy these; if it's recalling a wrong fact, even with a high score, that’s a failure at scale.
Lalam: The conclusion points out that you need to distinguish between those missing mappings and just retaining candidates for the repair.
Tom: It also shows that some models can replace assigned values with real credits even when told they shouldn't, which means we have to be careful about how we instruct them.
Jane: So, the big implication is that understanding *why* an AI makes a mistake—whether it’s because it’s bad at cleaning or because its knowledge base is misleading—is now just as important as the final score.
Lu: And that opens up a whole new way to test these systems, focusing on transfer and how they handle those kinds of memory effects across different data types.
Meng: It means we need to build our own tests that specifically look for this kind of knowledge dependence instead of just running them against some pre-made benchmark.
Minzu University of China · Renmin University of China
cs.DB
Submitted: 2026-11-05
Updated: 2026-11-05
Code: https://github.com/hinsonver/CoRe-Benchmark
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 81/100
The gist: High scores on public data-cleaning benchmarks can reflect knowledge of familiar entities, making it difficult to assess how well an LLM repairs unfamiliar data The authors present CoRe…
Key concepts
- CoRe Protocol
- A method used to compare how well different LLMs repair data cleaning errors. It compares repairs made on original data against repairs made using word-based and syllable-based fake names to test entity familiarity and word form knowledge.
- Knowledge-Dependent Repair
- This occurs when an LLM preserves a fact that is technically wrong but familiar or assigned, rather than correcting it based on external evidence. For example, an LLM might keep a specific date even if the reference material shows it's incorrect because the fact is 'known'.
- Benchmark-Row Memory
- This refers to whether an LLM relies on the specific structure or content of a benchmark row rather than general knowledge when performing repairs. The study tested this using a screened continuation probe and found no positive evidence of this memory in the tested models.
Terminology
Summary
High scores on public data-cleaning benchmarks can reflect knowledge of familiar entities, making it difficult to assess how well an LLM repairs unfamiliar data The authors present CoRe, a protocol comparing public benchmarks with fresh real records and paired name surrogates built from English words or syllables across 21 controlled tables and eight LLMs, repair performance varies substantially with entity names on Hospital and, for commercial models, Movies Renaming removes some information needed for repair, such as a film’s identifier mapping On Hospital and Movies, commercial models score higher on fresh records than on word-based surrogates A screened continuation probe finds no positive evidence of benchmark-row memory in the tested models
Pruning redundant reasoning tokens
The paper introduces CoRe to compare repairs on dirty/reference pairs under controlled entity replacements Each benchmark is compared with fresh real records, a word-based surrogate with fictitious English-word names, and a syllable-based surrogate with meaningless names The original and its surrogates share rows and error positions, while fresh records carry the same kinds of errors Paired substitutions test entity familiarity and word form
How the training signal is built
CoRe evaluates repair under four input conditions, including Fresh (F), Word-based surrogate (W), and Syllable-based surrogate (S) The three paired contrasts are O−W, W−S, and O−S These contrasts measure score changes under entity and word-form replacement Recovery depends on the target convention and the information shown to the cleaner
Where the gains come from
The analysis reveals that Movie identifiers lose their mappings after renaming, but year ranges retain the same candidates On Movies, DeepSeek-V4-Pro’s year accuracy falls from 91% on real films to 18% on word names and 13% on syllable names, even though the candidates are unchanged On Hospital, three commercial models lose 0.59 to 0.72 repair F1 after syllable replacement on identity columns Reference errors can also change which edits receive credit
Insights into knowledge dependence
The study characterizes both knowledge-dependent repair and directed rewriting of assigned values across eight LLMs A familiar fact can be the wrong value to preserve On Rayyan, 434/461 of TabClean’s credited repairs reproduce the reference date transformation, whereas an externally matched ISSN correction earns no credit The paired comparisons in CoRe retain error positions, while fresh records and a screened continuation probe examine transfer and benchmark-row memory separately
Lessons and recommendations
The authors suggest distinguishing missing mappings from retained candidates They recommend aligning labels, error positions and supporting records when comparing chunk-based and whole-table methods Furthermore, they advise setting source priority and testing value preservation when dealing with cross-source errors or assigned values They also recommend inspecting the edits rewarded by the reference to understand which transformations are truly correct
Applying CoRe to other benchmarks
Applying CoRe requires profiling errors and reference conventions, selecting a compatible fresh source, and building the two naming styles with acceptance checks The study shows that geographic codes can retain familiar meanings after names change Record which field relationships survive identity replacement when applying CoRe to other benchmarks
Artifact and result replay
The statistical core replays condition contrasts, conflict rates and Rayyan checks offline from Hospital/Beers tables, Movies numerical witnesses and re-scored continuation counts Authorized local validation checks historical movie inputs, replies and scoring-code hashes before re-scoring
Limitations
The datasets diagnose recovery mechanisms and reference transformations in entity-rich public tables, rather than estimate their prevalence across cleaning workloads Transfer to private data also depends on size, sparsity, task complexity and the availability of supporting records The study covers eight commercial and open-weight cleaners on English tables concerning mainly US entities, with decoding settings in Report §2; it does not cover Claude or Gemini The intervals describe variation across entities and replies within fixed constructions New backgrounds, regrouped prompts and repeated API calls introduce other sources of variation The intervals describe variation across entities and replies within fixed constructions
Conclusion
Renaming Movies lowers DeepSeek-V4-Pro’s year accuracy from 91% to 18% even though the candidate years remain in the input Identifier repairs decline too, but their mappings are no longer available On Hospital, GPT repairs 74 of 95 errors with a unique restoration under the constructor’s restricted syllable inventory The paired comparisons in CoRe retain error positions, while fresh records and a screened continuation probe examine transfer and benchmark-row memory separately The probe finds no positive evidence of row memory in its tested models The assigned-value and reference analyses reveal errors that conventional repair scores can miss Some models replace assigned directors with real credits even after an explicit table-evidence instruction On Rayyan, 434/461 of TabClean’s credited repairs reproduce a reference date transformation, while an externally matched ISSN correction earns no credit For evaluation, these cases call for tests with a specified source priority and inspection of the edits counted as correct The constructions and scoring code support comparisons with other data cleaners
References
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Transfer benchmark error mechanisms to new records and renamed identity values across different entity names or word forms to assess repair performance under varying levels of entity familiarity, as described in CoRe protocol.
-
Evaluate repair performance using three input conditions: Fresh records (F), Word-based surrogates (W), and Syllable-based surrogates (S) to distinguish between knowledge dependence, word form knowledge, and syllable knowledge in data cleaning tasks.
-
Implement a screening continuation probe to test for benchmark-row memory, which helps distinguish whether repair scores reflect genuine recovery or reliance on familiar entity facts that might conflict with the task.
-
Differentiate between different types of errors—locally reversible (R1), constrained (R2), and underdetermined (R3)—to determine what information is truly recoverable from the input when a target convention is specified.
-
Utilize explicit table-evidence instructions to test for directed rewriting, as commercial models rewrite assigned directors at rates of 0.000–0.404 on known films compared to 68%–100% under standard prompts.
-
Incorporate a reference scoring mechanism that distinguishes between repairs that match reference transformations (like Rayyan's ISSN correction) and those that do not, to prevent erroneous credit for irrelevant edits.
-
Develop strategies for preserving assigned values by constructing knowledge conflicts where target cells are paired with real-world facts, allowing the system to test whether a familiar name draws the cleaner toward the real-world fact rather than arbitrary edits.
-
Improve entity mapping recovery on records like Movies by explicitly supplying candidate lists when an identifier mapping is absent after renaming, rather than assuming a fictitious slug determines its random identifier.
-
Compare chunk-based and whole-table methods under aligned labels and error positions to determine if supporting records outside the main chunks can supply replacement values that are absent from the chunk itself.
Abstract
High scores on public data-cleaning benchmarks can reflect knowledge of familiar entities, making it difficult to assess how well an LLM repairs unfamiliar data. We present CoRE (Cleaning or Recalling), a protocol comparing public benchmarks with fresh real records and paired name surrogates built from English words or syllables. The surrogates retain error positions and non-identity values while transferring the benchmark's error mechanisms. Across 21 controlled tables and eight LLMs, repair performance varies substantially with entity names on Hospital and, for commercial models, Movies. Renaming removes some information needed for repair, such as a film's identifier mapping. It also affects repairs whose candidates remain available: replacing entity names with word-based surrogates lowers DeepSeek-V4-Pro's year accuracy from 91% to 18%. On Hospital and Movies, commercial models score higher on fresh records than on word-based surrogates. A screened continuation probe finds no positive evidence of benchmark-row memory in the tested models. Familiar facts can conflict with the task. Even with explicit table-evidence instructions, commercial models rewrite assigned directors at rates of 0.000-0.404 on known films. Reference scoring can also disagree with external records. On Rayyan, an ISSN correction matching a title-matched PubMed record receives no credit, whereas 94% of TabClean's credited repairs reproduce the reference date transformation. We provide controlled tables, scoring code and materials for offline replay.
Sources
- REIN: A Comprehensive Benchmark Framework for Data Cleaning Methods in ML Pipelines
- SEED: Domain-Specific Data Curation With Large Language Models
- DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
- Making LLMs Work for Enterprise Data Tasks
- The Llama 3 Herd of Models
- Qwen2.5 Technical Report
- TabClean: Reusable LLM-Synthesized Programs for Tabular Data Cleaning
- Benchmark Data Contamination of Large Language Models: A Survey
Related papers
- Beyond Overlap: Hierarchical Graph Abstraction for Out-of-Core Vector Search
- EMA: Approximate Nearest Neighbor Search with General Attribute Filtering and Dynamic Updates
- Open Biomedical Knowledge Graphs at Scale: Construction, Federation, and AI Agent Access with Samyama Graph Database
- Samyama: A Unified Graph-Vector Database with In-Database Optimization, Agentic Enrichment, and Hardware Acceleration
- GenTUS: Table Union Search via Constrained Generation of Semantic Table Identifiers
- Common-Neighbor-Count-Based Representative Possible World Finding on Uncertain Graphs