Soft Contamination Means Benchmarks Test Shallow Generalization
cs.LG, cs.AI
Submitted: 2026-02-12
Updated: 2026-11-07
Code: https://github.com/AriSpiesberger/Soft-Contamination-Prevelance
Terminology
Sources
- Program Synthesis with Large Language Models
- Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Tasks
- BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Training Verifiers to Solve Math Word Problems
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- Measuring Massive Multitask Language Understanding
- Investigating Data Contamination for Pre-training Language Models
- StarCoder: may the source be with you!
- Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling
- Artificial Intelligence Index Report 2025
- Olmo 3
- GPT-4 Technical Report
- Detecting Pretraining Data from Large Language Models
- The Mosaic Memory of Large Language Models
- Self-Instruct: Aligning Language Models with Self-Generated Instructions
- Rethinking Benchmark and Contamination for Language Models with Rephrased Samples
- HellaSwag: Can a Machine Really Finish Your Sentence?
- LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks
Related papers
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis
- (Mis)generalization of Helpful-Only Fine-tuning