Are AI Coders Snitches? An Empirical Study of Pretraining Data Detection on Code Large Language Models

arXiv:2507.17389 · cs.SE, cs.AI · Submitted 2025-07-23 · Read on arXiv

cs.SE, cs.AI

Submitted: 2025-07-23

Updated: 2026-10-01

Code: https://github.com/browser-use/browser-use

Project page: https://sites.google.com/view/codellms-tdd

Terminology

• Benchmark — 1× in this paper · explained in 2 episodes
• Robustness — 1× in this paper · explained in 5 episodes

Sources

• StarCoder: may the source be with you!
• Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
• Purifying Large Language Models by Ensembling a Small Language Model
• Investigating Data Contamination for Pre-training Language Models
• Membership Inference Attacks against Machine Learning Models
• Detecting Pretraining Data from Large Language Models
• Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models
• ReCaLL: Membership Inference via Relative Conditional Log-Likelihoods
• ACECode: A Reinforcement Learning Framework for Aligning Code Efficiency and Correctness in Code Language Models
• Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agents
• Towards Trustworthy LLMs for Code: A Data-Centric Synergistic Auditing Framework
• Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting
• Membership Inference Attacks against Language Models via Neighbourhood Comparison
• Gotcha! This Model Uses My Code! Evaluating Membership Leakage Risks in Code Models
• CodeBERT: A Pre-Trained Model for Programming and Natural Languages
• How Do Semantically Equivalent Code Transformations Impact Membership Inference on LLMs for Code?
• The Stack: 3 TB of permissively licensed source code
• DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
• SantaCoder: don't reach for the stars!
• Code Llama: Open Foundation Models for Code

Related papers

• Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
• GitSkills: A Dataset of Agent Skills on GitHub
• SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
• PackMonitor: Enabling Zero Package Hallucinations Through Decoding-Time Monitoring
• IntentCoding: Amplifying User Intent in Code Generation
• Incentives and Outcomes in Bug Bounties