Which Self-Improvements Should We Trust? Reliable Self-Improvement When Agents Reuse Their Benchmarks

arXiv:2609.33180 · stat.ML, cs.AI, cs.LG, stat.ME · Submitted 2026-09-27 · Read on arXiv

stat.ML, cs.AI, cs.LG, stat.ME

Submitted: 2026-09-27

Updated: 2026-09-29

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 17× in this paper · explained in 2 episodes
• Accuracy — 10× in this paper · explained in 1 episode
• International Conference on Machine Learning — 5× in this paper
• Generalization — 4× in this paper · explained in 13 episodes
• Regularization — 4× in this paper · explained in 3 episodes
• Adam — 3× in this paper · explained in 2 episodes
• Alignment — 3× in this paper · explained in 1 episode
• Self-Evolving Agents — 2× in this paper · explained in 2 episodes
• Ablation — 1× in this paper · explained in 1 episode
• Adaptation — 1× in this paper · explained in 1 episode
• Agentic System — 1× in this paper · explained in 2 episodes
• Algorithmic Stability — 1× in this paper · explained in 1 episode
• Benchmarking — 1× in this paper · explained in 3 episodes
• Calibration — 1× in this paper · explained in 7 episodes
• Core — 1× in this paper · explained in 2 episodes
• Differential Privacy — 1× in this paper · explained in 4 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Foundation Models — 1× in this paper · explained in 7 episodes
• In Conference on Language Modeling — 1× in this paper
• In International Conference on Learning Representations — 1× in this paper
• Post-Selection — 1× in this paper · explained in 2 episodes
• Reward Hacking — 1× in this paper · explained in 8 episodes
• Risk-controlled Evaluation Under Sequential Evolution — 1× in this paper
• Robustness — 1× in this paper · explained in 5 episodes
• Scaling Laws — 1× in this paper · explained in 4 episodes
• Selection Bias — 1× in this paper · explained in 4 episodes
• Theoretical Guarantees — 1× in this paper · explained in 2 episodes
• Verification — 1× in this paper · explained in 2 episodes

Sources

• What Fits (Into Few Tokens) Doesn't Overfit: Compression and Generalization in ML Research Agents
• Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops
• Optimal Inference After Model Selection
• Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents
• The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
• Categorizing Variants of Goodhart's Law
• Signed Compression Progress on a Sealed Audit is Goodhart-Resistant
• AlphaEvolve: A coding agent for scientific and algorithmic discovery
• Self-Evolving Agents with Anytime-Valid Certificates
• PACE: Anytime-Valid Acceptance Tests for Self-Evolving Agents
• SGM: A Statistical Godel Machine for Risk-Controlled Recursive Self-Modification
• Towards Reliable LLM Evaluation: Correcting the Winner's Curse in Adaptive Benchmarking
• Qwen2.5 Technical Report

Related papers

• Behavior of prediction performance metrics with rare events
• Optimal Estimation of Generic Dynamics by Path-Dependent Neural Jump ODEs
• A Posterior-Dynamics Framework for Imaging Inverse Problems with Pretrained Diffusion Priors
• Efficient quadratic entropy with distance sketches
• Embedding-Bias in Conditional Independence Testing
• From Log-Odds to Shapley Values: An Explanatory Geometry for the Weighted Naive Bayes Classifier