Can LLM Agents Discover? Evaluating Creativity on ML Engineering Tasks
cs.AI
Submitted: 2026-08-30
Updated: 2026-08-30
Comments: COLM 2026
Code: https://github.com/shitanshubhushan/Creativity-Evaluation-Framework
License: http://creativecommons.org/licenses/by/4.0/
The gist: Recent AI systems promise autonomous scientific discovery, claiming to discover algorithms and produce research papers, yet understanding whether they exhibit creativity, the capacity to produce
Terminology
Abstract
Recent AI systems promise autonomous scientific discovery, claiming to discover algorithms and produce research papers, yet understanding whether they exhibit creativity, the capacity to produce solutions that are both novel and useful, remains an open question. We present a framework for evaluating multi-turn LLM research agents' creativity using ML engineering tasks as a testbed, through three dimensions: P-Creativity (psychological novelty: novel relative to the agent's own prior solutions within a run), H-Creativity (historical novelty: novel relative to the corpus of human solutions), and Usefulness (task performance). Evaluating two agent frameworks, AIDE and AIRA-Dojo, on 10 Kaggle-style machine learning tasks from MLE-Bench, we develop an LLM-as-a-Judge pipeline and verify its strong correlation with human creativity judgments, providing a reliable automated metric for P-Creativity evaluation at scale. Applying this pipeline to agent trajectories, we find: (1) all agents exhibit declining P-Creativity as they transition from exploration to exploitation; (2) LLMs exhibit greater H-Creativity than medal-winning humans, yet achieve lower performance. Our findings reveal that current agents can explore novel regions of the solution space but lack the capacity to convert this novelty into improved task performance.
Sources
- What Does It Take to Be a Good AI Research Agent? Studying the Role of Ideation Diversity
- LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
- AGC-Bench: Measuring Artificial General Creativity
- Divergent Creativity in Humans and Large Language Models
- MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research
- CreativEval: Evaluating Creativity of LLM-Based Hardware Code Generation
- MMTEB: Massive Multilingual Text Embedding Benchmark
- CreativityPrism: A Cross-Domain Evaluation Framework for Large Language Model Creativity
- AIDE: AI-Driven Exploration in the Space of Code
- AutoResearchClaw: Self-Reinforcing Autonomous Research with Human-AI Collaboration
- The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
- Rethinking Creativity Evaluation: A Critical Analysis of Existing Creativity Evaluations
- EvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discovery
- Kosmos: An AI Scientist for Autonomous Discovery
- AlphaEvolve: A coding agent for scientific and algorithmic discovery
- MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering
- OpenAI GPT-5 System Card
- GLM-5: from Vibe Coding to Agentic Engineering
- Qwen3 Technical Report
- AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-bench
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems