MetaBench-Harness: Unlocking End-to-End Optimization of Benchmark Harnesses
cs.AI, cs.CL, cs.SE
Submitted: 2026-09-27
Updated: 2026-09-27
Comments: Work in progress
License: http://creativecommons.org/licenses/by/4.0/
The gist: Rapid progress in Large Language Models (LLMs) is saturating static benchmarks faster than they can be designed.
Terminology
Abstract
Rapid progress in Large Language Models (LLMs) is saturating static benchmarks faster than they can be designed. While existing automated evolution frameworks attempt to generate harder questions by perturbing individual tasks, they remain constrained by rigid, hard-coded generation rules. Moving beyond the evolution of isolated tasks, we propose to optimize the benchmark generation workflow itself end to end with MetaBench-Harness, a dual-loop search framework. Specifically, the inner loop utilizes a benchmark harness to generate a new benchmark in each round, while the outer meta-harness orchestration layer iteratively refines and searches over harness implementations based on historical evolution trajectories. By applying MetaBench-Harness to the competitive programming CodeContests and Olympiad mathematics AIME-2024 datasets, we demonstrate that the evolved benchmarks are challenging and discriminative for frontier models. Trajectory and quality analyses verify that MetaBench-Harness enables multi-dimensional evolution, steadily improving evolution reasonableness, benchmark competency, and evaluator robustness across successive rounds. Furthermore, case studies reveal its effective utilization of diverse difficulty levers to reframe problems and elevate required capabilities. Ultimately, this work provides a solution to the pressing challenge of benchmark saturation.
Sources
- GPT-4 Technical Report
- Kimi K2.5: Visual Agentic Intelligence
- A Survey on Data Contamination for Large Language Models
- DeepSeek-V3 Technical Report
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- TRACER: A Semantic-Aware Framework for Fine-Grained Contamination Detection in Code LLMs
- CodaRAG: Connecting the Dots with Associativity Inspired by Complementary Learning
- MemGPT: Towards LLMs as Operating Systems
- AutoEvoEval: An Automated Framework for Evolving Close-Ended LLM Evaluation Data
- Learning to Continually Learn via Meta-learning Agentic Memory Designs
- Recursive Language Models
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems