Trace2Eval: Contextualized Evaluation of Coding Agents from Real-World Traces
cs.SE
Submitted: 2026-11-04
Updated: 2026-11-04
Code: https://github.com/tatsu-lab/alpaca_eval
Terminology
Sources
- Program Synthesis with Large Language Models
- Evaluating Large Language Models Trained on Code
- TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks
- NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents
- CATArena: Evaluating Evolutionary Capabilities of Code Agents via Iterative Tournaments
- Measuring Coding Challenge Competence With APPS
- CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation
- DevEval: Evaluating Code Generation in Practical Software Projects
- Holistic Evaluation of Language Models
- G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
- Self-Refine: Iterative Refinement with Self-Feedback
- Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
- Proving Test Set Contamination in Black Box Language Models
- Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack
- SWE-Mirror: Scaling Issue-Resolving Datasets by Mirroring Issues Across Repositories
- Asuka-Bench: Benchmarking Code Agents on Underspecified User Intent and Multi-Round Refinement
- Large-Scale Terminal Agentic Trajectory Generation from Dockerized Environments
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters