ExpertCrowd-SWE: Measuring Frontier Coding Agents on Expert-Authored Engineering Tasks in Unfamiliar Codebases
cs.AI, cs.SE
Submitted: 2026-11-09
Updated: 2026-11-09
Code: https://github.com/hud-evals/hudpython
Terminology
Sources
- Evaluating Large Language Models Trained on Code
- SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
- DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks
- APEX-SWE
- The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason
- Quantifying Variance in Evaluation Benchmarks
- Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
- Agentless: Demystifying LLM-based Software Engineering Agents
- SWE-smith: Scaling Data for Software Engineering Agents
- Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving
- SWE-bench Goes Live!
- Agent-as-a-Judge: Evaluate Agents with Agents
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents