Coding-Agent Benchmarks Should Match Their Users' Task Flows
cs.LG, cs.AI, cs.CL, cs.SE
Submitted: 2026-10-07
Updated: 2026-10-11
Code: https://github.com/peteromallet/dataclaw
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- SWE-chat: Coding Agent Interactions From Real Users in the Wild
- GOPO: Policy Optimization using Ranked Rewards
- Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents
- SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution
- SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions
- SWE-Touch: Benchmarking Coding Agents When Users Touch the Code
- Programming by Chat: A Large-Scale Behavioral Analysis of 11,579 Real-World AI-Assisted IDE Sessions
- HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?
- SWE-Together: Evaluating Coding Agents in Interactive User Sessions
- SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
- RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists
- AMIR-GRPO: Inducing Implicit Preference Signals into GRPO
- A Unified Pair-GRPO Family: From Implicit to Explicit Preference Constraints for Stable and General RL Alignment
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks