Harvesting SWE Benchmarks from Wild Vibe Coding Sessions
cs.SE, cs.AI
Submitted: 2026-11-02
Updated: 2026-11-02
Code: https://github.com/bytedance/SessionFarmer
Terminology
Sources
- SWE-chat: Coding Agent Interactions From Real Users in the Wild
- SWE-Exp: Experience-Driven Software Issue Resolution
- How can we assess human-agent interactions? Case studies in software agent design
- SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
- DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks
- REAP: Automatic Curation of Coding Agent Benchmarks from Interactive Production Usage
- Probe to Generate: Program Variant-Guided Test Augmentation for Repository-Level Repair Benchmarks
- RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessions
- Training Software Engineering Agents and Verifiers with SWE-Gym
- SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution
- Vibe coding: programming through conversation with artificial intelligence
- How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions
- Agent Workflow Memory
- Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory
- SWE-Together: Evaluating Coding Agents in Interactive User Sessions
- SWE-bench Goes Live!
- TOM-SWE: User Mental Modeling For Software Engineering Agents
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters