A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents

SWE-RPG is a repository-level benchmark that combines executable patch evaluation with validated ground-truth references (GTs) for requirement clarification and implementation planning. It comprises 163 tasks from 31 Python and Java repositories, including 113 bug fixes and 50 feature additions. The benchmark evaluates coding agents across three stages: requirement clarification, implementation planning, and code generation, using a stage-aware diagnostic protocol.

The paper states: SWE-RPG comprises 163 tasks from 31 Python and Java repositories, including 113 bug fixes and 50 feature additions. It also notes that SWE-RPG combines implicit requirements, implementation-sufficient planning, and stage-aware evaluation.

The benchmark construction involved a rigorous filtering process: successive checks of task relevance, environment reproducibility, test stability, and GT validity reduce more than 2,000 real-world PR–issue candidates to 163 quality-controlled instances. The dataset includes substantial repository-scale complexity: codebases contain 272.51K lines of code (LOC) on average, with the largest reaching 1.59M LOC; and each task contains 2,248.56 functional test cases on average.

For requirement clarification, the benchmark provides gold clarification references across six practitioner-informed dimensions including C1: Functional Intent, C2: Business Semantics, C3: Technical Context, C4: Interface and Protocol Specifications, C5: Code Structure and Naming Conventions, and C6: Detailed Data-Structure Semantics. On average, Clarification GT recovers 4.53 implicit and implementation-critical requirement points.

For planning, each task provides implementation-sufficient gold plan steps with an average of 2.06 actionable implementation steps with 11.80 implementation constraints. The planning GT is validated for plan-to-code reproducibility: an accepted plan must provide sufficient guidance for a coding agent to produce a functionally equivalent implementation.

The evaluation results show that the evaluated popular coding agents still struggle to implement user requests in existing repositories, achieving an average resolved rate of only 31.5% on SWE-RPG. The paper reports: At the configuration level, OpenCode with MoonshotAI-Kimi-K3 achieves the highest resolved rate (49.7%). Thus, even the best configuration leaves roughly half of the tasks unresolved.

Failure diagnosis reveals that implicit requirement recovery as the main bottleneck, accounting for 24.5%–46.0% of agent runs. The paper states: Requirement failure is the largest component for most configurations (24.5–46.0%), followed by code-generation (7.4–37.4%) and planning (5.5–17.8%) failures. Additionally, 46.7% of agent runs failing primarily during requirement clarification or implementation planning.

The stage-aware analysis shows distinct profiles: "Similar resolve rates mask different bottleneck profiles. Codex– and Claude Code–DeepSeek-V4-Pro achieve similar resolve rates (40.5% versus 39.3%) but markedly different profiles: their requirement-failure rates are 26.4% and 40.5%, whereas their implementation/verification-failure rates are 23.9% and 7.4%."

For clarification coverage, the paper reports: "All three agents cover intent and scope most reliably, but weaken on the implementation-facing categories. Claude Code’s lowest clarification score is code structure (54.2%); Codex falls to 42.0% on structure and 41.9% on data semantics."

For planning coverage, the paper states: "For every agent, planning coverage decreases from target location to implementation approach and then to constraints and validation: 79.7% to 64.5% to 41.6% for Claude Code, 65.6% to 49.0% to 37.1% for Codex, and 43.8% to 31.1% to 24.8% for OpenCode."

The paper concludes: "SWE-RPG combines 163 real PR-derived tasks, executable patch oracles, and validated Clarification and Plan GTs. Beyond measuring final patch correctness, it supports full-trajectory, GT-aligned retrospective diagnosis of requirement, planning, implementation, and artifact gaps. These diagnostic profiles make otherwise identical unresolved outcomes more interpretable and suggest stage-specific hypotheses for future improvement."

Based on SWE-RPG, I can improve AI systems in the following specific ways:

1. Proactive Requirement Clarification Agent

• The improved system will automatically detect implicit, implementation-critical requirements before coding. It will generate clarification questions across the six dimensions (functional intent, business semantics, technical context, interface/protocol specs, code structure/naming, data-structure semantics). It will prioritize questions that recover the 4.53 missing requirement points on average, targeting the 24.5–46.0% failure rate caused by requirement gaps.

2. Implementation-Sufficient Plan Generator

• The system will produce plans with explicit, actionable steps (average 2.06 steps) and 11.80 constraints per task. It will ensure each plan is validated for plan-to-code reproducibility, meaning a downstream coding agent can execute it without additional inference. It will focus on covering target location (79.7% baseline), implementation approach (64.5%), and constraints/validation (41.6%)—with the goal of raising the weakest category (constraints/validation) to at least 70% coverage.

3. Stage-Aware Failure Diagnosis and Recovery

• The system will run a post-hoc diagnostic on any unresolved task, classifying failures into requirement, planning, code-generation, or verification/implementation gaps. It will then automatically retry with a stage-specific strategy: e.g., if requirement failure is detected (46.7% of runs), it will re-ask clarification questions; if planning failure, it will regenerate a more constrained plan; if code-generation failure, it will refine the patch against the executable test oracle.

4. Adaptive Configuration Selection

• The system will select the best agent-model configuration based on the task's bottleneck profile. For example, if a task is requirement-heavy, it will prefer a configuration with lower requirement-failure rate (e.g., Codex at 26.4% vs. Claude Code at 40.5%); if implementation-heavy, it will prefer one with lower verification-failure (e.g., Claude Code at 7.4%). This will push resolved rates beyond the current best of 49.7%.

5. Coverage-Aware Planning and Clarification Training

• The system will be trained to maximize coverage on implementation-facing categories (code structure, data semantics, constraints, validation) rather than just intent/scope. It will use the GT references to fine-tune its question-generation and plan-generation to close the gap between intent coverage (high) and structure/data-semantics coverage (as low as 41.9–54.2%).

6. Executable Patch Oracle Integration

• The system will use the 2,248.56 functional test cases per task as a real-time feedback loop during code generation. It will iteratively run tests, diagnose failures, and refine the patch until it passes the executable oracle, reducing the 7.4–37.4% code-generation failure rate.

7. Repository-Scale Context Handling

• The system will be enhanced to operate on codebases averaging 272.51K LOC (up to 1.59M LOC) by improving its ability to locate relevant files, understand cross-module dependencies, and generate patches that are consistent with the entire repository—not just isolated snippets. This will address the scale-induced planning and implementation failures observed in the benchmark.

8. Feature-Addition Specialization

• The system will have a distinct mode for feature additions (50 of 163 tasks) that explicitly handles new functionality, API design, and integration with existing code, separate from bug-fix mode. This will improve performance on the 30.7% of tasks that are not bug fixes.

Large language model-powered coding agents are increasingly used to modify existing code repositories, for example, by adding features or fixing bugs. Yet existing repository-level benchmarks typically evaluate only whether the final patch passes tests. Satisfying a user request requires a long chain of interdependent reasoning and decisions: an agent must recover explicit and implicit requirements, formulate a repository-grounded implementation plan, and translate it into correct code. A pass/fail outcome cannot characterize how an unsuccessful trajectory diverges from the requirements and implementation process needed for a correct patch. To address this gap, we introduce SWE-RPG, a repository-level benchmark that combines executable patch evaluation with validated ground-truth references (GTs) for (1) Requirement Clarification and (2) Implementation Planning. These intermediate GTs support retrospective, GT-aligned diagnosis of complete coding-agent trajectories across clarification, planning, code generation, and artifact submission. SWE-RPG comprises 163 tasks from 31 Python and Java repositories, including 113 bug fixes and 50 feature additions. We evaluate 3 coding agents, including Claude Code, Codex, and OpenCode, with 6 large language model backends, including Claude-Sonnet-5 and GPT-5.6-Terra. Results show that the evaluated popular coding agents still struggle to implement user requests in existing repositories, achieving an average resolved rate of only 31.5% on SWE-RPG. Intermediate-GT diagnosis further identifies implicit requirement recovery as the main bottleneck, accounting for 24.5%--46.0% of agent runs. This result suggests implicit-requirement recovery as a key candidate direction for improving coding agents. The benchmark data and evaluation code are available at https://github.com/Xin-Zhou-smu/SWE-RPG-Bench.