SWE-Time: Benchmarking Temporal Repository Understanding
cs.SE
Submitted: 2026-11-09
Updated: 2026-11-09
License: http://creativecommons.org/licenses/by/4.0/
The gist: Repository-level code-understanding benchmarks have progressively expanded the spatial scope of reasoning, from local code to entire repositories…
Terminology
Abstract
Repository-level code-understanding benchmarks have progressively expanded the spatial scope of reasoning, from local code to entire repositories, while largely treating each repository as a static snapshot. However, how models reason about repository evolution over time remains largely unexplored. We introduce SWE-Time, a benchmark for temporal repository understanding that makes repository evolution itself the reasoning target. SWE-Time evaluates two complementary temporal directions: retrospective understanding of repository evolution through historical regression analysis and prospective understanding of intended changes from pre-change repository states. To enable realistic yet reliable evaluation in these settings, we introduce the RealWorld-to-Verifiable Construction Framework, which transforms real-world developer queries and historical evidence into verifiable evaluation tasks. We evaluate 15 coding agents on 1,661 cases from 65 GitHub repositories. Within Claude Code, Claude Opus 4.8 achieves the highest SWE-Time Avg. of 61.9, leaving substantial headroom. Controlled analyses highlight the importance of historical code evidence and developer intent, while consistent reasoning across repository states remains challenging. These results highlight repository evolution as a distinct and challenging dimension of code understanding.
Sources
- CoReQA: Uncovering Potentials of Language Models in Code Repository Question Answering
- Evaluating Large Language Models Trained on Code
- SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios
- AgentSZZ: Teaching the LLM Agent to Play Detective with Bug-Inducing Commits
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters