SWE-Spot: Building Small Repo-Experts with Repository-Centric Learning

arXiv:2601.21649 · cs.LG, cs.AI, cs.CL, cs.SE · Submitted 2026-01-29 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "SWE-Spot: Building Small Repo-Experts with Repository-Centric Learning".

Jane: The gist The deployment of coding agents in privacy-sensitive and resource-constrained environments drives the demand for capable open-weight Small Language Models (SLMs) that lack strong inference-time generalization to work with…

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at the paper titled "SWE-Spot: Building Small Repo-Experts with Repository-Centric Learning," and the title itself suggests they’re aiming to build models that are specialists in one specific repository.

Jane: That really points to an idea that these agents aren't just general coders anymore; they are becoming experts in a particular software environment.

Lu: They argue that this approach lets small language models, which usually have limited size, acquire the necessary knowledge during training through deep parametric learning instead of just trying to recover it later when they're actually being used.

Meng: I see the name "Repo-Experts," and that implies a level of depth we don't always see with smaller models when they’re just being broadly exposed to a lot of different code.

Lalam: The core idea is that for these models, getting deep mastery over one specific repository is more effective than just having shallow exposure across many unrelated repositories.

The paper's summary: Tom: So what’s the actual mechanism they propose? They suggest a new way of training where we prioritize vertical depth in a single codebase over horizontal breadth across many tasks.

Jane: Instead of just feeding the model lots of different code examples randomly, they design this four-unit RepositoryCentric Experience to turn those static codebases into signals that actively teach the model during its training.

Lu: They call this four-unit experience the RCX framework, which includes things like software design, modeling contextual implementation, learning from evolutionary replay of historical changes, and aligning semantics with runtime behavior.

Meng: That sounds like a lot of structured data they’re feeding into the model during training to ensure it learns the structure instead of just memorizing surface-level code snippets.

Lalam: They are essentially encoding the repository’s operational rules as a reusable prior, which is how they think this deep parametric knowledge acquisition actually works for these small language models.

The paper's improvements: Tom: Now let's look at what they claim is better about this Repository-Centric Learning approach when compared to the Task-Centric Learning that they’re contrasting it with.

Jane: They show that SWE-Spot, even though it’s a smaller model, it performs better than open-weight models that are up to eight times larger and even matches or exceeds efficiency-comparable commercial models like GPT-five-nano and GPT-four point one-mini <ref:2601.21649#pg3>.

Lu: That performance difference comes with a shift in how the agent behaves because by internalizing the specific structure and dynamics of the repository, SWE-SPOT achieves better training sample efficiency and substantially lower inference time cost.

Meng: Lower inference time cost is important for deployment; if it runs faster or cheaper once you put it into production, that’s a tangible benefit we care about when scaling up.

Lalam: They point out that this performance gap shows deep parametric knowledge acquisition beyond just simple context memorization or superficial pattern matching because they show "Oracle Localization" becomes redundant for the model under this approach.

Conclusion: Tom: So to wrap up, Repository-Centric Learning moves us toward models that have a distinct dimension of repository mastery that complements their general coding capability.

Jane: It suggests that for small language models, deep mastery of one specific target environment is more valuable than broad but shallow exposure across countless unrelated codebases.

Lu: The ablation studies confirm that the four units of RCX are synergistic; removing any single unit degrades performance across multiple downstream tasks, so it’s about holistic exposure rather than isolated skills.

Meng: I think prioritizing performance within a specific working scope over trying to maintain average performance across the entire world of codebases is a very practical conclusion for deployment.

Lalam: The paper "SWE-Spot: Building Small Repo-Experts with Repository-Centric Learning" introduces this new dimension, showing that repo expertise is not just about memorizing syntax but about having robust representations that can affect each other.

Jinjun Peng, Magnus Saebo, Tianjun Zhong, Yi-Jie Cheng, Junfeng Yang, Baishakhi Ray

Columbia University · University of California, Los Angeles

cs.LG, cs.AI, cs.CL, cs.SE

Submitted: 2026-01-29

Updated: 2026-10-13

Comments: substantially revised

Code: https://github.com/anthropics/claude-code

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 88/100

The gist: The gist The deployment of coding agents in privacy-sensitive and resource-constrained environments drives the demand for capable open-weight Small Language Models (SLMs) that lack strong

Key concepts

Task-Centric Learning (TCL): This traditional method scales exposure by training models across many different, unrelated repositories and tasks. The paper found this fails for small models because it leads to surface-level pattern matching instead of deep understanding of a specific codebase's design and semantics.
Repository-Centric Learning (RCL): RCL is a new approach that prioritizes vertical depth—intense, repeated interaction within one single repository. It aims to teach the SLM the operational rules and design constraints of that specific software environment through focused, multi-step learning.
Repository-Centric Experience (RCX): This is a four-unit training framework used in RCL to turn static code into interactive signals. These units include analyzing design intent, modeling cross-file dependencies, learning from historical changes, and designing tests to check runtime behavior against specifications.

Terminology

Repository-Centric Learning — 2× in this paper
Generalization — 1× in this paper · explained in 13 episodes
Sample Efficiency — 1× in this paper · explained in 7 episodes

Summary

The gist The deployment of coding agents in privacy-sensitive and resource-constrained environments drives the demand for capable open-weight Small Language Models (SLMs) that lack strong inference-time generalization to work with complicated, unfamiliar codebases.

The Paradigm Shift

Repository-Centric Learning (RCL) is proposed as a paradigm shift that prioritizes vertical repository depth over horizontal task breadth, suggesting SLMs must internalize the “physics” of a target software environment through parametric knowledge acquisition rather than only attempting to recover it via costly inference-time search <ref:2601.21649#pg4>. This approach is motivated by the failure of Task-Centric Learning (TCL), which scales exposure across disparate repositories, to address the fundamental capability gap in SLMs, as they tend to overfit surface-level patterns rather than internalizing repository-specific design and semantics <ref:2601.21649#pg3>. RCL trains SLMs by grounding learning in repeated, multi-task interactions within a single codebase to encode the repository’s operational semantics as a reusable prior <ref:2601.21649#pg4>.

The Repository-Centric Experience (RCX)

To operationalize RCL, the paper designs a four-unit RepositoryCentric Experience (RCX) to transform static codebases into interactive learning signals <ref:2601.21649#pg4>. These units are:

Software design, where the agent infers architectural intent and module responsibilities through active code analysis <ref:2601.21649#pg4>.
Contextual Implementation, where it models cross-file dependencies, internal APIs, and implementation logic via context-aware completion <ref:2601.21649#pg4>.
Evolutionary Replay, where it learns historical changes to understand design trade-offs and constraints shaping the codebase over time <ref:2601.21649#pg4>.
Semantic-Runtime Alignment, where it designs tests to capture nuanced discrepancies between specifications and runtime behaviors <ref:2601.21649#pg4>.

Evaluation and Results

The model SWE-SPOT-4B, trained with RCL, consistently breaks the scaling trends established by TCL across multiple software engineering tasks <ref:2601.21649#pg6>. Despite its compact size, SWE-SPOT-4B outperforms open-weight TCL-trained models up to 8× larger and matches or exceeds efficiency-comparable commercial models such as GPT-5-nano and GPT-4.1-mini <ref:2601.21649#pg6>. The results show that SWE-SPOT4B achieves an average execution-based score of 17.12%, outperforming open-weight models up to 8× larger with a clear margin, including Qwen3-Coder-30B-A3B-Instruct, Gemma-3-27B-it, and CWM-32B <ref:2601.21649#pg7>.

Advantages of RCL over TCL

RCL demonstrates superior performance under controlled data budgets compared to TCL, showing that for SLMs with limited capacity deep mastery of the specific target environment is more valuable than broad but shallow exposure to diverse, unrelated codebases <ref:2601.21649#pg8>. Furthermore, RCL induces a qualitative shift in agent behavior by achieving higher training sample efficiency and substantially lower inference-time cost <ref:2601.21649#pg4>. The RCL model reduces the average number of turns by 23.0% and total token consumption by 11.1% compared to the TCL model, even though the RCL training trajectories are on average 12.6% longer <ref:2601.21649#pg8>.

Necessity of Deep Parametric Knowledge

The analysis suggests that RCL results in better performance because it involves deep parametric knowledge injection beyond simple memorization, as demonstrated by the lack of gain from Oracle Localization for the TCL model <ref:2601.21649#pg9>. This confirms that the role of repo-agnostic TCL on SLMs is likely fitting shared task-specific formats, achievable by low-capacity learning, whereas RCL involves substantial experience and knowledge storage regarding the repository’s specifications, requiring robust representations from full-weight updates <ref:2601.21649#pg9>. The full-unit RCL yields the best performance across all evaluated tasks, suggesting that for SLMs, repository expertise consists of synergistic representations that can affect each other <ref:2601.21649#pg9>.

Synergistic Experience Units

A comprehensive leave-one-out ablation study confirms that removing any single experience unit degrades performance across multiple downstream tasks, suggesting that for SLMs, repository expertise is possibly not a collection of isolated skills but consists of synergistic representations <ref:2601.21649#pg9>. The joint RCL model achieves synergy on django and sympy while suffering from a significant negative transfer on matplotlib, which validates the necessity of prioritizing performance within a specific and limited working scope over maintaining average performance across all codebases in the entire world <ref:2601.21649#pg10>.

Future Directions

Extensions to the RCL instantiation include implementing more advanced training techniques such as on-policy distillation or RLVR, which allows the agent to directly learn from self-produced RCX <ref:2601.21649#pg10>. The work leaves open questions regarding when and how repository characteristics lead to synergy or interference, and the applicability of RCL to larger models <ref:2601.21649#pg10>.

Conclusion

In this work, Repository-Centric Learning is introduced as a paradigm shift designed to bridge the inference-time capability gap in Small Language Models by parametrically internalizing the dynamics of target codebases <ref:2601.21649#pg11>. Using this approach, SWE-SPOT4B demonstrates that repository mastery is a distinct, necessary new dimension that complements general coding capability <ref:2601.21649#pg11>.

Improvements for AI systems

A new paradigm called Repository-Centric Learning (RCL) internalizes repository knowledge during training, transforming static codebases into interactive learning signals. This allows Small Language Models (SLMs) to internalize the 'physics' of a target software environment through parametric knowledge acquisition, rather than only attempting to recover it via costly inference-time search.
The improved system SWE-SPOT-4B can achieve repository mastery and demonstrate strong and more stable generalization across multiple software engineering tasks within a single codebase, unlike Task-Centric Learning (TCL) which tend[s] to overfit surface-level patterns.
SWE-SPOT can surpass or match efficiency-focused commercial models like GPT-5-nano and GPT-4.1-mini while yielding higher training sample efficiency and lower inference costs because it reduces the average number of turns by 23.0% and total token consumption by 11.1% compared to TCL models, even with longer training trajectories.
The system gains deep parametric knowledge acquisition beyond context memorization or superficial pattern matching, as demonstrated when Oracle Localization renders Oracle Localization redundant for the RCL model, suggesting it learns how to reason about the task and modify code in compliance with the target repository’s logic.
The RCX framework provides four specific learning experiences: (i) Software Design, (ii) Contextual Implementation, (iii) Evolutionary Replay, and (iv) Semantic-Runtime Alignment. This holistic exposure enables robust, multi-task performance within a single codebase by teaching the agent how to reason about why code is structured as it is.

Abstract

The deployment of coding agents in privacy-sensitive and resource-constrained environments drives the demand for capable open-weight Small Language Models (SLMs). However, they suffer from a fundamental capability gap: unlike frontier large models, they lack the inference-time strong generalization to handle complicated, unfamiliar codebases. We identify that the prevailing Task-Centric Learning (TCL) approach, which scales exposure across disparate repositories, fails to address this limitation. In response, we propose Repository-Centric Learning (RCL), a shift that prioritizes vertical, single-repository depth over horizontal, single-task breadth for SLM training. To accomplish this depth, we design a four-unit Repository-Centric Experience, transforming static codebases into interactive learning signals, to train SWE-Spot-4B, a family of highly compact models built as repo-specialized experts. Under controlled comparisons with the same teacher and data budget, RCL consistently outperforms TCL across tasks, teachers, model scales, and programming languages. On seven target repositories, SWE-Spot-4B experts match or exceed existing open-weight models up to 8 times larger (e.g., CWM by Meta, Qwen3-Coder-30B) and surpass or match efficiency-focused commercial models (e.g., GPT-4.1-mini, GPT-5-nano) across multiple SWE tasks. Further analysis reveals that RCL yields higher training and inference efficiency, deep knowledge acquisition, and can scale to repository-cluster experts. Our work emphasizes that for building efficient intelligence, repository mastery is a distinct, necessary, and practical dimension that complements general coding capability.