Scrouting: Cost-Aware Routing of Coding Agents by Scouting the Repository First

arXiv:2608.04804 · cs.SE, cs.AI · Submitted 2026-08-05 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Scrouting: Cost-Aware Routing of Coding Agents by Scouting the Repository First".

Title and authors: Tom: So we're looking at this paper today called "Scrouting: Cost-Aware Routing of Coding Agents by Scouting the Repository First". It’s about how to make using those big frontier language models for software issues cheaper.

Jane: Right, so the core idea is that instead of just looking at the problem description and picking a model from there, you first have a small searcher explore the actual code repository itself.

Lu: Exactly. They introduce this engage-then-route architecture where this 7B searcher model explores the repository and produces some kind of structured handoff before we even think about fixing anything <ref:2608.04804#pg1,explores the repository and produces>.

Meng: Wait, so it's like having a preliminary investigation phase before you hire a specialist to fix the bug? I need to know if that preliminary work actually saves time.

Tom: It does, Meng. The paper shows this searcher, SuperScout-7B, produces a handoff with reproduction claims that have been sandbox verified and false claims are stripped out before they get to the fixer models <ref:2608.04804#pg1>.

Jane: That’s important because it means the downstream models aren't wasting time on misleading or impossible tasks generated from a bad initial search.

Lu: And then this structured handoff, along with the task text, feeds into a résumé-based router that chooses one of four frontier fixers to actually do the work.

Tom: The results are pretty compelling: on SWE-bench Pro’s full Python slice with their official budget tier, SuperScout matches the best single model's solve rate at about a fifth of the total cost per solve.

Meng: A fifth? That sounds like a big saving, but I need to know if that accuracy gain is worth it against the extra compute for this scouting phase.

Jane: The paper suggests that this cost matching happens because the handoff itself seems to redistribute solving ability among different fixers, lifting some cheaper ones while only slightly hurting the strongest one.

Lu: That redistribution idea is key because it means the routing isn't just picking the best guess based on text alone; it’s picking based on where the knowledge already exists in that repository.

Title and authors: Tom: And they show that this routing decision is guided by two things: you have the task text, and you have those hidden states from SuperScout-7B, which are about three thousand five hundred eighty-four dimensions.

Jane: So it’s not just a simple lookup anymore; it’s using the searcher’s internal understanding of the repo to make a smarter routing call.

Meng: Using internal states for routing sounds interesting from an engineering standpoint, but how much extra compute is that searcher adding on top of what we already pay for?

Tom: The paper says the searcher’s compute adds less than half a cent of GPU time per task, which is pretty low given the benefit it provides to the overall cost.

Lu: And they built this searcher specifically by training it only on search-phase demonstrations from other agents, using nineteen thousand nine hundred eleven examples of full search trajectories as the final target.

Jane: So they focused the training signal only on that specific exploration step, which is a smart way to get the model focused on finding those good starting points.

Tom: And there’s this verification gate mentioned where they strip false reproduction claims before any fixer sees them, which helps neutralize overclaiming by making spontaneous handoffs much more likely to be genuine.

Meng: That sounds like a good safety mechanism for production; I worry about models just confidently claiming things that aren't true.

Lu: And regarding adding new fixers, they say you don’t need to retrain anything new because the résumé is built from twenty-five to fifty public outcomes and read at inference time.

Jane: So the system is built for a market where models are constantly changing, which is good because we don't have to keep retraining every time a new capable fixer comes along.

Tom: Exactly. The evaluation on SWE-bench Pro showed that the routed system, even without the router at all, sits three point six percentage points above the blind mixing line on two hundred sixty-six tasks <ref:2608.04804#pg1>.

Meng: That's a solid number for a baseline comparison; it shows that just having that verified handoff is better than random traffic splitting.

Title and authors: Lu: And to push on the cost aspect, they found that the calibration study showed that only fifty of those two hundred sixty-six handoffs were actually genuine, while one hundred seventy-four were demonstrably false, confirming the guard’s effectiveness.

Jane: It’s interesting how they framed this cost asymmetry; it suggests that engaging with the repository first pays for itself through a verified handoff rather than just by making a better routing decision.

Tom: And one thing they pointed out is that their feature design for the router shows that the hidden state contribution is mechanical, meaning it shifts when you trust the cheaper model more than it boosts accuracy per se.

Meng: So it’s less about making the cheap model smarter and more about trusting its output when you're worried about cost. That makes sense for deployment at scale.

Lu: And looking ahead, they suggest that natural next steps involve testing this across wider benchmark coverage in different languages and problem domains because they haven't tested cross-benchmark generality yet.

Jane: So the big picture here is that we’ve moved toward a system where exploration pays off through verification, but we still need to see if this works as well outside of the specific Python slice they used.

Tom: To wrap things up, it’s clear that the verified handoff, not just the routing decision itself, carries the result in this Scrouting paper.

Jane: So that’s our take on "Scrouting: Cost-Aware Routing of Coding Agents by Scouting the Repository First". It shows a way to use small searchers to prune expensive calls before dispatching tasks to frontier models.

Lu: It points toward an architecture where the exploration phase is valuable because it generates a trusted signal that informs cost-aware routing decisions.

Meng: For us in production, it suggests we can integrate this scouting step as a cheap pre-filter before hitting the most expensive model calls.

Tom: We’re leaving it there for now, but if you want to see how they handle different types of data scarcity or maybe how this scales beyond SWE-bench Pro, we'll be back next time.

The paper's summary: Tom: So, we’re talking about Scrouting today—this system that tries to make using these massive frontier models for software problems way cheaper by scouting the repository first before dispatching a task.

Jane: That’s right. The big idea is that instead of just looking at the problem description and hoping you pick the right model, you have this small searcher explore the actual code repository itself to build some kind of structured handoff.

Lu: And that handoff is verified in a sandbox—meaning they check if it actually works by checking out the files—and then it’s fed into a résumé router that picks one of four bigger fixers to do the actual work.

Meng: So, what does this actually do? Does it just make the AI smarter, or does it actually save money when you’re running these things in production?

Tom: It saves money because the paper shows that on a big benchmark called SWE-bench Pro, this routed system matches the best single frontier model's solve rate at about a fifth of its total cost per solve.

Jane: That’s a significant cost saving, but it seems like they aren't just relying on the routing decision itself making it faster.

Lu: They found that what carries the result is actually that verified handoff—the structured information from the repository exploration—not necessarily the router's final choice.

Tom: And to make routing smarter, they feed that searcher’s internal hidden states into the résumé router alongside the original task text.

Jane: So this means you’re using what the searcher *learned* about where things are in the code to help decide which fixer is best.

Meng: I’m curious about how much compute this extra searcher adds on top of what we already pay for when we run these things at scale.

Tom: They say the searcher’s compute adds less than half a cent of GPU time per task, which is pretty low given the benefit it provides to the overall cost efficiency.

Lu: And they built this searcher by training it only on demonstrations from other agents exploring repositories, using about twenty thousand examples of full search trajectories as its training data.

Jane: So they focused the learning signal specifically on that exploration step, which is a smart way to get the model focused on finding those good starting points in the code.

Tom: Plus, they have this guard where false reproduction claims are stripped out before any fixer even sees them, which helps neutralize overclaiming by making genuine handoffs much more likely.

Meng: That sounds like a solid safety mechanism for production; I worry about models just confidently claiming things that aren't true when they’re trying to fix code.

Lu: And another thing they mentioned is that you don't need to retrain anything new if you add a new fixer because its résumé is built from twenty five to fifty public outcomes and read at inference time.

Jane: That means this system is built for a market where models are constantly changing, which is good because we don't have to keep retraining every time a new capable fixer comes along.

Tom: They also pointed out in their calibration study that only about fifty of those two hundred sixty six handoffs were actually genuine, while one hundred seventy four were demonstrably false, confirming how effective that guard is.

Lu: The core idea here is that engaging with the repository first pays off through a verified handoff rather than just by making a better routing decision.

Jane: It suggests we’ve moved toward a system where exploration pays for itself because it generates a trusted signal that informs cost-aware routing decisions.

Tom: And they showed that the hidden state contribution to the router is mechanical, meaning it shifts when you trust the cheaper model more than it boosts accuracy per se.

Meng: So it’s less about making the cheap model smarter and more about trusting its output when you're worried about cost, which makes sense for deployment at scale.

Lu: Looking ahead, they suggest that natural next steps involve testing this across wider benchmark coverage in different languages and problem domains because they haven't tested cross-benchmark generality yet.

Jane: So the big picture here is that we’ve moved toward a system where exploration pays off through verification, but we still need to see if this works as well outside of the specific Python slice they used.

The paper's improvements: Tom: So we’re looking at how they suggest making this scouting system even better for the future than what's on page two of this paper.

Jane: They point out that the current searcher training is only focused on search phase demonstrations, so they want to see if you can expand that training signal.

Lu: Yeah, they are looking at ways to make that initial exploration phase more robust, maybe by incorporating more diverse data from different agents exploring different codebases.

Meng: If you get better data for the searcher, does that mean the handoff quality gets better? Is it just a bigger picture of where the code is?

Tom: It should definitely improve the handoff quality because if the searcher can explore more effectively, it’ll produce a more structured and accurate handoff.

Jane: And they suggest another way to tune the system: using those hidden states from the searcher as a feature space in that résumé router.

Lu: That makes sense because they found that those hidden states are important for cost routing on calibration labels, which is where it actually helps the system balance between accuracy and expense.

Meng: So they’re saying we can use internal knowledge to decide which fixer to trust when the cost matters most? I need to know if that’s something we can deploy reliably.

Tom: They suggest that this routing mechanism isn't just about picking a better model; it's about shifting trust toward cheaper models when you’re worried about the cost.

Jane: And they also make a point about how new fixers get added to the system without needing any retraining at all.

Lu: That’s because they design the résumé itself from twenty five to fifty public outcomes, and you just read that summary at inference time, so adding a new fixer is super fast.

Meng: So that's good for us in a production environment; we don't have to spend weeks retraining when a new tool comes along.

Tom: They also talked about the importance of the verification guard again, specifically how it neutralizes overclaiming by stripping false claims before any fixer sees them.

Jane: It’s like a quality control checkpoint that makes sure the initial exploration phase actually delivers reliable evidence instead of just making up stuff to sound confident.

Lu: They also hinted that we need to test this system across wider benchmark coverage, not just this single Python slice they used, because they haven't checked cross-benchmark generality yet.

Meng: That’s a fair point; if it only works on one type of code problem, it won't help us on the messy real-world stuff we actually run.

Tom: The main thing they are pointing toward is that this architecture lets you use a small model to prune expensive calls before dispatching tasks to the most powerful models.

Jane: So, the paper’s core improvement is moving away from a simple text-based router and making the routing decision dependent on a verified exploration of the codebase.

Conclusion: Tom: So to wrap up, Scrouting is essentially showing how you can use a small searcher model to explore code before dispatching tasks to big frontier models, and this scouting step actually pays off in terms of cost savings.

Jane: It really means that for anyone using these large language models for software work, you don't have to just rely on the problem description; you can use repository exploration as a way to get a more reliable signal before you spend money on a complex fix.

Lu: The implication is that we move from routing purely based on the prompt text to routing based on where the actual knowledge exists in the code, which should lead to more efficient resource allocation across different models.

Meng: For us at startup, it means if we can integrate this scouting step cheaply, we can significantly lower our operational costs when running these agents at scale.

Lalam: I’m confident in this direction because the evidence shows that the verified handoff carries the result, which is a very strong signal for reliable deployment.

Tom: We saw that they match the best single model's solve rate at about a fifth of its cost per solve on SWE-bench Pro, which is a solid number against what we can achieve with just text routing.

Jane: And even when you look closely at the numbers, like how many handoffs were actually genuine versus false, it confirms that this guard mechanism is working as intended.

Lu: They also showed that adding new fixers doesn't require retraining because their résumé is built from public outcomes, which is a huge practical benefit for keeping up with model developments.

Meng: It’s good to know that the system doesn't become totally obsolete every time a new powerful AI model comes out, as long as we can update those résumés easily.

Tom: So the final conclusion on Scrouting is that the verified handoff, not just the routing decision itself, is what carries the result in this paper.

Jane: It’s a smart way to build a system where exploration pays off through verification rather than just by making a better routing call based on text alone.

Lu: I think we still need to see how this holds up when we apply it across different types of code problems and maybe even different languages, because that cross-benchmark generality is still untested.

Meng: Yeah, I’m worried about running this on real traffic; the benchmark they used is just a slice of Python tasks, so we need to see how it handles the messiness of actual production environments.

Lalam: I don’t see any immediate flaws in their approach based on what we've seen in the literature, and I’m sure this method holds up well.

Tom: We’re leaving it there for now, but if you want to see how they handle different types of data scarcity or maybe how this scales beyond SWE-bench Pro, we'll be back next time.

Ishaan Bhola, Adithyan Krishnan, Mukunda NS

SuperAGI Research

cs.SE, cs.AI

Submitted: 2026-08-05

Updated: 2026-11-18

Code: https://github.com/TransformerOptimus/superscout

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 85/100

The gist: Frontier language models can resolve repository-level software issues, but each attempt is expensive, and existing routers select a model from the issue text alone<ref:2608.04804#pg2> SuperScout…

Key concepts

Scrouting
This is a new two-step process where a small searcher model first explores a software repository to find relevant files and diagnostic notes. Crucially, it performs sandbox verification on these findings to strip out any false claims before they are passed on to the next stage, ensuring downstream models receive reliable information.
Résumé Based Router
This component decides which of four large frontier fixers should handle a task. It makes its decision by looking at two main inputs: the original task text and the hidden states generated by the searcher model during its repository exploration phase.
Searcher's Hidden States
These are internal representations produced by the small 7B searcher model while it is exploring a repository. The paper shows these states are not just random data; they provide valuable information that helps the router make better cost-aware decisions, particularly when distinguishing between different fixer capabilities.
Cost-Aware Routing
This system aims to save money by matching the best single model's performance at a fraction of its cost. The routing decision is informed by features like the searcher's hidden states and task text, allowing the system to direct tasks toward cheaper models when appropriate, based on how well they perform on calibration data.

Terminology

Summary

Frontier language models can resolve repository-level software issues, but each attempt is expensive, and existing routers select a model from the issue text alone<ref:2608. p. 0

Where the paper says it

Scrouting: Cost-Aware Routing of Coding Agents by Scouting the Repository First Ishaan Bhola SuperAGI Research Adithyan Krishnan SuperAGI Research Mukunda NS SuperAGI Research Abstract—Frontier language models can resolve repository-level software issues, but each attempt is expensive, and existing routers select a model from the issue text alone. We present SuperScout, which routes after scouting the repository: a 7B searcher, SuperScout-7B, first explores the repository and produces a structured handoff whose reproduction claims are sandboxverified, with false claims stripped before delivery.

04804#pg2> SuperScout routes after scouting the repository: a 7B searcher, SuperScout-7B, first explores the repository and produces a structured handoff whose reproduction claims are sandboxverified, with false claims stripped before delivery<ref:2608. p. 0
Where the paper says it

Scrouting: Cost-Aware Routing of Coding Agents by Scouting the Repository First Ishaan Bhola SuperAGI Research Adithyan Krishnan SuperAGI Research Mukunda NS SuperAGI Research Abstract—Frontier language models can resolve repository-level software issues, but each attempt is expensive, and existing routers select a model from the issue text alone. We present SuperScout, which routes after scouting the repository: a 7B searcher, SuperScout-7B, first explores the repository and produces a structured handoff whose reproduction claims are sandboxverified, with false claims stripped before delivery.

04804#pg2> A résumé based router then dispatches the task to one of four frontier fixers using the searcher’s hidden states and the task text<ref:2608. p. 0
Where the paper says it

We present SuperScout, which routes after scouting the repository: a 7B searcher, SuperScout-7B, first explores the repository and produces a structured handoff whose reproduction claims are sandboxverified, with false claims stripped before delivery. The searcher’s hidden states, together with the task text, then feed a résumébased router that dispatches the task to one of four frontier fixers.

04804#pg2> On the full Python slice of SWE-bench Pro (266 tasks) under the benchmark’s official capped budget tier, SuperScout matches the best single model’s solve rate at about a fifth of the total cost per solve<ref:2608. p. 0
Where the paper says it

Adding a new fixer requires no retraining. On the full Python slice of SWE-bench Pro (266 tasks) under the benchmark’s official capped budget tier, SuperScout matches the best single model’s solve rate (159 of 266 for SuperScout, 158 for the best model) at about a fifth of the total cost per solve, and the reported configuration sits above the random traffic-splitting baseline.

04804#pg2> The searcher’s hidden states, together with the task text, then feed a résumé based router that dispatches the task to one of four frontier fixers<ref:2608. p. 0
Where the paper says it

We present SuperScout, which routes after scouting the repository: a 7B searcher, SuperScout-7B, first explores the repository and produces a structured handoff whose reproduction claims are sandboxverified, with false claims stripped before delivery. The searcher’s hidden states, together with the task text, then feed a résumébased router that dispatches the task to one of four frontier fixers.

04804#pg2>

Pruning redundant reasoning tokens

The paper introduces SuperScout, an engage-then-route architecture where a small searcher model explores the repository and produces a structured handoff, which is then consumed by a résumé router to select one of four frontier fixers<ref:2608.04804#pg2> This system is designed to route after engaging with the problem, rather than from the task text alone<ref:2608. p. 2

Where the paper says it

Rather than seeking accuracy gains by combining models, we aim to match the strongest available model’s solve rate at a materially lower total cost per solve. The thesis is that routing should happen after engaging with the problem, not from the task text alone.

04804#pg2> The core mechanism involves a 7B searcher, SuperScout-7B, which explores the repository and produces a structured handoff containing implicated files and diagnostic notes<ref:2608. p. 0
Where the paper says it

SuperScout routes after scouting, a pattern we call scrouting. A 7B searcher, SuperScout-7B, first explores the repository and produces a structured handoff: a list of implicated files, diagnostic notes, and a candidate reproduction test.

04804#pg2> This handoff undergoes a sandbox verification step where reproduction claims are stripped if they do not check out before being delivered to a downstream model<ref:2608. p. 0
Where the paper says it

A 7B searcher, SuperScout-7B, first explores the repository and produces a structured handoff: a list of implicated files, diagnostic notes, and a candidate reproduction test. These claims pass through a sandbox verification step, and claims that do not check out are stripped before a downstream model ever sees them.

04804#pg2> The résumé router selects among four frontier fixers using the task text together with the searcher’s own hidden states<ref:2608. p. 0
Where the paper says it

These claims pass through a sandbox verification step, and claims that do not check out are stripped before a downstream model ever sees them. A résumé router then selects among four frontier fixers using the task text together with the searcher’s own hidden states; adding a new fixer requires no retraining.

04804#pg2>

How the training signal is built

SuperScout-7B is trained exclusively for the search phase on search-phase demonstrations sliced from openly licensed agent trajectories produced by other systems<ref:2608.04804#pg2> The supervised training corpus consists of 19,911 examples, where each example is a full search trajectory with a synthesized handoff-emission turn as the final supervised target<ref:2608. p. 4

Where the paper says it

TABLE III: Composition of SuperScout-7B’s supervised search corpus. Each example is a full search trajectory with a synthesized handoff-emission turn as the final supervised step; loss is on assistant turns only.

04804#pg2> The loss function is applied only to assistant turns during this training process<ref:2608. p. 4
Where the paper says it

TABLE III: Composition of SuperScout-7B’s supervised search corpus. Each example is a full search trajectory with a synthesized handoff-emission turn as the final supervised step; loss is on assistant turns only.

04804#pg2> The model was fine-tuned using LoRA on Qwen2. p. 4
Where the paper says it

We fine-tune Qwen2. 5-Coder-7B [23] with LoRA [31] on a single GPU.

5-Coder-7B with a specific configuration involving linear projections and 32k token blocks<ref:2608. p. 4
Where the paper says it

Component Value Corpus Examples trained on 19,905 Built / frozen set 19,911 Held-back shelf (untrained) 9,478 Tokens per epoch 382. 4M Packed 32k blocks (fill) 12,723 (91.

04804#pg2> Decoding matters more than expected, as greedy decoding finds the right files at a rate of 0. p. 4
Where the paper says it

2 Decoding Matters More Than Expected The most consequential training-era finding was not about the training itself but about inference. On a 450-task held-out exam, greedy decoding finds the right files at a rate of 0.

1104, while sampling at temperature 0. p. 4
Where the paper says it

96× (Figure 4); the quality cost visible on the exam does not appear there, so the dial set is a mechanism check, not a replication. Greedy decoding, it turns out, makes the searcher reluctant to declare a result; sampling at moderate temperature recovers ability the model already has.

9 reaches 0. p. 4
Where the paper says it

A single sampled draw at temperature 0. 9 reaches 0.

3058, a gain after matching for infrastructure timeouts<ref:2608. p. 4
Where the paper says it

9 reaches 0. 65× gain after matching for infrastructure timeouts.

04804#pg2>

Where the gains come from

The primary cost-aware routing mechanism is driven by two distinct feature spaces fed into the résumé router: task text and SuperScout-7B’s 3,584-dimensional pre-decode hidden state<ref:2608.04804#pg2> The searcher’s hidden states improve cost routing on calibration labels while the handoff’s own text does not<ref:2608. p. 0

Where the paper says it

The searcher’s contribution to system cost is negligible: SuperScout-7B’s entire GPU bill for the evaluation was $1. A paired calibration study on 100 fresh tasks points to why this cost reduction holds: the handoff appears to redistribute rather than add solving ability, lifting the three cheaper fixers while slightly hurting the strongest, and the searcher’s hidden states improve cost routing where the handoff’s own text does arXiv:2608.

04804#pg2> A paired calibration study points to the mechanism, showing that the handoff appears to redistribute rather than add solving ability, lifting three cheaper fixers while slightly hurting the strongest<ref:2608. p. 0
Where the paper says it

The searcher’s contribution to system cost is negligible: SuperScout-7B’s entire GPU bill for the evaluation was $1. A paired calibration study on 100 fresh tasks points to why this cost reduction holds: the handoff appears to redistribute rather than add solving ability, lifting the three cheaper fixers while slightly hurting the strongest, and the searcher’s hidden states improve cost routing where the handoff’s own text does arXiv:2608.

04804#pg2> The router’s decision is governed by a logistic regression head blending these two feature sets by uniformly averaging their predicted probabilities<ref:2608. p. 3
Where the paper says it

In each feature space the router computes uncentered cosine similarities between the task and every résumé’s solved and failed centroids, their difference, and the base rate; a single logistic regression per feature space, shared across fixers, then scores P(solve) for each fixer from its résumé-relative features. The routing head blends the tasktext and hidden-state feature sets by uniformly averaging their predicted probabilities.

04804#pg2>

The evaluation on SWE-bench Pro showed that SuperScout matches the best single frontier model’s solve rate at about a fifth of its cost per solve, resolving 159 of 266 tasks versus 158, at a total cost of about one-fifth<ref:2608.04804#pg2> The no-router ablation ties the routed system on this benchmark, as the handoff rather than the routing decision carries the result<ref:2608. p. 0

Where the paper says it

On the full Python slice of SWE-bench Pro (266 tasks) under the benchmark’s official capped budget tier, SuperScout matches the best single model’s solve rate (159 of 266 for SuperScout, 158 for the best model) at about a fifth of the total cost per solve, and the reported configuration sits above the random traffic-splitting baseline. A no-router ablation, always the cheapest fixer with the handoff, ties the routed system on this benchmark, so the handoff rather than the routing decision carries the result.

04804#pg2> The calibration study established that fixer rankings survive handoff conditioning, with only one adjacent flip falling within a confidence interval<ref:2608. p. 8
Where the paper says it

At N=99 every per-fixer test is underpowered; the confidence intervals, not the point estimates, carry the information. What the pattern does establish is that fixer rankings survive handoff conditioning, with only one adjacent flip falling within the confidence interval.

04804#pg2>

The system achieves a total cost per solve of about 59.77% of the best solo model's cost at a matched solve rate, which is well above the blind-mixing line<ref:2608.04804#pg2> The searcher’s compute adds less than half a cent of GPU time per task, contributing to the overall efficiency<ref:2608. p. 0

Where the paper says it

A paired calibration study points to the mechanism: the handoff appears to redistribute rather than add solving ability, lifting the three cheaper fixers while slightly hurting the strongest, though at N=99 the per-fixer effects are directional only; the searcher’s hidden states improve cost routing on the calibration labels while the handoff’s own text does not. The searcher’s compute adds less than half a cent of GPU time per task.

04804#pg2> Adding a new fixer requires no retraining, as its résumé is built from 25–50 public outcomes and read at inference time<ref:2608. p. 3
Where the paper says it

4 Extensibility Adding a new fixer requires no retraining. A new model’s résumé consists of two averaged embeddings and a base rate, computed from its public outcomes.

04804#pg2> The verified handoff guard neutralizes overclaiming by stripping false claims before any fixer sees it, with spontaneous handoffs being more than twice as likely to be genuine<ref:2608. p. 7
Where the paper says it

These benchmark rates are worse than those observed during calibration (32% genuine, 56% false; §8), consistent with harder tasks producing more overclaiming. The guard neutralizes this overclaiming before any fixer sees it: 174 fixer prompts were stripped of confident misinformation that would otherwise have been treated as ground truth (full outcome classes in Appendix C); we did not run a pass-through arm, so the guard’s contribution to the headline is an inference from this claim census rather than a measurement (§9).

04804#pg2> The router’s feature design space was explored, showing that the hidden state’s contribution is mechanical, shifting when the cheap model is trusted rather than boosting accuracy per se<ref:2608. p. 7
Where the paper says it

561 for task text at N=99, Appendix F) is suggestive rather than decisive. The hidden state’s contribution is not boosting accuracy per se but shifting when the cheap model is trusted, consistent with its cost-side rather than accuracy-side value observed during calibration (§8).

04804#pg2>

The headline result rests on a single benchmark’s Python slice, comprising 266 tasks, and the robust evidence shows both the routed system and the no-router ablation sit +3.60 percentage points above the blind-mixing line<ref:2608. p. 6

Where the paper says it

Both system rows sit above the blind-mixing line defined in §6 by +3. 60 percentage points at their respective cost points (visible in Figure 1).

04804#pg2> The calibration label run showed that only 50 of 266 handoffs were genuine, while 174 were demonstrably false, confirming the guard’s effectiveness<ref:2608. p. 7
Where the paper says it

sandbox revealed that only 50 of them (20%) were genuine, while 174 (70%) were demonstrably false. The guard stripped every false claim before any fixer saw it.

04804#pg2> The system is built for a model market that changes quarter by quarter because adding a new fixer requires no retraining<ref:2608. p. 9
Where the paper says it

The searcher’s hidden states, meanwhile, change which tasks the router trusts to the cheap model, yet feeding the handoff’s text directly to the router hurts it. Because adding a new fixer requires no retraining, the system is built for a model market that changes quarter by quarter.

p. 9
Where the paper says it

The searcher’s hidden states, meanwhile, change which tasks the router trusts to the cheap model, yet feeding the handoff’s text directly to the router hurts it. Because adding a new fixer requires no retraining, the system is built for a model market that changes quarter by quarter.

p. 9
Where the paper says it

The searcher’s hidden states, meanwhile, change which tasks the router trusts to the cheap model, yet feeding the handoff’s text directly to the router hurts it. Because adding a new fixer requires no retraining, the system is built for a model market that changes quarter by quarter.

04804#pg2> Natural next steps include wider benchmark coverage across languages and problem domains, as cross-benchmark generality remains untested<ref:2608. p. 9
Where the paper says it

Because adding a new fixer requires no retraining, the system is built for a model market that changes quarter by quarter. Natural next steps include wider benchmark coverage across languages and problem domains.

p. 9
Where the paper says it

Because adding a new fixer requires no retraining, the system is built for a model market that changes quarter by quarter. Natural next steps include wider benchmark coverage across languages and problem domains.

04804#pg2> The cost accounting asymmetry is conservative in our disfavor, but it is still an accounting choice that readers should weigh<ref:2608. p. 9
Where the paper says it

Prices drift, and one fixer was served as a quantized build through a pinned provider, so cost conclusions are snapshots rather than stable constants. The cost-accounting asymmetry described in §6 (all-in for the system versus API-only for solo baselines) is conservative in our disfavor, but it is still an accounting choice that readers should weigh.

p. 9
Where the paper says it

Prices drift, and one fixer was served as a quantized build through a pinned provider, so cost conclusions are snapshots rather than stable constants. The cost-accounting asymmetry described in §6 (all-in for the system versus API-only for solo baselines) is conservative in our disfavor, but it is still an accounting choice that readers should weigh.

04804#pg2>

The system demonstrates that engaging with the repository before dispatch pays through a verified handoff, not the routing decision itself<ref:2608.04804#pg2> The handoff appears to redistribute solving ability, lifting cheaper models while slightly hurting the strongest, which is exactly the shape cost-aware routing needs<ref:2608. p. 0

Where the paper says it

The searcher’s contribution to system cost is negligible: SuperScout-7B’s entire GPU bill for the evaluation was $1. A paired calibration study on 100 fresh tasks points to why this cost reduction holds: the handoff appears to redistribute rather than add solving ability, lifting the three cheaper fixers while slightly hurting the strongest, and the searcher’s hidden states improve cost routing where the handoff’s own text does arXiv:2608.

04804#pg2> The searcher’s internal state helps the router by shifting which tasks it trusts to the cheap model, consistent with its cost-side rather than accuracy-side value<ref:2608. p. 7
Where the paper says it

561 for task text at N=99, Appendix F) is suggestive rather than decisive. The hidden state’s contribution is not boosting accuracy per se but shifting when the cheap model is trusted, consistent with its cost-side rather than accuracy-side value observed during calibration (§8).

04804#pg2> The system is built for a model market that changes quarter by quarter because adding a new fixer requires no retraining<ref:2608.04804#pg2> The final conclusion is that the verified handoff, not the routing decision, carries the result<ref:2608. p. 9
Where the paper says it

The zero-retrain onboarding property applies to adding new fixers, not to changing the searcher itself. 10 Conclusion SuperScout shows that engaging with the repository before dispatch pays, though not where we expected: the verified handoff, not the routing decision, carries the result.

p. 9
Where the paper says it

The zero-retrain onboarding property applies to adding new fixers, not to changing the searcher itself. 10 Conclusion SuperScout shows that engaging with the repository before dispatch pays, though not where we expected: the verified handoff, not the routing decision, carries the result.

04804#pg2>

SuperScout shows that engaging with the repository before dispatch pays, though not where we expected: the verified handoff, not the routing decision, carries the result<ref:2608.04804#pg2> A small trained searcher explores the repository and writes a structured handoff whose reproduction claims are verified in a sandbox (false claims are stripped), and whose hidden states feed a résumé router that selects the downstream fixer<ref:2608. p. 9

Where the paper says it

10 Conclusion SuperScout shows that engaging with the repository before dispatch pays, though not where we expected: the verified handoff, not the routing decision, carries the result. A small trained searcher, SuperScout-7B, explores the repository and writes a structured handoff whose reproduction claims are verified in a sandbox (false claims are stripped), and whose hidden states feed a résumé router that selects the downstream fixer.

04804#pg2> On the benchmark’s full Python slice, this pipeline solves 159 of 266 problems, matching the best single frontier model (158 of 266) at about a fifth of the total cost per solve, with the searcher’s own compute contributing a rounding error to the budget<ref:2608. p. 9
Where the paper says it

A small trained searcher, SuperScout-7B, explores the repository and writes a structured handoff whose reproduction claims are verified in a sandbox (false claims are stripped), and whose hidden states feed a résumé router that selects the downstream fixer. On the benchmark’s full Python slice, this pipeline solves 159 of 266 problems, matching the best single frontier model (158 of 266) at about a fifth of the total cost per solve, with the searcher’s own compute contributing a rounding error to the budget.

04804#pg2> The system is built for a model market that changes quarter by quarter because adding a new fixer requires no retraining<ref:2608.04804#pg2> Natural next steps include wider benchmark coverage across languages and problem domains, as cross-benchmark generality remains untested<ref:2608.04804#pg2> The cost accounting asymmetry is conservative in our disfavor, but it is still an accounting choice that readers should weigh<ref:2608.04804#pg2> The final conclusion is that the verified handoff, not the routing decision, carries the result<ref:2608.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. A searcher model should first explore a repository to produce a structured handoff whose reproduction claims are sandbox verified, with false claims stripped before delivery. This process, termed scrouting, ensures that downstream fixers receive only reliable evidence rather than potentially misleading information from the initial exploration phase.

  2. A résumé based router should dispatch tasks to one of four frontier fixers using the task text together with the searcher’s own hidden states. This mechanism allows for routing decisions to be informed by repository exploration without requiring retraining, as Adding a new fixer requires no retraining.

  3. The system can achieve cost savings by matching the best single model's solve rate at about a fifth of its total cost per solve on SWE-bench Pro, rather than relying on accuracy gains from combining models. This is achieved because The handoff appears to redistribute rather than add solving ability, lifting the three cheaper fixers while slightly hurting the strongest.

  4. The routing mechanism can be tuned by using the searcher’s hidden states as a feature space in the router, which improves cost routing on calibration labels where the searcher’s hidden states improve cost routing on the calibration labels while the handoff’s own text does not.

  5. New fixers can be added to the system without retraining by simply providing a résumé consisting of two averaged embeddings and a base rate, computed from 25–50 public per-task outcomes.

Abstract

Frontier language models can resolve repository-level software issues, but each attempt is expensive, and existing routers select a model from the issue text alone. We present SuperScout, which routes after scouting the repository: a 7B searcher, SuperScout-7B, first explores the repository and produces a structured handoff whose reproduction claims are sandbox-verified, with false claims stripped before delivery. The searcher's hidden states, together with the task text, then feed a resume-based router that dispatches the task to one of four frontier fixers. Adding a new fixer requires no retraining. On the full Python slice of SWE-bench Pro (266 tasks) under the benchmark's official capped budget tier, SuperScout matches the best single model's solve rate (159 of 266 for SuperScout, 158 for the best model) at about a fifth of the total cost per solve, and the reported configuration sits above the random traffic-splitting baseline. A no-router ablation, always the cheapest fixer with the handoff, ties the routed system on this benchmark, so the handoff rather than the routing decision carries the result. A paired calibration study points to the mechanism: the handoff appears to redistribute rather than add solving ability, lifting the three cheaper fixers while slightly hurting the strongest, though at N=99 the per-fixer effects are directional only; the searcher's hidden states improve cost routing on the calibration labels while the handoff's own text does not. The searcher's compute adds less than half a cent of GPU time per task.

Sources

Related papers