Scrouting: Cost-Aware Routing of Coding Agents by Scouting the Repository First
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "Scrouting: Cost-Aware Routing of Coding Agents by Scouting the Repository First".
Title and authors: Tom: So we're looking at this paper today called "Scrouting: Cost-Aware Routing of Coding Agents by Scouting the Repository First". It’s about how to make using those big frontier language models for software issues cheaper.
Jane: Right, so the core idea is that instead of just looking at the problem description and picking a model from there, you first have a small searcher explore the actual code repository itself.
Lu: Exactly. They introduce this engage-then-route architecture where this 7B searcher model explores the repository and produces some kind of structured handoff before we even think about fixing anything <ref:2608.04804#pg1,explores the repository and produces>.
Meng: Wait, so it's like having a preliminary investigation phase before you hire a specialist to fix the bug? I need to know if that preliminary work actually saves time.
Tom: It does, Meng. The paper shows this searcher, SuperScout-7B, produces a handoff with reproduction claims that have been sandbox verified and false claims are stripped out before they get to the fixer models <ref:2608.04804#pg1>.
Jane: That’s important because it means the downstream models aren't wasting time on misleading or impossible tasks generated from a bad initial search.
Lu: And then this structured handoff, along with the task text, feeds into a résumé-based router that chooses one of four frontier fixers to actually do the work.
Tom: The results are pretty compelling: on SWE-bench Pro’s full Python slice with their official budget tier, SuperScout matches the best single model's solve rate at about a fifth of the total cost per solve.
Meng: A fifth? That sounds like a big saving, but I need to know if that accuracy gain is worth it against the extra compute for this scouting phase.
Jane: The paper suggests that this cost matching happens because the handoff itself seems to redistribute solving ability among different fixers, lifting some cheaper ones while only slightly hurting the strongest one.
Lu: That redistribution idea is key because it means the routing isn't just picking the best guess based on text alone; it’s picking based on where the knowledge already exists in that repository.
Title and authors: Tom: And they show that this routing decision is guided by two things: you have the task text, and you have those hidden states from SuperScout-7B, which are about three thousand five hundred eighty-four dimensions.
Jane: So it’s not just a simple lookup anymore; it’s using the searcher’s internal understanding of the repo to make a smarter routing call.
Meng: Using internal states for routing sounds interesting from an engineering standpoint, but how much extra compute is that searcher adding on top of what we already pay for?
Tom: The paper says the searcher’s compute adds less than half a cent of GPU time per task, which is pretty low given the benefit it provides to the overall cost.
Lu: And they built this searcher specifically by training it only on search-phase demonstrations from other agents, using nineteen thousand nine hundred eleven examples of full search trajectories as the final target.
Jane: So they focused the training signal only on that specific exploration step, which is a smart way to get the model focused on finding those good starting points.
Tom: And there’s this verification gate mentioned where they strip false reproduction claims before any fixer sees them, which helps neutralize overclaiming by making spontaneous handoffs much more likely to be genuine.
Meng: That sounds like a good safety mechanism for production; I worry about models just confidently claiming things that aren't true.
Lu: And regarding adding new fixers, they say you don’t need to retrain anything new because the résumé is built from twenty-five to fifty public outcomes and read at inference time.
Jane: So the system is built for a market where models are constantly changing, which is good because we don't have to keep retraining every time a new capable fixer comes along.
Tom: Exactly. The evaluation on SWE-bench Pro showed that the routed system, even without the router at all, sits three point six percentage points above the blind mixing line on two hundred sixty-six tasks <ref:2608.04804#pg1>.
Meng: That's a solid number for a baseline comparison; it shows that just having that verified handoff is better than random traffic splitting.
Title and authors: Lu: And to push on the cost aspect, they found that the calibration study showed that only fifty of those two hundred sixty-six handoffs were actually genuine, while one hundred seventy-four were demonstrably false, confirming the guard’s effectiveness.
Jane: It’s interesting how they framed this cost asymmetry; it suggests that engaging with the repository first pays for itself through a verified handoff rather than just by making a better routing decision.
Tom: And one thing they pointed out is that their feature design for the router shows that the hidden state contribution is mechanical, meaning it shifts when you trust the cheaper model more than it boosts accuracy per se.
Meng: So it’s less about making the cheap model smarter and more about trusting its output when you're worried about cost. That makes sense for deployment at scale.
Lu: And looking ahead, they suggest that natural next steps involve testing this across wider benchmark coverage in different languages and problem domains because they haven't tested cross-benchmark generality yet.
Jane: So the big picture here is that we’ve moved toward a system where exploration pays off through verification, but we still need to see if this works as well outside of the specific Python slice they used.
Tom: To wrap things up, it’s clear that the verified handoff, not just the routing decision itself, carries the result in this Scrouting paper.
Jane: So that’s our take on "Scrouting: Cost-Aware Routing of Coding Agents by Scouting the Repository First". It shows a way to use small searchers to prune expensive calls before dispatching tasks to frontier models.
Lu: It points toward an architecture where the exploration phase is valuable because it generates a trusted signal that informs cost-aware routing decisions.
Meng: For us in production, it suggests we can integrate this scouting step as a cheap pre-filter before hitting the most expensive model calls.
Tom: We’re leaving it there for now, but if you want to see how they handle different types of data scarcity or maybe how this scales beyond SWE-bench Pro, we'll be back next time.
The paper's summary: Tom: So, we’re talking about Scrouting today—this system that tries to make using these massive frontier models for software problems way cheaper by scouting the repository first before dispatching a task.
Jane: That’s right. The big idea is that instead of just looking at the problem description and hoping you pick the right model, you have this small searcher explore the actual code repository itself to build some kind of structured handoff.
Lu: And that handoff is verified in a sandbox—meaning they check if it actually works by checking out the files—and then it’s fed into a résumé router that picks one of four bigger fixers to do the actual work.
Meng: So, what does this actually do? Does it just make the AI smarter, or does it actually save money when you’re running these things in production?
Tom: It saves money because the paper shows that on a big benchmark called SWE-bench Pro, this routed system matches the best single frontier model's solve rate at about a fifth of its total cost per solve.
Jane: That’s a significant cost saving, but it seems like they aren't just relying on the routing decision itself making it faster.
Lu: They found that what carries the result is actually that verified handoff—the structured information from the repository exploration—not necessarily the router's final choice.
Tom: And to make routing smarter, they feed that searcher’s internal hidden states into the résumé router alongside the original task text.
Jane: So this means you’re using what the searcher *learned* about where things are in the code to help decide which fixer is best.
Meng: I’m curious about how much compute this extra searcher adds on top of what we already pay for when we run these things at scale.
Tom: They say the searcher’s compute adds less than half a cent of GPU time per task, which is pretty low given the benefit it provides to the overall cost efficiency.
Lu: And they built this searcher by training it only on demonstrations from other agents exploring repositories, using about twenty thousand examples of full search trajectories as its training data.
Jane: So they focused the learning signal specifically on that exploration step, which is a smart way to get the model focused on finding those good starting points in the code.
Tom: Plus, they have this guard where false reproduction claims are stripped out before any fixer even sees them, which helps neutralize overclaiming by making genuine handoffs much more likely.
Meng: That sounds like a solid safety mechanism for production; I worry about models just confidently claiming things that aren't true when they’re trying to fix code.
Lu: And another thing they mentioned is that you don't need to retrain anything new if you add a new fixer because its résumé is built from twenty five to fifty public outcomes and read at inference time.
Jane: That means this system is built for a market where models are constantly changing, which is good because we don't have to keep retraining every time a new capable fixer comes along.
Tom: They also pointed out in their calibration study that only about fifty of those two hundred sixty six handoffs were actually genuine, while one hundred seventy four were demonstrably false, confirming how effective that guard is.
Lu: The core idea here is that engaging with the repository first pays off through a verified handoff rather than just by making a better routing decision.
Jane: It suggests we’ve moved toward a system where exploration pays for itself because it generates a trusted signal that informs cost-aware routing decisions.
Tom: And they showed that the hidden state contribution to the router is mechanical, meaning it shifts when you trust the cheaper model more than it boosts accuracy per se.
Meng: So it’s less about making the cheap model smarter and more about trusting its output when you're worried about cost, which makes sense for deployment at scale.
Lu: Looking ahead, they suggest that natural next steps involve testing this across wider benchmark coverage in different languages and problem domains because they haven't tested cross-benchmark generality yet.
Jane: So the big picture here is that we’ve moved toward a system where exploration pays off through verification, but we still need to see if this works as well outside of the specific Python slice they used.
The paper's improvements: Tom: So we’re looking at how they suggest making this scouting system even better for the future than what's on page two of this paper.
Jane: They point out that the current searcher training is only focused on search phase demonstrations, so they want to see if you can expand that training signal.
Lu: Yeah, they are looking at ways to make that initial exploration phase more robust, maybe by incorporating more diverse data from different agents exploring different codebases.
Meng: If you get better data for the searcher, does that mean the handoff quality gets better? Is it just a bigger picture of where the code is?
Tom: It should definitely improve the handoff quality because if the searcher can explore more effectively, it’ll produce a more structured and accurate handoff.
Jane: And they suggest another way to tune the system: using those hidden states from the searcher as a feature space in that résumé router.
Lu: That makes sense because they found that those hidden states are important for cost routing on calibration labels, which is where it actually helps the system balance between accuracy and expense.
Meng: So they’re saying we can use internal knowledge to decide which fixer to trust when the cost matters most? I need to know if that’s something we can deploy reliably.
Tom: They suggest that this routing mechanism isn't just about picking a better model; it's about shifting trust toward cheaper models when you’re worried about the cost.
Jane: And they also make a point about how new fixers get added to the system without needing any retraining at all.
Lu: That’s because they design the résumé itself from twenty five to fifty public outcomes, and you just read that summary at inference time, so adding a new fixer is super fast.
Meng: So that's good for us in a production environment; we don't have to spend weeks retraining when a new tool comes along.
Tom: They also talked about the importance of the verification guard again, specifically how it neutralizes overclaiming by stripping false claims before any fixer sees them.
Jane: It’s like a quality control checkpoint that makes sure the initial exploration phase actually delivers reliable evidence instead of just making up stuff to sound confident.
Lu: They also hinted that we need to test this system across wider benchmark coverage, not just this single Python slice they used, because they haven't checked cross-benchmark generality yet.
Meng: That’s a fair point; if it only works on one type of code problem, it won't help us on the messy real-world stuff we actually run.
Tom: The main thing they are pointing toward is that this architecture lets you use a small model to prune expensive calls before dispatching tasks to the most powerful models.
Jane: So, the paper’s core improvement is moving away from a simple text-based router and making the routing decision dependent on a verified exploration of the codebase.
Conclusion: Tom: So to wrap up, Scrouting is essentially showing how you can use a small searcher model to explore code before dispatching tasks to big frontier models, and this scouting step actually pays off in terms of cost savings.
Jane: It really means that for anyone using these large language models for software work, you don't have to just rely on the problem description; you can use repository exploration as a way to get a more reliable signal before you spend money on a complex fix.
Lu: The implication is that we move from routing purely based on the prompt text to routing based on where the actual knowledge exists in the code, which should lead to more efficient resource allocation across different models.
Meng: For us at startup, it means if we can integrate this scouting step cheaply, we can significantly lower our operational costs when running these agents at scale.
Lalam: I’m confident in this direction because the evidence shows that the verified handoff carries the result, which is a very strong signal for reliable deployment.
Tom: We saw that they match the best single model's solve rate at about a fifth of its cost per solve on SWE-bench Pro, which is a solid number against what we can achieve with just text routing.
Jane: And even when you look closely at the numbers, like how many handoffs were actually genuine versus false, it confirms that this guard mechanism is working as intended.
Lu: They also showed that adding new fixers doesn't require retraining because their résumé is built from public outcomes, which is a huge practical benefit for keeping up with model developments.
Meng: It’s good to know that the system doesn't become totally obsolete every time a new powerful AI model comes out, as long as we can update those résumés easily.
Tom: So the final conclusion on Scrouting is that the verified handoff, not just the routing decision itself, is what carries the result in this paper.
Jane: It’s a smart way to build a system where exploration pays off through verification rather than just by making a better routing call based on text alone.
Lu: I think we still need to see how this holds up when we apply it across different types of code problems and maybe even different languages, because that cross-benchmark generality is still untested.
Meng: Yeah, I’m worried about running this on real traffic; the benchmark they used is just a slice of Python tasks, so we need to see how it handles the messiness of actual production environments.
Lalam: I don’t see any immediate flaws in their approach based on what we've seen in the literature, and I’m sure this method holds up well.
Tom: We’re leaving it there for now, but if you want to see how they handle different types of data scarcity or maybe how this scales beyond SWE-bench Pro, we'll be back next time.
Ishaan Bhola, Adithyan Krishnan, Mukunda NS
SuperAGI Research
cs.SE, cs.AI
Submitted: 2026-08-05
Updated: 2026-11-18
Code: https://github.com/TransformerOptimus/superscout
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 85/100
The gist: Frontier language models can resolve repository-level software issues, but each attempt is expensive, and existing routers select a model from the issue text alone<ref:2608.04804#pg2> SuperScout…
Key concepts
- Scrouting
- This is a new two-step process where a small searcher model first explores a software repository to find relevant files and diagnostic notes. Crucially, it performs sandbox verification on these findings to strip out any false claims before they are passed on to the next stage, ensuring downstream models receive reliable information.
- Résumé Based Router
- This component decides which of four large frontier fixers should handle a task. It makes its decision by looking at two main inputs: the original task text and the hidden states generated by the searcher model during its repository exploration phase.
- Searcher's Hidden States
- These are internal representations produced by the small 7B searcher model while it is exploring a repository. The paper shows these states are not just random data; they provide valuable information that helps the router make better cost-aware decisions, particularly when distinguishing between different fixer capabilities.
- Cost-Aware Routing
- This system aims to save money by matching the best single model's performance at a fraction of its cost. The routing decision is informed by features like the searcher's hidden states and task text, allowing the system to direct tasks toward cheaper models when appropriate, based on how well they perform on calibration data.
Terminology
Summary
Frontier language models can resolve repository-level software issues, but each attempt is expensive, and existing routers select a model from the issue text alone<ref:2608. p. 0 Scrouting: Cost-Aware Routing of Coding Agents by
Scouting the Repository First
Ishaan Bhola
SuperAGI Research
Adithyan Krishnan
SuperAGI Research
Mukunda NS
SuperAGI Research
Abstract—Frontier language models can resolve repository-level
software issues, but each attempt is expensive, and existing routers
select a model from the issue text alone. We present SuperScout,
which routes after scouting the repository: a 7B searcher,
SuperScout-7B, first explores the repository and produces a
structured handoff whose reproduction claims are sandboxverified, with false claims stripped before delivery. Scrouting: Cost-Aware Routing of Coding Agents by
Scouting the Repository First
Ishaan Bhola
SuperAGI Research
Adithyan Krishnan
SuperAGI Research
Mukunda NS
SuperAGI Research
Abstract—Frontier language models can resolve repository-level
software issues, but each attempt is expensive, and existing routers
select a model from the issue text alone. We present SuperScout,
which routes after scouting the repository: a 7B searcher,
SuperScout-7B, first explores the repository and produces a
structured handoff whose reproduction claims are sandboxverified, with false claims stripped before delivery. We present SuperScout,
which routes after scouting the repository: a 7B searcher,
SuperScout-7B, first explores the repository and produces a
structured handoff whose reproduction claims are sandboxverified, with false claims stripped before delivery. The searcher’s
hidden states, together with the task text, then feed a résumébased router that dispatches the task to one of four frontier fixers. Adding a new fixer requires no retraining. On the full Python slice
of SWE-bench Pro (266 tasks) under the benchmark’s official
capped budget tier, SuperScout matches the best single model’s
solve rate (159 of 266 for SuperScout, 158 for the best model)
at about a fifth of the total cost per solve, and the reported
configuration sits above the random traffic-splitting baseline. We present SuperScout,
which routes after scouting the repository: a 7B searcher,
SuperScout-7B, first explores the repository and produces a
structured handoff whose reproduction claims are sandboxverified, with false claims stripped before delivery. The searcher’s
hidden states, together with the task text, then feed a résumébased router that dispatches the task to one of four frontier fixers.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
The paper introduces SuperScout, an engage-then-route architecture where a small searcher model explores the repository and produces a structured handoff, which is then consumed by a résumé router to select one of four frontier fixers<ref:2608.04804#pg2> This system is designed to route after engaging with the problem, rather than from the task text alone<ref:2608. p. 2 Rather than seeking accuracy gains by combining models, we
aim to match the strongest available model’s solve rate at a
materially lower total cost per solve. The thesis is that routing should happen after engaging with
the problem, not from the task text alone. SuperScout routes after scouting, a pattern we call scrouting. A 7B searcher, SuperScout-7B, first explores the repository
and produces a structured handoff: a list of implicated files,
diagnostic notes, and a candidate reproduction test. A 7B searcher, SuperScout-7B, first explores the repository
and produces a structured handoff: a list of implicated files,
diagnostic notes, and a candidate reproduction test. These
claims pass through a sandbox verification step, and claims
that do not check out are stripped before a downstream model
ever sees them. These
claims pass through a sandbox verification step, and claims
that do not check out are stripped before a downstream model
ever sees them. A résumé router then selects among four
frontier fixers using the task text together with the searcher’s
own hidden states; adding a new fixer requires no retraining.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
How the training signal is built
SuperScout-7B is trained exclusively for the search phase on search-phase demonstrations sliced from openly licensed agent trajectories produced by other systems<ref:2608.04804#pg2> The supervised training corpus consists of 19,911 examples, where each example is a full search trajectory with a synthesized handoff-emission turn as the final supervised target<ref:2608. p. 4 TABLE III: Composition of SuperScout-7B’s supervised
search corpus. Each example is a full search trajectory with a
synthesized handoff-emission turn as the final supervised step;
loss is on assistant turns only. TABLE III: Composition of SuperScout-7B’s supervised
search corpus. Each example is a full search trajectory with a
synthesized handoff-emission turn as the final supervised step;
loss is on assistant turns only. We fine-tune Qwen2. 5-Coder-7B [23] with LoRA [31] on
a single GPU. Component Value
Corpus
Examples trained on 19,905
Built / frozen set 19,911
Held-back shelf (untrained) 9,478
Tokens per epoch 382. 4M
Packed 32k blocks (fill) 12,723 (91. 2 Decoding Matters More Than Expected
The most consequential training-era finding was not about
the training itself but about inference. On a 450-task held-out
exam, greedy decoding finds the right files at a rate of 0. 96×
(Figure 4); the quality cost visible on the exam does not appear
there, so the dial set is a mechanism check, not a replication. Greedy decoding, it turns out, makes the searcher reluctant to
declare a result; sampling at moderate temperature recovers
ability the model already has. A single sampled draw at temperature 0. 9 reaches 0. 9 reaches 0. 65× gain after matching for infrastructure timeouts.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
The primary cost-aware routing mechanism is driven by two distinct feature spaces fed into the résumé router: task text and SuperScout-7B’s 3,584-dimensional pre-decode hidden state<ref:2608.04804#pg2> The searcher’s hidden states improve cost routing on calibration labels while the handoff’s own text does not<ref:2608. p. 0 The searcher’s contribution to system cost is
negligible: SuperScout-7B’s entire GPU bill for the evaluation
was $1. A paired calibration study on 100 fresh tasks points to why
this cost reduction holds: the handoff appears to redistribute
rather than add solving ability, lifting the three cheaper fixers
while slightly hurting the strongest, and the searcher’s hidden
states improve cost routing where the handoff’s own text does
arXiv:2608. The searcher’s contribution to system cost is
negligible: SuperScout-7B’s entire GPU bill for the evaluation
was $1. A paired calibration study on 100 fresh tasks points to why
this cost reduction holds: the handoff appears to redistribute
rather than add solving ability, lifting the three cheaper fixers
while slightly hurting the strongest, and the searcher’s hidden
states improve cost routing where the handoff’s own text does
arXiv:2608. In each feature space the router computes
uncentered cosine similarities between the task and every
résumé’s solved and failed centroids, their difference, and
the base rate; a single logistic regression per feature space,
shared across fixers, then scores P(solve) for each fixer from
its résumé-relative features. The routing head blends the tasktext and hidden-state feature sets by uniformly averaging their
predicted probabilities.Where the paper says it
Where the paper says it
Where the paper says it
The evaluation on SWE-bench Pro showed that SuperScout matches the best single frontier model’s solve rate at about a fifth of its cost per solve, resolving 159 of 266 tasks versus 158, at a total cost of about one-fifth<ref:2608.04804#pg2> The no-router ablation ties the routed system on this benchmark, as the handoff rather than the routing decision carries the result<ref:2608. p. 0 On the full Python slice
of SWE-bench Pro (266 tasks) under the benchmark’s official
capped budget tier, SuperScout matches the best single model’s
solve rate (159 of 266 for SuperScout, 158 for the best model)
at about a fifth of the total cost per solve, and the reported
configuration sits above the random traffic-splitting baseline. A
no-router ablation, always the cheapest fixer with the handoff, ties
the routed system on this benchmark, so the handoff rather than
the routing decision carries the result. At N=99 every per-fixer test is
underpowered; the confidence intervals, not the point estimates,
carry the information. What the pattern does establish is that fixer rankings survive
handoff conditioning, with only one adjacent flip falling within
the confidence interval.Where the paper says it
Where the paper says it
The system achieves a total cost per solve of about 59.77% of the best solo model's cost at a matched solve rate, which is well above the blind-mixing line<ref:2608.04804#pg2> The searcher’s compute adds less than half a cent of GPU time per task, contributing to the overall efficiency<ref:2608. p. 0 A paired calibration study
points to the mechanism: the handoff appears to redistribute
rather than add solving ability, lifting the three cheaper fixers
while slightly hurting the strongest, though at N=99 the per-fixer
effects are directional only; the searcher’s hidden states improve
cost routing on the calibration labels while the handoff’s own
text does not. The searcher’s compute adds less than half a cent
of GPU time per task. 4 Extensibility
Adding a new fixer requires no retraining. A new model’s
résumé consists of two averaged embeddings and a base rate,
computed from its public outcomes. These benchmark rates are worse than those observed
during calibration (32% genuine, 56% false; §8), consistent
with harder tasks producing more overclaiming. The guard
neutralizes this overclaiming before any fixer sees it: 174 fixer
prompts were stripped of confident misinformation that would
otherwise have been treated as ground truth (full outcome
classes in Appendix C); we did not run a pass-through arm, so
the guard’s contribution to the headline is an inference from
this claim census rather than a measurement (§9). 561 for task text at N=99, Appendix F) is suggestive rather
than decisive. The hidden state’s contribution is not boosting
accuracy per se but shifting when the cheap model is trusted,
consistent with its cost-side rather than accuracy-side value
observed during calibration (§8).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The headline result rests on a single benchmark’s Python slice, comprising 266 tasks, and the robust evidence shows both the routed system and the no-router ablation sit +3.60 percentage points above the blind-mixing line<ref:2608. p. 6 Both system rows sit above the blind-mixing line defined in
§6 by +3. 60 percentage points at their respective cost points
(visible in Figure 1). sandbox revealed that only 50 of them (20%) were genuine,
while 174 (70%) were demonstrably false. The guard stripped
every false claim before any fixer saw it. The searcher’s hidden states,
meanwhile, change which tasks the router trusts to the cheap
model, yet feeding the handoff’s text directly to the router
hurts it. Because adding a new fixer requires no retraining, the
system is built for a model market that changes quarter by
quarter. The searcher’s hidden states,
meanwhile, change which tasks the router trusts to the cheap
model, yet feeding the handoff’s text directly to the router
hurts it. Because adding a new fixer requires no retraining, the
system is built for a model market that changes quarter by
quarter. The searcher’s hidden states,
meanwhile, change which tasks the router trusts to the cheap
model, yet feeding the handoff’s text directly to the router
hurts it. Because adding a new fixer requires no retraining, the
system is built for a model market that changes quarter by
quarter. Because adding a new fixer requires no retraining, the
system is built for a model market that changes quarter by
quarter. Natural next steps include wider benchmark coverage across
languages and problem domains. Because adding a new fixer requires no retraining, the
system is built for a model market that changes quarter by
quarter. Natural next steps include wider benchmark coverage across
languages and problem domains. Prices drift, and one fixer was served as a quantized build
through a pinned provider, so cost conclusions are snapshots
rather than stable constants. The cost-accounting asymmetry
described in §6 (all-in for the system versus API-only for
solo baselines) is conservative in our disfavor, but it is still an
accounting choice that readers should weigh. Prices drift, and one fixer was served as a quantized build
through a pinned provider, so cost conclusions are snapshots
rather than stable constants. The cost-accounting asymmetry
described in §6 (all-in for the system versus API-only for
solo baselines) is conservative in our disfavor, but it is still an
accounting choice that readers should weigh.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The system demonstrates that engaging with the repository before dispatch pays through a verified handoff, not the routing decision itself<ref:2608.04804#pg2> The handoff appears to redistribute solving ability, lifting cheaper models while slightly hurting the strongest, which is exactly the shape cost-aware routing needs<ref:2608. p. 0 The searcher’s contribution to system cost is
negligible: SuperScout-7B’s entire GPU bill for the evaluation
was $1. A paired calibration study on 100 fresh tasks points to why
this cost reduction holds: the handoff appears to redistribute
rather than add solving ability, lifting the three cheaper fixers
while slightly hurting the strongest, and the searcher’s hidden
states improve cost routing where the handoff’s own text does
arXiv:2608. 561 for task text at N=99, Appendix F) is suggestive rather
than decisive. The hidden state’s contribution is not boosting
accuracy per se but shifting when the cheap model is trusted,
consistent with its cost-side rather than accuracy-side value
observed during calibration (§8). The zero-retrain onboarding property applies to adding
new fixers, not to changing the searcher itself. 10 Conclusion
SuperScout shows that engaging with the repository before
dispatch pays, though not where we expected: the verified
handoff, not the routing decision, carries the result. The zero-retrain onboarding property applies to adding
new fixers, not to changing the searcher itself. 10 Conclusion
SuperScout shows that engaging with the repository before
dispatch pays, though not where we expected: the verified
handoff, not the routing decision, carries the result.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
SuperScout shows that engaging with the repository before dispatch pays, though not where we expected: the verified handoff, not the routing decision, carries the result<ref:2608.04804#pg2> A small trained searcher explores the repository and writes a structured handoff whose reproduction claims are verified in a sandbox (false claims are stripped), and whose hidden states feed a résumé router that selects the downstream fixer<ref:2608. p. 9 10 Conclusion
SuperScout shows that engaging with the repository before
dispatch pays, though not where we expected: the verified
handoff, not the routing decision, carries the result. A small
trained searcher, SuperScout-7B, explores the repository and
writes a structured handoff whose reproduction claims are
verified in a sandbox (false claims are stripped), and whose
hidden states feed a résumé router that selects the downstream
fixer. A small
trained searcher, SuperScout-7B, explores the repository and
writes a structured handoff whose reproduction claims are
verified in a sandbox (false claims are stripped), and whose
hidden states feed a résumé router that selects the downstream
fixer. On the benchmark’s full Python slice, this pipeline solves
159 of 266 problems, matching the best single frontier model
(158 of 266) at about a fifth of the total cost per solve, with
the searcher’s own compute contributing a rounding error to
the budget.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
A searcher model should first explore a repository to produce a structured handoff whose reproduction claims are sandbox verified, with false claims stripped before delivery. This process, termed scrouting, ensures that downstream fixers receive only reliable evidence rather than potentially misleading information from the initial exploration phase.
-
A résumé based router should dispatch tasks to one of four frontier fixers using the task text together with the searcher’s own hidden states. This mechanism allows for routing decisions to be informed by repository exploration without requiring retraining, as
Adding a new fixer requires no retraining.
-
The system can achieve cost savings by matching the best single model's solve rate at about a fifth of its total cost per solve on SWE-bench Pro, rather than relying on accuracy gains from combining models. This is achieved because
The handoff appears to redistribute rather than add solving ability, lifting the three cheaper fixers while slightly hurting the strongest.
-
The routing mechanism can be tuned by using the searcher’s hidden states as a feature space in the router, which improves cost routing on calibration labels where
the searcher’s hidden states improve cost routing on the calibration labels while the handoff’s own text does not.
-
New fixers can be added to the system without retraining by simply providing a résumé consisting of
two averaged embeddings and a base rate, computed from 25–50 public per-task outcomes.
Abstract
Frontier language models can resolve repository-level software issues, but each attempt is expensive, and existing routers select a model from the issue text alone. We present SuperScout, which routes after scouting the repository: a 7B searcher, SuperScout-7B, first explores the repository and produces a structured handoff whose reproduction claims are sandbox-verified, with false claims stripped before delivery. The searcher's hidden states, together with the task text, then feed a resume-based router that dispatches the task to one of four frontier fixers. Adding a new fixer requires no retraining. On the full Python slice of SWE-bench Pro (266 tasks) under the benchmark's official capped budget tier, SuperScout matches the best single model's solve rate (159 of 266 for SuperScout, 158 for the best model) at about a fifth of the total cost per solve, and the reported configuration sits above the random traffic-splitting baseline. A no-router ablation, always the cheapest fixer with the handoff, ties the routed system on this benchmark, so the handoff rather than the routing decision carries the result. A paired calibration study points to the mechanism: the handoff appears to redistribute rather than add solving ability, lifting the three cheaper fixers while slightly hurting the strongest, though at N=99 the per-fixer effects are directional only; the searcher's hidden states improve cost routing on the calibration labels while the handoff's own text does not. The searcher's compute adds less than half a cent of GPU time per task.
Sources
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- RouteLLM: Learning to Route LLMs with Preference Data
- FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
- SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
- SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
- Agentless: Demystifying LLM-based Software Engineering Agents
- SWE-smith: Scaling Data for Software Engineering Agents
- R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents
- SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolution
- LocAgent: Graph-Guided LLM Agents for Code Localization
- SweRank: Software Issue Localization with Code Ranking
- Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing
- Large Language Model Routing with Benchmark Datasets
- EmbedLLM: Learning Compact Representations of Large Language Models
- ICL-Router: In-Context Learned Model Representations for LLM Routing
- SWE-Router: Routing in Multi-turn Agentic Software Engineering Tasks
- CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents
- ORACLE: Agentic AI Orchestrator Routing Via Adaptive Verifier Calibration Feedback
- Toward Training Superintelligent Software Agents through Self-Play SWE-RL
- Qwen2.5-Coder Technical Report
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters