Hard Issue, Weak Model, or Bad Run? Rethinking Failure Prediction for Coding Agents

arXiv:2611.08285 · cs.SE · Submitted 2026-11-05 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Hard Issue, Weak Model, or Bad Run? Rethinking Failure Prediction for Coding Agents".

Jane: Here's the gist of the paper. Trajectory-based failure predictors for software engineering (SE) agents read the partial trajectory of a run to flag whether it is going right or wrong On public SWE-bench Verified trajectories, where each model attempts each issue once, two run-independent factors dominate predictor performance Beyond difficulty and ability, what predictors gain from the trajectory is statistically equivalent to zero.

Tom: The Problem with Current Predictors Current predictors cannot tell the causes apart, as they mostly predict the issue and the model, not the run.

Jane: The paper questions three assumptions that current practice takes for granted regarding failure prediction.

Tom: These assumptions are that a predictor with a high AUROC has learned run-specific information, that the oracle gap reflects issue-specific strengths that switching can exploit, and that an alarm marks a bad run so that acting on it pays off.

Title and authors: Jane: So they're suggesting we look at three assumptions about current predictors and see if they hold up when you test them rigorously.

Lu: The paper tests three main assumptions: first, that a predictor with high performance has learned run-specific information, second, that the gap between models is due to issue-specific strengths we can exploit by switching models, and third, that an alarm actually points to a bad run worth retrying.

Tom: That's the core critique they're making against how we currently evaluate these things—that standard evaluations pool everything together and don't test these assumptions properly.

Meng: So if we can't prove those assumptions are true, then their high performance scores might just be misleading noise based on issue difficulty or model ability.

Lu: They show that a high pooled AUROC doesn't require learning about the run because it just points to issue difficulty, which is a reference score that never looks at the specific run itself and reaches zero point nine two AUROC <ref:2611.08285#pg1,a reference score that never looks at the>.

Jane: That zero point nine two AUROC for issue difficulty is quite high, but it doesn't help us understand why a specific run succeeded or failed in a particular way <ref:2611.08285#pg1>.

Tom: And then there’s the model ability part, where knowing which model ran on an issue predicts failure better than our trajectory predictors do—the predictors largely reproduce that overall ranking of models.

Meng: So even if we know the model is good, our predictor doesn't give us any extra signal to decide what to do next.

Lu: They show that adding the prediction score to issue difficulty and model identity changes pooled AUROC by an amount equivalent to zero in all eighteen configurations they tested.

The paper's summary: Tom: So, the paper’s main takeaway is that trajectory-based failure predictors mostly learn the issue and the model, but not the run itself.

Jane: That means when we get an alarm, it doesn't tell us *why* that specific run failed—it doesn't distinguish between a hard issue and a weak model.

Lu: They show that if you try to claim these predictors are learning something run-specific, the evidence for that idea is extremely weak; it's statistically equivalent to zero.

Meng: From an engineering standpoint, this means we can’t rely on these scores to decide whether we should stop the run or switch models based on a fine-grained prediction of failure.

Tom: The authors argue that alarms cannot reliably tell a bad run from a hard issue or a weak model, which is what they call the alarm problem.

Jane: They call for something different—failure prediction that explains *why* it failed so we can use the right action to fix it, like stopping for tokens saved or retrying with a better model.

Lu: They suggest that we need to build predictors that separate these three distinct failure causes: hard issue, weak model, and bad run.

The paper's improvements: Tom: The improvements they suggest are pretty clear: redesigning the predictors to explicitly distinguish between those three failure causes.

Jane: Instead of one score that just says "it might fail," we need a system that outputs which cause is likely responsible for the predicted failure.

Lu: This would allow for much more targeted action, so you can stop because it’s a bad run, or switch to a stronger model if it’s weak, or retry if it's just a hard issue.

Meng: That moves us from just getting a score to actually having an instruction on what to do next based on the prediction. It makes the decision loop much more useful for production systems.

Tom: Exactly. They want failure prediction linked directly to an appropriate action, meaning the predictor has to output not just a failure score, but also which cause led to that predicted failure.

Jane: That sounds like they are trying to move beyond just predicting success or failure and into prescriptive advice for agentic workflows.

Conclusion: Tom: So, to wrap up the paper "Hard Issue, Weak Model, or Bad Run? Rethinking Failure Prediction for Coding Agents," the main point is that current predictors mostly predict the issue and the model, not what happened in this specific run.

Jane: They found that pooled performance gains from learning about the run itself are statistically equivalent to zero when compared to just looking at issue difficulty and model ability scores.

Lu: They emphasize that we need a new kind of failure prediction that explains *why* a run fails so we can choose the right fix, whether it’s stopping, switching models, or retrying.

Meng: For deployment, this means we shouldn't rely on these trajectory predictors to guide our actions unless they provide more than just issue and model information.

Tom: The implication is that the way we evaluate these tools needs to change; we need methods that test those three core assumptions about run-specific learning properly.

Jane: It’s a call for prediction that actually helps us act on the failure, not just predicts it passively. That's what this paper suggests before we move on to another interesting piece of research.

NC State University

cs.SE

Submitted: 2026-11-05

Updated: 2026-11-05

Comments: 5 pages, 2 figures, 2 tables

Code: https://github.com/SWEagent/mini-swe-agent

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: Trajectory-based failure predictors for software engineering (SE) agents read the partial trajectory of a run to flag whether it is going right or wrong On public SWE-bench Verified trajectories…

Key concepts

Trajectory-based Predictors
These are tools that analyze only the partial sequence of an agent's work (its commands and outputs) as it runs. They are used to predict if a run will fail before it finishes, allowing the system to take action early, like stopping or switching models.
Issue Difficulty vs. Model Ability
The study found that predictors learn issue difficulty (how hard the problem is) and model ability (how good the specific model is at that task). These factors explain most of a predictor's success, showing it learns about the task and the tool, not unique run history.
Oracle Gap
This refers to the difference in performance between different models on individual issues. The paper argues this gap doesn't prove specialization is useful for switching; choosing a better model for each issue from past runs offers no measurable benefit on new runs.

Terminology

Summary

Trajectory-based failure predictors for software engineering (SE) agents read the partial trajectory of a run to flag whether it is going right or wrong On public SWE-bench Verified trajectories, where each model attempts each issue once, two run-independent factors dominate predictor performance Beyond difficulty and ability, what predictors gain from the trajectory is statistically equivalent to zero. p. 0

The Problem with Current Predictors

Current predictors cannot tell the causes apart, as they mostly predict the issue and the model, not the run. The paper questions three assumptions that current practice takes for granted regarding failure prediction. p. 0 These assumptions are that a predictor with a high AUROC has learned run-specific information, that the oracle gap reflects issue-specific strengths that switching can exploit, and that an alarm marks a bad run so that acting on it pays off. p. 0 Standard evaluations cannot test these assumptions because predictors are usually scored with metrics pooled over many issues and models. p. 0

Findings on Predictor Performance

The study finds that predictors learn mostly the issue and the model, not the run. Issue difficulty explains pooled AUROC, which is almost entirely a cross-issue measure. p. 2 Model ability explains within-issue AUROC, where knowing the model answers it better than the trajectory-based predictors do. p. 2 p. 2 The authors show that adding a predictor’s score to issue difficulty and model identity changes pooled AUROC by an amount equivalent to zero in all configurations. p. 2

The Oracle Gap and Model Switching

The oracle gap is not evidence of specialization, as the best single model solves 76.8% of the issues, whereas some model solves 86. p. 28%, an oracle gap of 10. p. 00 pp. The paper shows that choosing a model for each issue from past runs brings no measurable gain on new runs. p. 1 Specialization is real but sparse, accounting for only 4. p. 36% of the variation in outcomes across repeated runs. p. 3

The Alarm and Retrying Action

Alarms cannot tell a bad run from a hard issue or a weak model. A high score thus signals a hard issue or a weak model, where a retry is less likely to succeed. p. 3 The authors call for failure prediction that explains why a run fails so that each failure is met with the action that can fix it. p. 1

Future Directions

Future work should focus on building predictors that tell why a run fails, separating a hard issue, a weak model, and a bad run. This involves collecting repeated runs of recent models across more benchmarks and scaffolds so that run-to-run variation can be separated from issue difficulty and model ability. p. 3 The goal is to evaluate predictors in a closed loop at equal budgets, counting false alarms and the cost of monitoring, and judging each by the decisions it improves rather than by its pooled AUROC. p. 4

REFERENCES

[1] C. E. Jimenez, J. p. 4 Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. p. 4 Narasimhan, “SWE-bench: Can language models resolve real-world GithHub issues? p. 4” in International Conference on Learning Representations, 2024, pp. p. 4 54 107–54 157

[2] OpenAI, “Introducing SWE-bench verified,” https://openai.com/index/introducing-swe-bench-verified/, 2024, accessed: 2026

[3] C. Wang, Y. Lyu, J. He, Z. Yang, C. Zhong, Y. Harel, and D. p. 4 Lo, “FailFast, Restart-Smart: Early failure prediction and restart for SWE agentic tasks,” arXiv preprint arXiv:2608. p. 403222, 2026

[4] T. Mehtiyev and W. p. 4 Assunc¸ao, “Beyond resolution rates: Behaviors of coding agent success and failure,” arXiv preprint arXiv:2604. p. 402547, 2026

[5] D. Pham, K. Katevas, A. p. 4 S. Shamsabadi, and H. p. 4 Haddadi, “AgentStop: Terminating local AI agents early to save energy in consumer devices,” in Proceedings of the ACM Conference on AI and Agentic Systems Association for Computing Machinery, 2026

[6] Y. Shi, Z. Sun, J. Dong, C. Wan, D. Lo, and X. Gu, “EarlyEval: Cheaper agent evaluation via early outcome prediction,” arXiv preprint arXiv:2609. p. 402783, 2026

[7] N. Kim and S. Yoo, “Atropos: Improving cost-benefit trade-off of LLMbased agents under self-consistency with early termination and model hotswap,” Proceedings of the ACM on Software Engineering, vol. p. 4 3, no. ISSTA, pp. 1492–1512, 2026

[8] S. Son, S. Yoon, J. Tang, S. Wang, L. Wolf, and I. p. 4 Bogunovic, “SWERouter: Routing in multi-turn agentic software engineering tasks,” arXiv preprint arXiv:2607. p. 400053, 2026

[9] G. Rasch, Probabilistic models for some intelligence and attainment tests. p. 4 Chicago: MESA Press, 1993

[10] P. Rodriguez, J. p. 4 Barrow, A. Hoyle, J. P. Lalor, R. Jia, and J. p. 4 BoydGraber, “Evaluation examples are not equally informative: How should that change NLP leaderboards? p. 4” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021, pp. p. 4 4486–4503

[11] C. Ge, D. Kryvosheieva, D. p. 4 Fried, U. Girit, and K. p. 4 Hariharan, “Agent psychometrics: Task-level performance prediction in agentic coding benchmarks,” arXiv preprint arXiv:2604. p. 400594, 2026

[12] S. Yao, N. Shinn, P. Razavi, and K. p. 4 Narasimhan, “τ-bench: A benchmark for tool-agent-user interaction in real-world domains,” in The Thirteenth International Conference on Learning Representations, 2025, pp. p. 4 74 824–74 876

[13] B. Brown, J. Juravsky, R. p. 4 Ehrlich, R. p. 4 Clark, Q. V. Le, C. Re, and ´ A Mirhoseini, “Large language monkeys: Scaling inference compute with repeated sampling,” arXiv preprint arXiv:2407. p. 421787, 2024

[14] M. EsfandyariDoulabi, L. p. 4 Arkoh, B. Tadesse, V. p. 4 Patel, M. Sharma, M. d’Amorim, and W. p. 4 Assunc¸ao, “Disentangling task difficulty from run-level failure in agent failure prediction,” arXiv preprint arXiv:2610. p. 405572, 2026

[15] K. Zhang, W. Yao, Z. Liu, Y. Feng, Z. Liu, R. R. N. Murthy, T. Lan, L. Li, R. Lou, J Xu, B Pang

[16] K. Ruan, Z Huang, Z Zhou, Q Wei, J Lin

[17] A. Silva, H. Tu, and M. Monperrus, “Latent programming horizons in coding agents,” arXiv preprint arXiv:2607. p. 405188, 2026

[18] SWE-agent contributors, “mini-swe-agent,” https://github.com/SWEagent/mini-swe-agent, accessed: 2026

[19] tarsur385, “swebench-verified-trajectories,” https://huggingface.co/datasets/tarsur385/swebench-verified-trajectories, accessed: 2026

[20] ——, “swev-trm-trajectories-25models,” https://huggingface.co/datasets/tarsur385/swev-trm-trajectories-25models, accessed: 2026

[21] D. J. Schuirmann, “A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability,” Journal of Pharmacokinetics and Biopharmaceutics, vol. p. 4 15, no. 6, pp. 657–680, 1987

[22] S. Holm, “A simple sequentially rejective multiple test procedure,” Scandinavian Journal of Statistics, pp. p. 4 65–70, 1979

[23] G. Chamberlain, “Analysis of covariance with qualitative data,” The Review of Economic Studies, vol. p. 4 47, no. 1, pp. 225–238, 1980

[24] A. Birnbaum, “Some latent trait models and their use in inferring an examinee’s ability,” Statistical Theories of Mental Test Scores, 1968

[25] Nebius, “SWE-agent-trajectories,” https://huggingface.co/datasets/nebius/SWE-agent-trajectories, accessed: 2026

--- Page 1 ---

Hard Issue, Weak Model, or Bad Run? Rethinking Failure Prediction for Coding Agents

Xingyue Shi NC State University Raleigh, NC, USA xshi22@ncsu.edu Wesley K. p. 0 G. Assunc¸ao˜ NC State University Raleigh, NC, USA wguezas@ncsu. p. 0edu

Abstract—Trajectory-based failure predictors for software engineering (SE) agents read the partial trajectory of a run to flag whether it is going right or wrong If it is going wrong, the run can be stopped, retried, or switched to another model We ask whether these predictors learn anything about the run itself.

I. INTRODUCTION

Software engineering (SE) agents resolve repository issues by inspecting code, editing files, and running tests [1], [2] Agent runs are usually long and often fail [3], [4] A growing line of work therefore predicts failure while a run is still in progress and acts on the prediction: it stops the run to save tokens [5], [6], restarts the same model [3], or hands the issue to another model [7], [8]. We focus on trajectory-based predictors, which read only what the agent has produced so far (its reasoning, commands, and tool outputs) and therefore apply to closed frontier models. p. 0 A prediction can guide the choice of action only if it explains why the current run is likely to fail, using evidence from that run. p. 0 A run can fail for three different reasons: the issue may be hard, so that most models fail it; the model may be weak, so that it fails many issues; or this particular Easy Hard (a) Hard issue? p. 0 P(success) A B Any AA 0. p. 05 1.70.60.88.91 (b) Weak model? p. 0 Flagged Missed (hard) 0. p. 05 1.10.90

Fig. 1. A toy world in which each outcome depends only on the issue, the model, and a coin flip; values are probabilities, not empirical results (a) Success on easy and hard issues. p. 0 (b) Success with model A, model B, either model (Any), and A run twice (AA). p. 0 (c) Retry success after a failure that a difficulty score flags or misses

How the training signal is built

A predictor reads a run’s trajectory after t ∈ 5, 20, 40 completed steps, either the last eight steps (R8) or the full prefix (ALL) and outputs a failure score. Earlier observation points leave more runs to act on but give less evidence, and the two windows contrast recent context with the full history. p. 1 The authors train TF-IDF logistic regression, a choice also used in [14], with issue-grouped five-fold cross-validation. p. 1 To check that the findings are not specific to a simple text model, they also fine-tune Qwen3-0. p. 16B with LoRA on one fold, and pair each neural predictor with a logistic model on the same truncated text. p. 1

Where the gains come from

Issue difficulty explains pooled AUROC, which is almost entirely a cross-issue measure. Model ability explains within-issue AUROC, where knowing the model answers it better than the trajectory-based predictors do. The authors show that adding a predictor’s score to issue difficulty and model identity changes pooled AUROC by an amount equivalent to zero in all 18 configurations. p. 2

Conclusion

The authors conclude that they mostly predict the issue and the model, not the run. The evidence for acting on their alarms, whether by switching or by retrying, inherits the same confusion. p. 1 They call for failure prediction that explains why a run fails, so that each failure is met with the action that can fix it. p. 1

--- Page 2 ---

Fig. 1 shows how all three assumptions can fail in a toy world where each run is a coin flip whose success probability is set only by the issue and the model (a) A high pooled AUROC does not require learning about the run. p. 1 (b) An oracle gap does not require issue-specific strengths. p. 1 (c) An alarm need not mark a bad run. p. 1

Testing the Assumptions

The authors test whether predictors add anything beyond issue difficulty and model ability. They require the 90% interval of its AUROC gain to lie within a pre-. p. 1 They compare it with run-blind references for issue difficulty and model ability (Table I) and test what they add beyond both. p. 2

The authors test whether predictors add anything beyond issue difficulty and model ability. They require the 90% interval of its AUROC gain to lie within a pre-

The authors test whether predictors add anything beyond issue difficulty and model ability.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Redesign failure predictors to explicitly distinguish between three failure causes: hard issue, weak model, or bad run. This allows for a prediction that explains why a run fails, enabling specific actions like stopping, switching to a stronger model for weakness, or retrying for bad runs.

  2. Implement evaluation methodologies that test the three core assumptions of current predictors: whether they learn run-specific information beyond issue difficulty and model ability, whether the oracle gap reflects issue-specific strengths exploitable by switching, and whether alarms mark bad runs that a retry can fix.

  3. Develop a system where failure prediction is linked directly to an appropriate action. This means designing predictors that output not just a failure score, but also an indication of which cause led to the predicted failure, guiding the agent to the action that can fix it.

Sources

Related papers