Hard Issue, Weak Model, or Bad Run? Rethinking Failure Prediction for Coding Agents

summary

The gist

Trajectory-based failure predictors for software engineering (SE) agents read the partial trajectory of a run to flag whether it is going right or wrong On public SWE-bench Verified trajectories…

In short

This study investigates trajectory-based predictors for software engineering agents to determine if they learn run-specific failure information. Findings show predictors primarily learn issue difficulty and model ability, not the specific run itself. Current alarms fail because they cannot distinguish between a hard issue, a weak model, or a bad run, necessitating new predictors that explain the root cause to enable targeted fixes.

Key concepts

Trajectory-based Predictors
These are tools that analyze only the partial sequence of an agent's work (its commands and outputs) as it runs. They are used to predict if a run will fail before it finishes, allowing the system to take action early, like stopping or switching models.
Issue Difficulty vs. Model Ability
The study found that predictors learn issue difficulty (how hard the problem is) and model ability (how good the specific model is at that task). These factors explain most of a predictor's success, showing it learns about the task and the tool, not unique run history.
Oracle Gap
This refers to the difference in performance between different models on individual issues. The paper argues this gap doesn't prove specialization is useful for switching; choosing a better model for each issue from past runs offers no measurable benefit on new runs.

Terminology used across episodes

This episode discusses

The paper

Hard Issue, Weak Model, or Bad Run? Rethinking Failure Prediction for Coding Agents · Read on arXiv

NC State University

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Hard Issue, Weak Model, or Bad Run? Rethinking Failure Prediction for Coding Agents".

Jane: Here's the gist of the paper. Trajectory-based failure predictors for software engineering (SE) agents read the partial trajectory of a run to flag whether it is going right or wrong On public SWE-bench Verified trajectories, where each model attempts each issue once, two run-independent factors dominate predictor performance Beyond difficulty and ability, what predictors gain from the trajectory is statistically equivalent to zero.

Tom: The Problem with Current Predictors Current predictors cannot tell the causes apart, as they mostly predict the issue and the model, not the run.

Jane: The paper questions three assumptions that current practice takes for granted regarding failure prediction.

Tom: These assumptions are that a predictor with a high AUROC has learned run-specific information, that the oracle gap reflects issue-specific strengths that switching can exploit, and that an alarm marks a bad run so that acting on it pays off.

Title and authors: Jane: So they're suggesting we look at three assumptions about current predictors and see if they hold up when you test them rigorously.

Lu: The paper tests three main assumptions: first, that a predictor with high performance has learned run-specific information, second, that the gap between models is due to issue-specific strengths we can exploit by switching models, and third, that an alarm actually points to a bad run worth retrying.

Tom: That's the core critique they're making against how we currently evaluate these things—that standard evaluations pool everything together and don't test these assumptions properly.

Meng: So if we can't prove those assumptions are true, then their high performance scores might just be misleading noise based on issue difficulty or model ability.

Lu: They show that a high pooled AUROC doesn't require learning about the run because it just points to issue difficulty, which is a reference score that never looks at the specific run itself and reaches zero point nine two AUROC <ref:2611.08285#pg1,a reference score that never looks at the>.

Jane: That zero point nine two AUROC for issue difficulty is quite high, but it doesn't help us understand why a specific run succeeded or failed in a particular way <ref:2611.08285#pg1>.

Tom: And then there’s the model ability part, where knowing which model ran on an issue predicts failure better than our trajectory predictors do—the predictors largely reproduce that overall ranking of models.

Meng: So even if we know the model is good, our predictor doesn't give us any extra signal to decide what to do next.

Lu: They show that adding the prediction score to issue difficulty and model identity changes pooled AUROC by an amount equivalent to zero in all eighteen configurations they tested.

The paper's summary: Tom: So, the paper’s main takeaway is that trajectory-based failure predictors mostly learn the issue and the model, but not the run itself.

Jane: That means when we get an alarm, it doesn't tell us *why* that specific run failed—it doesn't distinguish between a hard issue and a weak model.

Lu: They show that if you try to claim these predictors are learning something run-specific, the evidence for that idea is extremely weak; it's statistically equivalent to zero.

Meng: From an engineering standpoint, this means we can’t rely on these scores to decide whether we should stop the run or switch models based on a fine-grained prediction of failure.

Tom: The authors argue that alarms cannot reliably tell a bad run from a hard issue or a weak model, which is what they call the alarm problem.

Jane: They call for something different—failure prediction that explains *why* it failed so we can use the right action to fix it, like stopping for tokens saved or retrying with a better model.

Lu: They suggest that we need to build predictors that separate these three distinct failure causes: hard issue, weak model, and bad run.

The paper's improvements: Tom: The improvements they suggest are pretty clear: redesigning the predictors to explicitly distinguish between those three failure causes.

Jane: Instead of one score that just says "it might fail," we need a system that outputs which cause is likely responsible for the predicted failure.

Lu: This would allow for much more targeted action, so you can stop because it’s a bad run, or switch to a stronger model if it’s weak, or retry if it's just a hard issue.

Meng: That moves us from just getting a score to actually having an instruction on what to do next based on the prediction. It makes the decision loop much more useful for production systems.

Tom: Exactly. They want failure prediction linked directly to an appropriate action, meaning the predictor has to output not just a failure score, but also which cause led to that predicted failure.

Jane: That sounds like they are trying to move beyond just predicting success or failure and into prescriptive advice for agentic workflows.

Conclusion: Tom: So, to wrap up the paper "Hard Issue, Weak Model, or Bad Run? Rethinking Failure Prediction for Coding Agents," the main point is that current predictors mostly predict the issue and the model, not what happened in this specific run.

Jane: They found that pooled performance gains from learning about the run itself are statistically equivalent to zero when compared to just looking at issue difficulty and model ability scores.

Lu: They emphasize that we need a new kind of failure prediction that explains *why* a run fails so we can choose the right fix, whether it’s stopping, switching models, or retrying.

Meng: For deployment, this means we shouldn't rely on these trajectory predictors to guide our actions unless they provide more than just issue and model information.

Tom: The implication is that the way we evaluate these tools needs to change; we need methods that test those three core assumptions about run-specific learning properly.

Jane: It’s a call for prediction that actually helps us act on the failure, not just predicts it passively. That's what this paper suggests before we move on to another interesting piece of research.

More episodes

← Home