Update from Hell: Can Coding Agents Survive Hidden Breakage in Dependency Upgrades?

arXiv:2608.30300 · cs.SE · Submitted 2026-08-31 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Update from Hell".

Jane: Here's the gist of the paper. Modern software systems rely heavily on thirdparty dependencies, but upgrading those dependencies remains a costly maintenance activity.

Tom: The best completed configuration solves only 104/203 tasks (51.2%), with substantial variation across agent harnesses, models, and ecosystems.

Jane: Benchmark Construction The paper introduces DEPBENCH, an executable repository-level benchmark that isolates dependency-upgrade repair through patch decomposition, held-out test patches, and a four-state oracle across five package ecosystems.

Tom: This benchmark is built from merged pull requests opened by dependency-update bots but does not treat a merged pull as a task directly, instead subjecting each candidate to patch decomposition.

Paper summary: Tom: So we're looking at this paper called "Update from Hell: Can Coding Agents Survive Hidden Breakage in Dependency Upgrades?". Basically, they’re asking if coding agents can actually handle the messy reality of updating third-party libraries when those updates break things in unexpected ways.

Jane: That sounds like a really practical problem. So the core idea seems to be that dependency upgrades don't always just keep things working smoothly; sometimes they change how things are supposed to work, like breaking function signatures or type systems, and developers have to manually fix the code themselves.

Lu: Right, and the paper focuses on how coding agents do when that breakage happens without clear instructions on how to fix it. They set up this benchmark called DEPBENCH.

Meng: A benchmark? What exactly is this DEPBENCH thing they built? Is it just a list of problems they made up, or something more concrete? We need to know if this is actually testable outside of their specific setup.

Tom: It’s not just a list of problems. They’re taking real dependency update pull requests and breaking them down into smaller pieces—like separating the manifest changes from the source code fixes—to isolate exactly when a failure is due to the dependency upgrade itself.

Jane: That separation sounds crucial. So instead of just giving an agent a finished fix, they give it separate parts: what changed in the files, what was written to repair it, and a hidden test case that shows where things went wrong.

Lu: Exactly. They use a four-state oracle to check if the repair actually fixes the problem caused by the upgrade and not just some unrelated change in the code. This makes sure they are only testing for true dependency upgrade failures, according to this paper on "Update from Hell: Can Coding Agents Survive Hidden Breakage in Dependency Upgrades?".

Meng: If it’s isolating that causal link, that sounds like a good start, but how do you handle the sheer variety of these dependencies? There are so many ecosystems out there.

Lalam: The paper tests this across five different package ecosystems. They mention npm, yarn, Cargo, and Maven as examples of where they tested these tasks >

Tom: That variety is a big deal because it shows that the difficulty isn't tied to just one type of software setup. The main point they make is that even with the strongest configurations, most agents don't solve all the problems.

Jane: So what’s this main failure mode they found? What’s holding back these coding agents when faced with these hidden breakage scenarios?

Lu: The dominant failure mode is incomplete repository-wide migration. Agents often find the right spot to fix a local piece of code, but they fail to update the contract across everything else, like wrappers or generated artifacts >

Paper summary: Meng: So it’s not just fixing the immediate error; it’s failing to maintain consistency throughout the entire codebase after an upgrade. That sounds like a nightmare for production deployment.

Tom: It really is. The paper points out that hard tasks usually combine a tricky ecosystem, a big set of tests, and they often fail to propagate that dependency upgrade contract through all the layers of the repository >

Jane: And what about those tests themselves? Are they just standard unit tests, or do they test specific behaviors related to how the library changed?

Lu: Most retained tests encode concrete upgrade-induced behavior rather than just random changes. They focus on updating expectations or mocks specifically for the upgraded dependency >

Meng: That makes sense. If a test is only checking if a function runs, but the library changed that function’s return type, it won't catch it unless the test expectation itself was updated to match the new type >

Tom: That’s what they found is hard. The difficulty isn't just finding *a* fix; it's making sure that fix carries through every single layer of the system where that dependency is used >

Jane: So if we look at why this matters for us in the real world, what does this mean for how we trust these AI coding assistants?

Lu: It means an agent needs to reason beyond just editing a few lines of code. It has to validate semantic consistency across the whole repository against the upgraded library's new rules >

Meng: From a deployment standpoint, if an agent misses that one wrapper or one generated file update, your entire system can fail in production because it’s not behaving as expected after the upgrade >

Tom: The paper suggests that effective upgrade agents need to reason about how the dependency contract flows through every layer of the codebase, not just where the direct API call is made >

Jane: So they are pushing for a level of reasoning that goes deeper than simple local edits, which is a big shift in what we expect from these tools.

Lu: They conclude that DEPBENCH provides two hundred three oracle-validated tasks across five ecosystems to show this difficulty > <ref:2608.30300#pg1>

Meng: It’s good they have those tests. But the paper also points out their limitation: making a truly repository-consistent source repair for the upgraded dependency is still very hard >

Tom: So, even with this detailed benchmark, we still see significant variation in how well different agents perform across those tasks >

Jane: It sounds like the next step isn't just more tests, but developing an agent that can truly reason about that full-repository semantic consistency.

Conclusion: Tom: So we’ve been looking at how AI agents handle those messy dependency upgrades in this paper called "Update from Hell: Can Coding Agents Survive Hidden Breakage in Dependency Upgrades?"

Jane: Right, so the main thing is that these agents aren't just fixing what's broken on the surface of a file. They’re dealing with hidden breakage caused by library updates that ripple through the whole system.

Lu: The authors built this benchmark, DEPBENCH, which takes real dependency update fixes and breaks them down to see if an AI can repair them correctly across different software setups.

Meng: But what’s the actual finding? Is it like, "the agent failed eighty percent of the time"? I need a number that means something when I think about deploying this stuff.

Tom: The results show that even the best configurations struggle. The top configuration only solved one hundred four out of two hundred three tasks, which is about fifty-one percent success rate on its own.

Jane: That’s tough for an AI system to get right when it comes to something as complex as dependency migration. What exactly is this "hidden breakage" they keep talking about?

Lalam: It's when the agent finds the right place in the code to fix a local issue, but it forgets to update all the related things like wrappers or generated files that depend on that library.

Lu: That incomplete repository-wide migration is what trips them up most often. The agents see one piece of the puzzle and miss how it connects to everything else in a large codebase.

Meng: From my side, if an AI misses that connection, it doesn't just fail locally; the whole production deployment breaks because the contract isn't consistent across all layers.

Tom: And this difficulty isn't just about one type of software. The paper shows it’s ecosystem-dependent; it’s much harder in some dependency systems than others, like npm or Cargo versus Java.

Jane: So what does this mean for us who are building these AI assistants? It means we can't just expect them to handle deep, architectural changes without a lot of extra guidance.

Lu: The authors say that effective upgrade agents need to reason way beyond simple text edits. They have to check if the entire system still makes sense after the library contract has changed.

Meng: So for me, it means we can't just ship an agent and hope it handles all these complex semantic requirements automatically when things change under the hood.

Tom: The paper sets up this benchmark to prove that simply having a strong AI model isn't enough; you need a method to test if the fix is actually complete across the whole repository.

Jane: It’s pushing us toward needing agents that understand how data and code interact at a much deeper level than what we see on the surface.

Lu: The next step, according to them, is showing that agents can validate this entire repository-wide semantic consistency, not just patch local errors.

Meng: If we can get an agent to handle that level of reasoning reliably across different ecosystems, then it could actually make dependency management much less painful for developers.

Zijian Luo, Runzhi He, Pengfei Gao, Yu Kang, Zeqi Lin, Minghua Ma, Qingwei Lin, Saravan Rajmohan, Yongqiang Tian

Monash University · Peking University (Department) · §Microsoft

cs.SE

Submitted: 2026-08-31

Updated: 2026-11-09

Code: https://github.com/harvard-lil/js-wacz

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 87/100

The gist: Modern software systems rely heavily on thirdparty dependencies, but upgrading those dependencies remains a costly maintenance activity.

Key concepts

DEPBENCH
An executable benchmark designed to isolate dependency-upgrade repair. It uses merged pull requests and patch decomposition across five package ecosystems to create tasks that test agents' ability to fix broken dependencies.
Oracle-Clean Release Tasks
Tasks generated by a pipeline combining bot mining, manual/LLM audits, and four-state oracle validation. These tasks ensure the ground truth for testing is accurate and free from errors, providing reliable targets for agent evaluation.
Incomplete Repository-Wide Migration
The dominant failure mode where agents fix a local dependency issue but fail to propagate the necessary changes through all parts of the repository, such as wrappers, types, or generated artifacts. This means the upgrade contract is not consistently applied across the entire codebase.
Propagation Depth
A measure of how far an agent must carry an upgraded API or semantic requirement. Hard tasks require agents to reason beyond simple local edits and ensure the upgraded dependency's changes are reflected in every layer exercised by hidden test cases.

Terminology

Summary

Modern software systems rely heavily on thirdparty dependencies, but upgrading those dependencies remains a costly maintenance activity. p. 0 The best completed configuration solves only 104/203 tasks (51. p. 02%), with substantial variation across agent harnesses, models, and ecosystems. p. 0

Benchmark Construction

The paper introduces DEPBENCH, an executable repository-level benchmark that isolates dependency-upgrade repair through patch decomposition, held-out test patches, and a four-state oracle across five package ecosystems. This benchmark is built from merged pull requests opened by dependency-update bots but does not treat a merged pull as a task directly, instead subjecting each candidate to patch decomposition. p. 0 The construction pipeline combines dependency-update bot mining, path-based hunk classification, manual and LLM-assisted audit, leakage controls, and four-state oracle validation to produce oracle-clean release tasks. p. 1

Evaluation Protocol

The evaluation phase uses these tasks to test coding agents in a full-upgrade setting where the agent starts from the base repository state and an upgrade instruction, while the developer repair and held-out test patch remain hidden until task verification. Each release task packages information for reproducible evaluation, including a pinned repository environment, an agent-facing upgrade instruction, dependency-change artifacts, the developer repair from the original pull request, a held-out test patch, a task verifier, and metadata. p. 2 The agent receives neither patch-derived repair descriptions nor the developer repair or held-out tests. p. 1

Performance Findings

The main result is that dependency-upgrade repair remains difficult even for the strongest completed configurations (Table II). The best configuration solves 104/203 tasks with Codex and GPT-5. p. 55; Copilot CLI with GPT-5. p. 45 solves 99/203 and OpenCode with GPT-5. p. 55 solves 80/203. p. 0 The dominant failure mode is incomplete repository-wide migration, where agents often identify a relevant dependency surface yet fail to propagate the upgraded dependency contract through wrappers, types, fixtures, generated artifacts, or behavioral outputs. p. 1 Difficulty is highly ecosystem-dependent (Table III). p. 5

Failure Analysis

The main obstacle in non-pass runs with a completed verifier outcome is making a repository-consistent source repair for the upgraded dependency. The most common listed agent-side root cause is incomplete migration, where agents often find the right migration area but leave wrappers, types, generated state, fixtures, or call sites inconsistent with the upgraded dependency. p. 8 Hard tasks tend to combine a difficult ecosystem, a broad or mixed-behavior hidden test patch, a repair that spans multiple repository layers, and agents that often fail to propagate the observed dependencyupgrade contract across the whole repository. p. 9

Test Behavior Encoding

Most retained tests encode concrete upgrade-induced behavior rather than arbitrary test churn, typically by updating expectations, mocks, snapshots, compile targets, or adding focused coverage for the upgraded dependency. Mixed and correction-heavy tests are the hardest primary category because they often force the agent to update several semantic surfaces at once. p. 8 The common failure across these categories is an incomplete repositorywide adaptation to the upgraded library. p. 7

Task Difficulty Factors

Task difficulty is not explained by a single factor; hard tasks combine ecosystem/tooling friction, rich held-out behavioral oracles, and repository-wide propagation requirements. Ecosystem is the most visible axis, with npm/yarn and Cargo being much harder than Maven/Java for the strongest rows. p. 5 Held-out test behavior is the second axis, where mixed direct-behavior tests are the hardest primary category. p. 8 Agent behavior is a fourth axis, where agents often fail to propagate the observed dependencyupgrade contract across the whole repository. p. 9 The fifth axis is propagation depth, where a plausible local edit is insufficient and the agent must carry the upgraded API or semantic requirement through every repository layer exercised by the hidden oracle. p. 9

Conclusion

DEPBENCH provides 203 oracle-validated, container-native dependency- upgrade tasks across five ecosystems. Effective upgrade agents must therefore reason beyond local API edits and validate repository-wide semantic consistency. p. 9 The benchmark establishes upgrade causality and uses post-upgrade tests rather than visible-test success alone as evidence of a complete repair. p. 6

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Create an executable repository-level benchmark called DEPBENCH that isolates dependency-upgrade repair through patch decomposition and a four-state oracle across five package ecosystems. This allows for causal validation of agent performance rather than relying on general issue repair datasets like SWE-bench or BUMP.

  2. Develop a construction pipeline combining dependency-update bot mining, path-based hunk classification, manual and LLM-assisted audit, leakage controls, and four-state oracle validation to produce oracle clean release tasks. This pipeline ensures that retained tasks are causally tied to the dependency upgrade and descriptions do not leak solution hints.

  3. Evaluate mainstream coding agents on DEPBENCH using a containerized evaluation harness that runs agents in a full-upgrade setting while applying hidden test patches via a task verifier to compute the final pass/non-pass outcome. This setup allows for reproducible testing of agent performance under realistic maintenance needs, showing how configuration, ecosystem, and failure modes shape results.

  4. Analyze agent behavior across four axes: external documentation lookup, local dependency API inspection, test running evidence, and explicit breaking-change hypotheses to understand where agents fail during repair. This reveals that incomplete migration is the main obstacle because agents often find the right dependency-upgrade surface but do not propagate the change through all wrappers, generated artifacts, lockfiles, types, or call sites.

  5. Improve agent performance by focusing on repository-wide semantic consistency rather than local API edits. Agents must be trained or guided to reason beyond local API edits and validate repository-wide semantic consistency to successfully solve hard tasks that require connecting a dependency version change to the observable output encoded by hidden tests.

Sources

Related papers