Where LLM Agents Get Stuck Debugging Databases: Access More Than Reasoning

summary

The gist

Large language model (LLM) coding agents are increasingly asked not only to write code but to debug running systems from live runtime data The agent’s wrong hypotheses were consistent with the…

In short

LLM coding agents struggle to debug production databases because they lack direct access to runtime state. The core finding is that access to production data, not reasoning ability, is the primary bottleneck for fixing concurrency issues like deadlocks. The solution proposed is a forensics layer providing read-only database insights.

Key concepts

Access Gap
The agent cannot query the production database directly; it can only see what's exposed through existing security controls. This lack of direct access prevents it from verifying hypotheses about complex runtime issues like locking and transactional consistency.
Forensics API
A proposed tool that gives the agent read-only access to critical database information, such as lock states and query plans. This would allow the agent to answer specific questions about why a transaction is blocked or what locks are held, moving beyond just reading logs.
False Start
A hypothesis generated by the agent that seems plausible based on available code but turns out to be incorrect when tested against runtime evidence. These false starts often stem from the inability to inspect real-time database state, leading to costly debugging cycles.
Reasoning vs. Access Bottleneck
The study proves that for concurrency-dependent incidents, the limitation is access, not reasoning. The agent's ability to form plausible hypotheses is sound; it fails when it cannot obtain the necessary runtime data to confirm or deny those hypotheses.

Terminology used across episodes

This episode discusses

The paper

Where LLM Agents Get Stuck Debugging Databases: Access More Than Reasoning · Read on arXiv

Gustavo Pinto

Federal University of Parı (UFPA)

Large language model (LLM) coding agents are increasingly asked not only to write code but to debug running systems from live runtime data. We report a field observation of one such agent (Claude Code) as it diagnosed eight database incidents in a production message-dispatch engine. In every incident whose root cause depended on runtime concurrency, the agent announced a cause, shipped a fix, and later walked it back, and each wrong guess cost a deploy-and-observe cycle. Mapping each incident to what the agent lacked, we find that access to production state was the most pervasive bottleneck. The missing access included live lock views, untruncated deadlock reports, read-only execution, and an explicit environment context. An access gap was present in all five concurrency-dependent incidents: alone in one, and in four combined with a reasoning error such as anchoring on an early cause, incomplete exploration of the configuration, or a misreading of the code's transaction semantics. The three incidents that did not depend on concurrency needed nothing beyond what the agent already had. Within the incidents, the agent's wrong hypotheses were consistent with the evidence it could see; they were not fabrications. We argue that the next step for production-debugging agents is a forensics layer that gives the agent the access it lacks inside existing security controls. Such a layer would combine a safe interrogation API, a lock-and-deadlock flight recorder, and a production replay harness.

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Where LLM Agents Get Stuck Debugging Databases".

Jane: Here's the gist of the paper. Large language model (LLM) coding agents are increasingly asked not only to write code but to debug running systems from live runtime data The agent’s wrong hypotheses were consistent with the evidence it could see; they were not fabrications We argue that the next step for production-debugging agents is a forensics layer that gives the agent the access it lacks inside existing security controls The core finding regarding agent limitations Access to production state was identified as the most pervasive bottleneck on an LLM coding agent’s diagnosis An access gap was present in all five concurrency-dependent incidents In one of them it was the only limit In the other four it combined with a reasoning error, and in each of those the error was corrected only through production evidence that the agent could not obtain directly The three incidents that did not depend on concurrency needed nothing beyond what the agent already had Analysis of incident patterns The study analyzed eight database incidents over a ten-day window, where six of them converged on the locking and transactional-consistency layer Table 2 summarizes the result The matrix splits cleanly along concurrency The three incidents that did not depend on concurrency needed nothing the agent did not already have Implications for future tooling and research The paper proposes a concrete agenda for tool builders, including a safe database forensics API, a lock-and-deadlock flight recorder, and a production replay harness The agent’s largest gap is that it cannot ask the production database anything beyond a read-only projection Standardizing such an API for agents is a concrete deliverable Limitations and future work The study is limited by being a retrospective field observation of a single agent on a single system Future work suggests turning the observation into a controlled comparison, crossing model capability with access The most direct next step is to turn our observation into a controlled comparison, crossing model capability with access The reasoning × access matrix makes the pattern explicit: every concurrency-dependent incident had an access gap, and no incident was limited by reasoning alone This finding changes where the next investment should go The implication runs against the assumption that the model is the ceiling The first bottleneck of today’s debugging agents is access, and it is the one that tooling can remove Page 5 --- Access More Than Reasoning The agent’s largest gap is that it cannot ask the production database anything beyond a read-only projection: “I do not have access to the database nor exec in prod” What it needs is a forensics API: a read-only surface exposing pg locks, pg stat activity, query plans, and row-level state under the safety guarantees that already protect production, but rich enough to answer “which statement holds this lock?” This is the runtime counterpart of what SWE-agent calls the agent–computer interface (Yang et al., 2024), and of Panda’s grounding and affordance components (Singh et al., 2024) Page 8 --- Access More Than Reasoning The matrix splits cleanly along concurrency The three incidents that did not depend on concurrency needed nothing the agent did not already have Page 9 --- Access More Than Reasoning The reasoning × access matrix makes the pattern explicit: every concurrency-dependent incident had an access gap, and no incident was limited by reasoning alone This finding changes where the next investment should go The implication runs against the assumption that the model is the ceiling Page 9 --- Access More Than Reasoning The agent’s diagnostic reasoning was largely sound: it formed plausible hypotheses, noticed its own contradictions, and retired wrong hypotheses through measurement Yet every incident whose root cause depended on runtime concurrency began with a false start The agent had the entire codebase, yet for these failures the code was not enough Page 9 --- Access More Than Reasoning The agent cannot run queries against the production database, so it must infer row-level state from logs: “I do not have access to the database nor exec in prod” Locks and deadlocks are transient by nature: by the time the agent looks, the lock is gone, and the deadlock detail arrives truncated “the DETAIL is STILL truncated...

Tom: at the SOURCE” A deadlock happens when two transactions each hold a lock the other needs The error’s DETAIL is the lock graph It names which transaction is waiting for which lock and which transaction holds it, and it is precisely what identifies the deadlock cycle Page 9 --- Access More Than Reasoning The information was not absent; it was out of reach Even obtaining access cost a deploy A full deadlock report existed in the Postgres server’s own log—which has log lock waits enabled and logs the complete deadlock report—but the agent had access only to the application log Page 9 --- Access More Than Reasoning The agent cannot run queries against the production database, so it must infer row-level state from logs It must guess, ship, and watch Each wrong hypothesis then costs a full deploy–observe cycle before it can be refuted The indexed Datadog logs that would settle a diagnosis sit behind a login the agent cannot pass “requires login” Page 9 --- Access More Than Reasoning A false start, as we use the term, is a hypothesis consistent with everything the agent had seen that turns out to be wrong A false start, as we use the term, is a hypothesis consistent with everything the agent had seen that turns out to be wrong A false start, as we use the term, is a hypothesis consistent with everything the agent had seen that turns out to be wrong Page 9 --- Access More Than Reasoning The two call for different remedies: hallucination calls for better grounding, and false starts call for better access This reading is consistent with evidence that current LLMs detect data races and deadlocks reliably when the whole program is in view Our agent could read the whole repository, but at about 406,000 lines it had to choose what to read Page 9 --- Access More Than Reasoning Within the incidents, our agent’s wrong hypotheses were not fabrications; they were explanations consistent with the evidence it could see Within the standard architecture of LLM-based agents (planning, memory, tools, action) (Wang et al., 2024a), all of these strands point to the tool and perception layer, at least as much as to the reasoning core The agent cannot run queries against the production database Page 9 --- Access More Than Reasoning The agent’s wrong hypotheses were not fabrications; they were explanations consistent with the evidence it could see Within the standard architecture of LLM-based agents (planning, memory, tools, action) (Wang et al., 2024a), all of these strands point to the tool and perception layer, at least as much as to the reasoning core The agent cannot run queries against the production database Page 9 --- Access More Than Reasoning The agent’s wrong hypotheses were not fabrications; they were explanations consistent with the evidence it could see Within the standard architecture of LLM-based agents (planning, memory, tools, action) (Wang et al.

Paper summary: Tom: So Jane, we're looking at this paper now called "Where LLM Agents Get Stuck Debugging Databases: Access More Than Reasoning." Basically, they’re testing how well these AI coding agents can fix problems in a live production database. The big idea is that the agent isn't failing because its reasoning is bad; it’s failing because it doesn't have the right access to what’s actually happening inside the system.

Jane: Exactly. The paper says that these agents are being asked not just to write code, but to debug systems using live runtime data, and they observed a specific issue with an agent called Claude Code diagnosing eight database incidents in a message dispatch engine. The core claim is that when the root cause depended on runtime concurrency—that’s when things happen in real time with multiple things happening at once—the agent announced a cause, tried to ship a fix, and then walked it back later.

Lu: What’s really interesting from the abstract is how they mapped each of those incidents to what the agent was missing. They found that access to production state was the most pervasive bottleneck for these agents, and this missing access included things like live lock views, untruncated deadlock reports, read-only execution privileges, and an explicit environment context.

Meng: So it sounds like the model itself isn't the problem; it’s what the model is allowed to see or touch. I mean, if we give a model all the code in a repository, it can reason through logic pretty well, but if it can't query the actual database state, how can it fix concurrency issues?

Lalam: Lalam here. The paper confirms that access to production state was the most pervasive bottleneck for these LLM coding agents (Where LLM Agents Get Stuck Debugging Databases: Access More Than Reasoning). They found an access gap present in all five incidents whose root cause depended on runtime concurrency, and that this missing access was the most pervasive bottleneck (reference <ref:2611.17304#pg1>).

Tom: And what they found is that this access gap showed up alone in one of the incidents, but in four of them, it combined with a reasoning error. That means sometimes the agent had a wrong idea, and then it needed that production evidence to correct itself—evidence it just couldn't get directly.

Paper summary: Jane: That’s a crucial detail. It suggests that when an agent is wrong, its reasoning isn't necessarily flawed; it’s often just based on incomplete information from the perspective of what the agent can see. The agent had the whole codebase, but for those failures, the code wasn't enough (reference <ref:2611.17304#pg9>).

Lu: It points toward a need for a forensics layer. The paper suggests that instead of just improving the reasoning core of the model, we need to give it access it lacks inside existing security controls. They propose things like a safe database forensics API, and even something like a lock-and-deadlock flight recorder (reference <ref:2611.17304#pg2>).

Meng: From an engineering standpoint, that’s what we need to build. Right now, if I want to debug a live issue at scale, I can't just ask the production database questions; I have to rely on logs or guess how the row state is changing. The paper highlights that the agent can't run queries against production because it doesn't have that direct access (reference <ref:2611.17304#pg9>).

Lalam: Lalam confirms that the agent’s largest gap is being unable to ask the production database anything beyond a read-only projection; it says, "I do not have access to the database nor exec in prod" (reference <ref:2611.17304#pg2>). What it needs is this forensics API that exposes things like pg locks and query plans under safety guarantees (reference <ref:2611.17304#pg5>).

Tom: So the research question they ask is really about whether the limiting factor for an LLM coding agent’s diagnosis of production database incidents is its reasoning or its access (reference <ref:2611.17304#pg2>). Their answer, based on this field observation, leans heavily toward access (reference <ref:2611.17304#pg9>).

Jane: And that finding shifts where we should invest our time and resources. It goes against the idea that the model itself is the ceiling for what these agents can do (reference <ref:2611.17304#pg9>). The implication is clear: we need to focus on providing better tools, like standardizing this forensics API for agents, because that’s a concrete deliverable (reference <ref:2611.17304#pg5>).

Lu: What the paper does well is making the pattern explicit using that reasoning times access matrix (reference <ref:2611.17304#pg9>). It shows that every concurrency-dependent incident had an access gap, and no incident was limited by reasoning alone (reference <ref:2611.17304#pg9>).

Meng: I agree with Lu on the pattern recognition part. My concern is the operational cost of building this access layer. The paper mentions that even obtaining that full deadlock report existed in the Postgres server’s own log, but because the agent only had access to the application logs, it couldn't see that detail (reference <ref:2611.17304#pg9>).

Paper summary: Tom: It shows how much information is out of reach even when it exists on the system. The agent has to guess, ship a fix, and watch—which costs a full deploy-and-observe cycle every time it guesses wrong (reference <ref:2611.17304#pg9>). That’s not scalable for production debugging.

Jane: So what does this mean for someone just listening to the show? It means that when we talk about AI fixing real system bugs, we have to stop assuming the model is smart enough to see everything in production automatically (reference <ref:2611.17304#pg9>). We have to acknowledge that the next big investment isn't just a bigger language model, but better tools that give the agent visibility into the live state of things.

Lu: The authors also point out their own limitation, which is important for us to keep in mind. They note that this study was a retrospective field observation of one agent on one system (reference <ref:2611.17304#pg3>). They suggest turning this observation into a controlled comparison, crossing model capability with access, as the most direct next step (reference <ref:2611.17304#pg3>).

Meng: That makes sense. We can’t just rely on one study to tell us everything about what an agent *can* do at scale. It has to be a controlled comparison that tests model capability alongside access capabilities (reference <ref:2611.17304#pg3>).

Lalam: Lalam is pretty sure of this direction too. The paper’s conclusion on reasoning versus access makes the pattern explicit, and that finding changes where the next investment should go (reference <ref:2611.17304#pg9>). The first bottleneck for today’s debugging agents is access, and it is the one that tooling can remove (reference <ref:2611.17304#pg9>).

Tom: So we're moving away from just chasing better reasoning models and toward building infrastructure that gives the agent real eyes inside the production environment. That’s a tangible engineering problem they’ve pointed out (reference <ref:2611.17304#pg5>). It moves debugging from pure guesswork to evidence-based diagnosis, even if the evidence comes from a specialized API (reference <ref:2611.17304#pg5>).

Jane: Right. We’re moving from hoping the model gets it right to making sure the model can see what it needs to get it right (reference <ref:2611.17304#pg9>). This paper, "Where LLM Agents Get Stuck Debugging Databases: Access More Than Reasoning," really hammers home that for AI agents working with live systems, the missing link isn't always the logic; sometimes it's just the right permission.

Conclusion: Tom: So, we're wrapping up on "Where LLM Agents Get Stuck Debugging Databases: Access More Than Reasoning." Basically, this paper shows that for AI coding agents trying to fix live database problems, the real blocker isn't just their brains; it’s what they are allowed to look at.

Jane: Right. The authors are pointing out that these agents have a huge gap because they can't ask the production database anything beyond just reading data. It says they don't have direct access to things like live locks or the full deadlock reports.

Lu: What this means is that when you have concurrency issues, which is when multiple things happen at once, the agent usually fails because it’s missing that real-time state information. The paper shows a pattern where every single concurrency incident had this access gap.

Meng: From my side, it's about deployment costs. If the agent can't just query the database state directly, it has to infer everything from application logs or guess row states. That means every wrong hypothesis requires a full deploy and observation cycle before you can even test something new.

Lalam: I agree with Meng on that cost issue. The research shows that while the AI’s reasoning might be sound, it gets stuck because the evidence needed to correct a false start isn't accessible to it within the existing security controls.

Tom: So, in simple terms, this paper is saying we need to build a forensics layer for these agents. It suggests that giving an agent access to things like pg locks and query plans is more important than just making its language model smarter about logic.

Jane: Exactly. The implication for the world is that the next big investment isn't necessarily in bigger models, but in better tools—specifically, standardized APIs that give agents the right kind of visibility into production systems.

Lu: The authors themselves admit they did this as a field observation on one system, so their next step is to run controlled comparisons between different models and different access levels. They aren't claiming this is a final answer yet.

Meng: I like that honesty in the conclusion. It keeps it grounded in reality for what we can actually build right now instead of making big claims about what future models will do automatically.

Lalam: And so, the core finding remains: reasoning times access is a matrix where every concurrency failure hits an access gap, and that’s where we need to focus our engineering efforts.

More episodes

← Home