Can Failed Trajectories Reveal Where Agents Go Wrong? Agentic Failure Attribution Without Step-Level Labels

arXiv:2610.13105 · cs.LG · Submitted 2026-10-09 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Can Failed Trajectories Reveal Where Agents Go Wrong? Agentic Failure Attribution Without Step-Level Labels".

Jane: The gist Failure attribution for LLM agents aims to identify decisive steps responsible for task failure,

Tom: First, who's behind it and why it matters.

Title and authors: Jane: Speaking of localization, what exactly is this SOAP framework doing? Tom, can you explain the core idea behind how they're trying to find that decisive step?

Tom: Well, SOAP has two main parts. First, they build a spectral reference space from the representations extracted by a frozen language model on those failure trajectories. They use singular value decomposition for that part.

Lu: So they create these orthogonal directions in a spectral reference space based on how the frozen language model represents every step in all the failure trajectories. Then they compute a base error score for each step by measuring its average squared projection onto a selected spectral band and inverting that quantity.

Tom: That sounds like it's exploiting an imbalance where ordinary steps dominate, so the decisive-error steps show lower projection energy within that chosen band.

Meng: Okay, but that sounds mathematical and abstract. How does the second part of SOAP deal with the contamination problem they mentioned? Downstream contamination seems like a big issue for any error scoring system.

Jane: That's where they introduce attention-guided rescoring to combat that downstream contamination. They derive soft dependency graphs from models' attention patterns, showing how much later steps rely on earlier ones.

Lalam: They calculate the dependency weight by aggregating post-softmax attention routed from tokens in step s to tokens in an earlier step s i, and they normalize those masses over the retained predecessors.

Tom: And then they use that to calculate a final rescoring score for an earlier step, S(s i), using the formula (s i) = S(s i) + gamma B i, where B i is the dependency-weighted average of downstream base scores.

Lu: That propagation mechanism promotes an earlier step when anomalous behavior is systematically built upon it, which should help pull the signal back to the source.

Meng: So it sounds like they are using attention to trace failure backward through time, instead of just looking at isolated scores for each step. That’s a clever way to handle that contamination issue.

The paper's summary: Jane: Let's recap the main problem again. The central challenge this paper addresses is identifying the decisive step in failed trajectories without step-level annotations, which is something that is usually very costly and often ambiguous in practice.

Tom: They are specifically studying failure attribution from step-unlabeled failure trajectories, where we know the trajectory failed, but no single step gets a label for being either decisive or non-decisive.

Lu: The paper notes that these failure-conditioned datasets directly expose the behaviors that arise when agents fail, unlike methods that require verified successful trajectories which are scarce.

Jane: And their main contribution is proposing SOAP, which they claim consistently improves decisive-step localization over competitive prompt-based and training-based baselines across different evaluation settings.

Tom: On a specific benchmark called TRACEELEPHANT with the DeepSeek-R1-Distill-Llama model, SOAP improved step-level accuracy by twenty-four point six six percent compared to the strongest baseline they tested <ref:2610.13105#pg3,step-level accuracy by 24.66>.

Meng: That's a significant number improvement on accuracy, which is what we actually care about when we move models into production. But how robust is this improvement across different agent types?

Lalam: They show strong generalization across different backbones and data distributions, even when they constructed their method on a source subset and tested it on a target subset.

Jane: That’s good because it suggests the method isn't just tuned to one specific model or one specific way of failing; it seems to capture a more general signal.

The paper's improvements: Tom: Beyond just getting better results on TRACEELEPHANT, they highlight two key technical contributions in their main summary. First, formulating the localization from step-unlabeled trajectories that bypasses the need for decisive-error annotations or carefully verified successful trajectories.

Lu: They explicitly identify downstream contamination as a central obstacle to this localization problem because later steps can show stronger failure signals just because they inherited an earlier mistake.

Jane: And their second major contribution is proposing SOAP itself, which combines spectral step scoring and attention-guided rescoring to solve this identification challenge.

Meng: So the improvement isn't just a new scoring function; it’s the entire pipeline that handles both the initial error estimation and the propagation of that error backward.

Lalam: They also emphasize that they require neither step-level training labels nor a separate success-only reference set to work, which makes it much more practical for real-world debugging scenarios.

Tom: And on the efficiency front, they say SOAP is lightweight because it doesn't need those extra labels. The computational cost for fitting their SVD on the unlabeled reference set is very low: zero point zero eight seconds on WW-AG and zero point two nine seconds on WW-HC, which is much faster than methods like OAT and StepFinder which can be two to three orders of magnitude slower.

Jane: That speed is important for practical deployment because it doesn't require massive computational overhead just to get a preliminary attribution score.

Conclusion: Jane: So, to wrap up, this paper presents the SOAP framework as an effective way to attribute failures in LLM-based multi-agent systems by scoring steps against a spectral reference and routing late contamination back using attention propagation.

Tom: It's lightweight, it requires no step-level annotation, and it consistently outperforms prompt-based and training-based baselines across benchmarks like WHO andWHEN, CORRECT-ERROR, and TRACEELEPHANT.

Lu: The main implication is that step-unlabeled failure trajectories are a viable source of localization signals for debugging agents without the need for extensive manual annotation.

Meng: For production environments, this means we can get faster feedback on where an agent went wrong without having to halt training or spend days manually annotating failures.

Lalam: I'm sure SOAP is quite confident in its answer because it works across different backbones and data distributions, showing high performance even when constructed on a source subset and evaluated on a target subset.

Tom: So, the full title is "Can Failed Trajectories Reveal Where Agents Go Wrong? Agentic Failure Attribution Without Step-Level Labels". It's a solid framework for finding that decisive step without needing those costly labels.

Jane: A very practical and efficient approach to debugging agentic systems that seems highly relevant right now. We'll take a quick break before we talk about some of the other papers we looked at today.

Nguyen-Thanh, Junlin Fang, Bing Li, Zhen Fang, Sean Du

Nanyang Technological University · Australian Artificial Intelligence Institute

cs.LG

Submitted: 2026-10-09

Updated: 2026-10-09

License: http://creativecommons.org/licenses/by/4.0/

The gist: The gist Failure attribution for LLM agents aims to identify decisive steps responsible for task failure, which is critical for debugging and improving agentic systems How it works The paper

Key concepts

Failure Attribution: This is the process of identifying the exact action or step an agent took that led to a task failing. The goal is debugging and improving agents by knowing precisely where the error occurred in their execution path.
SOAP Framework: SOAP stands for Spectral scOring with Attention-guided Propagation. It is a method designed to find the decisive error without needing step-level labels or success data. It works by first scoring steps based on their representation in a spectral space and then using attention patterns to propagate failure evidence backward.
Spectral Reference Space: This involves taking representations of all steps from failed trajectories and using Singular Value Decomposition (SVD) to create a new, organized mathematical space. Steps are then scored based on how much their representation projects onto specific directions in this space, helping to identify the most anomalous or decisive steps.
Attention-guided Rescoring: This technique uses the attention patterns within the LLM's model to determine dependencies between steps. It calculates how much a later step relies on an earlier one and uses this dependency weight to adjust scores, ensuring that downstream errors are correctly traced back to their likely upstream sources.

Summary

The gist Failure attribution for LLM agents aims to identify decisive steps responsible for task failure, which is critical for debugging and improving agentic systems

How it works

The paper introduces SOAP (Spectral scOring with Attention-guided Propagation), an effective framework for decisive-error localization from step-unlabeled failure trajectories SOAP consists of two main components: first, constructing a spectral reference space from step representations extracted by a frozen language model on the failure trajectories to assign each step a base error score Second, addressing downstream contamination through dependency-aware rescoring by deriving soft dependency graphs from models’ attention patterns and propagating downstream error evidence toward likely upstream sources

Key Challenges Addressed

The research addresses the challenge of identifying the decisive step in failed trajectories without step-level annotations, which is costly and often ambiguous A central obstacle identified is downstream contamination, where later steps may exhibit stronger failure signals because they inherit an earlier mistake, causing independent per-step scorers to systematically localize failures too late The paper formalizes the problem by assuming access to a disjoint reference collection of failed trajectories, modeling the marginal distribution of reference steps as an unlabeled contamination mixture

SOAP Framework Details

The SOAP framework involves several technical steps to achieve localization

** Spectral Reference Space Construction: This involves applying the representation extraction procedure to every step in the failure trajectories, obtaining representations for all steps, stacking these row-wise, and performing a compact singular value decomposition (SVD) to define orthogonal directions in the reference representation space The spectral base score is then computed by measuring each step's average squared projection onto a selected spectral band and inverting this quantity, resulting in a step-level signal without requiring decisive-error annotations This process exploits the imbalance where ordinary steps dominate the reference representations, allowing decisive-error steps to exhibit lower projection energy within the selected band The spectral band is selected as a contiguous range from the leading singular vectors, which are ordered by non-increasing singular value **

** Attention-guided Rescoring: To combat downstream contamination, SOAP derives a soft dependency graph from models’ attention patterns, where each edge approximates how strongly a later step draws on an earlier one It calculates the dependency weight by aggregating post-softmax attention routed from tokens in step s t to tokens in an earlier step s i, and normalizes these masses over the retained predecessors The final rescoring score is calculated as S̃(si) = S(si) + γBi, where Bi is the dependency-weighted average of downstream base scores, and γ controls the strength of this propagation This mechanism promotes an earlier step when anomalous downstream behavior is systematically built upon it The final decisive-error step and responsible agent are predicted as t= arg max S̃(si), a=at **

Experimental Results

Extensive experiments show that SOAP consistently improves attribution performance over competitive baselines across three failure-attribution benchmarks, including WHO&WHEN, CORRECT-ERROR, and TRACEELEPHANT Specifically, on TRACEELEPHANT with DeepSeek-R1-Distill-Llama-8B, SOAP improves step-level Accuracy by 24.66% over the strongest baseline The method demonstrates strong generalization across different backbones and data distributions, maintaining high performance even when constructed on a source subset and evaluated on a target subset Furthermore, ablation studies confirm that both spectral step scoring and attention-guided rescoring contribute significantly to the overall improvements

Practicality and Efficiency

SOAP is highlighted as a lightweight alternative because it requires neither step-level training labels nor a separate success-only reference set The computational cost is also favorable; SOAP fits its SVD on the unlabeled reference set in 0.08 s on WW-AG and 0.29 s on WW-HC, which is two to three orders of magnitude faster than methods like OAT and StepFinder The paper also shows that SOAP needs little reference data, with a third of the training split being sufficient to achieve accuracy within a few points of the full-data result

Conclusion

SOAP provides an effective framework for failure attribution in LLM-based multi-agent systems by scoring each step against a spectral reference and routing late contamination back to the decisive error with attention-guided propagation It is lightweight, requires no step-level annotation, and consistently outperforms prompt-based and training-based baselines across benchmarks, backbones, and settings The results demonstrate that step-unlabeled failure trajectories provide an effective source of localization signal

--- Page 32 ---

References

[1] Y. Bakish, A. Dudai, R. Ganz, O. Nuriel, E. Ben Avraham, M. S. Nacson, and R. Litman Adaptive influence graphs for failure attribution in multi-agent systems<ref:2610.13105#pg4>

[2] C. Bandi, R.-G. Dumitru, B. Hertzberg, D. Agarwal, G. Boo, T. Polakam, S. Hassaan J Da H Kim V Gupta M Sharma A Park M Dimakis E Hernandez Montoya D Rambado I Salazar R Cruz M Rezaei C Rane B Levin D Y Zhang B Kenstler and B Liu MCP-Atlas: A large-scale benchmark for tool-use competency with real MCP servers<ref:2610.13105#pg4>

[3] A. Banerjee et al. Where did it all go wrong? a hierarchical look into multi-agent error attribution arXiv preprint arXiv:2510.04886 2025<ref:2610.13105#pg4>

[4] S. Barke, A. Goyal, A. Khare, A. Singh, S. Nath, and C Bansal AgentRx: Diagnosing AI agent failures from execution trajectories arXiv preprint arXiv:2602.02475 2026<ref:2610.13105#pg4>

[5] M. Cemri, M. Z. Pan, S Yang, L A Agrawal B Chopra R Tiwari K Keutzer A Parameswaran D Klein K Ramchandran M Zaharia J E Gonzalez and I Stoica Why do multi-agent LLM systems fail? In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track 2026 URL https://openreview.net/forum?id=fAjbYBmonr<ref:2610.13105#pg4>

[6] M. Chen, J. Wang, F Mu, Y Wang Z Liu H Feng and Q Wang Seeing the whole elephant: A benchmark for failure attribution in LLM-based multi-agent systems In M Liakata V P Moreira J Zhang and D Jurgens editors Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics Volume 1 Long Papers pages 19888–19905 San Diego California United States July 2026 Association for Computational Linguistics ISBN 979-8-89176-390-6 doi: 10.18653/v1/2026.acl-long.912 URL https://aclanthology.org/2026.acl-long.912<ref:2610.

Improvements for AI systems

No step-level training labels are required for failure attribution, which bypasses costly annotation methods and allows for failure attribution from step-unlabeled failure trajectories. This enables debugging without needing to manually annotate every error in successful or failed runs.
The system can identify the decisive step that introduced the failure by constructing a spectral reference space from LLM representations and assigning a base error score according to its projection behavior, which is then corrected using attention-guided rescoring.
The improved system can mitigate downstream contamination by deriving soft dependencies from attention patterns and propagating evidence backward, as described in the paper: an earlier step is therefore promoted when anomalous downstream behavior is systematically built upon it.
The framework provides a method for localization that requires no separate success-only reference set, addressing the limitation of prior methods like OAT which rely on verified successful trajectories.
The system can be applied to proprietary agents where internal states are inaccessible, as it uses a frozen language model M without fine-tuning on failure data, enabling analysis of trajectories generated by proprietary agents whose internal states are inaccessible.

Abstract

Failure attribution for LLM agents aims to identify decisive steps responsible for task failure, which is critical for debugging and improving agentic systems. Existing methods rely on costly LLM prompting, step-level error annotations, or verified successful trajectories that lack failure signals for assistance. In this paper, we study failure attribution from step-unlabeled failure trajectories, where each trajectory is known to have failed but its decisive step is unknown. We propose SOAP (Spectral scOring with Attention-guided Propagation), an effective framework requiring neither step-level training labels nor a separate success-only reference set. SOAP first constructs a spectral reference space from LLM representations and assigns each step a base error score according to its projection behavior. Importantly, later steps may exhibit stronger failure signals because they inherit an earlier mistake, causing independent scorers to favor downstream consequences over the decisive error. To address this downstream contamination, SOAP derives soft step dependencies from attention patterns and propagates downstream error evidence toward its likely upstream sources. Extensive experiments show that SOAP consistently improves attribution performance over competitive baselines.