TRACE: TRajectory Attribution for Automated Context Engineering

Summary

This paper presents Trace (TRajectory Attribution for Automated Context Engineering), an automated feedback loop system designed to diagnose and remediate context failures in production AI agents. The authors—Yikai Zhao, Pradeep Kumar Misra, and Saurabh Pandey from Amazon—address the problem that Production AI agents fail when their context sources—system prompts, knowledge bases, tool descriptions, and procedural skills—contain errors or gaps, and that Current maintenance approaches rely on manual log review and ad-hoc debugging, creating a scalability bottleneck as interaction volume grows.

The core insight is that "agent trajectories are rich with implicit dissatisfaction signals—user corrections, rephrasing patterns, abandonment cues—that reveal precisely where context sources failed, without requiring explicit feedback collection. Unlike model fine-tuning approaches, Trace operates on the context layer, enabling rapid iteration without retraining."

The system makes four contributions:

1. Trajectory Mining for Context Engineering: a novel framework for systematically mining historical agent trajectories—an overlooked resource—to diagnose and remediate context failures.

2. Implicit Signal Exploitation: We exploit implicit user feedback signals—corrections, rephrasing patterns, abandonment cues—that typically go unnoticed in production logs.

3. End-to-End Attribution with Exploratory Verification: "We extend textual gradients from monolithic prompt optimization to multi-component context sources (prompts, KB, tools, Skills), and introduce exploratory verification where the Recommender agent actively reads context source files before generating recommendations."

4. Verifiable Evaluation Benchmark and Simulation Methodology: we introduce a systematic three-layer simulation methodology for generating synthetic agent traces with perfect ground truth, including a six-category fault taxonomy, traces of varying complexity (2–16 nodes) with documented cascade effects, complete attribution ground truth, and a cross-layer verification protocol.

The architecture consists of three specialized agents operating in sequence:

• Detector Agent: identifies dissatisfaction signals from agent trajectories, distinguishing between explicit signals (direct feedback) and implicit signals (linguistic cues). It outputs a binary DSAT determination along with a confidence score, with sessions exceeding the confidence threshold escalated to the Root Cause.

• Root Cause Agent: analyzes the conversation trajectory to identify where and why the agent failed, generating textual gradients that attribute responsibility to specific components. It uses delta-guided holistic attribution where the delta—the discrepancy between what the user expected and what the agent produced—serves as the 'loss signal' that guides attribution. The LLM walks backward (response → tool outputs → thinking → inputs) to locate the earliest node whose content is inconsistent with the delta.

• Recommender Agent: treats the Root Cause's root cause analysis as a hypothesis to verify, not a conclusion to accept. It conducts its own comprehensive exploration of the agent's context sources before producing recommendations, using tools like search kb, read kb entry, list skills, read skill, read system prompt, and read tool prompt. It outputs structured CRUD recommendations specifying the operation (CREATE, UPDATE, DELETE, or NO ACTION), target path, recommended change, and supporting evidence.

The evaluation uses a synthetic dataset of 75 conversation traces: 60 DSAT traces: Conversations containing user dissatisfaction signals with known root causes and 15 control traces: Successful conversations without failures (negative examples). The traces span three complexity tiers (simple: 2–8 nodes, complex: 9–11 nodes, difficult: 15–16 nodes) and six fault categories: SKILL FILE STALE (n=12), KB CONTENT STALE (n=12), KB CONTENT GAP (n=11), TOOL PROMPT ERROR (n=9), SYSTEM PROMPT GAP (n=6), and RETRIEVAL FAILURE (n=10).

Key results:

• Detector Performance: Both Detector and a vanilla LLM baseline achieve perfect binary DSAT detection (precision=1.0, recall=1.0, F1=1.0).

• Root Cause Agent Performance: Node Accuracy (Acc@1) of 72.7% with 95% CI [59%, 85%], and Acc@3 of 85.0% with 95% CI [73%, 95%].

• Recommender Performance: Operation Accuracy of 96% with 95% CI [87%, 100%], and Path Accuracy of 82% with 95% CI [69%, 92%].

• End-to-End Pipeline: Node Attribution of 72.7% and Fix Effectiveness of 82% with 95% CI [69%, 92%].

Two ablation studies validate key design decisions:

• Holistic vs. Iterative Attribution: The holistic approach achieves 2× higher node accuracy with 16× fewer LLM calls. The iterative approach fails because when faulty content propagates through multiple nodes, each appears to 'contain the error,' but only the first node introduced it.

• Exploration for CRUD Operation: exploration achieves 83% vs 33% operation accuracy on these faults (KB content faults specifically). Without exploration, the system cannot distinguish GAP faults (requiring CREATE) from STALE faults (requiring UPDATE)—it must guess. The Recovery Rate of 67% shows that when the Root Cause misidentifies the root cause, exploration still produces correct recommendations two-thirds of the time.

The paper concludes: "Evaluation on 60 dissatisfaction traces spanning three complexity tiers (up to 16 execution nodes) achieves 72.7% root cause node attribution and 82% end-to-end fix effectiveness—demonstrating that over 80% of context-layer failures could be automatically diagnosed and correctly remediated by mining historical agent trajectories, an overlooked resource in production systems. Future work could extend single-session analysis with cross-session pattern detection to aggregate recurring failure modes across user populations."

Improvements to AI Systems:

1. Automated Context-Layer Debugging: AI agents can now self-diagnose and repair failures in their own system prompts, knowledge bases, tool descriptions, and procedural skills without retraining, using historical trajectory mining to identify dissatisfaction signals (corrections, rephrasing, abandonment) as implicit error indicators.

2. Delta-Guided Root Cause Attribution: AI systems can perform backward-chaining analysis (response → tool outputs → thinking → inputs) to pinpoint the earliest node where content diverges from user expectations, enabling precise localization of faulty context components rather than treating all downstream errors as independent failures.

3. Exploratory Verification Before Remediation: AI systems can actively read and verify context source files (knowledge base entries, skill definitions, tool prompts) before generating fixes, distinguishing between missing content (requiring CREATE) and stale content (requiring UPDATE) with 83% vs 33% accuracy compared to non-exploratory approaches.

4. Structured CRUD-Based Self-Improvement: AI agents can output actionable, machine-readable recommendations (CREATE, UPDATE, DELETE, NO ACTION) with target paths and supporting evidence, enabling automated or semi-automated context updates with 96% operation accuracy and 82% path accuracy.

5. Confidence-Scaled Escalation: AI systems can assign confidence scores to dissatisfaction detections and only escalate high-confidence cases for root cause analysis, reducing noise from ambiguous user signals while maintaining perfect detection precision/recall.

6. Holistic Attribution Efficiency: AI systems can perform holistic (single-pass) attribution instead of iterative per-node analysis, achieving 2× higher accuracy with 16× fewer computational calls—critical for production-scale log processing.

7. Cross-Session Failure Pattern Aggregation: Future AI systems can aggregate recurring failure modes across user populations, enabling proactive context updates before failures occur rather than reactive fixes after dissatisfaction signals.

What the Improved AI System Can Do:

• Automatically detect user dissatisfaction from conversation logs without explicit feedback forms

• Identify the exact faulty component (prompt, KB entry, tool description, skill) causing agent failure

• Generate and execute context updates (create/update/delete) with evidence-based justification

• Recover correct fixes even when initial root cause attribution is wrong (67% recovery rate)

• Scale maintenance from manual log review to automated pipeline processing for high-volume interactions

• Achieve 82% end-to-end fix effectiveness on context-layer failures, reducing need for model retraining or human intervention

Production AI agents fail when their context sources -- system prompts, knowledge bases, tool descriptions, and procedural skills -- contain errors or gaps. Current maintenance relies on manual log review and ad-hoc debugging, creating a scalability bottleneck as interaction volume grows. We present TRACE (TRajectory Attribution for Automated Context Engineering), an automated feedback loop that mines historical agent trajectories to diagnose and remediate context failures. Our key insight is that trajectories are rich with implicit dissatisfaction signals -- user corrections, rephrasing, abandonment cues -- that reveal precisely where context sources failed, without explicit feedback collection. Unlike model fine-tuning, TRACE operates on the context layer, enabling rapid iteration without retraining. We make four contributions: (1) a trajectory mining framework that systematically extracts diagnostic information from historical agent executions; (2) multi-component causal attribution that extends textual gradients from monolithic prompt optimization to heterogeneous context sources (skills, knowledge bases, tools, prompts); (3) exploratory verification, where agents actively read context sources to distinguish content gaps requiring CREATE from stale content requiring UPDATE, achieving 96% operation accuracy; and (4) a reusable simulation methodology and verifiable benchmark addressing the absence of open datasets for context debugging, with a six-category fault taxonomy, ground truth annotations, and a cross-layer verification protocol. On 60 dissatisfaction traces spanning three complexity tiers (up to 16 execution nodes), TRACE achieves 72.7% root cause attribution and 82% end-to-end fix effectiveness, showing that over 80% of context-layer failures can be automatically diagnosed and remediated by mining historical trajectories, an overlooked resource in production systems.