Replay Fidelity in LLM Agent Evaluation: Randomness, Hidden State, and Test Oracles in AppWorld
cs.SE, cs.AI
Submitted: 2026-11-08
Updated: 2026-11-08
Terminology
Sources
- AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents
- ReplayRCA: Measuring Post-hoc Diagnosis and Live Oversight of Agent Failures by Counterfactual Replay
- Repair or Resample? Rethinking Failure Debugging in LLM Multi-Agent Systems
- Chronicle: Cut-Point Replay for Regression Testing of LLM Agents
- Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training
- The Replay Gap: Static Evaluation of Model Switching in LLM Agents Scores the Wrong World
- When Agent Evaluation Becomes Unreliable: An Empirical Study of Oracle Failures and Configuration-Dependent Measurement in an LLM Agent Benchmark
- CAREFUL-Bench: Effect-Aware, Judge-Free Evaluation of Multi-Turn LLM Function Calling
- Lightweight User-Space Record And Replay
- Attributing Agent Failures to Their History Needs an Executed Counterfactual: Toward a Reporting Standard
- Replayable Financial Agents: A Determinism-Faithfulness Assurance Harness for Tool-Using LLM Agents
- DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems
- TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories
- Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems
- Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?
- Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failures
- EigenData: A Self-Evolving Multi-Agent Platform for Function-Calling Data Synthesis, Auditing, and Repair
Related papers
- ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
- TestPrism: Rethinking Test Evaluation Beyond a Single Reference
- Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents
- Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
- GitSkills: A Dataset of Agent Skills on GitHub
- For Coding Agents, the Repository Matters