Monitoring Web Agents Without Internal Signals: Observable Trajectories and Key-Step Supervision
cs.AI
Submitted: 2026-09-02
Updated: 2026-09-02
Comments: preprint
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reliable web-agent monitoring is difficult when model-internal uncertainty signals such as token logits are unavailable.
Terminology
Abstract
Reliable web-agent monitoring is difficult when model-internal uncertainty signals such as token logits are unavailable. In this work, we study prefix-level risk prediction for web agents using observable trajectory signals: given an evolving prefix, estimate whether the current execution remains on track or is tending toward failure. We derive two observable trajectory representations: Macro features summarize cross-step agent--environment behavior and feedback, while Micro features measure the consistency of intention, action, and anticipated state change through repeated black-box queries. Instead of inheriting the final result label, we label the first critical error that remains uncorrected in the observed continuation and is associated with final failure as a key-step boundary, preserving valid early prefixes of failed trajectories as on track. Across WebArena-Lite and Online Mind2Web web agent benchmarks with five open- and closed-source backbones, observable trajectory signals are competitive with internal-signal baselines. The resulting predictors also support early intervention under fixed false-cut budgets and transfer across held-out website categories. These findings show that observable trajectory signals support valuable risk prediction abilities.
Sources
- UProp: Investigating the Uncertainty Propagation of LLMs in Multi-Step Agentic Decision-Making
- AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents
- REHEARSE: Experiential Rehearsal for Verbal Confidence Calibration in Large Language Models
- Qwen3-VL Technical Report
- Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers
- PrefixGuard: Online Failure Warning and Trace-Grounded Diagnosis for LLM Agents
- Language Models (Mostly) Know What They Know
- AgentRx: Diagnosing AI Agent Failures from Execution Trajectories
- Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs
- Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation
- VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
- Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attribution
- An Illusion of Progress? Assessing the Current State of Web Agents
- Shared Prefixes, Better Credit: Adaptive Routing for Multi-Agent Reasoning
- Qwen3 Technical Report
- Uncertainty Estimation in Autoregressive Structured Prediction
- ReAct: Synergizing Reasoning and Acting in Language Models
- AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systems
- Agentic Uncertainty Quantification
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems