ProbGuard: Proactive Runtime Monitoring for LLM Agent Safety via Probabilistic Prediction
cs.AI, cs.SE
Submitted: 2025-08-01
Updated: 2026-08-03
Comments: Accepted by the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE 2026)
Journal ref: Proc. ASE'26, pages 1470-1482. ACM, 2026
Code: https://github.com/haoyuwang99/ProbGuard
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large Language Model (LLM) agents increasingly operate across domains such as robotics, virtual assistants, and web automation.
Terminology
Abstract
Large Language Model (LLM) agents increasingly operate across domains such as robotics, virtual assistants, and web automation. However, their stochastic decision-making introduces safety risks that are difficult to anticipate during execution. Existing runtime monitoring frameworks, such as AgentSpec, primarily rely on reactive safety rules that detect violations only when unsafe behavior is imminent or has already occurred, limiting their ability to handle long-horizon dependencies. We present ProbGuard, a proactive runtime monitoring framework for LLM agents that anticipates safety violations through probabilistic risk prediction. ProbGuard abstracts agent executions into symbolic states and learns a Discrete-Time Markov Chain (DTMC) from execution traces to model behavioral dynamics. At runtime, the monitor estimates the probability that execution will remain safe from the current state, and triggers an intervention when this probability falls below a user-defined threshold. To improve robustness, ProbGuard incorporates semantic validity constraints in the abstraction and admits a PAC-style analysis that characterizes the sample complexity required to certify the learned model under standard assumptions. We evaluate ProbGuard in two safety-critical domains: autonomous driving and embodied household agents. Across evaluated scenarios, ProbGuard consistently predicts traffic law violations and collisions in advance, with warnings up to 15.84 seconds at a threshold yielding no false alarms, and up to 38.66 seconds at stricter thresholds. In embodied agent tasks, ProbGuard's re-prompting intervention mode reduces unsafe behavior by 65.37% relative to the unmonitored baseline while retaining 80.4% of the baseline task completion; a stricter halting configuration reduces unsafe behavior by 93.60% at a larger cost in completion.
Sources
- LLM-Empowered Embodied Agent for Memory-Augmented Task Planning in Household Robotics
- Understanding the planning of LLM agents: A survey
- AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents
- $\mu$Drive: User-Controlled Autonomous Driving
- SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents
- The Fusion of Large Language Models and Formal Methods for Trustworthy AI Agents: A Roadmap
- LLM-enabled Applications Require System-Level Threat Monitoring
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems