PASTABench: Proactive Assessment of Sequential Trajectories for Agent Safety

arXiv:2609.28197 · cs.AI, cs.CL · Submitted 2026-09-23 · Read on arXiv

cs.AI, cs.CL

Submitted: 2026-09-23

Updated: 2026-09-23

Comments: EMNLP 2026

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 31× in this paper · explained in 2 episodes
• Actor — 16× in this paper · explained in 3 episodes
• Ground Truth — 11× in this paper
• Optimal Intervention Window — 9× in this paper
• Verification — 9× in this paper · explained in 2 episodes
• Ablation — 6× in this paper · explained in 1 episode
• Calibration — 4× in this paper · explained in 7 episodes
• Causal Reasoning — 4× in this paper · explained in 2 episodes
• Temporal Reasoning — 4× in this paper · explained in 2 episodes
• Adam — 3× in this paper · explained in 2 episodes
• Agentic System — 3× in this paper · explained in 2 episodes
• Benchmarking — 3× in this paper · explained in 3 episodes
• Robustness — 3× in this paper · explained in 5 episodes
• Accuracy — 2× in this paper · explained in 1 episode
• Alignment — 2× in this paper · explained in 1 episode
• Grounding — 2× in this paper · explained in 3 episodes
• Large Language Models (LLMs) — 2× in this paper · explained in 6 episodes
• Reinforcement Learning — 2× in this paper · explained in 4 episodes
• AI Agents — 1× in this paper · explained in 2 episodes
• Accountability — 1× in this paper · explained in 2 episodes
• Agentic Workflows — 1× in this paper · explained in 1 episode
• Core — 1× in this paper · explained in 2 episodes
• Failure Modes — 1× in this paper · explained in 3 episodes
• Fine-Tuning — 1× in this paper · explained in 5 episodes
• Initial Data — 1× in this paper · explained in 2 episodes
• LLM-as-a-Judge — 1× in this paper · explained in 3 episodes
• Multi-agent Collaboration — 1× in this paper · explained in 2 episodes
• Multimodal Inputs — 1× in this paper · explained in 2 episodes
• Trace — 1× in this paper · explained in 2 episodes

Sources

• AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
• YETI (YET to Intervene) Proactive Interventions by Multimodal AI Agents in Augmented Reality Tasks
• The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation
• Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversations
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• The Llama 3 Herd of Models
• RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
• A Safety and Security Framework for Real-World Agentic Systems
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• GPT-4o System Card
• OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents
• GSPR: Aligning LLM Safeguards as Generalizable Safety Policy Reasoners
• AgentDoG: A Diagnostic Guardrail Framework for AI Agent Safety and Security
• HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
• ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedback
• Chemist Eye: A Visual Language Model-Powered System for Safety Monitoring and Robot Decision-Making in Self-Driving Laboratories
• Qwen2.5 Technical Report
• CaveAgent: Transforming LLMs into Stateful Runtime Operators
• SemEval-2025 Task 9: The Food Hazard Detection Challenge
• SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• Agents' Last Exam
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems