LPS-Bench: Benchmarking Safety Awareness of Computer-Use Agents in Long-Horizon Planning under Benign and Adversarial Scenarios

arXiv:2602.03255 · cs.AI · Submitted 2026-02-03 · Read on arXiv

cs.AI

Submitted: 2026-02-03

Updated: 2026-10-02

Comments: 50 pages. Accepted at NeurIPS 2026, Evaluations and Datasets Track. Revised benchmark description, skill-augmented evaluation, evaluator validation, and appendices

Code: https://github.com/tychenn/LPS-Bench

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 2× in this paper · explained in 2 episodes
• Adversarial scenarios — 1× in this paper · explained in 1 episode
• Benchmarking — 1× in this paper · explained in 4 episodes
• LLM-as-a-Judge — 1× in this paper · explained in 3 episodes

Sources

• WebArena: A Realistic Web Environment for Building Autonomous Agents
• Large Language Model-Brained GUI Agents: A Survey
• What Prompts Don't Say: Understanding and Managing Underspecification in LLM Prompts
• Active Task Disambiguation with LLMs
• Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering
• Check Yourself Before You Wreck Yourself: Selectively Quitting Improves LLM Agent Safety
• Prompt Injection Attack to Tool Selection in LLM Agents
• ToolTweak: An Attack on Tool Selection in LLM-based Agents
• Attractive Metadata Attack: Inducing LLM Agents to Invoke Malicious Tools
• MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP Servers
• RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environments
• RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agents
• Identifying the Risks of LM Agents with an LM-Emulated Sandbox
• Agent-SafetyBench: Evaluating the Safety of LLM Agents
• MCP-SafetyBench: A Benchmark for Safety Evaluation of Large Language Models with Real-World MCP Servers
• AgentLAB: Benchmarking LLM Agents against Long-Horizon Attacks
• Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agents
• AgentHazard: A Benchmark for Evaluating Harmful Behavior in Computer-Use Agents
• ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces
• Monitoring Decomposition Attacks in LLMs with Lightweight Sequential Monitors

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• Agents' Last Exam
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems