Securing Agentic AI: From Per-Action Checks to Trajectory Assurance
Alireza Lotfi, Subangkar Karmaker Shanto, Imtiaz Karim, Elisa Bertino
cs.AI, cs.CR, cs.MA
Submitted: 2026-08-03
Comments: 6 pages. Accepted to the ACM AI Leadership Summit 2026 (Visionary Track)
License: http://creativecommons.org/licenses/by/4.0/
The gist: Autonomous agents are increasingly used to execute consequential tasks in environments governed by operational constraints, organizational policies, regulatory requirements, and technical standards.
Terminology
Abstract
Autonomous agents are increasingly used to execute consequential tasks in environments governed by operational constraints, organizational policies, regulatory requirements, and technical standards. Their safety is therefore determined not by the correctness of individual actions, but by whether their overall behavior remains consistent with the rules and invariants of the systems in which they operate. As large language model (LLM)-based agents become more autonomous and increasingly delegate tasks across organizational boundaries, securing them evolves from a single challenge into a broad and interconnected landscape spanning the entire agentic stack. At the single-agent level, untrusted inputs through prompts, memory, retrieved knowledge, and tool interfaces create attack surfaces. In multi-agent settings, delegation and communication introduce challenges related to identity, trust, capability control, and decision transparency, while the underlying model routing and execution control plane remains vulnerable to manipulation and to unverified model provenance. Perhaps the most fundamental challenge is behavioral containment: sequences of individually permissible actions may collectively violate system-level constraints and safety invariants. At the broader level, supply-chain integrity, provenance, accountability, and end-to-end observability remain largely open problems. A common principle unifies these directions: security must become a verifiable property of the architectures, protocols, and runtimes that govern agent behavior, rather than an optional layer of guidance. Charting these challenges provides a roadmap toward trustworthy autonomous agent deployment.
Sources
- Security Threat Modeling for Emerging AI-Agent Protocols: A Comparative Analysis of MCP, A2A, Agora, and ANP
- From Inference Routing to Agent Orchestration: Declarative Policy Compilation with Cross-Layer Verification
- AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases
- Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applications
- Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges
- Defeating Prompt Injections by Design
- AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
- A survey of agent interoperability protocols: Model Context Protocol (MCP), Agent Communication Protocol (ACP), Agent-to-Agent Protocol (A2A), and Agent Network Protocol (ANP)
- Don't Make Models Guess Security and Safety: Symbolic Guardrails for Domain-Specific AI Agents
- RepetitionCurse: Measuring and Understanding Router Imbalance in Mixture-of-Experts LLMs under DoS Stress
- Enforcing Temporal Constraints for LLM Agents
- A Survey of Agentic AI and Cybersecurity: Challenges, Opportunities and Use-case Prototypes
- Life-Cycle Routing Vulnerabilities of LLM Router
- VeriGuard: Enhancing LLM Agent Safety via Verified Code Generation
- Security Considerations for Multi-agent Systems
- Under the Hood of SKILL.md: Semantic Supply-chain Attacks on AI Agent Skill Registry
- Rerouting LLM Routers
- Prompt Injection Attack to Tool Selection in LLM Agents
- Progent: Securing AI Agents with Privilege Control
- Route to Rome Attack: Directing LLM Routers to Expensive Models via Adversarial Suffix Optimization
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems