Where Rules End and Judges Begin: Measuring the Judgment Boundary in Multi-Agent Systems Security
cs.AI, cs.CL, cs.CR, cs.MA
Submitted: 2026-10-06
Updated: 2026-10-11
Comments: 26 pages, 20 figures, 24 tables
Code: https://github.com/shaswata09/DEFER
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: LLM-based multi-agent systems (MAS) engage tools, share memory, and delegate tasks, often encountering adversarial content.
Terminology
Abstract
LLM-based multi-agent systems (MAS) engage tools, share memory, and delegate tasks, often encountering adversarial content. Current defenses for MAS are typically evaluated in isolation, focusing on one attack type at a time, which can lead to costly and hard-to-audit outcomes. This study organizes defenses into five principles, implementing them as DEFER1 (DEterministic-First Enforcement with Residual judgment), which includes a cascade of 28 checks that blocks what it can and refers the rest to a panel of four judges. In independent testing across four domains, attack success rates drop from about 30.0% to approximately 3.0%, with 78% of blocked attacks handled by deterministic checks. Only a quarter of proposals reach the judges in the security-operations domain, illustrating that the rules provide security for attacks violating clear policies, while judges manage those that only misrepresent intent. Both systems have weaknesses, such as a risk-score approval gate that inaccurately approves most attack proposals but few legitimate ones, highlighting the challenges in assessing threats accurately.
Sources
- Intelligent AI Delegation
- Multi-Agent Systems Execute Arbitrary Malicious Code
- A-MemGuard: A Proactive Defense Framework for LLM-Based Agent Memory
- Evaluating LLM Agent Collusion in Double Auctions
- SAGA: A Security Architecture for Governing AI Agentic Systems
- Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agents
- BlockA2A: Towards Secure and Verifiable Agent-to-Agent Interoperability
- AgentSafe: Safeguarding Large Language Model-based Multi-agent Systems via Hierarchical Data Management
- Collaborative Memory: Multi-User Memory Sharing in LLM Agents with Dynamic Access Control
- LlamaFirewall: An open source guardrail system for building secure AI agents
- Evaluating a Layered Prompt-Injection Defence for the Model Context Protocol: A Record-Level Audit of Decision Conventions, Corpus Provenance and Reproducibility
- Provably Secure Agent Guardrail
- Don't Make Models Guess Security and Safety: Symbolic Guardrails for Domain-Specific AI Agents
- Autoformalization of Agent Instructions into Policy-as-Code
- Defeating Prompt Injections by Design
- Design Patterns for Securing LLM Agents against Prompt Injections
- AgentTrust: A Self-Improving Trust Layer for AI-Agent Actions
- Large Language Model based Multi-Agents: A Survey of Progress and Challenges
- Security Threat Modeling for Emerging AI-Agent Protocols: A Comparative Analysis of MCP, A2A, Agora, and ANP
- Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems