Why Does Agentic Safety Fail to Generalize Across Tasks?
cs.LG, stat.ML
Submitted: 2026-05-07
Updated: 2026-09-27
Code: https://github.com/Tomerslortau/agentic-safety-generalization
License: http://creativecommons.org/licenses/by/4.0/
The gist: AI agents are increasingly deployed in multi-task settings, where the task to perform is specified at test time, and the agent must generalize to unseen tasks.
Terminology
Abstract
AI agents are increasingly deployed in multi-task settings, where the task to perform is specified at test time, and the agent must generalize to unseen tasks. A major concern in such settings is safety: often, an agent must not only execute unseen tasks, but do so while avoiding risks and handling ones that materialize. Empirical evidence suggests that even when the ability to execute generalizes to unseen tasks, the ability to do so safely frequently does not. This paper provides theory and experiments indicating that failures of agentic safety to generalize across tasks are not merely due to limitations of training methods, but can also reflect an inherent property of safety itself: the relationship between a task and its safe execution is more complex than the relationship between a task and its execution alone. Theoretically, we analyze linear-quadratic control with H infinity-robustness, and prove that in many cases, the mapping from task specification to an optimal controller has higher Lipschitz constant with safety requirements than without, yielding a Lipschitz bound of independent interest. Empirically, we demonstrate our conclusions in simulated quadcopter navigation with a neural network agent and in CRM with an LLM agent, showing that imitating safe and unsafe teachers is similarly straightforward on tasks seen in training, yet generalizing across tasks is considerably more difficult with the safe teacher. Our findings suggest that current efforts to enhance agentic safety may be insufficient, and point to a need for fundamentally different approaches.
Sources
- GitHub's Copilot Code Review: Can AI Spot Security Flaws Before You Commit?
- AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- Constitutional AI: Harmlessness from AI Feedback
- A System for Human-AI collaboration for Online Customer Support
- DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- Safe RLHF: Safe Reinforcement Learning from Human Feedback
- Safe Exploration in Continuous Action Spaces
- RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
- WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models
- Large Language Model based Multi-Agents: A Survey of Progress and Challenges
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- Generalizing from a few environments in safety-critical reinforcement learning
- A CMDP-within-online framework for Meta-Safe Reinforcement Learning
- Adam: A Method for Stochastic Optimization
- Safety Generalization Under Distribution Shift in Safe Reinforcement Learning: A Diabetes Testbed
- In-context Reinforcement Learning with Algorithm Distillation
- ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents
- Deep Reinforcement Learning: An Overview
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks