Security in the Age of AI Teammates: An Empirical Study of Agentic Pull Requests on GitHub
cs.CR, cs.SE
Submitted: 2026-01-01
Updated: 2026-09-03
Comments: Submitted to Information and Software Technology Journal, currently under minor revision
License: http://creativecommons.org/licenses/by/4.0/
The gist: Autonomous coding agents are increasingly deployed as AI teammates in modern software engineering, independently authoring pull requests (PRs) that modify production code at scale.
Terminology
Abstract
Autonomous coding agents are increasingly deployed as AI teammates in modern software engineering, independently authoring pull requests (PRs) that modify production code at scale. This study aims to systematically characterize how autonomous coding agents contribute to software security in practice, how these security-related contributions are reviewed and accepted, and which observable signals are associated with PR rejection. We conduct a large-scale empirical analysis of agent-authored PRs using the AIDev dataset, comprising of over 33,000 curated PRs from popular GitHub repositories. Security-relevant PRs are identified using a keyword filtering strategy, followed by manual validation, resulting in 1,293 confirmed security-related agentic-PRs. We then analyze prevalence, acceptance outcomes, and review latency across autonomous agents, programming ecosystems, and types of code changes. Moreover, we apply qualitative open coding to identify recurring security-related actions and underlying intents, and examine review metadata to identify early signals associated with PR rejection. Security-related Agentic-PRs constitute a meaningful share of agent activity (approximately 4%). Rather than focusing solely on narrow vulnerability fixes, agents most frequently perform supportive security hardening activities, including testing, documentation, configuration, and improved error handling. Compared to non-security PRs, security-related Agentic-PRs exhibit lower merge rates and longer review latency, reflecting heightened human scrutiny, with variation across agents and programming ecosystems. PR rejection is more strongly associated with PR complexity and verbosity than with explicit security topics.
Sources
- Training language models to follow instructions with human feedback
- GPT-4 Technical Report
- The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering
- Agentic Software Engineering: Foundational Pillars and a Research Roadmap
- ReAct: Synergizing Reasoning and Acting in Language Models
- WebArena: A Realistic Web Environment for Building Autonomous Agents
- On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub
- Evaluating Large Language Models Trained on Code
- A Survey on Large Language Models for Software Engineering
- Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions
- Formalizing the Safety, Security, and Functional Properties of Agentic AI Systems
- Taming Uncertainty via Automation: Observing, Analyzing, and Optimizing Agentic AI Systems
- ProofWright: Towards Agentic Formal Verification of CUDA
- Toward Agentic Software Engineering Beyond Code: Framing Vision, Values, and Vocabulary
- Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI
- Agentsway -- Software Development Methodology for AI Agents-based Teams
- CodePori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology
- Experimenting with Multi-Agent Software Development: Towards a Unified Platform
- AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities
- ChatCollab: Exploring Collaboration Between Humans and AI Agents in Software Teams
Related papers
- SoK: AI-Augmented Binary Reversing
- Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
- Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
- Efficient Fuzzy PSI under One-Sided Assumptions
- Sealing the Audit-Runtime Gap for LLM Skills
- Token Composition: A Graph Based on EVM Logs