CyberForge: Verified Vulnerability Injection at Repository Level for Cybersecurity Agent Training
Amine Lbath, Manan Suri, Aurelien Delaitre, Vadim Okun, Massih-Reza Amini, Ram D. Sriram, Dinesh Manocha
cs.CR, cs.AI, cs.SE
Submitted: 2026-08-06
Updated: 2026-08-10
Code: https://github.com/google/oss-fuzz
Project page: https://cyb3rforge.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: Despite recent advances, frontier large language model (LLM) agents remain limited in discovering and patching complex vulnerabilities in real-world software.
Terminology
Abstract
Despite recent advances, frontier large language model (LLM) agents remain limited in discovering and patching complex vulnerabilities in real-world software. Generally available agents can already aid attackers, who only need to find one exploitable weakness, while defenders must continuously identify and patch all vulnerabilities across fast-growing codebases. Stronger defensive agents would help close this gap, yet the scarcity of security training data with reproducible build and execution environments remains a bottleneck. We present CyberForge, a framework that synthesizes executable, repository-level security training data by injecting vulnerabilities into real C/C++ projects. It validates each instance dynamically: the injected build must pass the project's unit tests, and generated proof-of-vulnerability (PoV) must trigger on the injected build and not on the clean one. CyberForge is not limited by the availability of disclosed vulnerabilities, therefore it can scale in comparison to data augmentation techniques which rely on historic CVE data. The resulting corpus holds 1034 validated vulnerabilities across 80 projects and 63 weakness categories, with edit locality similar to real CVE patches under a real-versus-real noise floor. Fine-tuning on trajectories collected over this corpus improves SEC-bench patch repair by +3.3 to +14.7 points, in all six configurations of three model scales and two teachers, with the 31B student reaching its GPT-5.4-mini teacher, 72.7% against 74.0%. These gains generalize out of distribution to PatchEval, a corpus containing other programming languages, where every configuration also improves and the 31B student passes its teacher.
Sources
- PATCHEVAL: A New Benchmark for Evaluating LLMs on Patching Real-World Vulnerabilities
- R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents
- AVIATOR: Towards AI-Agentic Vulnerability Injection Workflow for High-Fidelity, Large-Scale Code Security Dataset
- SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks
- CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability
- VGX: Large-Scale Sample Generation for Boosting Learning-Based Software Vulnerability Analyses
- Training Software Engineering Agents and Verifiers with SWE-Gym
- Frontier AI's Impact on the Cybersecurity Landscape
- Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection
- From CVE Entries to Verifiable Exploits: An Automated Multi-Agent Framework for Reproducing CVEs
- ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?
- CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale
- ProSec: Fortifying Code LLMs with Proactive Security Alignment
- SWE-smith: Scaling Data for Software Engineering Agents
- BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems
- CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities
- To Defend Against Cyber Attacks, We Must Teach AI Agents to Hack
- Cyber-Zero: Training Cybersecurity Agents without Runtime
- Training Language Model Agents to Find Vulnerabilities with CTF-Dojo
Related papers
- SoK: AI-Augmented Binary Reversing
- Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
- Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
- Estimating the Social Cost of Corporate Data Breaches
- Efficient Fuzzy PSI under One-Sided Assumptions
- Sealing the Audit-Runtime Gap for LLM Skills