BypassGym: A Benchmark for Evaluating AI Agents' Endpoint Defense Evasion Capabilities
summary
The gist
Artificial intelligence agents increasingly automate cybersecurity tasks, including offensive operations whether AI agents can evade it remains unclear BypassGym is introduced as the first benchmark…
In short
BypassGym is a benchmark designed to test if AI agents can create software loaders that bypass modern endpoint security systems like EDRs on real Windows machines. The study found that even simple models can generate successful submissions, and specific evasion knowledge and detailed feedback help agents improve their attempts against defenses.
Key concepts
- BypassGym
- This is a benchmark used to systematically evaluate an AI agent's ability to create a loader that executes malicious code on an endpoint protected by EDR. It tests the agent's end-to-end capability, from generating the loader to ensuring the payload runs successfully.
- EDR
- Endpoint Detection and Response is security software deployed on computers that monitors activities for signs of malicious behavior. In this benchmark, it acts as a defense system that an AI agent must try to evade while deploying its custom loader.
- Evasion Knowledge Skill
- This is curated information provided to the AI agent containing established techniques for evading security measures, such as specific methods for code obfuscation or bypassing detection mechanisms. This knowledge gives the agent workable strategies instead of relying solely on general training.
- Agent Scaffold
- The scaffold refers to the structure or framework given to an AI agent, which dictates how it examines inputs, builds potential solutions, and handles errors. Different scaffolds affect success rates depending on the specific AI model being used.
Terminology used across episodes
This episode discusses
- BypassGym: A Benchmark for Evaluating AI Agents' Endpoint Defense Evasion Capabilities · Paper Radio
- AgentCyberRange: Benchmarking Frontier AI Systems in Realistic Cyber Ranges
- MalGEN: A Testbed for Modeling and Evaluating Malware Behaviors
- CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models
- Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models
The paper
BypassGym: A Benchmark for Evaluating AI Agents' Endpoint Defense Evasion Capabilities · Read on arXiv
Weifeng Yuan, Zifan Chen, Yucheng Zhang, Feng Dong, Peng Gao, Haoyu Wang
Huazhong University of Science and Technology · Virginia Polytechnic Institute and State University
Artificial intelligence agents increasingly automate cybersecurity tasks, including offensive operations. Frequent attacks across the web make endpoint detection and response (EDR) a critical defense, but whether AI agents can evade it remains unclear. Benchmarks evaluate the offensive capabilities of large language model (LLM) agents in areas such as penetration testing and vulnerability discovery. One important capability that these benchmarks do not address is defense evasion, specifically whether an agent can develop a program that executes a payload without being blocked or terminated by a deployed EDR. We introduce BypassGym, to our knowledge the first benchmark to systematically evaluate this capability on real endpoints. Given command-and-control (C2) shellcode, an agent must generate and compile a loader that executes the payload without being blocked or terminated. A supplementary evasion knowledge skill provides domain knowledge and supports comparisons with and without this input. BypassGym is built on Proxmox VE(PVE), so the evaluation environment runs real Windows systems and EDR products inside virtual machines and returns structured detection feedback to the agent. We evaluate 5 lightweight model backends against 6 EDR products and 3 payloads, and find that even these simple models generate successful submissions, with DeepSeek-V4.1 Flash reaching 17/18 successful runs and Bitdefender allowing only 5 of 15 runs to pass. These results provide a systematic measurement of current AI agents' endpoint defense evasion capabilities and identify areas for defenders to investigate.
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: I'm Nadia.
Elias: And I'm Elias, here with Priya.
Nadia: We're looking at the paper: "BypassGym: A Benchmark for Evaluating AI Agents' Endpoint Defense Evasion Capabilities".
Elias: Here's the gist of the paper. Artificial intelligence agents increasingly automate cybersecurity tasks, including offensive operations whether AI agents can evade it remains unclear BypassGym is introduced as the first benchmark to systematically evaluate this capability on real endpoints and finds that even these simple models generate successful submissions.
Nadia: Benchmark Design and Scope BypassGym measures whether an AI agent can generate a loader that executes a supplied shellcode payload on a Windows endpoint protected by a deployed EDR The benchmark fixes the payload, available tools, and iteration budget while leaving the loader implementation open.
Elias: It consists of three modules: the Evasion Task Setup module provides a fixed shellcode payload from an open-source command-and-control (C2) framework and a curated evasion knowledge skill, the Shellcode Loader Development module gives the agent a Windows compilation toolchain for building and revising a shellcode loader, then packaging it as a bundle with one entry executable and any required dependencies, and the EDR Evaluation and Feedback module first runs the bundle on a Windows 11 VM without EDR, referred to as Windows 11 NoEDR, to verify that the payload executes and the process remains alive.
Nadia: Evaluation Methodology The evaluation oracle combines two complementary check groups listed in Table 2, which are the Endpoint Execution Check and the C2 Functionality Check.
Title and authors: Nadia: So, we're talking about this paper called "BypassGym: A Benchmark for Evaluating AI Agents' Endpoint Defense Evasion Capabilities." Basically, they are trying to figure out if these AI agents can actually sneak past security defenses on real computer systems.
Elias: Right. It’s focused on whether an agent can develop a program that executes shellcode payload on a Windows endpoint protected by an EDR system. The core question is whether AI agents can evade detection when the defense is active.
Priya: So, the implication here for us as privacy folks is figuring out how easy it is for these models to bypass security controls when they are trying to operate in a real environment.
Nadia: Exactly. The authors set up this benchmark by fixing the shellcode payload and the tools available, but they leave the actual loader implementation open for the AI agent to develop. They also include a specific set of evasion knowledge skills that help guide the agent's thinking.
Elias: And then you have three main modules in this BypassGym setup. First, there’s the Evasion Task Setup module which provides a fixed shellcode payload from an open-source command and control framework and that curated evasion knowledge skill.
Priya: That sounds like they are giving the agent a starting point for what kind of evasion techniques it should be trying to use, rather than letting it just wander around randomly.
Nadia: Then you have the Shellcode Loader Development module which gives the AI agent a Windows compilation toolchain so it can build and revise that shellcode loader. After that, they package it up as a bundle with one entry executable and any required dependencies.
Elias: And finally, there's the EDR Evaluation and Feedback module. This part first runs the bundle on a Windows eleven virtual machine without any EDR installed, which they call Windows eleven NoEDR, just to make sure that the payload actually executes and that the process stays alive <ref:2611.13684#pg1>.
Priya: So it’s a two-stage test before hitting the real target system. That second stage is where you run those candidates on a separate Windows virtual machine with the actual target EDR installed to see if they get blocked or terminated.
Nadia: The evaluation itself uses two checks: an Endpoint Execution Check and a C2 Functionality Check, which verifies end-to-end execution by checking the traffic on the C2 server during the target EDR stage.
Elias: And what they found is that even these simple models can generate successful submissions when tested against this setup. The results show that seventy one out of ninety runs—that's seventy eight point nine percent—generated a passing submission in total.
Priya: That’s a high success rate, which tells us that the evasion techniques being generated are quite effective against these specific defenses they used. But we have to remember the context here is on simulated Windows systems, not necessarily real-world attacks.
Title and authors: Nadia: True. The paper also points out some things that improve this benchmark setup, like making sure you have a clean endpoint state between tests to keep the outcomes comparable across different runs.
Elias: They also measure the impact of domain knowledge, noting that agents might lack specialized evasion knowledge, which can limit their performance unless you provide that supplementary domain knowledge as a controllable input.
Priya: So it’s not just about how smart the model is, but what specific evasion techniques they are being allowed to use to guide their search.
Nadia: And the feedback mechanism is also crucial here. The paper suggests using detailed endpoint feedback instead of just a simple pass or fail signal so agents can actually see where they failed and revise their submissions.
Elias: That’s important because it helps them learn how to fix the loader when it gets stopped by something specific, like a private read write execute allocation or an inline patch.
Priya: So, for someone just listening here, the main point is that we now have a systematic way to test if AI agents can bypass endpoint detection and response systems on Windows without relying on just vague guesses about what might work.
Nadia: Right. And this gives us a concrete benchmark to measure how capable these models are at developing actual malicious code execution chains against modern defenses.
Elias: It’s interesting because the results show that even low cost backends like Flash and Lite can match or exceed results from larger models like GPT six luna in some areas of evasion generation.
Priya: That suggests that the complexity of the underlying model isn't the only factor; how well those specific AI agents are guided by their input, like that curated evasion knowledge skill, really matters.
Nadia: So to wrap up this section, BypassGym gives us a structured way to test evasion capabilities by combining setup modules and detailed feedback loops. It shows that even with limited knowledge and budget constraints, certain models can achieve high success rates against endpoint defenses.
Elias: And it highlights that precise block or alert reasons help defenders understand exactly what mechanism was triggered, which helps them pinpoint the interception point for future attempts.
Priya: It means if a defender gets detailed feedback on why a loader failed, they can use that specific detail to design the next attempt to bypass it.
Nadia: So we’ve seen how this benchmark structures the problem and what kind of performance we're seeing from different models in terms of evasion success. Next up, we’re going to look at how those results translate into practical defense strategies, which is where I think things get really interesting for us.
The paper's summary: Elias: The big finding is that even these simple AI models can generate successful submissions when tested against this setup. Seventy-one out of ninety runs, that’s seventy-eight point nine percent success rate overall.
Nadia: That success rate is high, which tells us the evasion techniques being generated are quite effective against these specific defenses they used. But we gotta remember this was on simulated systems, not necessarily real attacks in the wild.
Priya: Exactly. We need to be careful about how much we trust those numbers when it’s all behind a virtual machine and a controlled setup. What does the methodology actually look like?
Elias: They set up three main parts for BypassGym, first you have the Evasion Task Setup module which gives the AI agent a fixed shellcode payload and some curated evasion knowledge skill. Then there’s the Shellcode Loader Development module where they give it a Windows compiler to build and revise its loader. And finally, there’s the EDR Evaluation and Feedback module.
Nadia: That third part is key because it runs those generated bundles on a Windows eleven virtual machine without any EDR at first, just to check if the payload actually executes and stays alive. Then they run it against a real target endpoint with an EDR installed to see if it gets blocked or terminated.
Priya: I’m thinking about what that feedback loop means for us. The paper mentions using detailed endpoint feedback instead of just a simple pass or fail signal, so the AI can actually see which checks failed and revise its submission accordingly.
Elias: That’s the point, Priya, it helps them learn how to fix things when something specific stops them. And they also found that external evasion knowledge really helps agents solve the task by giving them workable strategies and code they can reuse.
Nadia: It shows that the scaffold—that framework you give an agent—actually matters because it affects how the agent looks at inputs, builds candidates, and handles failures; different scaffolds yield similar success rates depending on what’s in there.
Priya: So this tells us that we don't just need a super smart model; we need to guide that model with specific evasion knowledge and detailed feedback to get usable results.
Elias: It also points out how reasoning effort impacts the results, for example, one model showed a range of success rates depending on how much reasoning effort you give it.
Nadia: Ultimately, the implication is that low-cost defense evasion is already working in practice with these setups. And what they found is that precise block or alert reasons help defenders pinpoint exactly where the interception happened so they can design the next attempt better.
Priya: If a defender gets that specific detail on why a loader failed, they can use that to design the next attempt to bypass it more effectively.
Nadia: This gives us a structured way to test evasion capabilities by combining all those setup modules and feedback loops. And it shows how capable these models are at developing actual malicious code execution chains against modern defenses.
Elias: It’s interesting because the results show that even low cost backends like Flash and Lite can match or exceed results from larger models like GPT six luna in some areas of evasion generation.
Nadia: So the complexity of the underlying model isn't the only factor; how well those specific AI agents are guided by their input really matters for these kinds of tasks.
Elias: And this leads us to thinking about what’s next, because we need to know how these findings translate into practical defense strategies.
The paper's improvements: Nadia: So, we're looking at how they suggest improving this BypassGym benchmark to make it even better for testing AI evasion skills. The paper points out several ways to make the evaluation process more useful for researchers and defenders alike.
Elias: They are suggesting an automated EDR evaluation service that can take those generated loaders and run them on real endpoints, assess their behavior under deployed EDR systems, and then return structured execution results.
Priya: That sounds way better than just a pass or fail signal because it lets the AI agents revise their submissions based on specific failure points instead of just getting a binary outcome.
Nadia: Exactly, Priya. It moves the feedback from a simple verdict to actionable data that helps guide the revision process directly.
Elias: And they emphasize that adding evasion knowledge skills gives agents workable strategies, which is important because the value of those specific scaffolds depends on the model you're using and how much reasoning effort you want to put in.
Priya: That means developers can actually select a scaffold optimized for a specific model's strengths in generating evasive code, instead of just throwing tools at it blindly.
Nadia: They also highlight that detailed EDR feedback is better than binary feedback because it tells the agents exactly which checks failed, which helps them target their revisions more effectively.
Elias: For instance, one agent revised a loader after a failure by considering specific mechanisms like private read write execute allocation or an inline patch to fix the issue.
Priya: That specificity is what makes this research valuable for privacy and measurement because it shows us *how* the evasion actually works at the system level.
Nadia: It also shows that agents can adapt their behavior based on their reasoning effort; for example, one model showed a range of success rates depending on how much reasoning effort you gave it, like from eleven to eighteen successful runs across different levels.
Elias: That tells us that tailoring the cognitive load you put on the AI can actually improve success rates depending on its sensitivity.
Priya: So what this means for us is that we need to look beyond just the final success rate and really dig into those specific failure modes to understand how these agents operate in real-world security environments.
Nadia: It also confirms that low cost backends, like Flash and Lite, can match or even exceed results from larger models like GPT six luna when it comes to evasion generation, which is a significant finding.
Elias: So the conclusion here is that while large models are powerful, the way you guide them with specific knowledge and feedback can make a smaller model highly effective for certain tasks.
Priya: The limitation they mention is that this benchmark is still on simulated systems, so we need to keep an eye on how these results translate when they hit actual production environments.
Nadia: Right, and that brings us right back to the next topic: how these evasion capabilities actually change what defenses are needed in the real world.
Conclusion: Nadia: So to wrap up, we’ve seen how the BypassGym benchmark systematically tests if AI agents can bypass endpoint defense systems on Windows using a loader that executes shellcode.
Elias: The core takeaway is that even low cost backends can match or exceed results from larger models when it comes to generating evasion techniques under these specific testing conditions.
Priya: From what the data shows, this gives us a much clearer picture of the actual capabilities of these agents when they are constrained by specific tools and knowledge skills.
Nadia: And the implication is that defenders need to focus on getting that detailed endpoint feedback because it tells them exactly where to look for the vulnerability they need to patch.
Elias: It suggests that understanding how an AI agent reasons about building a loader isn't just about its raw power but also about the specific structure of the problem you present it with.
Priya: I think what this means for measurement is that we need more frameworks like Honeyval to see how well these agents perform when they are used in more complex, real-world testing scenarios.
Nadia: Exactly, Priya. and that brings us to a huge question: if AI can do this kind of evasion reliably on simulated systems, how much real-world risk does that actually pose?
Elias: It opens the door for us to think about how we design defenses that are not just looking for known signatures but are looking for novel execution behaviors.
Priya: That’s what I want to explore next: what specific detection methods can actually catch these sophisticated, AI-generated loaders before they get a chance to execute payload?
Nadia: We’ll look at the research coming out on that very soon.
More episodes
- 2608.27496-ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection
- Daily Summary for 2026-11-19
- 2607.07085-Is Randomness Necessary for Adaptive Data Analysis?
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2609.10601-Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows
- 2610.38786-Adversarial Process Disturbances at a Deployable Alarm Budget: Detector Evaluation on a Simulator and a Real Plant
- 2511.08660-Binary and Multiclass Cyberattack Classification on GeNIS Dataset
- 2604.27302-Static Attribution of Android Residential Proxy Malware Using Graph Kernels
- Daily Summary for 2026-11-18