Closing the Sim-to-Real Gap: An Evaluation Framework for Autonomous Cyber Defense Configuration of Commercial EDR

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "Closing the Sim-to-Real Gap".

Elias: Here's the gist of the paper. The gist: This paper presents an evaluation framework for autonomous defense agents hardening commercial EDRs by instantiating it in a Game of Active Directory lab with Horizon3.ai’s NodeZero as the autonomous pentester and Microsoft Defender XDR as the EDR, reporting three lessons learned that highlight a sim-to-real gap for enterprise defense and motivate evaluation methodology for benchmarking autonomous defense agents in environments with black-box, autonomous tools.

Nadia: The need for an evaluation framework stems from several critical gaps in existing literature regarding commercial EDRs and cyber defense configurations In this section, we motivate the need for an evaluation framework that integrates commercial EDR by examining gaps in MITRE ATT&CK technique coverage, differences in vendor-claimed and observed EDR behavior, false positives and lack of per-policy attribution, and black-box, autonomous features Attack Coverage Gap: Recent research has demonstrated that commercial EDRs exhibit coverage gaps of the MITRE ATT&CK techniques

13: and multi-step, chained attacks EDR systems do not natively reason about kill-chain causality and fragment multi-step attacks into single-step alerts

14: ,

15: In 2024, Virkud et al.

16: found that EDR detection rule sets from Carbon Black, Splunk, and Elastic only cover about 48% to 55% of MITRE ATT&CK techniques and that coverage of an ATT&CK technique does not guarantee coverage of the same real-world attacks Expected vs.

Nadia: Observed Behavior Gap: Vendor-claimed, expected EDR effectiveness diverges substantially from observed performance under real-world attacks Karantzas and Patsakis

17: empirically assessed eleven commercial EDR products against four Advanced Persistent Threat (APT) scenarios and found that most products neither block nor log the majority of attacks

17: They also echo the need for holistic event analysis for effective defense and identify assessing true-positive, false-negative, false-positive results produced by EDRs and the response time of EDRs during real-world scenarios as a critical gap in the literature

17: False Positives and Per-Policy Attribution Gap: In a four-year longitudinal study of real-world network alert logs from Security Operations Centers (SOCs), Yang et al.

Title and authors: Nadia: So this paper is called "Closing the Sim-to-Real Gap: An Evaluation Framework for Autonomous Cyber Defense Configuration of Commercial EDR". It sounds like they’re trying to figure out how to test defenses that are now running on their own, not just following simple rules.

Elias: Yeah, that's the core idea. They’re looking at these commercial endpoint detection and response products where the AI components are starting to operate alongside or even instead of what a human operator sets up.

Nadia: It means we're moving from tuning a static tool to dealing with something black-box that can make vendor-specific decisions on its own.

Elias: Exactly, they’re testing those autonomous defense agents by putting them in a Game of Active Directory lab with NodeZero as the attacker and Microsoft Defender XDR as the EDR.

Nadia: So, what's the big deal about that setup? It sounds like a very specific test bed for seeing how these agents actually perform under pressure.

Elias: Well, the whole point is to report on three lessons learned that neither simulation nor open-source EDR evaluation can surface. They want to see what happens when you combine an autonomous attacker with a commercial EDR in this way.

Priya: I’m interested in those lessons, because right now we don't really know how much we can trust the results from these automated configurations when they are operating like that.

Nadia: It sounds like they’re setting up a system to see if these autonomous agents can actually harden the EDR effectively in a way that matters in the real world.

Elias: That's right, the framework itself is about integrating commercial EDR by looking at coverage gaps, observed versus claimed behavior, and those tricky false positives.

The paper's summary: Nadia: Okay, so the summary of this paper explains that there are several big gaps in how we currently evaluate commercial EDRs. They point out issues like not enough MITRE ATT andCK technique coverage for what these tools actually detect, and they don't account for how multi-step attacks get fragmented into single alerts.

Elias: And then there’s the gap between what vendors claim their EDR can do and what it actually does when you run real attacks. They also highlight the problem of false positives overwhelming analysts, which is a huge issue in SOCs today.

Priya: That aspect about false positives is really important for me because if an agent makes a change that just floods the system with noise, then it’s not helping defense at all.

Nadia: Right. And they also point out that EDR policy optimization is this incredibly complex, combinatorially intractable problem when you have so many settings available in something like Microsoft Defender for Endpoint.

Elias: That leads into the final gap they address, which is these black-box autonomous features embedded in things like Microsoft Defender XDR. You can have automation toggles that are all one thing, and you don't really get visibility into what’s driving the response decisions.

Nadia: So they’re basically saying we need a way to evaluate these autonomous systems, not just their individual components, but how they work together in this feedback loop.

Elias: Precisely. The paper proposes a closed-loop evaluation framework where an agent proposes a policy change, you run it through a commercial pentest like NodeZero, and then the results feed back to the agent for another round of decisions.

The paper's improvements: Nadia: So they outline some improvements they suggest for this whole process. One big thing is decoupling experiment orchestration from the policy-decision algorithms using a pluggable proposer interface.

Elias: That way, you can test different AI architectures or models against the same testing harness without rewriting the whole framework every time, which makes benchmarking across different agentic designs much cleaner.

Priya: And they also suggest incorporating a deterministic gating mechanism before deployment to check transition cost budgets and game value.

Nadia: That means they want to make sure that when an agent proposes a policy change, it actually fits within defined operational budgets and doesn't just open the door for more attacker gain.

Elias: It also points toward being able to provide per-policy attribution, separating what the defense agent does from what the autonomous EDR does.

Priya: That would be useful because it lets us quantify exactly how much of the observed block came from our policy tuning versus just the vendor's built-in autonomous features.

Nadia: And they also suggest accounting for variable EDR behavior during evaluation windows by treating that evolving layer as a measurement axis, meaning you have to run ablation studies on things like automation levels.

Conclusion: Elias: So to wrap up the "Closing the Sim-to-Real Gap: An Evaluation Framework for Autonomous Cyber Defense Configuration of Commercial EDR" paper, it’s about establishing a structured way to evaluate these autonomous defense agents by running them in a controlled loop against a real attacker.

Nadia: They found that this closed-loop process, using NodeZero and Microsoft Defender XDR in the GOAD lab, gives us concrete lessons about what we don't know about commercial EDRs right now.

Priya: From my side, it sounds like the most important thing is moving toward having per-policy attribution so we can actually measure if our tuning efforts are paying off in terms of risk reduction.

Elias: And they emphasize that this framework helps us see how these black-box autonomous features interact with operator choices and whether the agent can actually make good decisions under those constraints.

Nadia: So, it’s a way to test hardening without risking actual production systems by simulating the real-world impact of tuning an automated defense tool.

Priya: I think this is a solid step toward making sure that when we deploy these autonomous agents, we have a method to verify their behavior and not just trust the vendor's marketing materials.