Defender Policy Evaluation and Resource Allocation Using MITRE ATT&CK Evaluations Data

arXiv:2107.04075 · cs.CR, cs.GT · Submitted 2021-07-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: I'm Nadia.

Elias: And I'm Elias, here with Priya.

Nadia: We're looking at the paper: "Defender Policy Evaluation and Resource Allocation Using MITRE ATT&CK Evaluations Data".

Elias: Here's the gist of the paper. Defender Policy Evaluation and Resource Allocation Using MITRE ATT&CK Evaluations Data presents a novel methodology that applies game-theoretic approaches to attack, attacker, and defender data derived from MITRE’s ATT&CK® Framework to enable defender policy optimization and resource allocation against uncertain multi-step attacks.

Nadia: The paper introduces a novel methodology that applies a game-theoretic approach to the attack, attacker, and defender data derived from MITRE’s ATT&CK® Framework to enable defender policy optimization and resource allocation against uncertain multi-step attacks.

Elias: Time to complete attack steps is drawn from a probability distribution determined by attacker and defender strategies and capabilities.

Nadia: By approximating attacker-defender games as Markov processes, the authors represent the attacker-defender interaction, estimate the attack success parameters, determine the effects of attacker and defender strategies, and maximize opportunities for defender strategy improvements against an uncertain stream of attacks.

Paper summary: Nadia: To recap, this paper presents a novel methodology that uses game theory to analyze attack, attacker, and defender data from the MITRE ATT andCK Framework. It claims this approach allows defenders to optimize policy and allocate resources against uncertain multi-step attacks by modeling interactions as Markov processes.

Elias: Exactly. The core thesis is that you can use these models to estimate attack success parameters based on attacker and defender strategies, even when those steps have uncertain durations or timing. They approximate the attacker-defender games as Markov processes to represent the interaction dynamics.

Priya: So, I'm hearing that they are taking real-world data from MITRE ATT andCK Evaluations and using it to feed into this game theory model for policy evaluation, which is what makes it practical for real systems?

Nadia: That's right. They incorporate empirical observations from ATT andCK Evaluations into the attack success metrics and defender response evaluations. This lets them aggregate data across different parts of an attack, not just look at isolated steps.

Elias: The method specifically involves translating a GPLADD game, which represents the attacker-defender game through an attack graph, into a discretetime Markov chain representation to model these dynamics accurately.

Priya: It sounds complex because of the modeling—the GPLADD game and the Markov chain approximation—but what's the practical value over just using simpler detection rules? What's the data really showing us about this approach?

Nadia: The value lies in quantifying the effects of specific defender actions. They calculate sensitivity metrics to see how changing resource allocation at one step impacts a metric like time to success distribution or steady state distribution.

Elias: Another key output they focus on is that fraction of time an attacker can achieve in a completed or ready state, which they call the "Ready" state residence time, as you heard earlier. This helps evaluate the defender's strategy directly against that uncertainty.

Priya: So if I had to boil it down for someone who isn't deep in game theory: this paper gives us a way to systematically test and compare different defense resource allocation strategies by feeding them real attack data from ATT andCK Evaluations into a probabilistic model.

Conclusion: Nadia: Looking at the title, "Defender Policy Evaluation and Resource Allocation Using MITRE ATT andCK Evaluations Data," it really tells you what this work is doing. It’s not just about finding new malware; it’s about optimizing how defenders spend their money and effort given the complexity of multi-step threats.

Elias: And the authors, Outkin, Schulz, Tarman, and Pinar, they are clearly focused on bridging game theory with real threat data from ATT andCK to create a framework for proactive defense planning. It's about moving from reaction to informed allocation.

Priya: What does this mean for the broader security field if we take the conclusion they draw—that optimized effort allocation might be more important than just making a single step substantially harder? What’s that saying practically?

Nadia: It means defenses shouldn't focus on a massive, expensive fix for just one vulnerability. Instead, it suggests distributing resources across different parts of the system to manage the attack progression dynamically. You allocate based on where the attacker is most likely to succeed next.

Elias: That shifts the focus from brute-force defense to strategic management of uncertainty. It’s about managing your response capabilities across a whole process, not just patching one specific hole in it.

Priya: So, in simple terms, this paper provides a mathematical tool that lets security teams evaluate how their current policies would fare against known attack patterns before they even deploy those policies in the real world. It's a simulation for defensive policy.

Nadia: Precisely. It offers a way to use empirical data to guide where you should invest your limited defense budget for maximum risk reduction against complex attacks documented in ATT andCK.

Sandia National Laboratories

cs.CR, cs.GT

Submitted: 2021-07-08

Updated: 2021-07-08

DOI: 10.1109/TDSC.2022.3165624

Code: https://github.com/mitre-attack/attack-evals

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 75/100

The gist: Defender Policy Evaluation and Resource Allocation Using MITRE ATT&CK Evaluations Data presents a novel methodology that applies game-theoretic approaches to attack, attacker…

Key concepts

Game-Theoretic Approach
This involves modeling the interaction between an attacker and a defender as a strategic game. It helps determine the best possible strategies for both parties by considering their capabilities and objectives. This allows researchers to predict how an attacker will behave and how a defender should respond optimally in complex, uncertain attack scenarios.
Markov Chain Representation
The paper translates the complex attack-defender interaction into a Markov chain model. This simplifies the dynamics by defining states as success conditions within an attack graph. It allows for the estimation of probabilities related to moving from one state (e.g., an attack step) to another, making it easier to analyze long, multi-step attacks.
Time-to-Success Distribution
This metric measures the probability distribution of how long it takes an attacker to complete a specific attack step. By analyzing this distribution, defenders can calculate success probabilities and detection probabilities at each stage. This helps in understanding the timing and likelihood of an attack progressing through its chain.
Ready State Residence Time
This is a key metric introduced to evaluate defender strategy. It measures the fraction of time an attacker spends in a 'ready' or completed state before achieving their ultimate objective. Analyzing this time helps defenders assess how effectively they can interrupt the attack progression at various points.

Terminology

Summary

Defender Policy Evaluation and Resource Allocation Using MITRE ATT&CK Evaluations Data presents a novel methodology that applies game-theoretic approaches to attack, attacker, and defender data derived from MITRE’s ATT&CK® Framework to enable defender policy optimization and resource allocation against uncertain multi-step attacks. p. 0

Methodology Overview

The paper introduces a novel methodology that applies a game-theoretic approach to the attack, attacker, and defender data derived from MITRE’s ATT&CK® Framework to enable defender policy optimization and resource allocation against uncertain multi-step attacks. Time to complete attack steps is drawn from a probability distribution determined by attacker and defender strategies and capabilities. p. 0 By approximating attacker-defender games as Markov processes, the authors represent the attacker-defender interaction, estimate the attack success parameters, determine the effects of attacker and defender strategies, and maximize opportunities for defender strategy improvements against an uncertain stream of attacks. p. 0

Modeling Attack Dynamics

The core of their approach involves translating a GPLADD game into a Markov chain representation to model attack dynamics. A GPLADD game represents the attacker-defender game as a general stochastic process determined jointly by the attack graph and the attacker and defender capabilities and strategies. p. 3 The attack is defined by an attack graph representing success conditions where the attacker must achieve a set of terminal conditions for success. p. 4 The dynamics are approximated using Markov chains where states are the success conditions in the attack graph. p. 1

Attack Metrics and Sensitivity

The methodology allows for quantitative evaluation of defender policy against multi-step attacks by calculating attack success metrics such as time-to-success distribution and steady state distribution. The authors quantify the effects of a defender allocating more resources for detection at specific attack steps by calculating the sensitivity of attack success metrics to such changes. p. 0 A key metric introduced is the fraction of the time that an attacker can achieve in the completed or ready state (“Ready” state residence time) to evaluate defender strategy. p. 1

Empirical Application and Findings

The approach is demonstrated using data from MITRE’s APT3 ATT&CK® Evaluations overlaid on an attack scenario involving a cyber-controlled power grid. The analysis shows that the performance of different defender models against APT3 notional attack varied greatly and could mean the difference in the attacker always achieving their objectives vs. p. 2 the attacker spending the most of their time trying to regain a foothold in the system and rarely accomplishing their objectives. p. 0 The paper concludes that optimized effort allocation by defenders across different parts of their system may be more important than making a single step substantially more difficult. p. 8

Conclusion and Implications

This approach can be applied to a large set of attacks, including other attacks and attacker capabilities documented in the MITRE ATT&CK Framework to develop or improve the defender policy for a given system. The resulting attack metrics assessment can be used to develop or improve the defender policy for a given system, or to change the system if necessary to achieve cybersecurity risk reduction or other defender objectives. p. 0

References

[1] D. Agrawal, S. Baktir, D. Karakoyunlu, P. Rohatgi, and B. Sunar., Trojan detection using ic fingerprinting. p. 2, Proc. IEEE Symp. Security Privacy, pages 296–310, 2007. p. 0

[2] Tansu Alpcan and Tamer Ba¸sar., A game theoretic approach to decision and analysis in network and intrusion detection. p. 0, In Proceedings of the 42nd IEEE Conference on Decision and Control, Hawaii USA, December, 2003 IEEE, 2003. p. 0

[3] Tansu Alpcan and Tamer Ba¸sar., A game theoretic analysis of intrusion detection in access control systems. p. 6, In 43rd IEEE Conference on Decision and Control, December 14-17, 2004 Atlantis, Paradise Island, Bahamas IEEE, 2004. p. 0

[4] Sven Banisch., Markov Chain Aggregation for AgentBased Models. p. 4, Springer, 2015.

[5] R Reid Bishop, Alexander V Outkin, Brandon K Eames, and Chelsea C White. p. 0, Increasing trust in development processes using robust, data-driven markov games: An application to prestige. p. 2, IEEE Transactions on Computational Social Systems, 2020. p. 0

[6] Robert Reid Bishop., Developing Trust and Managing Uncertainty in Partially Observable Sequential DecisionMaking Environments., PhD thesis, Georgia Institute of Technology, 2019.

[7] Yu-Cheng Chen, Dustin Campbell, Vincent Mooney, Santiago Grijalva, Alexander Outkin Brandon Eames, Eric Vugrin, Ryan Helinski, and Benjamin Anthony. p. 0, Power grid bad data injection attack modeling in prestige. p. 1, In GOMACTech 2019, 2019.

[8] Elizabeth DeWaard, Casey Deccio, David Jakob Fritz, and Thomas D Tarman. p. 0, Research directions for cyber experimentation: Workshop discussion analysis. p. 7, Technical report, Sandia National Lab. p. 0 p. 0(SNL-NM), Albuquerque, NM (United States), 2017. p. 1 p. 1

[9] Xiaotao Feng, Zizhan Zheng, Prasant Mohapatra, and Derya Cansever., A stackelberg game and markov modeling of moving target defense., In Stefan Rass, Bo An, Christopher Kiekintveld, Fei Fang, and Stefan Schauer, editors, Decision and Game Theory for Security. p. 0, pages 315–335, Cham, 2017 Springer International Publishing.

[10] MITRE ATT&CK Framework. p. 0, Attack Chain Information, 2020 (accessed July 27, 2020. p. 0

[11] Jonathan Graf., Toward optimal hardware trojan detection through security economics and game theory. p. 0, In Proceedings of the 41st Annual GOMACTech Conference, pages 187–190, 2016.

[12] Jonathan Graf., Trust games: How game theory can guide the development of hardware trojan detection methods. p. 2, In Proceedings of the 2016 IEEE International Symposium on Hardware Oreinted Security and Trust (HOST), pages 91–96, IEEE, 2016. p. 0

[13] James P Hobert, Galin L Jones, and Christian P Robert., Using a markov chain to construct a tractable approximation of an intractable probability distribution. p. 1, Scandinavian Journal of Statistics, 33(1):37–51, 2006.

[14] Hannes Holm., A large-scale study of the time required to compromise a computer system. p. 3, IEEE Trans. p. 0 Dependable Secur. Comput., 11(1):2–15, January 2014.

[15] Zhisheng Hu, Minghui Zhu, and Peng Liu., Online algorithms for adaptive cyber defense on bayesian attack graphs. p. 0, In Proceedings of the 2017 Workshop on moving target defense, pages 99–109, 2017. p. 1

[16] Linan Huang and Quanyan Zhu., A dynamic games approach to proactive defense strategies against advanced persistent threats in cyber-physical systems. p. 2, Computers & Security, 89:101660, 2020. p. 0

[17] Kyle Ingols, Matthew Chu, Richard Lippmann, Seth Webster, and Stephen Boyer., Modeling modern network attacks and countermeasures using attack graphs. p. 0, In Computer Security Applications Conference. p. 1 ACSAC ’09, 2009.

[18] Y. Jin and Y. Makris., Hardware trojan detection using path delay fingerprint. p. 2, In Proc. IEEE Int. Workshop Hardw.-Orient. Security Trust, pages 51–57, 2008. p. 2

[19] Stephen Jones, Alexander Outkin, Jared Gearhart, Jacob Hobbs, John Siirola, Cindy Phillips, Stephen Verzi, Daniel Tauritz, Samuel Mulder, and Asmeret Naugle., Evaluating moving target defense with pladd. p. 1, Technical report, Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States), 2015. p. 0

[20] Stephen T Jones, Kasimir G Gabert, and Thomas D Tarman. p. 0, Evaluating emulation-based models of distributed computing systems. p. 5, Technical report, Sandia National Lab.(SNL-NM), Albuquerque, NM (United States), 2017.

[21] C. A. Kamhoua, H. Zhao, M. Rodriguez, and K. A. Kwiat., A game-theoretic approach for testing for hardware trojans. p. 0, IEEE Transactions on Multi-Scale Computing Systems, 2(3):199–210, 2016. p. 0

[22] Eric M Hutchins, Michael J Cloppert, and Rohan M Amin., Intelligence-driven computer network defense informed by analysis of adversary campaigns and intrusion kill chains. p. 9, Leading Issues in Information Warfare & Security Research, 1, 01 2011. p. 7

[23] J. P. McDermott., Attack net penetration testing. p. 0, In Proceedings of the 2000 Workshop on New Security Paradigms, pages pp. p. 915–21, New York, 2001 ACM Press. p. 9

[24] Erik Miehling, Mohammad Rasouli, and Demosthenis Teneketzis., A pomdp approach to the dynamic defense of large-scale cyber networks. p. 1, IEEE Transactions on Information Forensics and Security, 13(10):2490–2505, 2018. p. 0

[25] R. Ellison Moore and R. Linger., Attack modeling for information security and survivability. p. 2, Technical report, CMU Software Engineering Institute, 2001. p. 5

[26] Kien C Nguyen, Tansu Alpcan, and Tamer Ba¸sar., A decentralized bayesian attack detection algorithm for network security. p. 2, In IFIP International Information Security Conference, pages 413–428, Springer, 2008. p. 0

[27] STEVEN NOEL, SUSHIL JAJODIA, LINGYU WANG, and ANOOP SINGHAL., Measuring security risk of networks using attack graphs. p. 0, International Journal of NextGeneration Computing, 1(1), 2010.

[28] Alexander V. p. 0 Outkin, Brandon K. Eames, Meghan A. Galiardi, Sarah Walsh, Eric D. Vugrin, Byron Heersink, Jacob Hobbs, and Gregory D. Wyss., Gpladd: Quantifying trust in government and commercial systems a game-theoretic approach. p. 2, ACM Trans. Priv. Secur., 22(3):18:1–18:27, June 2019.

[29] Marcus Pendleton, Richard Garcia-Lebron, Jin-Hee Cho, and Shouhuai Xu., A survey on systems security metrics. p. 2, ACM Comput. Surv., 49(4):62:1–62:35, December 2016.

[30] David Pentrack, Neal Levine, Jon Lloyd, and Aman Gahoonia., Quantifying system trust and microeletronics integrity. p. 2, In Proceedings of GOMACTech-15 Government Microcircuit Applications and Critical Technology Conference, pages 211–214, 2015. p. 1

[31] Nayot Poolsappasit, Rinku Dewri, and Indrakshi Ray., Dynamic security risk management using bayesian attack graphs. p. 0, Dependable and Secure Computing, IEEE Transactions on, 9:61 – 74, 03 2012. p. 0

[32] B. Schneier., Attack trees. p. 2, Dr. Dobbs Journal, December, 1999.

[33] Branka Stojanovi´c, Katharina Hofer-Schmitz, and Ulrike Kleb., Apt datasets and attack modeling for automated detection methods: A review. p. 2, Computers & Security, 92:101734, 2020. p. 0

[34] Blake E Strom, Andy Applebaum, Doug P Miller, Kathryn C Nickels, Adam G Pennington, and Cody B Thomas. p. 0, Mitre att&ck: Design and philosophy. p. 0, MITRE Product MP, pages 18–0944, 2018. p. 2

[35] Blake E Strom, Joseph A Battaglia, Michael S Kemmerer, William Kupersanin, Douglas P Miller, Craig Wampler, Sean M Whitley, and Ross D Wolf., Finding cyber threats with att&ck-based analytics. p. 2, The MITRE Corporation, Bedford, MA Technical Report No. p. 0 MTR170202, 2017.

[36] L. P. Swiler, C. Phillips, D. Ellis, and S. Chakerian., Computer-attack graph generation tool. p. 5, In Proceedings of the Second DARPA Information Survivability Conference & Exposition (DISCEX II), volume vol. p. 5 II, pages pp 307–321, Los Alamitos, California, 2001 IEEE Computer Society. p. 0

[37] Vincent Urias and Brian Van Leeuwen., Experimental methods for control system security research. p. 2, In Cybersecurity of SCADA and Other Industrial Control Systems, pages 253–277, Springer, 2016. p. 2

[38] Brian Van Leeuwen, Vincent Urias, John Eldridge, Charles Villamarin, and Ron Olsberg., Cyber security analysis testbed: Combining real, emulation, and simulation. p. 7, In 44th Annual 2010 IEEE International Carnahan Conference on Security Technology, pages 121–126 IEEE, 2010. p. 0

[39] Brian P Van Leeuwen, William MS Stout, and Vincent E Urias., Empirical assessment of network-based moving target defense approaches., In MILCOM 2016-2016 IEEE Military Communications Conference, pages 764–769 IEEE, 2016. p. 0

[40] Neal Wagner, Cem S¸ S¸ahin, Jaime Pena, and William W Streilein., Automatic generation of cyber architectures optimized for security, cost, and mission performance: A nature-inspired approach. p. 2, In Advances in Nature-Inspired Computing and Applications, pages 1–25 Springer, 2019. p. 0

[41] Emma J Williams, Joanne Hinds, and Adam N Joinson., Exploring susceptibility to phishing in the workplace. p. 0, International Journal of Human-Computer Studies, 120:1–13, 2018. p. 9

[42] Daniel Wilt, Richard Maitzler, and John DeVale., Metrics for trust in integrated circuits. p. 2, In Proceedings of GOMACTech-08 Government Microcircuit Applications and Critical Technology Conference, pages 233–236, 2008. p. 1

[43] Allan Wollaber, Jaime Pe˜nna, Benjamin Blease, Leslie Shing, Kenneth Alperin, Serge Vilvovsky, Pierre Trepagnier, Neal Wagner, and Leslie Leonard., Proactive cyber situation awareness via high performance computing. p. 2, In 2019 IEEE High Performance Extreme Computing Conference (HPEC), pages 1–7 IEEE, 2019. p. 0

[44] J. Zhang, F. Yuan, L. Wei, Y. Liu, and Q. Xu., Veritrust: Verification for hardware trust. p. 2, IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS, 34(7):1148–1161, 2015.

[45] Quanyan Zhu, Hamidou Tembine, and Tamer Ba¸sar., Distributed strategic learning with application to network security. p. 3, In 2011 American Control Conference on O’Farrell Street, San Francisco, CA, USA June 29 - July 01, 2011, 2008.

[46] Robert Reid Bishop., Developing Trust and Managing Uncertainty in Partially Observable Sequential DecisionMaking Environments., PhD thesis, Georgia Institute of Technology, 2019.

How the training signal is built

The paper builds the Markov chain representation of the game dynamics using two methods: one based on integrating time-to-success distributions and another based directly on MITRE ATT&CK Evaluations data. The first method integrates the time-to-success distributions at each time step to estimate success probability and detection probability. p. 4 The second method estimates Markov chain parameters directly from the MITRE ATT&CK Evaluations data for the APT3 attack emulation. p. 5

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Attack success metrics calculation can be used to quantify where we will get the best return from improving detection by calculating the sensitivity of attack success metrics to such changes. This allows defenders to understand where to allocate resources for maximum impact against uncertain attacks.

  2. The system can incorporate real-world data, "such as the results of MITRE ATT&CK Evaluations, into its core metrics, allowing it to estimate attacker success parameters and defender response evaluations" based on empirical observations.

  3. The AI can model multi-step attacks by representing them as Markov processes to analyze their dynamics, including estimating the time-to-success distribution for the entire attack chain. This provides a quantitative measure of how long an attacker can remain in a dangerous state before success or detection.

  4. Defender resource allocation can be optimized by evaluating tradeoffs between defender actions, such as detection efforts at different parts of the system and at different steps of the attack. This enables policy optimization by comparing how changing resource allocation affects attack success metrics and transition diagrams.

  5. The system can generate a Markov chain representation from an attack graph, allowing it to visualize the attacker-defender game progress quickly and explicitly, facilitating better understanding of the interaction dynamics between attacker strategies and defender capabilities.

Related papers