MEvince: Knowledge-Constrained LLM Reasoning for Technique Identification from Malware Reports
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "MEvince: Knowledge-Constrained LLM Reasoning for Technique Identification from Malware Reports".
Elias: Here's the gist of the paper. Mapping behavioral intelligence in malware analysis reports to ATT&CK techniques supports threat analysis and defense but reliable identification remains challenging The paper presents MEvince, a knowledge-constrained LLM reasoning framework for report-level malware technique identification Its core is a shared behavioral representation connecting report observations to technique definitions, specifying what evidence to extract and how it supports technique mapping Pruning redundant reasoning tokens The paper introduces MEvince, a knowledge-constrained LLM reasoning framework for report-level malware technique identification The key principle is that external knowledge guides interpretation while every assignment must be supported by observed report evidence This representation is built through agentic mining of technique documentation and supports two online stages These stages are 1) primitive-guided extraction which uses behavioral primitives to structure LLM-extracted observations, link them to source passages, and group dispersed evidence; and 2) evidence-aligned retrieval which matches this evidence to technique profiles through the same concepts Controls at both stages validate evidence references and check technique-specific requirements How the training signal is built The core of MEvince is a shared concept foundation built through offline agentic mining of technique description documentation This foundation connects report observations with technique definitions through behavioral primitives that provide a common basis on which evidence mining and retrieval can be built Behavioral primitives are defined as anchors as consistent concept labels mined from reports that can identify behavior or an artifact in context The vocabulary is the disjoint union C = CA ∪CO ∪CG ∪CM of action, object, goal, and mechanism concepts The knowledge base K is defined as K = C, G, Pt t ∈ T, (1) where G contains reusable semantic groups A technique t has a profile Pt = (Φt, St, Rt, Qt), where Φt contains its concept sets Where the gains come from The framework achieves macro F1 scores of 0.310 at exact-ID granularity and 0.386 at parent-technique granularity, outperforming the baselines under the same evaluation setting MEvince improves exact-ID F1 over the strongest baseline by 17.4% Concept-guided retrieval increases goldlabel candidate coverage from 31.6% to 47.9% Controlled ablations show that concept-guided retrieval improves candidate coverage beyond evidence selection alone The evidence-memory revision yields further gains without updating model parameters Output Control and Verification MEvince checks whether extracted observations can be traced back to the report and whether proposed technique assignments have sufficient support in the report The two controls operate after extraction and mapping, so LLM responses are not treated as final decisions Evidence-unit control normalizes and source-aligns the extracted ledger, assigning evidential statuses like observed or contextual Claim verification resolves these references and checks the claim against the report evidence and requirements Qt This verification process ensures that acceptance requires observed behavior concerning the malware under analysis or explicit capability evidence permitted by a matching rule Evidence-Memory Adaptation MEvince optionally uses labeled training reports to adjust which retrieved techniques the LLM must explicitly judge This optional block uses an evidence memory learned from labeled examples to refine candidate selection without fine-tuning model parameters The memory can move an alternative already in the retrieved packet into the mandatory decision queue, replacing its weakest-scoring member while keeping the base concept vocabulary unchanged This adaptation yields a net gain of 0.316 for exact-ID and 0.394 for parent-level macro F1 scores The framework improves candidate coverage while reducing false-positive assignments, and provenance audits confirm that accepted assignments remain traceable to source evidence We presented MEvince, a knowledge-constrained LLM reasoning framework for report-level malware technique identification The framework aligns dispersed report observations with ATT&CK technique definitions through a shared behavioral representation while maintaining a clear distinction among source evidence, interpretive knowledge, and unverified hypotheses Evaluations against 11 baselines on public malware reports demonstrate that MEvince improves fine-grained technique identification The framework improves candidate coverage while reducing false-positive assignments and provenance audits confirm that accepted assignments remain traceable to source evidence References
1: Gbadebo Ayoade, Swarup Chandra, Latifur Khan, Kevin Hamlen, and Bhavani Thuraisingham.
Elias: Automated threat report classification over multi-source data.
Nadia: In 2018 IEEE 4th International Conference on Collaboration and Internet Computing (CIC).
Paper summary: Nadia: So we're looking at this paper called "MEvince: Knowledge-Constrained LLM Reasoning for Technique Identification from Malware Reports". The main idea here is trying to map the behavioral intelligence found in malware reports to ATT andCK techniques, but it's really hard because the evidence you need is usually scattered across a long, messy report.
Elias: Exactly. They're pointing out that while large language models can use external knowledge to interpret these reports, they often end up assigning techniques based on just plausible associations instead of having enough solid evidence.
Priya: So what's the core claim here? What exactly does this paper propose that's different from just using a standard retrieval system like RAG or direct prompting?
Nadia: They introduce MEvince, which is a knowledge-constrained LLM reasoning framework. The key principle they are pushing is that every assignment has to be supported by the actual observed evidence in the report, guided by external knowledge.
Elias: It builds this representation through two main online stages. First, there's primitive-guided extraction where they use behavioral primitives to structure what the LLM extracts, link those observations back to specific passages in the source document, and then group up that dispersed evidence.
Priya: And then there's the second stage where it does evidence-aligned retrieval, matching this structured evidence against established technique profiles using the same concepts. That sounds like a very specific way to handle that scattered data.
Nadia: It’s designed to control both stages by validating those references and checking if the proposed technique assignments actually meet all of the requirements for that specific technique.
Elias: The foundation for this whole thing is built offline through agentic mining of technical documentation. They create a shared concept foundation using behavioral primitives, which are essentially anchor labels mined from reports that identify a behavior or an artifact in context.
Priya: So those primitives are like the common language that connects the observations in the report to the definitions of techniques? What kind of vocabulary are they talking about?
Elias: They define this vocabulary as a disjoint union of action, object, goal, and mechanism concepts. This knowledge base is defined by these concepts and technique profiles where each technique has a profile with concept sets.
Paper summary: Nadia: The results they show are pretty interesting too. They get macro F1 scores of zero point three hundred ten at the exact-ID level and zero point three hundred eighty-six at the parent-technique level, which they say outperforms baselines under the same testing conditions.
Elias: They note that MEvince improves that exact-ID F1 score over the strongest baseline by a factor of seventeen point four percent. But they also show that concept-guided retrieval helps goldlabel candidate coverage go up from thirty-one point six percent to forty-seven point nine percent.
Priya: That jump in candidate coverage is significant, but I want to know what the real data shows us about precision versus recall here. RAG gets the highest exact-ID recall at zero point four hundred twenty-four but its precision only hits about zero point one hundred ninety-one.
Nadia: MEvince actually manages to double that precision to around zero point three hundred eighty-four while keeping that high recall. They also show that this evidence-memory revision gives further gains without needing to update the actual model parameters.
Elias: That adaptation uses labeled training reports to tell the LLM exactly which retrieved techniques it has to judge, and it refines candidate selection by moving a weaker member into the mandatory decision queue. This optional block yielded a net gain of zero point three hundred sixteen for exact-ID and zero point three hundred ninety-four for parent-level macro F1 scores.
Priya: That sounds like they are using learned examples to guide the decision process without needing a full fine-tuning session on the model itself. What about verification? How do we know this isn't just hallucination?
Nadia: They have output control and verification steps that happen after extraction and mapping. These checks ensure that every extracted observation can be traced back to the original report, and that any technique assignment has sufficient support in the report itself.
Elias: Specifically, they have evidence-unit control which normalizes the ledger of extracted data, assigning statuses like observed or contextual to each piece of evidence. Then there's claim verification which resolves those references and checks the claim against both the report evidence and those required technique elements.
Paper summary: Priya: So if we look at what this means for us in practice, what's the real-world implication of having a framework that prioritizes traceable, supported assignments over just high recall numbers?
Nadia: It means that when we are trying to automate threat analysis from these massive reports, we can get better results because the assignments aren't just plausible guesses; they are tied directly to verifiable facts in the source material.
Elias: The paper, "MEvince: Knowledge-Constrained LLM Reasoning for Technique Identification from Malware Reports," suggests that by building this shared behavioral representation and controlling both extraction and retrieval stages, we can get better results than standard RAG methods which often produce too many false positives.
Priya: It seems like a way to bridge the gap between the LLM's ability to use external knowledge and the need for grounded evidence.
Nadia: Right, so for anyone doing threat intelligence analysis, this framework suggests that grounding your AI reasoning in explicit report evidence is what actually moves you past just getting high recall figures.
Elias: And they also highlight a limitation: the framework relies on that shared concept foundation built from offline agentic mining of documentation. If those initial behavioral primitives aren't good, the whole system starts there.
Priya: That makes sense, since those primitives are what define the common language for everything else in MEvince.
Nadia: So, to wrap up this paper on "MEvince: Knowledge-Constrained LLM Reasoning for Technique Identification from Malware Reports," it’s a framework that uses structured reasoning and explicit control to ensure AI assignments are traceable and supported by evidence from the report.
Elias: The implication is that we can improve fine-grained technique identification while reducing those false positive assignments you see in retrieval-augmented generation methods.
Priya: It gives us a tool where we can audit *why* the AI picked a certain technique, which is crucial when you need to trust the output for defensive action.
Nadia: And it shows that even without updating the core model parameters, adapting it with evidence memory can still give you measurable improvements in performance metrics.
Elias: So, as we look at this work on "MEvince: Knowledge-Constrained LLM Reasoning for Technique Identification from Malware Reports," the main point is using a knowledge-constrained approach to make LLM reasoning more grounded and verifiable.
Conclusion: Nadia: So we've been looking at this paper called "MEvince: Knowledge-Constrained LLM Reasoning for Technique Identification from Malware Reports."
Elias: Yeah, it’s about using a knowledge-constrained framework to figure out which ATT andCK technique is in a malware report.
Priya: It seems like they're trying to solve that big problem where the AI just makes up techniques instead of finding the real ones.
Nadia: Exactly. They build this system so every answer has to be backed up by actual evidence from the report, guided by some external knowledge base they create themselves.
Elias: The authors say their core idea is connecting what you see in the report directly to what a technique actually is through these behavioral concepts.
Priya: And they show that this approach works better than just letting the AI wander around with general knowledge. They get higher scores, like zero point three hundred ten at the exact identification level.
Nadia: The paper highlights that this method improves candidate coverage, meaning it finds more potential techniques to consider before narrowing down to the right one.
Elias: It’s interesting because they show that even when you use their optional memory adaptation, which uses labeled examples, you still get those gains without having to re-train the main model parameters.
Priya: So what does this mean for a cybersecurity analyst who is trying to automate threat intelligence? It suggests that grounding the AI's reasoning in traceable facts makes its output way more reliable.
Nadia: Right. It moves it from just being a fancy guess to something you can actually audit because you can trace the assignment back to the source document.
Elias: The conclusion of this paper is that MEvince successfully aligns those scattered report observations with the definitions of ATT andCK techniques using this shared behavioral representation.
Priya: It’s a framework that clearly separates what's observed evidence from what's just interpretive knowledge or unverified hypotheses, which is really important for privacy and measurement research.
Nadia: It sets a new standard for how we can constrain these large language models to perform high-stakes tasks like threat mapping in practice.
Elias: We’ll be looking at how these concept foundations they built from mining technique documentation shape the future of AI reasoning systems next.
Hongyu She, Yiling He, Mengda Xie, Yiping Liu, Xingzhi Qian, Zhan Qin, Lorenzo Cavallaro
Zhejiang University · University College London · Jiangsu Police Institute · Alibaba Group
cs.CR, cs.CL, cs.IR
Submitted: 2026-11-05
Updated: 2026-11-05
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 100/100
The gist: Mapping behavioral intelligence in malware analysis reports to ATT&CK techniques supports threat analysis and defense but reliable identification remains challenging The paper presents MEvince…
Key concepts
- Behavioral Primitives
- These are consistent concept labels, such as 'action' or 'object,' mined from reports that describe specific behaviors or artifacts. They serve as anchors to connect raw report observations with the formal definitions of malware techniques, providing a common vocabulary for evidence mining.
- Shared Concept Foundation (Knowledge Base K)
- This is the core knowledge base built by mining technique documentation offline. It connects report observations to technique definitions using behavioral primitives. This foundation ensures that both evidence extraction and retrieval stages operate on a unified semantic basis, making the mapping process more robust.
- Evidence-Aligned Retrieval
- This is the second stage of MEvince where extracted evidence is matched against technique profiles. It uses the shared concepts to find relevant technique definitions. This stage validates that proposed technique assignments are directly supported by observed evidence in the report, improving accuracy over simple retrieval.
- Output Control and Verification
- This control mechanism checks two things after mapping: first, if extracted observations can be traced back to the original report text, and second, if the proposed technique assignment has sufficient support. This ensures that final decisions are grounded in observed behavior rather than just LLM interpretation.
Terminology
Summary
Mapping behavioral intelligence in malware analysis reports to ATT&CK techniques supports threat analysis and defense but reliable identification remains challenging The paper presents MEvince, a knowledge-constrained LLM reasoning framework for report-level malware technique identification Its core is a shared behavioral representation connecting report observations to technique definitions, specifying what evidence to extract and how it supports technique mapping
Pruning redundant reasoning tokens
The paper introduces MEvince, a knowledge-constrained LLM reasoning framework for report-level malware technique identification The key principle is that external knowledge guides interpretation while every assignment must be supported by observed report evidence This representation is built through agentic mining of technique documentation and supports two online stages These stages are 1) primitive-guided extraction which uses behavioral primitives to structure LLM-extracted observations, link them to source passages, and group dispersed evidence; and 2) evidence-aligned retrieval which matches this evidence to technique profiles through the same concepts Controls at both stages validate evidence references and check technique-specific requirements
How the training signal is built
The core of MEvince is a shared concept foundation built through offline agentic mining of technique description documentation This foundation connects report observations with technique definitions through behavioral primitives that provide a common basis on which evidence mining and retrieval can be built Behavioral primitives are defined as anchors as consistent concept labels mined from reports that can identify behavior or an artifact in context The vocabulary is the disjoint union C = CA ∪CO ∪CG ∪CM of action, object, goal, and mechanism concepts The knowledge base K is defined as K = C, G, Pt t ∈ T, (1) where G contains reusable semantic groups A technique t has a profile Pt = (Φt, St, Rt, Qt), where Φt contains its concept sets
Where the gains come from
The framework achieves macro F1 scores of 0.310 at exact-ID granularity and 0. p. 1 MEvince achieves macro F1 scores of 0. 310
at exact-ID granularity and 0. 386 at parent-technique granularity,
outperforming the baselines under the same evaluation setting. Controlled ablations show that concept-guided retrieval improves
candidate coverage beyond evidence selection alone, while claim
verification reduces false-positive assignments. MEvince improves exact-ID F1 over the strongest
baseline by 17. 4%, and its concept-guided retrieval increases goldlabel candidate coverage from 31.Where the paper says it
Where the paper says it
Where the paper says it
Output Control and Verification
MEvince checks whether extracted observations can be traced back to the report and whether proposed technique assignments have sufficient support in the report The two controls operate after extraction and mapping, so LLM responses are not treated as final decisions Evidence-unit control normalizes and source-aligns the extracted ledger, assigning evidential statuses like observed or contextual Claim verification resolves these references and checks the claim against the report evidence and requirements Qt This verification process ensures that acceptance requires observed behavior concerning the malware under analysis or explicit capability evidence permitted by a matching rule
Evidence-Memory Adaptation
MEvince optionally uses labeled training reports to adjust which retrieved techniques the LLM must explicitly judge This optional block uses an evidence memory learned from labeled examples to refine candidate selection without fine-tuning model parameters The memory can move an alternative already in the retrieved packet into the mandatory decision queue, replacing its weakest-scoring member while keeping the base concept vocabulary unchanged This adaptation yields a net gain of 0.316 for exact-ID and 0. p. 0 nnnnnnn
We evaluate MEvince against 11 baselines spanning four method
families using a dataset of 610 public malware reports referenced by
MITRE ATT&CK. MEvince improves exact-ID F1 over the strongest
baseline by 17.Where the paper says it
Conclusion
We presented MEvince, a knowledge-constrained LLM reasoning framework for report-level malware technique identification The framework aligns dispersed report observations with ATT&CK technique definitions through a shared behavioral representation while maintaining a clear distinction among source evidence, interpretive knowledge, and unverified hypotheses Evaluations against 11 baselines on public malware reports demonstrate that MEvince improves fine-grained technique identification The framework improves candidate coverage while reducing false-positive assignments and provenance audits confirm that accepted assignments remain traceable to source evidence
References
[1] Gbadebo Ayoade, Swarup Chandra, Latifur Khan, Kevin Hamlen, and Bhavani Thuraisingham. 2018. Automated threat report classification over multi-source data. p. 8 References
[1] Gbadebo Ayoade, Swarup Chandra, Latifur Khan, Kevin Hamlen, and Bhavani
Thuraisingham. Automated threat report classification over multi-source
data. Automated threat report classification over multi-source
data. In 2018 IEEE 4th International Conference on Collaboration and Internet
Computing (CIC).Where the paper says it
Where the paper says it
[2] Markus Bayer, Philipp Kuehn, Ramin Shanehsaz, and Christian Reuter. 2024. Cysecbert: A domain-adapted language model for the cybersecurity domain. p. 8 [2] Markus Bayer, Philipp Kuehn, Ramin Shanehsaz, and Christian Reuter. Cysecbert: A domain-adapted language model for the cybersecurity domain.Where the paper says it
[3] Marvin Büchel, Tommaso Paladini, Stefano Longari, Michele Carminati, Stefano Zanero, Hodaya Binyamini, Gal Engelberg, Dan Klein, Giancarlo Guizzardi, Marco Caselli, et al. 2025. SoK: Automated TTP Extraction from CTI Reports–Are We There Yet? p. 8 [3] Marvin Büchel, Tommaso Paladini, Stefano Longari, Michele Carminati, Stefano
Zanero, Hodaya Binyamini, Gal Engelberg, Dan Klein, Giancarlo Guizzardi, Marco
Caselli, et al. SoK: Automated TTP Extraction from CTI Reports–Are We
There Yet? SoK: Automated TTP Extraction from CTI Reports–Are We
There Yet? In 34th USENIX security symposium (USENIX Security 25).Where the paper says it
Where the paper says it
[4] Feng Chen, Jiajia Liu, Kaixiang Ji, Wang Ren, Jian Wang, and Jingdong Chen. 2023. Learning implicit entity-object relations by bidirectional generative alignment for multimodal ner. p. 8 [4] Feng Chen, Jiajia Liu, Kaixiang Ji, Wang Ren, Jian Wang, and Jingdong Chen. Learning implicit entity-object relations by bidirectional generative alignment
for multimodal ner. Learning implicit entity-object relations by bidirectional generative alignment
for multimodal ner. In Proceedings of the 31st ACM international conference on
multimedia.Where the paper says it
Where the paper says it
[5] Minghao Chen, Kaijie Zhu, Bin Lu, Ding Li, Qingjun Yuan, and Yuefei Zhu. 2025. AECR: Automatic attack technique intelligence extraction based on fine-tuned large language model. p. 8 [5] Minghao Chen, Kaijie Zhu, Bin Lu, Ding Li, Qingjun Yuan, and Yuefei Zhu. AECR: Automatic attack technique intelligence extraction based on fine-tuned
large language model.Where the paper says it
[6] Yutong Cheng, Osama Bajaber, Saimon Amanuel Tsegai, Dawn Song, and Peng Gao. 2024. CTINEXUS: leveraging optimized LLM in-context learning for constructing cybersecurity knowledge graphs under data scarcity. p. 8 [6] Yutong Cheng, Osama Bajaber, Saimon Amanuel Tsegai, Dawn Song, and Peng
Gao. CTINEXUS: leveraging optimized LLM in-context learning for constructing cybersecurity knowledge graphs under data scarcity. CTINEXUS: leveraging optimized LLM in-context learning for constructing cybersecurity knowledge graphs under data scarcity. arXiv preprint
arXiv:2410.Where the paper says it
Where the paper says it
[7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. p. 8 [7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT:
Pre-training of Deep Bidirectional Transformers for Language Understanding. BERT:
Pre-training of Deep Bidirectional Transformers for Language Understanding. In
Proceedings of the 2019 Conference of the North American Chapter of the Association
for Computational Linguistics: Human Language Technologies, Volume 1 (Long
and Short Papers).Where the paper says it
Where the paper says it
[8] Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. The Faiss library. p. 8 18653/v1/N19-1423
[8] Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy,
Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The Faiss library. The Faiss library. arXiv preprint arXiv:2401.Where the paper says it
Where the paper says it
[9] B. Efron. 1979. Bootstrap Methods: Another Look at the Jackknife. p. 8 08281
[9] B. Bootstrap Methods: Another Look at the Jackknife.Where the paper says it
[10] Reza Fayyazi, Rozhina Taghdimi, and Shanchieh Jay Yang. 2024. Advancing TTP analysis: Harnessing the power of large language models with retrieval augmented generation. p. 8 1214/aos/1176344552
[10] Reza Fayyazi, Rozhina Taghdimi, and Shanchieh Jay Yang. Advancing
TTP analysis: Harnessing the power of large language models with retrieval
augmented generation. Advancing
TTP analysis: Harnessing the power of large language models with retrieval
augmented generation. In 2024 Annual Computer Security Applications Conference
Workshops (ACSAC Workshops).Where the paper says it
Where the paper says it
[11] Romy Fieblinger, Md Tanvirul Alam, and Nidhi Rastogi. 2024. Actionable cyber threat intelligence using knowledge graphs and large language models. p. 8 [11] Romy Fieblinger, Md Tanvirul Alam, and Nidhi Rastogi. Actionable cyber
threat intelligence using knowledge graphs and large language models. Actionable cyber
threat intelligence using knowledge graphs and large language models. In 2024
IEEE European symposium on security and privacy workshops (EuroS&PW).Where the paper says it
Where the paper says it
[12] Keisuke Furumoto, Tomohiro Morikawa, Antti Kolehmainen, Bilhanan Silverajan, Takeshi Takahashi, and Daisuke Inoue. 2025. A Comprehensive Survey of Threat Intelligence Research: A Measurement-Based Study. p. 8 [12] Keisuke Furumoto, Tomohiro Morikawa, Antti Kolehmainen, Bilhanan Silverajan,
Takeshi Takahashi, and Daisuke Inoue. A Comprehensive Survey of Threat
Intelligence Research: A Measurement-Based Study. A Comprehensive Survey of Threat
Intelligence Research: A Measurement-Based Study. Surveys 58, 6,
Article 153 (Dec.Where the paper says it
Where the paper says it
[13] Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. 2023. RARR: Researching and Revising What Language Models Say Using Language Models. p. 8 1145/3772280
[13] Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and
Kelvin Guu. RARR: Researching and Revising What Language Models Say,
Using Language Models. RARR: Researching and Revising What Language Models Say,
Using Language Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).Where the paper says it
Where the paper says it
[14] Peng Gao, Fei Shao, Xiaoyuan Liu, Xusheng Xiao, Zheng Qin, Fengyuan Xu, Prateek Mittal, Sanjeev R. Kulkarni, and Dawn Song. p. 8 910
[14] Peng Gao, Fei Shao, Xiaoyuan Liu, Xusheng Xiao, Zheng Qin, Fengyuan Xu,
Prateek Mittal, Sanjeev R. Kulkarni, and Dawn Song. Kulkarni, and Dawn Song. Enabling Efficient Cyber Threat Hunting With Cyber Threat Intelligence. Enabling Efficient Cyber Threat Hunting With Cyber Threat Intelligence. In 2021 IEEE 37th
International Conference on Data Engineering (ICDE).Where the paper says it
Where the paper says it
Where the paper says it
[15] Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling Large Language Models to Generate Text with Citations. p. 8 00024
[15] Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling Large
Language Models to Generate Text with Citations. Enabling Large
Language Models to Generate Text with Citations. In Proceedings of the 2023
Conference on Empirical Methods in Natural Language Processing.Where the paper says it
Where the paper says it
[16] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. p. 8 398
[16] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek
Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex
Vaughan, et al. The llama 3 herd of models. The llama 3 herd of models. arXiv preprint arXiv:2407.Where the paper says it
Where the paper says it
[17] Wenbo Guo, Chengwei Liu, Limin Wang, Yiran Zhang, Jiahui Wu, Zhengzi Xu, and Yang Liu. 2026. IntelliRadar: A Comprehensive Platform to Pinpoint Malicious Package Information from Cyber Intelligence. p. 8 [17] Wenbo Guo, Chengwei Liu, Limin Wang, Yiran Zhang, Jiahui Wu, Zhengzi
Xu, and Yang Liu. IntelliRadar: A Comprehensive Platform to Pinpoint
Malicious Package Information from Cyber Intelligence. IntelliRadar: A Comprehensive Platform to Pinpoint
Malicious Package Information from Cyber Intelligence. In 2026 IEEE/ACM 48th
International Conference on Software Engineering (ICSE).Where the paper says it
Where the paper says it
[18] Kyla Guru, Robert J Moss, and Mykel J Kochenderfer. 2025. On Technique Identification and Threat-Actor Attribution using LLMs and Embedding Models. p. 8 [18] Kyla Guru, Robert J Moss, and Mykel J Kochenderfer. On Technique
Identification and Threat-Actor Attribution using LLMs and Embedding Models. On Technique
Identification and Threat-Actor Attribution using LLMs and Embedding Models. In 2025 International Conference on Cybersecurity and AI-Based Systems (Cyber-AI).Where the paper says it
Where the paper says it
[19] Yiling He, Junchi Lei, Zhan Qin, Kui Ren, and Chun Chen. 2025. Combating concept drift with explanatory detection and adaptation for android malware classification. p. 8 [19] Yiling He, Junchi Lei, Zhan Qin, Kui Ren, and Chun Chen. Combating
concept drift with explanatory detection and adaptation for android malware
classification. Combating
concept drift with explanatory detection and adaptation for android malware
classification. In Proceedings of the 2025 ACM SIGSAC Conference on Computer
and Communications Security.Where the paper says it
Where the paper says it
[20] Liangyi Huang and Xusheng Xiao. 2024. CTIKG: LLM-Powered Knowledge Graph Construction from Cyber Threat Intelligence. p. 8 [20] Liangyi Huang and Xusheng Xiao. CTIKG: LLM-Powered Knowledge
Graph Construction from Cyber Threat Intelligence.Where the paper says it
[21] Ghaith Husari, Ehab Al-Shaer, Mohiuddin Ahmed, Bill Chu, and Xi Niu. 2017. TTPDrill: Automatic and accurate extraction of threat actions from unstructured text of cti sources. p. 8 [21] Ghaith Husari, Ehab Al-Shaer, Mohiuddin Ahmed, Bill Chu, and Xi Niu. TTPDrill: Automatic and accurate extraction of threat actions from unstructured
text of cti sources. TTPDrill: Automatic and accurate extraction of threat actions from unstructured
text of cti sources. In Proceedings of the 33rd annual computer security applications
conference.Where the paper says it
Where the paper says it
[22] Ghaith Husari, Xi Niu, Bill Chu, and Ehab Al-Shaer. 2018. Using entropy and mutual information to extract threat actions from cyber threat intelligence. p. 8 [22] Ghaith Husari, Xi Niu, Bill Chu, and Ehab Al-Shaer. Using entropy and
mutual information to extract threat actions from cyber threat intelligence. Using entropy and
mutual information to extract threat actions from cyber threat intelligence. In
2018 IEEE international conference on intelligence and security informatics (ISI).Where the paper says it
Where the paper says it
[23] Youngjin Jin, Eugene Jang, Jian Cui, Jin-Woo Chung, Yongjae Lee, and Seungwon Shin. 2023. Darkbert: A language model for the dark side of the internet. p. 8 [23] Youngjin Jin, Eugene Jang, Jian Cui, Jin-Woo Chung, Yongjae Lee, and Seungwon
Shin. Darkbert: A language model for the dark side of the internet. RARR: Researching and Revising What Language Models Say,
Using Language Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).Where the paper says it
Where the paper says it
[24] Damjan Kalajdzievski. 2023. A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA. p. 8 [24] Damjan Kalajdzievski. A Rank Stabilization Scaling Factor for Fine-Tuning
with LoRA. A Rank Stabilization Scaling Factor for Fine-Tuning
with LoRA. arXiv preprint arXiv:2312.Where the paper says it
Where the paper says it
[555] We reserve 55 reports for optional in-domain adaptation (5,592 assignments) and use the remaining 55 for development evaluation (503 assignments across 164 technique identifiers)
[609] The knowledge base contains 651 technique and sub-technique profiles from ATT&CK Enterprise v19.2
[705] We compare the online LLM workload of Direct Prompt, RAG, CTIKG, and MEvince on the shortest, median-length, and longest reports among the 55 evaluation reports
[323] Direct Prompt uses one generation call per report and has the lowest token consumption
[794] CTIKG requires fewer calls and reduces generation-token usage by 62.1–83.8%
[355] MEvince uses more tokens than RAG, as it runs separate LLM stages for evidence extraction and technique mapping
[992] The memory contains 9,440 positive prototypes from the 555-report training partition
MEvince achieves the highest macro F1 and Jaccard at both granularities Its exact-ID F1 reaches 0.310, exceeding the strongest baseline, Llama-3. p. 6 Its exact-ID F1 reaches 0. 310, exceeding the strongest baseline, Llama-3. 310, exceeding the strongest baseline, Llama-3. 264), by 17. The advantage is more pronounced at the exact-ID level,
where evaluation requires preserving distinctions between techniques and sub-techniques. Among the context-driven LLM methods, RAG improves on
Direct Prompt, suggesting the value of external technique knowledge. RAG achieves the highest exact-ID recall (0. 424), but its precision
is only 0. 424), but its precision
is only 0. In contrast, MEvince approximately doubles precision to 0. In contrast, MEvince approximately doubles precision to 0. 384 while retaining the second-highest recall (0.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Knowledge-Constrained Reasoning Framework for Technique Identification: The system can align dispersed report observations with ATT&CK technique definitions through a shared behavioral representation that distinguishes between
source evidence, interpretive knowledge, and unverified hypotheses.
-
Primitive-Guided Evidence Extraction: This module structures LLM-extracted observations by defining an extraction schema based on the
action, object, goal, and mechanism concepts
to guide the LLM in preserving specific behavioral information needed for mapping. -
Evidence-Aligned Retrieval with Candidate Comparison: The retrieval module matches structured evidence against technique profiles using shared concepts while retaining
likely candidates and confusable alternatives for LLM comparison of distinguishing behaviors.
-
Deterministic Verification Controls: The system implements two controls—
evidence-unit control
andclaim verification
—which validate source references and enforce technique-specific requirements before acceptance, ensuring assignments aregrounded in the report.
-
Parameter-Free Adaptation: An optional evidence memory module can be used to refine candidate selection without updating model parameters, improving performance on unseen reports by promoting techniques based on positive training examples.
-
Structured Output for Analyst Workflows: The framework produces structured outputs that allow analysts to review accepted or abstained techniques with
source URL, quoted spans, evidence status, matched mechanism, and proof gaps,
aiding in knowledge-base curation and hypothesis generation.
Abstract
Mapping behavioral intelligence in malware analysis reports to ATT&CK techniques supports threat analysis and defense, but reliable identification remains challenging. Supporting evidence is often scattered across long, heterogeneous reports, while closely related techniques share behavioral cues and differ only in subtle but decisive details. While large language models (LLMs) can use external knowledge to interpret reports, they may assign techniques based on plausible associations rather than sufficient evidence. We present MEvince, a knowledge-constrained LLM reasoning framework for report-level malware technique identification. Its core is a shared behavioral representation connecting report observations to technique definitions. This representation is built through agentic mining of technique documentation and supports two online stages: 1) primitive-guided extraction uses behavioral primitives to structure LLM-extracted observations, link them to source passages, and group dispersed evidence; 2) evidence-aligned retrieval matches this evidence to technique profiles through the same concepts, retaining likely candidates and confusable alternatives for LLM comparison of distinguishing behaviors. Controls at both stages validate evidence references and check technique-specific requirements. The representation also supports adaptation without parameter updates, where offline agentic audits diagnose concept weaknesses and refine extraction or retrieval policies. We evaluate MEvince against 11 baselines spanning four method families using a dataset of 610 public malware reports referenced by MITRE ATT&CK. MEvince improves exact-ID F1 over the strongest baseline by 17.4%, and its concept-guided retrieval increases gold-label candidate coverage from 31.6% to 47.9%. With a small adaptation set, offline refinement yields further gains and outperforms baselines that update model parameters.
Sources
- CTINexus: Automatic Cyber Threat Intelligence Knowledge Graph Construction Using Large Language Models
- The Faiss library
- The Llama 3 Herd of Models
- A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA
- Automated Retrieval of ATT&CK Tactics and Techniques for Cyber Threat Reports
- RoBERTa: A Robustly Optimized BERT Pretraining Approach
- Qwen2.5 Technical Report
- EXTRACTOR: Extracting Attack Behavior from Threat Reports
- Identifying Adversary Tactics and Techniques in Malware Binaries with an LLM Agent
- Qwen2 Technical Report
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data