Minerva: Reinforcement Learning with Verifiable Rewards for Cyber Threat Intelligence LLMs
cs.LG
Submitted: 2026-01-31
Updated: 2026-11-06
Code: https://github.com/center-for-threat-informed-defense/mappings-explorer
Project page: https://ctid.mitre.org/projects/threat-report-attck-mapper-tram
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Cyber threat intelligence (CTI) analysts routinely convert noisy, unstructured security artifacts into standardized, automation-ready representations.
Terminology
Abstract
Cyber threat intelligence (CTI) analysts routinely convert noisy, unstructured security artifacts into standardized, automation-ready representations. Although large language models (LLMs) show promise for this task, existing approaches remain brittle when producing structured CTI outputs and have largely relied on supervised fine-tuning (SFT). In contrast, CTI standards and community-maintained resources define canonical identifiers and schemas that enable deterministic verification of model outputs. We leverage this structure to study reinforcement learning with verifiable rewards (RLVR) for CTI tasks. We introduce Minerva, a unified dataset and training pipeline spanning multiple CTI subtasks, each paired with task-specific verifiers that score structured outputs and identifier predictions. To address reward sparsity during rollout, we propose MinervaRL, an auxiliary self-training mechanism that generates additional verified trajectories and distills them back into the model. Averaged across four backbones and 12 CTI benchmarks, MinervaRL improves the mean score by 15.8 percentage points over the corresponding base models and by 4.3 points over GRPO. The project page is available at https://maveryn.github.io/minerva-tmlr/.
Sources
- AthenaBench: A Dynamic Benchmark for Evaluating LLMs in Cyber Threat Intelligence
- CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Models
- Reasoning with Exploration: An Entropy Perspective
- CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- SEvenLLM: Benchmarking, Eliciting, and Enhancing Abilities of Large Language Models in Cyber Threat Intelligence
- From Text to Actionable Intelligence: Automating STIX Entity and Relationship Extraction
- CyberPal.AI: Empowering LLMs with Expert-Driven Cybersecurity Instructions
- Toward Cybersecurity-Expert Small Language Models
- The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning
- Generalization of RLVR Using Causal Reasoning as a Testbed
- SQL-R1: Training Natural Language to SQL Reasoning Model By Reinforcement Learning
- Proximal Policy Optimization Algorithms
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains
- Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
- The Invisible Leash: Why RLVR May or May Not Escape Its Origin
- Rule-ATT&CK Mapper (RAM): Mapping SIEM Rules to TTPs Using LLMs
- Large Language Models for Cyber Security: A Systematic Literature Review
- Learning to Reason under Off-Policy Guidance
Related papers
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis
- (Mis)generalization of Helpful-Only Fine-tuning