HackerSignal: A Multi-Source, CVE-Linked Corpus and Prospective Benchmark for Cyber Threat Intelligence
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "HackerSignal: A Multi-Source, CVE-Linked Corpus and Prospective Benchmark for Cyber Threat Intelligence".
Elias: Here's the gist of the paper. HackerSignal introduces a benchmark for temporal out-of-distribution cyber threat intelligence and cross-source CVE linkage by aggregating millions of documents from diverse public sources to map the full exploit trajectory from hacker community discourse to vulnerability fixes Dataset Scope and Structure HackerSignal aggregates 7.45 million exact-deduplicated documents spanning eight source layers and a 36-year window from 1990 to 2026.
Nadia: The corpus is unified through a shared CVE identifier space, allowing for cross-source joins while preserving source-specific release modes for AI analytics tasks.
Elias: The eight source layers include hacker communities, exploit archives, vulnerability references, exploit Q&A references, advisories, bug-bounty disclosures, fix commits, and exploitation references.
Nadia: The largest subcorpus is the hacker community layer with 6.97 million rows spanning 56 forums across darkweb marketplaces and security discussion boards.
Title and authors: Nadia: Now we’re diving into the specific details of this work. The paper is titled HackerSignal: A Multi-Source, CVE-Linked Corpus and Prospective Benchmark for Cyber Threat Intelligence. We’ve already touched on the scale and structure, but let's unpack what that title actually means for us in practice.
Elias: It signals that this isn't just another dataset of forum posts or vulnerability descriptions; it’s a system designed to link hacker community discourse directly to the CVE vulnerability lifecycle. The focus on "multi-source" tells us we’re not relying on a single data stream, which is key for building robust models.
Priya: And the "CVE-linked corpus" part is important because it moves the goal from just identifying mentions of a vulnerability to tracing that mention all the way to its official advisory and eventually its fix commit. That’s where we get real intelligence.
Nadia: Right, Priya, because if you only have text about an exploit but no link to the actual patch code or advisory, you’re stuck in a very shallow understanding of what happened. HackerSignal aims to bridge that gap completely by creating those connections through a shared CVE identifier space.
Elias: And they emphasize "prospective benchmark," meaning the goal isn't just retrospective analysis of what already happened; it's testing how well an AI can predict or ground information about threats that haven't appeared in the training period yet.
Priya: That prospective aspect is where I see a lot of the real value for predictive security; being able to test if a system can anticipate future threat patterns based on historical context without simply memorizing past examples.
Nadia: Exactly, and they achieve this by setting up those three specific tasks: CVE linkage retrieval, exploit type classification, and temporal generalization. Those tasks define exactly what we want to measure about a CTI system’s ability to operate in a dynamic environment.
Elias: They also detail the eight source layers they aggregated across that thirty six-year window from nineteen ninety to two thousand twenty-six which is what gives them that massive temporal coverage and diversity. It covers everything from dark web marketplaces to official fix commits.
Priya: The diversity is what makes it rich for measurement, I think, because a model trained on just forum posts might fail when asked to classify an exploit type mentioned in a formal advisory document.
Nadia: That’s right; they want to see if the model can handle that cross-domain shift. The paper lays out the data statistics too, showing things like the percentage of rows with CVE strings present and how long those documents are, which gives us a good feel for the quality and complexity of what we’re feeding into these systems.
Elias: The data shows that while there's a decent presence of CVE strings—about four point eight percent—the sheer volume of hacker community posts is massive, with about six point nine seven million rows spanning fifty six forums alone, which is the largest subcorpus they have.
Priya: That large hacker community subcorpus is interesting because it suggests that even though the formal records are less numerous, the actual raw discussion from the underground communities holds a lot of contextual information.
Nadia: So, to summarize this segment, HackerSignal isn't just a big dump of text; it’s a structured, temporally grounded system built specifically to test AI’s ability to trace the full exploit trajectory across diverse data types and time periods.
Elias: And the key takeaway is that they are creating a comprehensive benchmark for evaluating systems on their ability to ground information reliably across disparate sources, which is what makes it so significant in the CTI space.
Priya: It sounds like we're looking at a really detailed foundation for measuring how effective AI can be at connecting the dots between raw chatter and concrete security fixes.
Nadia: Exactly, and that foundation is what allows us to start asking better questions about how quickly threats evolve and how effectively we can defend against them in the future.
The paper's summary: Nadia: Moving on to the actual summary of this paper, HackerSignal’s core contribution is providing a temporally structured, multi-source corpus of seven point four five million documents from sixty four public forum and source identifiers spanning eight different layers over a thirty six year window.
Elias: And the main message they are driving home is that previous publicly accessible datasets have been lacking in linking hacker posts to formal vulnerability records, so HackerSignal addresses this gap by creating these explicit CVE linkage metadata connecting the vulnerability identifiers to the texts that discuss them.
Priya: What I find compelling about their summary is how they frame it as addressing a known absence in the public literature—that structured datasets linking forum posts to formal vulnerability records are largely absent, and HackerSignal steps in to fill that hole.
Nadia: That’s right; they acknowledge that previous work focused heavily on advisory records or exploit archives, but this paper shows the operational gap between those and the raw community discussion that actually drives the initial discovery.
Elias: They explicitly state their three key contributions: first, building this temporally grounded corpus with all those links, second, creating a companion survey of cybercrime community research to corroborate that structured datasets are missing, and third, developing this benchmark for downstream CTI analytics tasks.
Priya: I’m interested in the benchmark part because it means they are not just presenting data; they are providing a tool for researchers to rigorously test their AI systems against these real-world intelligence problems.
Nadia: Precisely; they aren't just giving us a dataset to train on; they’re giving us a rigorous way to measure how well an AI can perform specific tasks like linking entities, classifying exploit types, and generalizing temporally.
Elias: They also detail the nature of their data quality diagnostics—showing things like the percentage of documents with CVE strings present and the distribution of document lengths—to give researchers a clearer picture of what they are working with.
Priya: That diagnostic information is crucial because it allows us to understand the inherent noise and signal in this corpus before we even start training any models on top of it. It helps us manage expectations about the data's quality and biases.
Nadia: So, to wrap up, HackerSignal’s summary is that they have filled a major gap by creating a dataset that maps the full potential exploit trajectory from hacker community discourse through exploits and fixes using explicit CVE links across time.
Elias: It’s positioning itself as the definitive resource for testing systems that need to operate robustly in the messy, evolving world of cyber threat intelligence.
Priya: It sounds like a necessary step toward making AI-driven threat intelligence more reliable because it grounds those predictions in verifiable, traceable data points from both the community and the fix side.
The paper's improvements: Nadia: Let's talk about the specific improvements they propose for HackerSignal. They aren't just presenting a static dataset; they are proposing ways to make this corpus even more useful for downstream analytics by suggesting enhancements to how we process and query it.
Elias: One major suggestion is around retrieval tasks, where they propose using a hybrid BM25 plus dense retrieval approach for reranking candidates, which combines the lexical score with the dense cosine score.
Priya: That makes sense; mixing those methods might give us better recall at k because we’re getting the best of both worlds—the keyword matching and the contextual understanding. But I wonder if that just adds complexity without a real gain.
Nadia: The authors suggest that this hybrid approach could actually improve recall at k when compared to symmetric bi-encoders, suggesting it's a worthwhile experiment to run on this data. They are also pointing out that asymmetric prefix training is particularly effective for cross-source CVE grounding.
Elias: That points back to the classification tasks, where they’re finding that asymmetric prefix training helps them achieve better results when grounding information across different sources, which is an interesting parameter tuning suggestion for those NLP models.
Priya: So, the improvement isn't just about the data itself, but about providing these specific training and retrieval strategies that are known to work well in this context, which is really helpful guidance for researchers.
Nadia: It’s about moving beyond just feeding a model raw data; it’s suggesting specific ways to train and query it so that its ability to cross-source link things improves, which is a practical application of the research.
Elias: And they also have these governance features mentioned, like explicit checks for temporal and source leakage during their audit process to keep the benchmark claims from being inflated. They are basically saying, we need rigorous internal validation if we want to trust these results.
Priya: That rigor is essential because without it, any benchmark can just be a vanity metric; checking for that kind of leakage shows they’re serious about maintaining data integrity throughout the entire process.
Nadia: So, the improvements are a combination of methodological tweaks—like hybrid retrieval and specific training techniques—and strong governance checks to ensure the integrity of the large-scale benchmark.
Elias: They are essentially saying, this is how you build a better system for this kind of complex CTI problem by paying attention to both the data architecture and the model training parameters.
Priya: It sounds like they’re really trying to give researchers a complete toolkit, not just a finished product, which is what we need in applied research.
Conclusion: Nadia: So, wrapping up this discussion on HackerSignal: This corpus and benchmark are designed to test if AI can actually perform the three key tasks—CVE linkage retrieval, exploit type classification, and temporal generalization—on truly complex, messy cyber threat data.
Elias: The implication is that if these models perform well here under those strict temporal constraints, then we gain confidence that they have learned a general understanding of the entire vulnerability lifecycle rather than just memorizing patterns from a specific time period.
Priya: For us, it means we have a more rigorous standard for evaluating predictive systems because we’re looking at how they handle both the sheer volume and the temporal shifts in threat data.
Nadia: And what’s next is seeing if this corpus can truly guide new methods for proactive CTI by showing us where AI is succeeding and where it’s still falling short when facing unseen threats.
Elias: The full scope of HackerSignal is that it provides a massive, multi-source data foundation for testing cross-source CVE grounding, exploit type classification, and prospective generalization in a way that was previously missing.
Priya: It’s a big step toward making AI-driven threat intelligence more reliable because it grounds those predictions in verifiable, traceable data points from both the community and the fix side.
Nadia: We've covered a lot about how this work is structured, from the governance checks to the specific results on task three, which showed an mrr of zero point six zero for prospective generalization.
Elias: And I’m glad we could discuss how asymmetric prefix training helps with cross-source grounding because that seems like a key mechanism for success in this domain.
Priya: It’s about having the tools to evaluate these systems properly, which is what matters most when we're building defenses against unknown risks.
Nadia: So, HackerSignal is positioned as a very comprehensive resource for anyone looking to test the limits of current AI in understanding and predicting cyber threats across time and different data sources.
Elias: It’s a lot of work, but it creates a high-quality standard for how we measure CTI capability moving forward.
Georgia State University · Indiana University Bloomington
cs.CR
Submitted: 2026-05-04
Updated: 2026-11-05
Comments: Published at NeurIPS 2026 (40th Conference on Neural Information Processing Systems), Track on Evaluations and Datasets. Camera-ready version. 19 pages, 4 figures, 13 tables. Data: https://huggingface.co/datasets/BenAmpel/HackerSignal; code: https://github.com/BenAmpel/hackersignal
Code: https://github.com/BenAmpel/hackersignal
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 89/100
The gist: HackerSignal introduces a benchmark for temporal out-of-distribution cyber threat intelligence and cross-source CVE linkage by aggregating millions of documents from diverse public sources to map the…
Key concepts
- CVE Linkage Retrieval
- This task tests if an AI can correctly find the specific Common Vulnerabilities and Exposures (CVE) identifier for a vulnerability by searching across different sources like hacker forums, exploit archives, and official advisories. It requires grounding a query in the correct CVE entity from disparate documents.
- Exploit Type Classification
- This involves training an AI to categorize exploit text into one of eight specific vulnerability types. The evaluation uses a range of models, from simple bag-of-words methods up to advanced transformer models like SecBERT, to see how well the model recognizes the exploit's nature.
- Temporal Generalization
- This benchmark assesses a model's ability to predict vulnerabilities that have not yet been discovered or fixed. It enforces a strict rule: the test set must contain CVEs that are entirely disjoint from those in the training set, ensuring the model generalizes beyond memorized historical patterns.
- Source Layer Aggregation
- The dataset is built by combining eight distinct layers of data, including hacker communities, exploit archives, and fix commits. This aggregation allows AI systems to learn relationships across these different types of information simultaneously for more comprehensive threat intelligence.
Terminology
Summary
HackerSignal introduces a benchmark for temporal out-of-distribution cyber threat intelligence and cross-source CVE linkage by aggregating millions of documents from diverse public sources to map the full exploit trajectory from hacker community discourse to vulnerability fixes
Dataset Scope and Structure
HackerSignal aggregates 7.45 million exact-deduplicated documents spanning eight source layers and a 36-year window from 1990 to 2026. p. 0 HackerSignal aggregates
7. 45 million exact-deduplicated documents from 64 public forum/source identifiers spanning eight source layers and a 36-year window (1990–2026). HackerSignal creates these linkages through a shared CVE identifier space while
preserving source-specific release modes to support a range of unique Artificial
Intelligence (AI)-enabled cybersecurity analytics tasks. In this paper, we summarize HackerSignal and illustrate three selected benchmark tasks it uniquely
supports: (1) CVE linkage retrieval (cross-source temporal out-of-distribution
entity grounding); (2) exploit type classification (8-class vulnerability type prediction with temporal OOD evaluation); and (3) temporal generalization (prospective
CVE-disjoint evaluation where Ctrain ∩ Ctest = ∅). A temporally grounded CTI corpus of 7. 45M documents from 64 public forum/source
identifiers across eight source layers (hacker communities, exploit archives, vulnerability
references, exploit Q&A references, advisories, bug-bounty disclosures, fix commits, and
exploitation references), with CVE linkage metadata connecting vulnerability identifiers to
the texts that discuss them. 45M documents from 64 public forum/source
identifiers across eight source layers (hacker communities, exploit archives, vulnerability
references, exploit Q&A references, advisories, bug-bounty disclosures, fix commits, and
exploitation references), with CVE linkage metadata connecting vulnerability identifiers to
the texts that discuss them. 97M-row hacker community subcorpus alone (spanning
56 forums across darkweb marketplaces, security discussion boards, and multilingual hacker
communities) supports diverse NLP tasks beyond our three benchmarks, including exploit
code summarization, security content classification, topic modeling, community interaction
analysis, and cross-lingual CTI extraction. 45M documents from 64 public forum/source
identifiers across eight source layers (hacker communities, exploit archives, vulnerability
references, exploit Q&A references, advisories, bug-bounty disclosures, fix commits, and
exploitation references), with CVE linkage metadata connecting vulnerability identifiers to
the texts that discuss them. 97M-row hacker community subcorpus alone (spanning
56 forums across darkweb marketplaces, security discussion boards, and multilingual hacker
communities) supports diverse NLP tasks beyond our three benchmarks, including exploit
code summarization, security content classification, topic modeling, community interaction
analysis, and cross-lingual CTI extraction.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Benchmark Tasks
The paper illustrates three selected benchmark tasks that HackerSignal uniquely supports for temporal out-of-distribution generalization. These tasks include:
-
CVE linkage retrieval (cross-source temporal out-of-distribution entity grounding). p. 0
Where the paper says it
HackerSignal creates these linkages through a shared CVE identifier space while preserving source-specific release modes to support a range of unique Artificial Intelligence (AI)-enabled cybersecurity analytics tasks. In this paper, we summarize HackerSignal and illustrate three selected benchmark tasks it uniquely supports: (1) CVE linkage retrieval (cross-source temporal out-of-distribution entity grounding); (2) exploit type classification (8-class vulnerability type prediction with temporal OOD evaluation); and (3) temporal generalization (prospective CVE-disjoint evaluation where Ctrain ∩ Ctest = ∅).
-
Exploit type classification (8-class vulnerability type prediction with temporal OOD evaluation). p. 0
Where the paper says it
HackerSignal creates these linkages through a shared CVE identifier space while preserving source-specific release modes to support a range of unique Artificial Intelligence (AI)-enabled cybersecurity analytics tasks. In this paper, we summarize HackerSignal and illustrate three selected benchmark tasks it uniquely supports: (1) CVE linkage retrieval (cross-source temporal out-of-distribution entity grounding); (2) exploit type classification (8-class vulnerability type prediction with temporal OOD evaluation); and (3) temporal generalization (prospective CVE-disjoint evaluation where Ctrain ∩ Ctest = ∅).
-
Temporal generalization (prospective CVE-disjoint evaluation where Ctrain ∩ Ctest = ∅). p. 0
Where the paper says it
HackerSignal creates these linkages through a shared CVE identifier space while preserving source-specific release modes to support a range of unique Artificial Intelligence (AI)-enabled cybersecurity analytics tasks. In this paper, we summarize HackerSignal and illustrate three selected benchmark tasks it uniquely supports: (1) CVE linkage retrieval (cross-source temporal out-of-distribution entity grounding); (2) exploit type classification (8-class vulnerability type prediction with temporal OOD evaluation); and (3) temporal generalization (prospective CVE-disjoint evaluation where Ctrain ∩ Ctest = ∅).
Methodology and Evaluation
All tasks utilize temporal splits to evaluate prospective generalization because real CTI systems must generalize to threats that did not exist at training time. The evaluation uses a heterogeneous baseline ladder spanning lexical, dense, and hybrid methods for retrieval tasks. p. 5 Following Thakur et al. (2021), we evaluate each task with a heterogeneous baseline ladder
spanning lexical, dense, and hybrid methods. Three benchmark tasks with temporal train/val/test splits that test progressively harder
generalization in (1) cross-source CVE linkage retrieval, (2) 8-class exploit type classification, and (3) CVE-disjoint temporal generalization (Ctrain ∩ Ctest = ∅), accompanied by
shortcut/leakage diagnostics. A full model ladder and governance artifacts from BM25 through four dense bi-encoders
(MiniLM, mpnet, BGE, E5), bag-of-words classifiers (Decision Tree, TF-IDF+LR, SVM),
recurrent neural networks (RNN, GRU, LSTM, BiLSTM), a domain-specific transformer
(SecBERT), and hybrid sparse+dense retrieval, alongside source-shortcut diagnostics,
manual-audit packets, and a release-governance addendum. Temporal splits make
it harder as exploit names and vulnerability patterns evolve, preventing models from memorizing
era-specific features. 3 Task 3: Temporal Generalization (TG)
Identical retrieval formulation to Task 1, but with a strict CVE-disjointness constraint: Ctrain ∩ Ctest =
∅, where Cs = {c
∗
i
: qi ∈ split s} is the set of gold CVE targets in split s.Where the paper says it
Where the paper says it
Where the paper says it
Audit and Governance
The authors explicitly audit the dataset for temporal and source leakage because benchmark claims are otherwise easy to inflate. Integrity checks include verifying correct CVE assignment, temporal split adherence, and absence of trivial textual overlap between query and target. p. 6 For each query q ∈ Q, the task is to retrieve the correct CVE
entry c
∗ ∈ C such that the source metadata links q to c
∗
. This is cross-source temporal out-of-domain
entity grounding, akin to knowledge-intensive entity linking (Petroni et al. We pseudonymize author IDs via
SHA-256, preserve provenance, and support takedown requests, downgrading ambiguous sources
accordingly. HackerSignal highlights vulnerabilities for defensive research only, prohibiting finetuning automated exploit models, de-anonymizing forum users, and automated incident response
without human review. The public release categorizes
sources into redistributable, research-only, metadata/pointer, and excluded. A governance addendum
details release modes, contact info, versioning, checksums, and takedown procedures.Where the paper says it
Where the paper says it
Where the paper says it
Results Summary
The model ladder establishes baselines for cross-source CVE grounding, exploit type classification, and prospective generalization with results such as an MRR of 0.60 for Task 3. p. 1 For cross-source
linkage, an attention-based deep-structured semantic model linked dark-web exploits to known
vulnerabilities, yielding 52,590 linkages (Samtani et al. HackerSignal differs by releasing a
governed, temporally split public benchmark that exposes source provenance, weak-label caveats,
and a quality-controlled CVE-linkage retrieval task. We report macro-F1 and per-class F1 for the three hardest classes in Table 5. The best bag-of-words baseline (SVM) attains macro-F1 of 0. Models generally
maintain or improve performance compared to
Task 1, confirming that our temporal split already prevents meaningful memorization. The
consistent E5 advantage across both tasks suggests that asymmetric prefix training is particularly effective for cross-source CVE grounding, where queries and targets have different
discourse design, as already demonstrated in
a rigorous prospective evaluation.Where the paper says it
Where the paper says it
Where the paper says it
References
Akhtar, M., et al. (2024). Croissant: A Metadata Format for ML-Ready Datasets. p. 9 References
Akhtar, M. Croissant: A Metadata Format for ML-Ready Datasets.Where the paper says it
Ampel, B., Samtani, S. p. 0 Mack Robinson College of Business, Georgia State University
bampel@gsu. edu
Sagar Samtani
Department of Operations and Decision Technologies
Kelley School of Business, Indiana University Bloomington
ssamtani@iu. Mack Robinson College of Business, Georgia State University
bampel@gsu. edu
Sagar Samtani
Department of Operations and Decision Technologies
Kelley School of Business, Indiana University Bloomington
ssamtani@iu. Mack Robinson College of Business, Georgia State University
bampel@gsu. edu
Sagar Samtani
Department of Operations and Decision Technologies
Kelley School of Business, Indiana University Bloomington
ssamtani@iu. Mack Robinson College of Business, Georgia State University
bampel@gsu. edu
Sagar Samtani
Department of Operations and Decision Technologies
Kelley School of Business, Indiana University Bloomington
ssamtani@iu. Mack Robinson College of Business, Georgia State University
bampel@gsu. edu
Sagar Samtani
Department of Operations and Decision Technologies
Kelley School of Business, Indiana University Bloomington
ssamtani@iu. Mack Robinson College of Business, Georgia State University
bampel@gsu. edu
Sagar Samtani
Department of Operations and Decision Technologies
Kelley School of Business, Indiana University Bloomington
ssamtani@iu. , Samtani, S. , Ullman, S. , Ullman, S. , & Chen, H. , Ullman, S. , & Chen, H. , Ullman, S. , & Chen, H. , Ullman, S. , & Chen, H. , Ullman, S. , & Chen, H. , Ullman, S. , & Chen, H. , Ullman, S. , & Chen, H. , Ullman, S. , & Chen, H. , & Chen, H. Labeling Hacker Exploits for Proactive Cyber
Threat Intelligence: A Deep Transfer Learning Approach. Labeling Hacker Exploits for Proactive Cyber
Threat Intelligence: A Deep Transfer Learning Approach. IEEE Intelligence and Security Informatics (ISI). Labeling Hacker Exploits for Proactive Cyber
Threat Intelligence: A Deep Transfer Learning Approach. IEEE Intelligence and Security Informatics (ISI). Labeling Hacker Exploits for Proactive Cyber
Threat Intelligence: A Deep Transfer Learning Approach. IEEE Intelligence and Security Informatics (ISI). Labeling Hacker Exploits for Proactive Cyber
Threat Intelligence: A Deep Transfer Learning Approach. IEEE Intelligence and Security Informatics (ISI). IEEE Intelligence and Security Informatics (ISI). https://doi. IEEE Intelligence and Security Informatics (ISI). https://doi. IEEE Intelligence and Security Informatics (ISI). https://doi. IEEE Intelligence and Security Informatics (ISI). https://doi. IEEE Intelligence and Security Informatics (ISI). https://doi. IEEE Intelligence and Security Informatics (ISI). https://doi. IEEE Intelligence and Security Informatics (ISI). https://doi. IEEE Intelligence and Security Informatics (ISI). https://doi. IEEE Intelligence and Security Informatics (ISI). https://doi. IEEE Intelligence and Security Informatics (ISI). https://doi. IEEE Intelligence and Security Informatics (ISI). https://doi. IEEE Intelligence and Security Informatics (ISI). https://doi. IEEE Intelligence and Security Informatics (ISI). https://doi. IEEE Intelligence and Security Informatics (ISI). https://doi. IEEE Intelligence and Security Informatics (ISI). https://doi. IEEE Intelligence and Security Informatics (ISI). https://doi. IEEE Intelligence and Security Informatics (ISI). https://doi. https://doi. 1109/ISI49825.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Ampel, B., Samtani, S., Zhu, H., & Chen, H. (2024). Creating Proactive Cyber Threat Intelligence with Hacker Exploit Labels: A Deep Transfer Learning Approach. p. 9 , & Chen, H. Creating Proactive Cyber Threat Intelligence with
Hacker Exploit Labels: A Deep Transfer Learning Approach. Creating Proactive Cyber Threat Intelligence with
Hacker Exploit Labels: A Deep Transfer Learning Approach. MIS Quarterly, 48(1), 137–166. IEEE Intelligence and Security Informatics (ISI). https://doi.Where the paper says it
Where the paper says it
Where the paper says it
Robertson, S., & Jones, K. p. 9 Robertson, S. , & Jones, K. , & Jones, K. Simple, proven approaches to text retrieval.Where the paper says it
Where the paper says it
Thakur, N., Reimers, N. p. 9 Technical report, University of
Cambridge. , Reimers, N. , Reimers, N. , Rücklé, A. , Rücklé, A. , Srivastava, A. , Srivastava, A. , & Gurevych, I. , Srivastava, A. , & Gurevych, I. , & Gurevych, I. BEIR: A Heterogeneous Benchmark
for Zero-shot Evaluation of Information Retrieval Models. BEIR: A Heterogeneous Benchmark
for Zero-shot Evaluation of Information Retrieval Models. NeurIPS 2021 D&B Track. BEIR: A Heterogeneous Benchmark
for Zero-shot Evaluation of Information Retrieval Models. NeurIPS 2021 D&B Track. NeurIPS 2021 D&B Track. arXiv:2104.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
CISA. (2021–2026). p. 2 ,
2021) links CVEs to fix commits in open-source repositories. NVD (NIST, 2002–2026), CISA
KEV (CISA, 2021–2026), and GitHub Advisory (GitHub, 2017–2026) are official advisory sources. (2021–2026). Known Exploited Vulnerabilities Catalog. Known Exploited Vulnerabilities Catalog. https://www. Known Exploited Vulnerabilities Catalog. https://www.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Pastrana, S., Thomas, D. p. 9 Pastrana, S. , Thomas, D. , Thomas, D. , Hutchings, A. , Thomas, D. , Hutchings, A. , Hutchings, A. , & Clayton, R. , & Clayton, R. CrimeBB: Enabling Cybercrime Research on
Underground Forums at Scale.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Deliu, I., Leichter, C. p. 9 CrimeBB: Enabling Cybercrime Research on
Underground Forums at Scale. , Leichter, C. , Leichter, C. , & Franke, K. , & Franke, K. Extracting Cyber Threat Intelligence from Hacker Forums: Support
Vector Machines versus Convolutional Neural Networks. Extracting Cyber Threat Intelligence from Hacker Forums: Support
Vector Machines versus Convolutional Neural Networks. IEEE International Conference on Big Data. https://doi. 1109/BigData.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Bhandari, G., Naseer, A. p. 9 Bhandari, G. , Naseer, A. , Naseer, A. , & Moonen, L. , & Moonen, L. CVEfixes: Automated Collection of Vulnerabilities and Their
Fixes from Open-Source Software. CVEfixes: Automated Collection of Vulnerabilities and Their
Fixes from Open-Source Software. PROMISE 2021.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Ranade, P., Piplai, A. p. 9 PROMISE 2021. , Piplai, A. , Piplai, A. , Joshi, A. , Joshi, A. , & Finin, T. , & Finin, T. CyBERT: Contextualized Embeddings for the Cybersecurity Domain. CyBERT: Contextualized Embeddings for the Cybersecurity Domain. IEEE International Conference on Big Data, 3334–3342. https://doi. 1109/
BigData52589.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Devlin, J., Chang, M. p. 9 1109/
BigData52589. , Chang, M. , Chang, M. , & Toutanova, K. , & Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Ebrahimi, M. p. 2 More general hacker-forum text classification work compared SVM and CNN
models for extracting CTI from forum posts (Deliu et al. , 2017), while Ebrahimi et al. , & Chen, H. Detecting Cyber Threats in Non-English Hacker Forums:
An Adversarial Cross-Lingual Knowledge Transfer Approach. Detecting Cyber Threats in Non-English Hacker Forums:
An Adversarial Cross-Lingual Knowledge Transfer Approach. IEEE Security and Privacy Workshops, 20–26. https://doi. 1109/SPW50608.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Offensive Security. (2003–26). p. 2 Exploit and vulnerability databases. ExploitDB (Offensive Security, 2003–2026) is the salient
public exploit archive (50K+ entries, many with CVE IDs). 1 Introduction
Understanding how cyber threats emerge, propagate, and relate to vulnerabilities is a central challenge
in modern applied cybersecurity. Signals that tracks this trajectory is distributed across heterogeneous
text-based sources, including: (1) hacker forums where exploits are first discussed, (2) public exploit
databases where proof-of-concept code is released, (3) official vulnerability advisories such as the
National Vulnerability Database, CISA’s Known Exploited Vulnerabilities, and GitHub Advisory
where threats are cataloged, and (4) open-source repositories where fixes are committed. 45 million exact-deduplicated documents from 64 public forum/source identifiers spanning eight source layers and a 36-year window (1990–2026). In contrast
to other publicly accessible cybersecurity datasets, HackerSignal is among the
first public benchmark datasets that maps the full potential exploit to vulnerability trajectory from hacker community discourse, exploit databases with working
and proof of concept exploits, vulnerability advisories, and software fix commits. exploit-db. Licensed CC BY-SA
4.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Gayanku. (2020). Hacker Forum Posts Dataset. p. 2 Underground community datasets. Research has also focused on the general crawling of hacker
communities, like a credentialized dataset of darkweb cybercrime forum posts widely used for
crime and exploit prediction (Pastrana et al.Where the paper says it
Gebru, T., et al. (2021). Datasheets for Datasets. p. 9 com/gayanku. Datasheets for Datasets.Where the paper says it
Cohen, J. (1960). A coefficient of agreement for nominal scales. p. 9 Communications of the ACM, 64(12), 86–92. A coefficient of agreement for nominal scales. A coefficient of agreement for nominal scales. Educational and Psychological Measurement,
20(1), 37–46.Where the paper says it
Where the paper says it
GitHub. (2017–26). p. 1 Early hacker-forum CTI work treated underground
communities as a source of proactive intelligence about tools, actors, and emerging attack capabilities (Samtani et al. , 2015, 2017). 1 Introduction
Understanding how cyber threats emerge, propagate, and relate to vulnerabilities is a central challenge
in modern applied cybersecurity. Signals that tracks this trajectory is distributed across heterogeneous
text-based sources, including: (1) hacker forums where exploits are first discussed, (2) public exploit
databases where proof-of-concept code is released, (3) official vulnerability advisories such as the
National Vulnerability Database, CISA’s Known Exploited Vulnerabilities, and GitHub Advisory
where threats are cataloged, and (4) open-source repositories where fixes are committed. GitHub Advisory Database. https://github.Where the paper says it
Where the paper says it
Where the paper says it
Grisham, J., Samtani, S., Patton, M. p. 9 , Samtani, S. , Patton, M. , & Chen, H. Identifying Mobile Malware and Key Threat Actors in
Online Hacker Forums for Proactive Cyber Threat Intelligence.Where the paper says it
Where the paper says it
Otto, K., Ampel, B. p. 0 HACKERSIGNAL: A Large-Scale Multi-Source Dataset
Linking Hacker Community Discourse to the CVE
Vulnerability Lifecycle
Benjamin M. Ampel
Department of Computer Information Systems
J. , & Chen, H. Exploring the Evolution of Exploit-Sharing
Hackers: An Unsupervised Graph Embedding Approach. https://doi. 1109/ISI53945.Where the paper says it
Where the paper says it
Where the paper says it
NIST. (2002–26). p. 2 ,
2021) links CVEs to fix commits in open-source repositories. NVD (NIST, 2002–2026), CISA
KEV (CISA, 2021–2026), and GitHub Advisory (GitHub, 2017–2026) are official advisory sources. 1 Introduction
Understanding how cyber threats emerge, propagate, and relate to vulnerabilities is a central challenge
in modern applied cybersecurity. Signals that tracks this trajectory is distributed across heterogeneous
text-based sources, including: (1) hacker forums where exploits are first discussed, (2) public exploit
databases where proof-of-concept code is released, (3) official vulnerability advisories such as the
National Vulnerability Database, CISA’s Known Exploited Vulnerabilities, and GitHub Advisory
where threats are cataloged, and (4) open-source repositories where fixes are committed. National Vulnerability Database. https://nvd.Where the paper says it
Where the paper says it
Where the paper says it
Petroni, F., Piktus, A. p. 9 Petroni, F. , Piktus, A. , Piktus, A. , Lewis, P. , Lewis, P. , Yazdani, M. , Yazdani, M. KILT: A Benchmark for Knowledge
Intensive Language Tasks. KILT: A Benchmark for Knowledge
Intensive Language Tasks. NAACL 2021.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Nunes, E., Diab, A., Gunn, A., Marin, E. p. 9 naacl-main. , Marin, E. , Marin, E. , Mishra, V. , Mishra, V. , Paliath, V. 25300/MISQ/2023/17316. Robertson, S. , Robertson, J. , Shakarian, J. , Shakarian, J. , Thart, A. , Robertson, J. , Shakarian, J. , &
Shakarian, P. Darknet and Deepnet Mining for Proactive Cybersecurity Threat Intelligence.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Jackaduma. (2022). SecBERT: A Pre-trained Language Model for Cybersecurity. p. 7 SecBERT (0. 846) surpasses GRU/LSTM
but underperforms BiLSTM, suggesting that domain-specific pre-training offers modest benefits,
and that bidirectionality is key. The full schema specification is provided in the accompanying datasheet (Gebru et al. 3 Collection and Preprocessing
Data was collected between January and April 2026 via (i) official APIs (NVD, GitHub Advisory,
CISA KEV, HackerOne HuggingFace Hub), (ii) ingestion of existing public datasets (e.Where the paper says it
Where the paper says it
Krippendorff, K. (2008). Systematic and random disagreement and the reliability of nominal data. p. 9 Communications of the ACM, 64(12), 86–92. A coefficient of agreement for nominal scales. The closest predecessor to HackerSignal is in the
field of exploit labeling research. Prior methods labeled hacker-forum exploit source code via deep
transfer learning from professional exploit repositories (Ampel et al.Where the paper says it
Where the paper says it
Gwet, K. L. (2014). Handbook of Inter-Rater Reliability. Advanced Analytics
Kapoor, S., & Narayanan, A. p. 1 Temporal splits are necessary in our context as real CTI systems must
generalize to threats that did not exist at training time (Koh et al. We explicitly audit the
dataset for temporal and source leakage (Kapoor and Narayanan, 2023) because benchmark claims
are otherwise easy to inflate (discussed in Section 3. , 2021) demonstrated that weak-supervision evaluation is sensitive to hidden protocol choices,
requiring explicit reporting of label sources and aggregation rules. Kapoor and Narayanan (2023)
documented widespread data leakage in ML-for-science benchmarks and proposed structured reporting to make integrity checks legible.Where the paper says it
Where the paper says it
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M. p. 2 , 2021) formalized natural arising distribution shifts
(including temporal shifts) as first-class benchmark objects. At the same time, WRENCH (Zhang
et al. WILDS (Koh et al. , 2021) formalized natural arising distribution shifts
(including temporal shifts) as first-class benchmark objects.Where the paper says it
Where the paper says it
Han, Z., Li, X., Xing, Z., Liu, H., & Feng, Z. (2017). Learning to Predict Severity of Software Vulnerability Using Only Vulnerability Description. p. 6 7) reranks BM25 top-100
candidates and outperforms symmetric bi-encoders but not the prefix-trained E5. 2 Task 2: Exploit Type Classification (ETC)
Given an exploit text x, predict its vulnerability type y ∈ Y where |Y| = 8 classes: injection (SQLi,
command/code injection, XXE), XSS (reflected, stored, DOM), memory corruption (buffer/heap overflow, use-after-free, format string), denial of service, file inclusion (LFI/RFI, path traversal, arbitrary
file operations), authentication & access control (privilege escalation, auth bypass, CSRF, IDOR),
remote code execution (RCE, deserialization), and information disclosure (SSRF, open redirect, data
exposure).Where the paper says it
Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. p. 9 , & Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.Where the paper says it
Samtani, S., Chinn, R., & Chen, H. (2015). Exploring Hacker Assets in Underground Forums. p. 1 , 2015, 2017). The AZSecure Hacker Assets Portal operationalized this as a
situational awareness platform (Samtani et al. Labeling Hacker Exploits for Proactive Cyber
Threat Intelligence: A Deep Transfer Learning Approach. IEEE Intelligence and Security Informatics (ISI).Where the paper says it
Where the paper says it
Samtani, S., Chinn, R., Chen, H., & Nunamaker, J. F. (2017). Exploring Emerging Hacker Assets and Key Hackers for Proactive Cyber Threat Intelligence. p. 1 2 Related Work
Hacker-community cyber threat intelligence. Early hacker-forum CTI work treated underground
communities as a source of proactive intelligence about tools, actors, and emerging attack capabilities (Samtani et al. HACKERSIGNAL: A Large-Scale Multi-Source Dataset
Linking Hacker Community Discourse to the CVE
Vulnerability Lifecycle
Benjamin M. Ampel
Department of Computer Information Systems
J.Where the paper says it
Where the paper says it
Samtani, S., Zhu, H., & Chen, H. (2020). Proactively Identifying Emerging Hacker Threats from the Dark Web: A Diachronic Graph Embedding Framework. p. 1 The AZSecure Hacker Assets Portal operationalized this as a
situational awareness platform (Samtani et al. , 2021), while parallel work modeled threat emergence
over time via diachronic graph embeddings (Samtani et al. Detecting Cyber Threats in Non-English Hacker Forums:
An Adversarial Cross-Lingual Knowledge Transfer Approach. IEEE Security and Privacy Workshops, 20–26.Where the paper says it
Where the paper says it
Samtani, S., Li, W., Benjamin, V. p. 0 HACKERSIGNAL: A Large-Scale Multi-Source Dataset
Linking Hacker Community Discourse to the CVE
Vulnerability Lifecycle
Benjamin M. Ampel
Department of Computer Information Systems
J. The AZSecure Hacker Assets Portal operationalized this as a
situational awareness platform (Samtani et al. , 2021), while parallel work modeled threat emergence
over time via diachronic graph embeddings (Samtani et al. com/BenAmpel/hackersignal (code). 1 Introduction
Understanding how cyber threats emerge, propagate, and relate to vulnerabilities is a central challenge
in modern applied cybersecurity.Where the paper says it
Where the paper says it
Where the paper says it
Zhang, J., Yu, Y., Li, Y., Wang, Y., Yang, Y., Yang, M., & Ratner, A. (2021). WRENCH: A Comprehensive Benchmark for Weak Supervision. p. 2 At the same time, WRENCH (Zhang
et al. , 2021) demonstrated that weak-supervision evaluation is sensitive to hidden protocol choices,
requiring explicit reporting of label sources and aggregation rules. A survey of CTI extraction from text
Preprint. arXiv:2605.Where the paper says it
Where the paper says it
Rahman, M. R., Mahdavi-Hezaveh, R., & Williams, L. (2023). What Are the Attackers Doing Now? Automating Cyberthreat Intelligence Extraction from Text on Pace with the Changing Threat Landscape: A Survey. p. 1 Rahman et al. (2023) survey automated CTI
extraction from text, observing that collections often concentrate on the advisory and threat-report tier,
while exploit-archive and PoC-focused benchmarks remain scarce. Datasheets for Datasets. Communications of the ACM, 64(12), 86–92.Where the paper says it
Where the paper says it
Hughes, J., Pastrana, S. p. 0 Signals that tracks this trajectory is distributed across heterogeneous
text-based sources, including: (1) hacker forums where exploits are first discussed, (2) public exploit
databases where proof-of-concept code is released, (3) official vulnerability advisories such as the
National Vulnerability Database, CISA’s Known Exploited Vulnerabilities, and GitHub Advisory
where threats are cataloged, and (4) open-source repositories where fixes are committed. Extant
public datasets in this domain often span a single or subset of these layers (Pastrana et al. naacl-main. , Marin, E. identifies advisory records and threat reports as the dominant labeled sources studied, with exploitarchive and PoC-focused benchmarks remaining scarce (Rahman et al. , 2023); a companion survey
of cybercrime community research corroborates that structured datasets linking forum posts to formal
vulnerability records are largely absent from the public literature (Hughes et al. Datasheets for Datasets. Communications of the ACM, 64(12), 86–92.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
A Source Collection Details
Table 7: Per-source collection details
Table 1: Corpus quality diagnostics
Figure 1: HackerSignal pipeline overview
Figure 2: Temporal coverage by source layer (log10 scale)
Figure 3: Pairwise vocabulary overlap across 46 hacker community forums (5% deterministic token sample, hierarchically clustered)
Table 2: Dataset statistics after exact deduplication
Table 3: Benchmark task summary
Table 4: Task 1 (CVE-R) test results (1,990 queries, 340K corpus)
Table 5: Task 2 (ETC) test results on 1,735 temporally held-out samples. Macro-F1 and per-class F1 for the three hardest classes
Table 6: Task 3 (TG) results (1,898 test queries) under strict CVE disjointness (Ctrain ∩ Ctest = ∅).
Figure 4: Radar chart summary of baseline performance across all three tasks
Table 8: Comparison of HackerSignal with prior CTI datasets across key dimensions = yes, = no, ∼ = partial.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Cross-source temporal out-of-distribution entity grounding allows AI systems to retrieve correct CVE entries from exploit archives, advisories, and fix commits based on hacker community discourse queries.
-
Exploit type classification enables AI systems to predict an 8-class vulnerability type from exploit text using a model ladder that spans bag of words classifiers up to domain-specific transformers like SecBERT.
-
Temporal generalization enables AI systems to perform prospective CVE disjoint evaluation by testing their ability to generalize to wholly unseen vulnerabilities that postdate the training data, ensuring models do not memorize era-specific features.
-
The hybrid BM25 plus dense retrieval approach can be used in retrieval tasks to rerank candidates using a weighted combination of lexical and dense cosine scores, potentially improving Recall@k compared to symmetric bi-encoders.
Abstract
Cyber threat intelligence (CTI) systems must link exploit code, advisories, and fix commits to the vulnerabilities they concern, and they must do so for vulnerabilities disclosed after training. HackerSignal is a public corpus and benchmark built for this prospective setting. It aggregates 7.45 million exact-deduplicated documents from 64 public sources across eight source layers (hacker forums, exploit archives, exploit question-and-answer sites, vulnerability references, advisories, bug-bounty disclosures, fix commits, and known-exploited-vulnerability records), spanning 1988--2026. All layers are keyed to a shared CVE identifier space with source provenance and release-mode governance. We define three benchmark tasks with temporal splits: (1) CVE linkage retrieval over 340,536 National Vulnerability Database descriptions, (2) exploit type classification into eight classes, and (3) temporal generalization, where training and test CVE sets do not overlap (C train C test =). Zero-shot dense and hybrid retrievers reach a mean reciprocal rank (MRR) of 0.71 on Task 1, and the top models are statistically tied under paired bootstrap tests. Cybersecurity-pretrained encoders lead exploit type classification (macro-F1 0.90). A two-annotator audit of 240 linkage labels finds 79.5% precision (Cohen's 0.94), with per-source precision from 20% to 100%. The hacker-community layer contributes 6.97 million posts but only about 2,000 CVE-linked records. A masked forum-post grounding pilot over these records reaches MRR 0.08, about nine times lower than the structured-source tasks. Grounding informal hacker discourse to vulnerabilities therefore remains an open problem that the release is designed to support. Code, data, and Croissant metadata are available at https://huggingface.co/datasets/BenAmpel/HackerSignal and https://github.com/BenAmpel/hackersignal.
Sources
- BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
- WRENCH: A Comprehensive Benchmark for Weak Supervision
Related papers
- From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment
- Privacy Leakage from Recurrent State in LLM Inference
- Incrementally Verifiable Oracle Aided Computation
- Privilege and confidentiality in generative AI workflows
- BadMoE: Backdooring Mixture-of-Experts LLMs via Routing-Aware Triggers and Task-Underutilized Experts
- CoVault: Secure, Scalable Analytics of Personal Data