HackerSignal: A Multi-Source, CVE-Linked Corpus and Prospective Benchmark for Cyber Threat Intelligence

arXiv:2605.03158 · cs.CR · Submitted 2026-05-04 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "HackerSignal: A Multi-Source, CVE-Linked Corpus and Prospective Benchmark for Cyber Threat Intelligence".

Elias: Here's the gist of the paper. HackerSignal introduces a benchmark for temporal out-of-distribution cyber threat intelligence and cross-source CVE linkage by aggregating millions of documents from diverse public sources to map the full exploit trajectory from hacker community discourse to vulnerability fixes Dataset Scope and Structure HackerSignal aggregates 7.45 million exact-deduplicated documents spanning eight source layers and a 36-year window from 1990 to 2026.

Nadia: The corpus is unified through a shared CVE identifier space, allowing for cross-source joins while preserving source-specific release modes for AI analytics tasks.

Elias: The eight source layers include hacker communities, exploit archives, vulnerability references, exploit Q&A references, advisories, bug-bounty disclosures, fix commits, and exploitation references.

Nadia: The largest subcorpus is the hacker community layer with 6.97 million rows spanning 56 forums across darkweb marketplaces and security discussion boards.

Title and authors: Nadia: Now we’re diving into the specific details of this work. The paper is titled HackerSignal: A Multi-Source, CVE-Linked Corpus and Prospective Benchmark for Cyber Threat Intelligence. We’ve already touched on the scale and structure, but let's unpack what that title actually means for us in practice.

Elias: It signals that this isn't just another dataset of forum posts or vulnerability descriptions; it’s a system designed to link hacker community discourse directly to the CVE vulnerability lifecycle. The focus on "multi-source" tells us we’re not relying on a single data stream, which is key for building robust models.

Priya: And the "CVE-linked corpus" part is important because it moves the goal from just identifying mentions of a vulnerability to tracing that mention all the way to its official advisory and eventually its fix commit. That’s where we get real intelligence.

Nadia: Right, Priya, because if you only have text about an exploit but no link to the actual patch code or advisory, you’re stuck in a very shallow understanding of what happened. HackerSignal aims to bridge that gap completely by creating those connections through a shared CVE identifier space.

Elias: And they emphasize "prospective benchmark," meaning the goal isn't just retrospective analysis of what already happened; it's testing how well an AI can predict or ground information about threats that haven't appeared in the training period yet.

Priya: That prospective aspect is where I see a lot of the real value for predictive security; being able to test if a system can anticipate future threat patterns based on historical context without simply memorizing past examples.

Nadia: Exactly, and they achieve this by setting up those three specific tasks: CVE linkage retrieval, exploit type classification, and temporal generalization. Those tasks define exactly what we want to measure about a CTI system’s ability to operate in a dynamic environment.

Elias: They also detail the eight source layers they aggregated across that thirty six-year window from nineteen ninety to two thousand twenty-six which is what gives them that massive temporal coverage and diversity. It covers everything from dark web marketplaces to official fix commits.

Priya: The diversity is what makes it rich for measurement, I think, because a model trained on just forum posts might fail when asked to classify an exploit type mentioned in a formal advisory document.

Nadia: That’s right; they want to see if the model can handle that cross-domain shift. The paper lays out the data statistics too, showing things like the percentage of rows with CVE strings present and how long those documents are, which gives us a good feel for the quality and complexity of what we’re feeding into these systems.

Elias: The data shows that while there's a decent presence of CVE strings—about four point eight percent—the sheer volume of hacker community posts is massive, with about six point nine seven million rows spanning fifty six forums alone, which is the largest subcorpus they have.

Priya: That large hacker community subcorpus is interesting because it suggests that even though the formal records are less numerous, the actual raw discussion from the underground communities holds a lot of contextual information.

Nadia: So, to summarize this segment, HackerSignal isn't just a big dump of text; it’s a structured, temporally grounded system built specifically to test AI’s ability to trace the full exploit trajectory across diverse data types and time periods.

Elias: And the key takeaway is that they are creating a comprehensive benchmark for evaluating systems on their ability to ground information reliably across disparate sources, which is what makes it so significant in the CTI space.

Priya: It sounds like we're looking at a really detailed foundation for measuring how effective AI can be at connecting the dots between raw chatter and concrete security fixes.

Nadia: Exactly, and that foundation is what allows us to start asking better questions about how quickly threats evolve and how effectively we can defend against them in the future.

The paper's summary: Nadia: Moving on to the actual summary of this paper, HackerSignal’s core contribution is providing a temporally structured, multi-source corpus of seven point four five million documents from sixty four public forum and source identifiers spanning eight different layers over a thirty six year window.

Elias: And the main message they are driving home is that previous publicly accessible datasets have been lacking in linking hacker posts to formal vulnerability records, so HackerSignal addresses this gap by creating these explicit CVE linkage metadata connecting the vulnerability identifiers to the texts that discuss them.

Priya: What I find compelling about their summary is how they frame it as addressing a known absence in the public literature—that structured datasets linking forum posts to formal vulnerability records are largely absent, and HackerSignal steps in to fill that hole.

Nadia: That’s right; they acknowledge that previous work focused heavily on advisory records or exploit archives, but this paper shows the operational gap between those and the raw community discussion that actually drives the initial discovery.

Elias: They explicitly state their three key contributions: first, building this temporally grounded corpus with all those links, second, creating a companion survey of cybercrime community research to corroborate that structured datasets are missing, and third, developing this benchmark for downstream CTI analytics tasks.

Priya: I’m interested in the benchmark part because it means they are not just presenting data; they are providing a tool for researchers to rigorously test their AI systems against these real-world intelligence problems.

Nadia: Precisely; they aren't just giving us a dataset to train on; they’re giving us a rigorous way to measure how well an AI can perform specific tasks like linking entities, classifying exploit types, and generalizing temporally.

Elias: They also detail the nature of their data quality diagnostics—showing things like the percentage of documents with CVE strings present and the distribution of document lengths—to give researchers a clearer picture of what they are working with.

Priya: That diagnostic information is crucial because it allows us to understand the inherent noise and signal in this corpus before we even start training any models on top of it. It helps us manage expectations about the data's quality and biases.

Nadia: So, to wrap up, HackerSignal’s summary is that they have filled a major gap by creating a dataset that maps the full potential exploit trajectory from hacker community discourse through exploits and fixes using explicit CVE links across time.

Elias: It’s positioning itself as the definitive resource for testing systems that need to operate robustly in the messy, evolving world of cyber threat intelligence.

Priya: It sounds like a necessary step toward making AI-driven threat intelligence more reliable because it grounds those predictions in verifiable, traceable data points from both the community and the fix side.

The paper's improvements: Nadia: Let's talk about the specific improvements they propose for HackerSignal. They aren't just presenting a static dataset; they are proposing ways to make this corpus even more useful for downstream analytics by suggesting enhancements to how we process and query it.

Elias: One major suggestion is around retrieval tasks, where they propose using a hybrid BM25 plus dense retrieval approach for reranking candidates, which combines the lexical score with the dense cosine score.

Priya: That makes sense; mixing those methods might give us better recall at k because we’re getting the best of both worlds—the keyword matching and the contextual understanding. But I wonder if that just adds complexity without a real gain.

Nadia: The authors suggest that this hybrid approach could actually improve recall at k when compared to symmetric bi-encoders, suggesting it's a worthwhile experiment to run on this data. They are also pointing out that asymmetric prefix training is particularly effective for cross-source CVE grounding.

Elias: That points back to the classification tasks, where they’re finding that asymmetric prefix training helps them achieve better results when grounding information across different sources, which is an interesting parameter tuning suggestion for those NLP models.

Priya: So, the improvement isn't just about the data itself, but about providing these specific training and retrieval strategies that are known to work well in this context, which is really helpful guidance for researchers.

Nadia: It’s about moving beyond just feeding a model raw data; it’s suggesting specific ways to train and query it so that its ability to cross-source link things improves, which is a practical application of the research.

Elias: And they also have these governance features mentioned, like explicit checks for temporal and source leakage during their audit process to keep the benchmark claims from being inflated. They are basically saying, we need rigorous internal validation if we want to trust these results.

Priya: That rigor is essential because without it, any benchmark can just be a vanity metric; checking for that kind of leakage shows they’re serious about maintaining data integrity throughout the entire process.

Nadia: So, the improvements are a combination of methodological tweaks—like hybrid retrieval and specific training techniques—and strong governance checks to ensure the integrity of the large-scale benchmark.

Elias: They are essentially saying, this is how you build a better system for this kind of complex CTI problem by paying attention to both the data architecture and the model training parameters.

Priya: It sounds like they’re really trying to give researchers a complete toolkit, not just a finished product, which is what we need in applied research.

Conclusion: Nadia: So, wrapping up this discussion on HackerSignal: This corpus and benchmark are designed to test if AI can actually perform the three key tasks—CVE linkage retrieval, exploit type classification, and temporal generalization—on truly complex, messy cyber threat data.

Elias: The implication is that if these models perform well here under those strict temporal constraints, then we gain confidence that they have learned a general understanding of the entire vulnerability lifecycle rather than just memorizing patterns from a specific time period.

Priya: For us, it means we have a more rigorous standard for evaluating predictive systems because we’re looking at how they handle both the sheer volume and the temporal shifts in threat data.

Nadia: And what’s next is seeing if this corpus can truly guide new methods for proactive CTI by showing us where AI is succeeding and where it’s still falling short when facing unseen threats.

Elias: The full scope of HackerSignal is that it provides a massive, multi-source data foundation for testing cross-source CVE grounding, exploit type classification, and prospective generalization in a way that was previously missing.

Priya: It’s a big step toward making AI-driven threat intelligence more reliable because it grounds those predictions in verifiable, traceable data points from both the community and the fix side.

Nadia: We've covered a lot about how this work is structured, from the governance checks to the specific results on task three, which showed an mrr of zero point six zero for prospective generalization.

Elias: And I’m glad we could discuss how asymmetric prefix training helps with cross-source grounding because that seems like a key mechanism for success in this domain.

Priya: It’s about having the tools to evaluate these systems properly, which is what matters most when we're building defenses against unknown risks.

Nadia: So, HackerSignal is positioned as a very comprehensive resource for anyone looking to test the limits of current AI in understanding and predicting cyber threats across time and different data sources.

Elias: It’s a lot of work, but it creates a high-quality standard for how we measure CTI capability moving forward.

Georgia State University · Indiana University Bloomington

cs.CR

Submitted: 2026-05-04

Updated: 2026-11-05

Comments: Published at NeurIPS 2026 (40th Conference on Neural Information Processing Systems), Track on Evaluations and Datasets. Camera-ready version. 19 pages, 4 figures, 13 tables. Data: https://huggingface.co/datasets/BenAmpel/HackerSignal; code: https://github.com/BenAmpel/hackersignal

Code: https://github.com/BenAmpel/hackersignal

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 89/100

The gist: HackerSignal introduces a benchmark for temporal out-of-distribution cyber threat intelligence and cross-source CVE linkage by aggregating millions of documents from diverse public sources to map the…

Key concepts

CVE Linkage Retrieval
This task tests if an AI can correctly find the specific Common Vulnerabilities and Exposures (CVE) identifier for a vulnerability by searching across different sources like hacker forums, exploit archives, and official advisories. It requires grounding a query in the correct CVE entity from disparate documents.
Exploit Type Classification
This involves training an AI to categorize exploit text into one of eight specific vulnerability types. The evaluation uses a range of models, from simple bag-of-words methods up to advanced transformer models like SecBERT, to see how well the model recognizes the exploit's nature.
Temporal Generalization
This benchmark assesses a model's ability to predict vulnerabilities that have not yet been discovered or fixed. It enforces a strict rule: the test set must contain CVEs that are entirely disjoint from those in the training set, ensuring the model generalizes beyond memorized historical patterns.
Source Layer Aggregation
The dataset is built by combining eight distinct layers of data, including hacker communities, exploit archives, and fix commits. This aggregation allows AI systems to learn relationships across these different types of information simultaneously for more comprehensive threat intelligence.

Terminology

Summary

HackerSignal introduces a benchmark for temporal out-of-distribution cyber threat intelligence and cross-source CVE linkage by aggregating millions of documents from diverse public sources to map the full exploit trajectory from hacker community discourse to vulnerability fixes

Dataset Scope and Structure

HackerSignal aggregates 7.45 million exact-deduplicated documents spanning eight source layers and a 36-year window from 1990 to 2026. p. 0

Where the paper says it

HackerSignal aggregates 7. 45 million exact-deduplicated documents from 64 public forum/source identifiers spanning eight source layers and a 36-year window (1990–2026).

The corpus is unified through a shared CVE identifier space, allowing for cross-source joins while preserving source-specific release modes for AI analytics tasks. p. 0
Where the paper says it

HackerSignal creates these linkages through a shared CVE identifier space while preserving source-specific release modes to support a range of unique Artificial Intelligence (AI)-enabled cybersecurity analytics tasks. In this paper, we summarize HackerSignal and illustrate three selected benchmark tasks it uniquely supports: (1) CVE linkage retrieval (cross-source temporal out-of-distribution entity grounding); (2) exploit type classification (8-class vulnerability type prediction with temporal OOD evaluation); and (3) temporal generalization (prospective CVE-disjoint evaluation where Ctrain ∩ Ctest = ∅).

The eight source layers include hacker communities, exploit archives, vulnerability references, exploit Q&A references, advisories, bug-bounty disclosures, fix commits, and exploitation references. p. 1
Where the paper says it

A temporally grounded CTI corpus of 7. 45M documents from 64 public forum/source identifiers across eight source layers (hacker communities, exploit archives, vulnerability references, exploit Q&A references, advisories, bug-bounty disclosures, fix commits, and exploitation references), with CVE linkage metadata connecting vulnerability identifiers to the texts that discuss them.

The largest subcorpus is the hacker community layer with 6. p. 1
Where the paper says it

45M documents from 64 public forum/source identifiers across eight source layers (hacker communities, exploit archives, vulnerability references, exploit Q&A references, advisories, bug-bounty disclosures, fix commits, and exploitation references), with CVE linkage metadata connecting vulnerability identifiers to the texts that discuss them. 97M-row hacker community subcorpus alone (spanning 56 forums across darkweb marketplaces, security discussion boards, and multilingual hacker communities) supports diverse NLP tasks beyond our three benchmarks, including exploit code summarization, security content classification, topic modeling, community interaction analysis, and cross-lingual CTI extraction.

97 million rows spanning 56 forums across darkweb marketplaces and security discussion boards. p. 1
Where the paper says it

45M documents from 64 public forum/source identifiers across eight source layers (hacker communities, exploit archives, vulnerability references, exploit Q&A references, advisories, bug-bounty disclosures, fix commits, and exploitation references), with CVE linkage metadata connecting vulnerability identifiers to the texts that discuss them. 97M-row hacker community subcorpus alone (spanning 56 forums across darkweb marketplaces, security discussion boards, and multilingual hacker communities) supports diverse NLP tasks beyond our three benchmarks, including exploit code summarization, security content classification, topic modeling, community interaction analysis, and cross-lingual CTI extraction.

Benchmark Tasks

The paper illustrates three selected benchmark tasks that HackerSignal uniquely supports for temporal out-of-distribution generalization. These tasks include:

  1. CVE linkage retrieval (cross-source temporal out-of-distribution entity grounding). p. 0

    Where the paper says it

    HackerSignal creates these linkages through a shared CVE identifier space while preserving source-specific release modes to support a range of unique Artificial Intelligence (AI)-enabled cybersecurity analytics tasks. In this paper, we summarize HackerSignal and illustrate three selected benchmark tasks it uniquely supports: (1) CVE linkage retrieval (cross-source temporal out-of-distribution entity grounding); (2) exploit type classification (8-class vulnerability type prediction with temporal OOD evaluation); and (3) temporal generalization (prospective CVE-disjoint evaluation where Ctrain ∩ Ctest = ∅).

  2. Exploit type classification (8-class vulnerability type prediction with temporal OOD evaluation). p. 0

    Where the paper says it

    HackerSignal creates these linkages through a shared CVE identifier space while preserving source-specific release modes to support a range of unique Artificial Intelligence (AI)-enabled cybersecurity analytics tasks. In this paper, we summarize HackerSignal and illustrate three selected benchmark tasks it uniquely supports: (1) CVE linkage retrieval (cross-source temporal out-of-distribution entity grounding); (2) exploit type classification (8-class vulnerability type prediction with temporal OOD evaluation); and (3) temporal generalization (prospective CVE-disjoint evaluation where Ctrain ∩ Ctest = ∅).

  3. Temporal generalization (prospective CVE-disjoint evaluation where Ctrain ∩ Ctest = ∅). p. 0

    Where the paper says it

    HackerSignal creates these linkages through a shared CVE identifier space while preserving source-specific release modes to support a range of unique Artificial Intelligence (AI)-enabled cybersecurity analytics tasks. In this paper, we summarize HackerSignal and illustrate three selected benchmark tasks it uniquely supports: (1) CVE linkage retrieval (cross-source temporal out-of-distribution entity grounding); (2) exploit type classification (8-class vulnerability type prediction with temporal OOD evaluation); and (3) temporal generalization (prospective CVE-disjoint evaluation where Ctrain ∩ Ctest = ∅).

Methodology and Evaluation

All tasks utilize temporal splits to evaluate prospective generalization because real CTI systems must generalize to threats that did not exist at training time. The evaluation uses a heterogeneous baseline ladder spanning lexical, dense, and hybrid methods for retrieval tasks. p. 5

Where the paper says it

Following Thakur et al. (2021), we evaluate each task with a heterogeneous baseline ladder spanning lexical, dense, and hybrid methods.

For classification tasks, the ladder spans bag-of-words classifiers through a domain-specific transformer like SecBERT. p. 1
Where the paper says it

Three benchmark tasks with temporal train/val/test splits that test progressively harder generalization in (1) cross-source CVE linkage retrieval, (2) 8-class exploit type classification, and (3) CVE-disjoint temporal generalization (Ctrain ∩ Ctest = ∅), accompanied by shortcut/leakage diagnostics. A full model ladder and governance artifacts from BM25 through four dense bi-encoders (MiniLM, mpnet, BGE, E5), bag-of-words classifiers (Decision Tree, TF-IDF+LR, SVM), recurrent neural networks (RNN, GRU, LSTM, BiLSTM), a domain-specific transformer (SecBERT), and hybrid sparse+dense retrieval, alongside source-shortcut diagnostics, manual-audit packets, and a release-governance addendum.

The temporal generalization task enforces a strict CVE-disjointness constraint where Ctrain ∩ Ctest = ∅. p. 7
Where the paper says it

Temporal splits make it harder as exploit names and vulnerability patterns evolve, preventing models from memorizing era-specific features. 3 Task 3: Temporal Generalization (TG) Identical retrieval formulation to Task 1, but with a strict CVE-disjointness constraint: Ctrain ∩ Ctest = ∅, where Cs = {c ∗ i : qi ∈ split s} is the set of gold CVE targets in split s.

Audit and Governance

The authors explicitly audit the dataset for temporal and source leakage because benchmark claims are otherwise easy to inflate. Integrity checks include verifying correct CVE assignment, temporal split adherence, and absence of trivial textual overlap between query and target. p. 6

Where the paper says it

For each query q ∈ Q, the task is to retrieve the correct CVE entry c ∗ ∈ C such that the source metadata links q to c ∗ . This is cross-source temporal out-of-domain entity grounding, akin to knowledge-intensive entity linking (Petroni et al.

Ethical considerations mandate that the dataset is for academic and defensive use only, prohibiting finetuning automated exploit models or de-anonymizing forum users without human review. p. 8
Where the paper says it

We pseudonymize author IDs via SHA-256, preserve provenance, and support takedown requests, downgrading ambiguous sources accordingly. HackerSignal highlights vulnerabilities for defensive research only, prohibiting finetuning automated exploit models, de-anonymizing forum users, and automated incident response without human review.

The release governance addendum details release modes, checksums, and takedown procedures for all sources. p. 8
Where the paper says it

The public release categorizes sources into redistributable, research-only, metadata/pointer, and excluded. A governance addendum details release modes, contact info, versioning, checksums, and takedown procedures.

Results Summary

The model ladder establishes baselines for cross-source CVE grounding, exploit type classification, and prospective generalization with results such as an MRR of 0.60 for Task 3. p. 1

Where the paper says it

For cross-source linkage, an attention-based deep-structured semantic model linked dark-web exploits to known vulnerabilities, yielding 52,590 linkages (Samtani et al. HackerSignal differs by releasing a governed, temporally split public benchmark that exposes source provenance, weak-label caveats, and a quality-controlled CVE-linkage retrieval task.

The best bag-of-words baseline for Task 2 attains a macro-F1 of 0. p. 6
Where the paper says it

We report macro-F1 and per-class F1 for the three hardest classes in Table 5. The best bag-of-words baseline (SVM) attains macro-F1 of 0.

799. The consistent E5 advantage across both tasks suggests that asymmetric prefix training is particularly effective for cross-source CVE grounding. p. 7
Where the paper says it

Models generally maintain or improve performance compared to Task 1, confirming that our temporal split already prevents meaningful memorization. The consistent E5 advantage across both tasks suggests that asymmetric prefix training is particularly effective for cross-source CVE grounding, where queries and targets have different discourse design, as already demonstrated in a rigorous prospective evaluation.

The paper concludes by positioning HackerSignal against prior CTI datasets, emphasizing its scale and governance features.

References

Akhtar, M., et al. (2024). Croissant: A Metadata Format for ML-Ready Datasets. p. 9

Where the paper says it

References Akhtar, M. Croissant: A Metadata Format for ML-Ready Datasets.

NeurIPS 2024 D&B Track

Ampel, B., Samtani, S. p. 0

Where the paper says it

Mack Robinson College of Business, Georgia State University bampel@gsu. edu Sagar Samtani Department of Operations and Decision Technologies Kelley School of Business, Indiana University Bloomington ssamtani@iu.

p. 0
Where the paper says it

Mack Robinson College of Business, Georgia State University bampel@gsu. edu Sagar Samtani Department of Operations and Decision Technologies Kelley School of Business, Indiana University Bloomington ssamtani@iu.

p. 0
Where the paper says it

Mack Robinson College of Business, Georgia State University bampel@gsu. edu Sagar Samtani Department of Operations and Decision Technologies Kelley School of Business, Indiana University Bloomington ssamtani@iu.

p. 0
Where the paper says it

Mack Robinson College of Business, Georgia State University bampel@gsu. edu Sagar Samtani Department of Operations and Decision Technologies Kelley School of Business, Indiana University Bloomington ssamtani@iu.

p. 0
Where the paper says it

Mack Robinson College of Business, Georgia State University bampel@gsu. edu Sagar Samtani Department of Operations and Decision Technologies Kelley School of Business, Indiana University Bloomington ssamtani@iu.

p. 0
Where the paper says it

Mack Robinson College of Business, Georgia State University bampel@gsu. edu Sagar Samtani Department of Operations and Decision Technologies Kelley School of Business, Indiana University Bloomington ssamtani@iu.

, Zhu, H., Ullman, S. p. 9
Where the paper says it

, Samtani, S. , Ullman, S.

, & Chen, H. p. 9
Where the paper says it

, Ullman, S. , & Chen, H.

p. 9
Where the paper says it

, Ullman, S. , & Chen, H.

p. 9
Where the paper says it

, Ullman, S. , & Chen, H.

p. 9
Where the paper says it

, Ullman, S. , & Chen, H.

p. 9
Where the paper says it

, Ullman, S. , & Chen, H.

p. 9
Where the paper says it

, Ullman, S. , & Chen, H.

p. 9
Where the paper says it

, Ullman, S. , & Chen, H.

p. 9
Where the paper says it

, Ullman, S. , & Chen, H.

(2020). Labeling Hacker Exploits for Proactive Cyber Threat Intelligence: A Deep Transfer Learning Approach. p. 9
Where the paper says it

, & Chen, H. Labeling Hacker Exploits for Proactive Cyber Threat Intelligence: A Deep Transfer Learning Approach.

IEEE Intelligence and Security Informatics (ISI). p. 9
Where the paper says it

Labeling Hacker Exploits for Proactive Cyber Threat Intelligence: A Deep Transfer Learning Approach. IEEE Intelligence and Security Informatics (ISI).

p. 9
Where the paper says it

Labeling Hacker Exploits for Proactive Cyber Threat Intelligence: A Deep Transfer Learning Approach. IEEE Intelligence and Security Informatics (ISI).

p. 9
Where the paper says it

Labeling Hacker Exploits for Proactive Cyber Threat Intelligence: A Deep Transfer Learning Approach. IEEE Intelligence and Security Informatics (ISI).

p. 9
Where the paper says it

Labeling Hacker Exploits for Proactive Cyber Threat Intelligence: A Deep Transfer Learning Approach. IEEE Intelligence and Security Informatics (ISI).

https://doi. p. 9
Where the paper says it

IEEE Intelligence and Security Informatics (ISI). https://doi.

p. 9
Where the paper says it

IEEE Intelligence and Security Informatics (ISI). https://doi.

p. 9
Where the paper says it

IEEE Intelligence and Security Informatics (ISI). https://doi.

p. 9
Where the paper says it

IEEE Intelligence and Security Informatics (ISI). https://doi.

p. 9
Where the paper says it

IEEE Intelligence and Security Informatics (ISI). https://doi.

p. 9
Where the paper says it

IEEE Intelligence and Security Informatics (ISI). https://doi.

p. 9
Where the paper says it

IEEE Intelligence and Security Informatics (ISI). https://doi.

p. 9
Where the paper says it

IEEE Intelligence and Security Informatics (ISI). https://doi.

p. 9
Where the paper says it

IEEE Intelligence and Security Informatics (ISI). https://doi.

p. 9
Where the paper says it

IEEE Intelligence and Security Informatics (ISI). https://doi.

p. 9
Where the paper says it

IEEE Intelligence and Security Informatics (ISI). https://doi.

p. 9
Where the paper says it

IEEE Intelligence and Security Informatics (ISI). https://doi.

p. 9
Where the paper says it

IEEE Intelligence and Security Informatics (ISI). https://doi.

p. 9
Where the paper says it

IEEE Intelligence and Security Informatics (ISI). https://doi.

p. 9
Where the paper says it

IEEE Intelligence and Security Informatics (ISI). https://doi.

p. 9
Where the paper says it

IEEE Intelligence and Security Informatics (ISI). https://doi.

p. 9
Where the paper says it

IEEE Intelligence and Security Informatics (ISI). https://doi.

org/10.1109/ISI49825. p. 9
Where the paper says it

https://doi. 1109/ISI49825.

2020.9280548

Ampel, B., Samtani, S., Zhu, H., & Chen, H. (2024). Creating Proactive Cyber Threat Intelligence with Hacker Exploit Labels: A Deep Transfer Learning Approach. p. 9

Where the paper says it

, & Chen, H. Creating Proactive Cyber Threat Intelligence with Hacker Exploit Labels: A Deep Transfer Learning Approach.

MIS Quarterly, 48(1), 137–166. p. 9
Where the paper says it

Creating Proactive Cyber Threat Intelligence with Hacker Exploit Labels: A Deep Transfer Learning Approach. MIS Quarterly, 48(1), 137–166.

https: //doi. p. 9
Where the paper says it

IEEE Intelligence and Security Informatics (ISI). https://doi.

org/10.25300/MISQ/2023/17316

Robertson, S., & Jones, K. p. 9

Where the paper says it

Robertson, S. , & Jones, K.

S. (1994). Simple, proven approaches to text retrieval. p. 9
Where the paper says it

, & Jones, K. Simple, proven approaches to text retrieval.

Technical report, University of Cambridge

Thakur, N., Reimers, N. p. 9

Where the paper says it

Technical report, University of Cambridge. , Reimers, N.

, Rücklé, A. p. 9
Where the paper says it

, Reimers, N. , Rücklé, A.

, Srivastava, A. p. 9
Where the paper says it

, Rücklé, A. , Srivastava, A.

, & Gurevych, I. p. 9
Where the paper says it

, Srivastava, A. , & Gurevych, I.

p. 9
Where the paper says it

, Srivastava, A. , & Gurevych, I.

(2021). BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. p. 9
Where the paper says it

, & Gurevych, I. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models.

NeurIPS 2021 D&B Track. p. 9
Where the paper says it

BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS 2021 D&B Track.

p. 9
Where the paper says it

BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS 2021 D&B Track.

arXiv:2104. p. 9
Where the paper says it

NeurIPS 2021 D&B Track. arXiv:2104.

08663

CISA. (2021–2026). p. 2

Where the paper says it

, 2021) links CVEs to fix commits in open-source repositories. NVD (NIST, 2002–2026), CISA KEV (CISA, 2021–2026), and GitHub Advisory (GitHub, 2017–2026) are official advisory sources.

Known Exploited Vulnerabilities Catalog. p. 9
Where the paper says it

(2021–2026). Known Exploited Vulnerabilities Catalog.

https://www. p. 9
Where the paper says it

Known Exploited Vulnerabilities Catalog. https://www.

p. 9
Where the paper says it

Known Exploited Vulnerabilities Catalog. https://www.

cisa.gov/known-exploited-vulnerabilities-catalog

Pastrana, S., Thomas, D. p. 9

Where the paper says it

Pastrana, S. , Thomas, D.

R., Hutchings, A. p. 9
Where the paper says it

, Thomas, D. , Hutchings, A.

p. 9
Where the paper says it

, Thomas, D. , Hutchings, A.

, & Clayton, R. p. 9
Where the paper says it

, Hutchings, A. , & Clayton, R.

(2018). CrimeBB: Enabling Cybercrime Research on Underground Forums at Scale. p. 9
Where the paper says it

, & Clayton, R. CrimeBB: Enabling Cybercrime Research on Underground Forums at Scale.

WWW 2018

Deliu, I., Leichter, C. p. 9

Where the paper says it

CrimeBB: Enabling Cybercrime Research on Underground Forums at Scale. , Leichter, C.

, & Franke, K. p. 9
Where the paper says it

, Leichter, C. , & Franke, K.

(2017). Extracting Cyber Threat Intelligence from Hacker Forums: Support Vector Machines versus Convolutional Neural Networks. p. 9
Where the paper says it

, & Franke, K. Extracting Cyber Threat Intelligence from Hacker Forums: Support Vector Machines versus Convolutional Neural Networks.

IEEE International Conference on Big Data. p. 9
Where the paper says it

Extracting Cyber Threat Intelligence from Hacker Forums: Support Vector Machines versus Convolutional Neural Networks. IEEE International Conference on Big Data.

https://doi.org/10.1109/BigData. p. 9
Where the paper says it

https://doi. 1109/BigData.

2017.8258359

Bhandari, G., Naseer, A. p. 9

Where the paper says it

Bhandari, G. , Naseer, A.

, & Moonen, L. p. 9
Where the paper says it

, Naseer, A. , & Moonen, L.

(2021). CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software. p. 9
Where the paper says it

, & Moonen, L. CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software.

PROMISE 2021. p. 9
Where the paper says it

CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software. PROMISE 2021.

https://doi.org/10.1145/357698

Ranade, P., Piplai, A. p. 9

Where the paper says it

PROMISE 2021. , Piplai, A.

, Joshi, A. p. 9
Where the paper says it

, Piplai, A. , Joshi, A.

, & Finin, T. p. 9
Where the paper says it

, Joshi, A. , & Finin, T.

(2021). CyBERT: Contextualized Embeddings for the Cybersecurity Domain. p. 9
Where the paper says it

, & Finin, T. CyBERT: Contextualized Embeddings for the Cybersecurity Domain.

IEEE International Conference on Big Data, 3334–3342. p. 9
Where the paper says it

CyBERT: Contextualized Embeddings for the Cybersecurity Domain. IEEE International Conference on Big Data, 3334–3342.

https://doi.org/10.1109/BigData52589. p. 9
Where the paper says it

https://doi. 1109/ BigData52589.

2021

Devlin, J., Chang, M. p. 9

Where the paper says it

1109/ BigData52589. , Chang, M.

-W., Lee, K., & Toutanova, K. p. 9
Where the paper says it

, Chang, M. , & Toutanova, K.

(2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. p. 9
Where the paper says it

, & Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.

NAACL 2019. p. 9
Where the paper says it

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019.

Ebrahimi, M. p. 2

Where the paper says it

More general hacker-forum text classification work compared SVM and CNN models for extracting CTI from forum posts (Deliu et al. , 2017), while Ebrahimi et al.

, Samtani, S., Chai, Y., & Chen, H. (2020). Detecting Cyber Threats in Non-English Hacker Forums: An Adversarial Cross-Lingual Knowledge Transfer Approach. p. 9
Where the paper says it

, & Chen, H. Detecting Cyber Threats in Non-English Hacker Forums: An Adversarial Cross-Lingual Knowledge Transfer Approach.

IEEE Security and Privacy Workshops, 20–26. p. 9
Where the paper says it

Detecting Cyber Threats in Non-English Hacker Forums: An Adversarial Cross-Lingual Knowledge Transfer Approach. IEEE Security and Privacy Workshops, 20–26.

https://doi.org/10.1109/SPW50608. p. 9
Where the paper says it

https://doi. 1109/SPW50608.

2020

Offensive Security. (2003–26). p. 2

Where the paper says it

Exploit and vulnerability databases. ExploitDB (Offensive Security, 2003–2026) is the salient public exploit archive (50K+ entries, many with CVE IDs).

Exploit Database. p. 0
Where the paper says it

1 Introduction Understanding how cyber threats emerge, propagate, and relate to vulnerabilities is a central challenge in modern applied cybersecurity. Signals that tracks this trajectory is distributed across heterogeneous text-based sources, including: (1) hacker forums where exploits are first discussed, (2) public exploit databases where proof-of-concept code is released, (3) official vulnerability advisories such as the National Vulnerability Database, CISA’s Known Exploited Vulnerabilities, and GitHub Advisory where threats are cataloged, and (4) open-source repositories where fixes are committed.

https://www.exploit-db. p. 0
Where the paper says it

45 million exact-deduplicated documents from 64 public forum/source identifiers spanning eight source layers and a 36-year window (1990–2026). In contrast to other publicly accessible cybersecurity datasets, HackerSignal is among the first public benchmark datasets that maps the full potential exploit to vulnerability trajectory from hacker community discourse, exploit databases with working and proof of concept exploits, vulnerability advisories, and software fix commits.

com/. Licensed CC BY-SA 4. p. 9
Where the paper says it

exploit-db. Licensed CC BY-SA 4.

0

Gayanku. (2020). Hacker Forum Posts Dataset. p. 2

Where the paper says it

Underground community datasets. Research has also focused on the general crawling of hacker communities, like a credentialized dataset of darkweb cybercrime forum posts widely used for crime and exploit prediction (Pastrana et al.

Kaggle

Gebru, T., et al. (2021). Datasheets for Datasets. p. 9

Where the paper says it

com/gayanku. Datasheets for Datasets.

Communications of the ACM, 64(12), 86–92

Cohen, J. (1960). A coefficient of agreement for nominal scales. p. 9

Where the paper says it

Communications of the ACM, 64(12), 86–92. A coefficient of agreement for nominal scales.

Educational and Psychological Measurement, 20(1), 37–46. p. 9
Where the paper says it

A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.

https://doi.org/10.1177/0013164460020001

GitHub. (2017–26). p. 1

Where the paper says it

Early hacker-forum CTI work treated underground communities as a source of proactive intelligence about tools, actors, and emerging attack capabilities (Samtani et al. , 2015, 2017).

GitHub Advisory Database. p. 0
Where the paper says it

1 Introduction Understanding how cyber threats emerge, propagate, and relate to vulnerabilities is a central challenge in modern applied cybersecurity. Signals that tracks this trajectory is distributed across heterogeneous text-based sources, including: (1) hacker forums where exploits are first discussed, (2) public exploit databases where proof-of-concept code is released, (3) official vulnerability advisories such as the National Vulnerability Database, CISA’s Known Exploited Vulnerabilities, and GitHub Advisory where threats are cataloged, and (4) open-source repositories where fixes are committed.

https://github. p. 9
Where the paper says it

GitHub Advisory Database. https://github.

com/advisories

Grisham, J., Samtani, S., Patton, M. p. 9

Where the paper says it

, Samtani, S. , Patton, M.

, & Chen, H. (2017). Identifying Mobile Malware and Key Threat Actors in Online Hacker Forums for Proactive Cyber Threat Intelligence. p. 9
Where the paper says it

, & Chen, H. Identifying Mobile Malware and Key Threat Actors in Online Hacker Forums for Proactive Cyber Threat Intelligence.

IEEE Intelligence and Security Informatics (ISI). https://doi.org/10.1145/3639362

Otto, K., Ampel, B. p. 0

Where the paper says it

HACKERSIGNAL: A Large-Scale Multi-Source Dataset Linking Hacker Community Discourse to the CVE Vulnerability Lifecycle Benjamin M. Ampel Department of Computer Information Systems J.

, Zhu, H., Samtani, S., & Chen, H. (2021). Exploring the Evolution of Exploit-Sharing Hackers: An Unsupervised Graph Embedding Approach. p. 9
Where the paper says it

, & Chen, H. Exploring the Evolution of Exploit-Sharing Hackers: An Unsupervised Graph Embedding Approach.

IEEE Intelligence and Security Informatics (ISI). https://doi.org/10.1109/ISI53945. p. 9
Where the paper says it

https://doi. 1109/ISI53945.

2021

NIST. (2002–26). p. 2

Where the paper says it

, 2021) links CVEs to fix commits in open-source repositories. NVD (NIST, 2002–2026), CISA KEV (CISA, 2021–2026), and GitHub Advisory (GitHub, 2017–2026) are official advisory sources.

National Vulnerability Database. p. 0
Where the paper says it

1 Introduction Understanding how cyber threats emerge, propagate, and relate to vulnerabilities is a central challenge in modern applied cybersecurity. Signals that tracks this trajectory is distributed across heterogeneous text-based sources, including: (1) hacker forums where exploits are first discussed, (2) public exploit databases where proof-of-concept code is released, (3) official vulnerability advisories such as the National Vulnerability Database, CISA’s Known Exploited Vulnerabilities, and GitHub Advisory where threats are cataloged, and (4) open-source repositories where fixes are committed.

https://nvd. p. 9
Where the paper says it

National Vulnerability Database. https://nvd.

nist.gov/. Public domain

Petroni, F., Piktus, A. p. 9

Where the paper says it

Petroni, F. , Piktus, A.

, Fan, A., Lewis, P. p. 9
Where the paper says it

, Piktus, A. , Lewis, P.

, Yazdani, M. p. 9
Where the paper says it

, Lewis, P. , Yazdani, M.

, et al. (2021). KILT: A Benchmark for Knowledge Intensive Language Tasks. p. 9
Where the paper says it

, Yazdani, M. KILT: A Benchmark for Knowledge Intensive Language Tasks.

NAACL 2021. p. 9
Where the paper says it

KILT: A Benchmark for Knowledge Intensive Language Tasks. NAACL 2021.

https://doi.org/10.18653/v1/2021

Nunes, E., Diab, A., Gunn, A., Marin, E. p. 9

Where the paper says it

naacl-main. , Marin, E.

, Mishra, V. p. 9
Where the paper says it

, Marin, E. , Mishra, V.

, Paliath, V. p. 9
Where the paper says it

, Mishra, V. , Paliath, V.

, Robertson, J. p. 9
Where the paper says it

25300/MISQ/2023/17316. Robertson, S.

, Shakarian, J. p. 9
Where the paper says it

, Robertson, J. , Shakarian, J.

, Thart, A. p. 9
Where the paper says it

, Shakarian, J. , Thart, A.

, & Shakarian, P. p. 9
Where the paper says it

, Robertson, J. , Shakarian, J.

(2016). Darknet and Deepnet Mining for Proactive Cybersecurity Threat Intelligence. p. 9
Where the paper says it

, & Shakarian, P. Darknet and Deepnet Mining for Proactive Cybersecurity Threat Intelligence.

IEEE Intelligence and Security Informatics (ISI). https://doi.org/10.1145/3409289

Jackaduma. (2022). SecBERT: A Pre-trained Language Model for Cybersecurity. p. 7

Where the paper says it

SecBERT (0. 846) surpasses GRU/LSTM but underperforms BiLSTM, suggesting that domain-specific pre-training offers modest benefits, and that bidirectionality is key.

https://huggingface. p. 4
Where the paper says it

The full schema specification is provided in the accompanying datasheet (Gebru et al. 3 Collection and Preprocessing Data was collected between January and April 2026 via (i) official APIs (NVD, GitHub Advisory, CISA KEV, HackerOne HuggingFace Hub), (ii) ingestion of existing public datasets (e.

co/jackaduma/SecBERT

Krippendorff, K. (2008). Systematic and random disagreement and the reliability of nominal data. p. 9

Where the paper says it

Communications of the ACM, 64(12), 86–92. A coefficient of agreement for nominal scales.

Communication Methods and Measures, 2(4), 323–338. p. 1
Where the paper says it

The closest predecessor to HackerSignal is in the field of exploit labeling research. Prior methods labeled hacker-forum exploit source code via deep transfer learning from professional exploit repositories (Ampel et al.

https://doi.org/10.1080/19312450802467134

Gwet, K. L. (2014). Handbook of Inter-Rater Reliability. Advanced Analytics

Kapoor, S., & Narayanan, A. p. 1

Where the paper says it

Temporal splits are necessary in our context as real CTI systems must generalize to threats that did not exist at training time (Koh et al. We explicitly audit the dataset for temporal and source leakage (Kapoor and Narayanan, 2023) because benchmark claims are otherwise easy to inflate (discussed in Section 3.

(2023). Leakage and the Reproducibility Crisis in Machine-Learning-Based Science. p. 2
Where the paper says it

, 2021) demonstrated that weak-supervision evaluation is sensitive to hidden protocol choices, requiring explicit reporting of label sources and aggregation rules. Kapoor and Narayanan (2023) documented widespread data leakage in ML-for-science benchmarks and proposed structured reporting to make integrity checks legible.

Patterns, 4(9), 100804

Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M. p. 2

Where the paper says it

, 2021) formalized natural arising distribution shifts (including temporal shifts) as first-class benchmark objects. At the same time, WRENCH (Zhang et al.

, et al. (2021). WILDS: A Benchmark of in-the-Wild Distribution Shifts. p. 2
Where the paper says it

WILDS (Koh et al. , 2021) formalized natural arising distribution shifts (including temporal shifts) as first-class benchmark objects.

ICML 2021, PMLR 139

Han, Z., Li, X., Xing, Z., Liu, H., & Feng, Z. (2017). Learning to Predict Severity of Software Vulnerability Using Only Vulnerability Description. p. 6

Where the paper says it

7) reranks BM25 top-100 candidates and outperforms symmetric bi-encoders but not the prefix-trained E5. 2 Task 2: Exploit Type Classification (ETC) Given an exploit text x, predict its vulnerability type y ∈ Y where |Y| = 8 classes: injection (SQLi, command/code injection, XXE), XSS (reflected, stored, DOM), memory corruption (buffer/heap overflow, use-after-free, format string), denial of service, file inclusion (LFI/RFI, path traversal, arbitrary file operations), authentication & access control (privilege escalation, auth bypass, CSRF, IDOR), remote code execution (RCE, deserialization), and information disclosure (SSRF, open redirect, data exposure).

ICSME 2017, 125–136

Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. p. 9

Where the paper says it

, & Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.

EMNLP 2019

Samtani, S., Chinn, R., & Chen, H. (2015). Exploring Hacker Assets in Underground Forums. p. 1

Where the paper says it

, 2015, 2017). The AZSecure Hacker Assets Portal operationalized this as a situational awareness platform (Samtani et al.

IEEE Intelligence and Security Informatics (ISI), 31–36. p. 9
Where the paper says it

Labeling Hacker Exploits for Proactive Cyber Threat Intelligence: A Deep Transfer Learning Approach. IEEE Intelligence and Security Informatics (ISI).

https://doi.org/10.1109/ISI.2015.7165935

Samtani, S., Chinn, R., Chen, H., & Nunamaker, J. F. (2017). Exploring Emerging Hacker Assets and Key Hackers for Proactive Cyber Threat Intelligence. p. 1

Where the paper says it

2 Related Work Hacker-community cyber threat intelligence. Early hacker-forum CTI work treated underground communities as a source of proactive intelligence about tools, actors, and emerging attack capabilities (Samtani et al.

Journal of Management Information Systems, 34(4), 1023–1053. p. 0
Where the paper says it

HACKERSIGNAL: A Large-Scale Multi-Source Dataset Linking Hacker Community Discourse to the CVE Vulnerability Lifecycle Benjamin M. Ampel Department of Computer Information Systems J.

https://doi.org/10.1080/07421222.2017.1394049

Samtani, S., Zhu, H., & Chen, H. (2020). Proactively Identifying Emerging Hacker Threats from the Dark Web: A Diachronic Graph Embedding Framework. p. 1

Where the paper says it

The AZSecure Hacker Assets Portal operationalized this as a situational awareness platform (Samtani et al. , 2021), while parallel work modeled threat emergence over time via diachronic graph embeddings (Samtani et al.

ACM Transactions on Privacy and Security, 23(4), Article 21. p. 9
Where the paper says it

Detecting Cyber Threats in Non-English Hacker Forums: An Adversarial Cross-Lingual Knowledge Transfer Approach. IEEE Security and Privacy Workshops, 20–26.

https://doi.org/10.1145/3409289

Samtani, S., Li, W., Benjamin, V. p. 0

Where the paper says it

HACKERSIGNAL: A Large-Scale Multi-Source Dataset Linking Hacker Community Discourse to the CVE Vulnerability Lifecycle Benjamin M. Ampel Department of Computer Information Systems J.

, & Chen, H. (2021). Informing Cyber Threat Intelligence through Dark Web Situational Awareness: The AZSecure Hacker Assets Portal. p. 1
Where the paper says it

The AZSecure Hacker Assets Portal operationalized this as a situational awareness platform (Samtani et al. , 2021), while parallel work modeled threat emergence over time via diachronic graph embeddings (Samtani et al.

Digital Threats: Research and Practice, 2(4), Article 27. p. 0
Where the paper says it

com/BenAmpel/hackersignal (code). 1 Introduction Understanding how cyber threats emerge, propagate, and relate to vulnerabilities is a central challenge in modern applied cybersecurity.

https://doi.org/10.1145/3450972

Zhang, J., Yu, Y., Li, Y., Wang, Y., Yang, Y., Yang, M., & Ratner, A. (2021). WRENCH: A Comprehensive Benchmark for Weak Supervision. p. 2

Where the paper says it

At the same time, WRENCH (Zhang et al. , 2021) demonstrated that weak-supervision evaluation is sensitive to hidden protocol choices, requiring explicit reporting of label sources and aggregation rules.

NeurIPS 2021 D&B Track. arXiv:2109. p. 0
Where the paper says it

A survey of CTI extraction from text Preprint. arXiv:2605.

11377

Rahman, M. R., Mahdavi-Hezaveh, R., & Williams, L. (2023). What Are the Attackers Doing Now? Automating Cyberthreat Intelligence Extraction from Text on Pace with the Changing Threat Landscape: A Survey. p. 1

Where the paper says it

Rahman et al. (2023) survey automated CTI extraction from text, observing that collections often concentrate on the advisory and threat-report tier, while exploit-archive and PoC-focused benchmarks remain scarce.

ACM Computing Surveys, 55(12), Article 241. p. 9
Where the paper says it

Datasheets for Datasets. Communications of the ACM, 64(12), 86–92.

https://doi.org/10.1145/3571726

Hughes, J., Pastrana, S. p. 0

Where the paper says it

Signals that tracks this trajectory is distributed across heterogeneous text-based sources, including: (1) hacker forums where exploits are first discussed, (2) public exploit databases where proof-of-concept code is released, (3) official vulnerability advisories such as the National Vulnerability Database, CISA’s Known Exploited Vulnerabilities, and GitHub Advisory where threats are cataloged, and (4) open-source repositories where fixes are committed. Extant public datasets in this domain often span a single or subset of these layers (Pastrana et al.

, Hutchings, A., Afroz, S., Samtani, S., Li, W., & Marin, E. p. 9
Where the paper says it

naacl-main. , Marin, E.

S. (2024). The Art of Cybercrime Community Research. p. 1
Where the paper says it

identifies advisory records and threat reports as the dominant labeled sources studied, with exploitarchive and PoC-focused benchmarks remaining scarce (Rahman et al. , 2023); a companion survey of cybercrime community research corroborates that structured datasets linking forum posts to formal vulnerability records are largely absent from the public literature (Hughes et al.

ACM Computing Surveys, 56(6), Article 155. p. 9
Where the paper says it

Datasheets for Datasets. Communications of the ACM, 64(12), 86–92.

https://doi.org/10.1145/3639362

A Source Collection Details

Table 7: Per-source collection details

Table 1: Corpus quality diagnostics

Figure 1: HackerSignal pipeline overview

Figure 2: Temporal coverage by source layer (log10 scale)

Figure 3: Pairwise vocabulary overlap across 46 hacker community forums (5% deterministic token sample, hierarchically clustered)

Table 2: Dataset statistics after exact deduplication

Table 3: Benchmark task summary

Table 4: Task 1 (CVE-R) test results (1,990 queries, 340K corpus)

Table 5: Task 2 (ETC) test results on 1,735 temporally held-out samples. Macro-F1 and per-class F1 for the three hardest classes

Table 6: Task 3 (TG) results (1,898 test queries) under strict CVE disjointness (Ctrain ∩ Ctest = ∅).

Figure 4: Radar chart summary of baseline performance across all three tasks

Table 8: Comparison of HackerSignal with prior CTI datasets across key dimensions = yes, = no, ∼ = partial.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Cross-source temporal out-of-distribution entity grounding allows AI systems to retrieve correct CVE entries from exploit archives, advisories, and fix commits based on hacker community discourse queries.

  2. Exploit type classification enables AI systems to predict an 8-class vulnerability type from exploit text using a model ladder that spans bag of words classifiers up to domain-specific transformers like SecBERT.

  3. Temporal generalization enables AI systems to perform prospective CVE disjoint evaluation by testing their ability to generalize to wholly unseen vulnerabilities that postdate the training data, ensuring models do not memorize era-specific features.

  4. The hybrid BM25 plus dense retrieval approach can be used in retrieval tasks to rerank candidates using a weighted combination of lexical and dense cosine scores, potentially improving Recall@k compared to symmetric bi-encoders.

Abstract

Cyber threat intelligence (CTI) systems must link exploit code, advisories, and fix commits to the vulnerabilities they concern, and they must do so for vulnerabilities disclosed after training. HackerSignal is a public corpus and benchmark built for this prospective setting. It aggregates 7.45 million exact-deduplicated documents from 64 public sources across eight source layers (hacker forums, exploit archives, exploit question-and-answer sites, vulnerability references, advisories, bug-bounty disclosures, fix commits, and known-exploited-vulnerability records), spanning 1988--2026. All layers are keyed to a shared CVE identifier space with source provenance and release-mode governance. We define three benchmark tasks with temporal splits: (1) CVE linkage retrieval over 340,536 National Vulnerability Database descriptions, (2) exploit type classification into eight classes, and (3) temporal generalization, where training and test CVE sets do not overlap (C train C test =). Zero-shot dense and hybrid retrievers reach a mean reciprocal rank (MRR) of 0.71 on Task 1, and the top models are statistically tied under paired bootstrap tests. Cybersecurity-pretrained encoders lead exploit type classification (macro-F1 0.90). A two-annotator audit of 240 linkage labels finds 79.5% precision (Cohen's 0.94), with per-source precision from 20% to 100%. The hacker-community layer contributes 6.97 million posts but only about 2,000 CVE-linked records. A masked forum-post grounding pilot over these records reaches MRR 0.08, about nine times lower than the structured-source tasks. Grounding informal hacker discourse to vulnerabilities therefore remains an open problem that the release is designed to support. Code, data, and Croissant metadata are available at https://huggingface.co/datasets/BenAmpel/HackerSignal and https://github.com/BenAmpel/hackersignal.

Sources

Related papers