Chinese Whispers About Vulnerabilities
summary
The gist
The paper demonstrates that the referenced online information sources tend to be mostly dissimilar statistically Given this variety and a dataset of about 382 thousand online references for about 55…
In short
The study analyzed over 382 thousand online references for 55 thousand software vulnerabilities to determine how similar or dissimilar these sources are. The results show that the online information sources are statistically mostly dissimilar, suggesting a knowledge gap in tracking vulnerability traceability.
Key concepts
- Online Information Sources
- These are the various online places where information about software vulnerabilities is discussed, such as bug reports or security advisories. The paper examines how different sources discussing the same vulnerability compare to each other statistically.
- Similarity Metrics (COSSIMm, COSSIMd, JACSIMm, JACSIMd)
- These are mathematical tools used to quantify how alike two or more online references are. They measure the degree of overlap between the information found in different sources about a specific vulnerability.
- Dissimilarity Hypothesis
- The hypothesis suggests that as the number of online sources discussing a vulnerability increases, the information within those sources becomes more dissimilar. This is metaphorically linked to 'Chinese whispers,' implying that messages get distorted when passed through multiple people.
Terminology used across episodes
This episode discusses
- Chinese Whispers About Vulnerabilities · Paper Radio
- CyberThreat-Eval: Can Large Language Models Automate Real-World Threat Research?
- CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning
- Uncovering Vulnerabilities of LLM-Assisted Cyber Threat Intelligence
- Predicting known Vulnerabilities from Attack News: A Transformer-Based Approach
- CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
- Characterizing and Modeling the GitHub Security Advisories Review Pipeline
- "An Endless Stream of AI Slop": How Developers Discuss the Burden of AI-Assisted Software Development
The paper
Chinese Whispers About Vulnerabilities · Read on arXiv
Jukka Ruohonen
University of Southern Denmark
The dissemination of information about known software vulnerabilities occurs online in various ways. Everything from email and development systems to social media are used. Given this variety and a dataset of about 382 thousand online references for about 55 thousand vulnerabilities, the paper demonstrates that the referenced online information sources tend to be mostly dissimilar statistically. The mean cosine similarity between them is only 0.22. Interestingly, the amounts of online references neither increase nor decrease similarity between the online information sources. In contrast, vulnerabilities primarily associated with GitHub and programming language ecosystems tend to have more dissimilar online references than other vulnerabilities. Also vulnerability reports, whether in bug tracking systems or bug bounty platforms, tend to increase dissimilarity. With these empirical results, the paper contributes to the efforts to narrow an enduring knowledge gap related to the online tracking and associated traceability of software vulnerabilities.
DOI: 10.1109/IDSTA70607.2026.11771624
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: I'm Nadia.
Elias: And I'm Elias, here with Priya.
Nadia: We're looking at the paper: "Chinese Whispers About Vulnerabilities".
Elias: Here's the gist of the paper. The paper demonstrates that the referenced online information sources tend to be mostly dissimilar statistically Given this variety and a dataset of about 382 thousand online references for about 55 thousand vulnerabilities, the paper demonstrates that the referenced online information sources tend to be mostly dissimilar statistically This finding contributes to efforts to narrow an enduring knowledge gap related to the online tracking and associated traceability of software vulnerabilities Research Questions The paper poses two primary research questions regarding the similarity or dissimilarity of online information sources about software vulnerabilities The first question is how similar or dissimilar different online information sources about software vulnerabilities are The second question asks what can statistically explain the similarities or dissimilarities and how well Regarding the second research question, the hypothesis is that "the more there are online sources discussing a software vulnerability, the more dissimilar the information in overall" This idea is metaphorically linked to "Chinese whispers," which refers to a popular children’s game where a message is whispered and then compared with an assumption that it has changed while traveling between whispering persons Data Collection and Pre-processing The dataset examined is based on vulnerability information extracted from the distributed Open Source Vulnerabilities (OSV) database The raw data included over 422 thousand unique vulnerability entries delivered in the JavaScript Object Notation (JSON) format Four pre-processing restrictions were imposed upon the vulnerability entries qualified into the sample analyzed These restrictions included: 1.
Nadia: only vulnerabilities with Common Vulnerabilities and Exposures (CVE) identifiers were included in the sample 2.
Elias: the CVE-referenced vulnerability entries were required to have Common Vulnerability Scoring System (CVSS) severity entries directly available from the OSV database 3.
Nadia: only vulnerability entries with at least two references to external sources were included 4.
Paper summary: Nadia: To get into the specifics of "Chinese Whispers About Vulnerabilities", we need to understand their setup. They looked at vulnerability information extracted from the Open Source Vulnerabilities database, which had over four hundred twenty-two thousand unique entries > <ref:2611.15462#pg1>
Elias: The researchers imposed a few restrictions on that big dataset to qualify what they were analyzing. They only included entries with Common Vulnerabilities and Exposures, or CVE, identifiers >
Priya: And they made sure those CVE entries had Common Vulnerability Scoring System, or CVSS, severity entries directly available from the database >
Nadia: They also filtered for vulnerabilities that had at least two references to external sources and only included those for which at least two external online sources could be successfully retrieved >
Elias: That brought the final dataset down to about fifty-five thousand vulnerabilities across those four hundred twenty-two thousand entries > <ref:2611.15462#pg1>
Priya: Once that data was pre-processed, they used four main dependent metrics to measure similarity: mean and median cosine similarities between two or more online references, and mean and median Jaccard similarities between two or more online references >
Nadia: They also used eleven independent metrics to model those four dependent ones, like the number of successfully retrieved online references for each vulnerability >
Elias: And they used censored regression because all those similarity metrics are on that unit interval from zero to one, which means standard regression wouldn't work well because it would be biased >
Priya: The whole point of this section is setting up the empirical analysis before they get to the main findings >
Nadia: So, what we just heard is how they built their sandbox for testing these ideas >
Conclusion: Elias: So, looking at "Chinese Whispers About Vulnerabilities", the authors are essentially saying that online information about software vulnerabilities is mostly dissimilar across the board >
Nadia: That means if you're trying to follow a vulnerability from one forum to another, you can't assume everything you read will be accurate or identical >
Priya: They found that for all four similarity metrics, the scores were below zero point five when they analyzed the data >
Elias: And their regression analysis showed that their prior hypothesis didn't hold up because "the effect of REFCOUNTi is small" when you account for estimation noise >
Nadia: But they did find some specific subsets where things were different, like GitHub and programming language ecosystem vulnerabilities having more dissimilar references than others >
Priya: They also noted that bug reports, bug bounty reports, and other related reports tend to increase dissimilarity in the information sources >
Elias: It sounds like the paper is contributing to a clearer picture of the challenges around online tracking and associated traceability of software vulnerabilities >
Nadia: It’s about showing that the way information gets spread online for these flaws is quite messy, and it helps researchers narrow that knowledge gap >
More episodes
- 2608.27496-ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection
- Daily Summary for 2026-11-19
- 2607.07085-Is Randomness Necessary for Adaptive Data Analysis?
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2609.10601-Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows
- 2610.38786-Adversarial Process Disturbances at a Deployable Alarm Budget: Detector Evaluation on a Simulator and a Real Plant
- 2511.08660-Binary and Multiclass Cyberattack Classification on GeNIS Dataset
- 2604.27302-Static Attribution of Android Residential Proxy Malware Using Graph Kernels
- Daily Summary for 2026-11-18