Chinese Whispers About Vulnerabilities

arXiv:2611.15462 · cs.CR · Submitted 2026-11-11 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: I'm Nadia.

Elias: And I'm Elias, here with Priya.

Nadia: We're looking at the paper: "Chinese Whispers About Vulnerabilities".

Elias: Here's the gist of the paper. The paper demonstrates that the referenced online information sources tend to be mostly dissimilar statistically Given this variety and a dataset of about 382 thousand online references for about 55 thousand vulnerabilities, the paper demonstrates that the referenced online information sources tend to be mostly dissimilar statistically This finding contributes to efforts to narrow an enduring knowledge gap related to the online tracking and associated traceability of software vulnerabilities Research Questions The paper poses two primary research questions regarding the similarity or dissimilarity of online information sources about software vulnerabilities The first question is how similar or dissimilar different online information sources about software vulnerabilities are The second question asks what can statistically explain the similarities or dissimilarities and how well Regarding the second research question, the hypothesis is that "the more there are online sources discussing a software vulnerability, the more dissimilar the information in overall" This idea is metaphorically linked to "Chinese whispers," which refers to a popular children’s game where a message is whispered and then compared with an assumption that it has changed while traveling between whispering persons Data Collection and Pre-processing The dataset examined is based on vulnerability information extracted from the distributed Open Source Vulnerabilities (OSV) database The raw data included over 422 thousand unique vulnerability entries delivered in the JavaScript Object Notation (JSON) format Four pre-processing restrictions were imposed upon the vulnerability entries qualified into the sample analyzed These restrictions included: 1.

Nadia: only vulnerabilities with Common Vulnerabilities and Exposures (CVE) identifiers were included in the sample 2.

Elias: the CVE-referenced vulnerability entries were required to have Common Vulnerability Scoring System (CVSS) severity entries directly available from the OSV database 3.

Nadia: only vulnerability entries with at least two references to external sources were included 4.

Paper summary: Nadia: To get into the specifics of "Chinese Whispers About Vulnerabilities", we need to understand their setup. They looked at vulnerability information extracted from the Open Source Vulnerabilities database, which had over four hundred twenty-two thousand unique entries > <ref:2611.15462#pg1>

Elias: The researchers imposed a few restrictions on that big dataset to qualify what they were analyzing. They only included entries with Common Vulnerabilities and Exposures, or CVE, identifiers >

Priya: And they made sure those CVE entries had Common Vulnerability Scoring System, or CVSS, severity entries directly available from the database >

Nadia: They also filtered for vulnerabilities that had at least two references to external sources and only included those for which at least two external online sources could be successfully retrieved >

Elias: That brought the final dataset down to about fifty-five thousand vulnerabilities across those four hundred twenty-two thousand entries > <ref:2611.15462#pg1>

Priya: Once that data was pre-processed, they used four main dependent metrics to measure similarity: mean and median cosine similarities between two or more online references, and mean and median Jaccard similarities between two or more online references >

Nadia: They also used eleven independent metrics to model those four dependent ones, like the number of successfully retrieved online references for each vulnerability >

Elias: And they used censored regression because all those similarity metrics are on that unit interval from zero to one, which means standard regression wouldn't work well because it would be biased >

Priya: The whole point of this section is setting up the empirical analysis before they get to the main findings >

Nadia: So, what we just heard is how they built their sandbox for testing these ideas >

Conclusion: Elias: So, looking at "Chinese Whispers About Vulnerabilities", the authors are essentially saying that online information about software vulnerabilities is mostly dissimilar across the board >

Nadia: That means if you're trying to follow a vulnerability from one forum to another, you can't assume everything you read will be accurate or identical >

Priya: They found that for all four similarity metrics, the scores were below zero point five when they analyzed the data >

Elias: And their regression analysis showed that their prior hypothesis didn't hold up because "the effect of REFCOUNTi is small" when you account for estimation noise >

Nadia: But they did find some specific subsets where things were different, like GitHub and programming language ecosystem vulnerabilities having more dissimilar references than others >

Priya: They also noted that bug reports, bug bounty reports, and other related reports tend to increase dissimilarity in the information sources >

Elias: It sounds like the paper is contributing to a clearer picture of the challenges around online tracking and associated traceability of software vulnerabilities >

Nadia: It’s about showing that the way information gets spread online for these flaws is quite messy, and it helps researchers narrow that knowledge gap >

Jukka Ruohonen

University of Southern Denmark

cs.CR

Submitted: 2026-11-11

Updated: 2026-11-11

Comments: Proceedings of the Seventh International Conference on Intelligent Data Science Technologies and Applications (IDSTA 2026), Bucharest, IEEE, 2026, pp. 11-17

DOI: 10.1109/IDSTA70607.2026.11771624

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 66/100

The gist: The paper demonstrates that the referenced online information sources tend to be mostly dissimilar statistically Given this variety and a dataset of about 382 thousand online references for about 55…

Key concepts

Online Information Sources
These are the various online places where information about software vulnerabilities is discussed, such as bug reports or security advisories. The paper examines how different sources discussing the same vulnerability compare to each other statistically.
Similarity Metrics (COSSIMm, COSSIMd, JACSIMm, JACSIMd)
These are mathematical tools used to quantify how alike two or more online references are. They measure the degree of overlap between the information found in different sources about a specific vulnerability.
Dissimilarity Hypothesis
The hypothesis suggests that as the number of online sources discussing a vulnerability increases, the information within those sources becomes more dissimilar. This is metaphorically linked to 'Chinese whispers,' implying that messages get distorted when passed through multiple people.

Terminology

Summary

The paper demonstrates that the referenced online information sources tend to be mostly dissimilar statistically Given this variety and a dataset of about 382 thousand online references for about 55 thousand vulnerabilities, the paper demonstrates that the referenced online information sources tend to be mostly dissimilar statistically This finding contributes to efforts to narrow an enduring knowledge gap related to the online tracking and associated traceability of software vulnerabilities

Research Questions

The paper poses two primary research questions regarding the similarity or dissimilarity of online information sources about software vulnerabilities The first question is how similar or dissimilar different online information sources about software vulnerabilities are The second question asks what can statistically explain the similarities or dissimilarities and how well Regarding the second research question, the hypothesis is that the more there are online sources discussing a software vulnerability, the more dissimilar the information in overall This idea is metaphorically linked to Chinese whispers, which refers to a popular children’s game where a message is whispered and then compared with an assumption that it has changed while traveling between whispering persons

Data Collection and Pre-processing

The dataset examined is based on vulnerability information extracted from the distributed Open Source Vulnerabilities (OSV) database The raw data included over 422 thousand unique vulnerability entries delivered in the JavaScript Object Notation (JSON) format Four pre-processing restrictions were imposed upon the vulnerability entries qualified into the sample analyzed These restrictions included:

  1. only vulnerabilities with Common Vulnerabilities and Exposures (CVE) identifiers were included in the sample

  2. the CVE-referenced vulnerability entries were required to have Common Vulnerability Scoring System (CVSS) severity entries directly available from the OSV database

  3. only vulnerability entries with at least two references to external sources were included

  4. only those vulnerabilities were included for which at least two external online sources were successfully retrieved

Methodology and Metrics

The empirical analysis operates with four dependent metrics, including COSSIMm and COSSIMd, mean and median cosine similarities between two or more online references for the i:th CVE-referenced vulnerability Additionally, JACSIMm and JACSIMd are used to measure mean and median Jaccard similarities between two or more online references for the i:th CVE-referenced vulnerability Eleven independent metrics are used to model these four dependent metrics, such as REFCOUNTi, which is the number of successfully retrieved online references for the i:th vulnerability The analysis uses censored regression because all values of the four dependent similarity metrics are on the unit interval [0, 1], which makes conventional ordinary least squares regression estimates biased

Key Findings and Interpretation

The descriptive statistics indicate that the answer to RQ1 is on the side of dissimilarity, as the masses of all four distributions are clearly below 0.5 The regression analysis results show that the prior hypothesis does not hold, as the effect of REFCOUNTi is small when estimation noise is taken into account In contrast, a few of the independent metrics do how visible effects in the similarity subsets Specifically, vulnerabilities primarily associated with GitHub and programming language ecosystems tend to have more dissimilar online information sources than other vulnerabilities Furthermore, bug reports, bug bounty reports, and other related reports tend to increase dissimilarity Finally, the amounts of online references tend to neither increase nor decrease similarity

Limitations and Future Work

Four limitations are mentioned, including the lack of overall traceability with online traceability links due to link rot Generalizability remains a question mark as almost always in this field of research Semantic text similarity metrics could provide further insights and reassure the validity of the results reported The sole reliance on text similarity metrics is another limitation, necessitating human evaluation to deduce whether online information sources are actually dissimilar in a sense of being valuable

Conclusion

The paper concludes that the online information sources are dissimilar in overall for all four similarity metrics, with scores below 0.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improved vulnerability coordination by analyzing online information sources to identify dissimilar references using metrics like COSSIMm and JACSIMm, which are found to be below 0.23 across all vulnerabilities. This allows AI systems to prioritize tracking information from more distinct sources when investigating a specific CVE-referenced vulnerability.

  2. Enhanced cross-ecosystem threat intelligence by using dummy variables such as GITHUBi and PROGECOi in regression models to quantify how vulnerabilities associated with GitHub or specific programming language ecosystems tend to have more dissimilar online references than other vulnerabilities.

  3. Optimized bug bounty report triage by using the EVIDENCE and REPORT metrics to identify reports that increase dissimilarity, which helps distinguish between valuable, diverse information and redundant or similar reporting sources.

Abstract

The dissemination of information about known software vulnerabilities occurs online in various ways. Everything from email and development systems to social media are used. Given this variety and a dataset of about 382 thousand online references for about 55 thousand vulnerabilities, the paper demonstrates that the referenced online information sources tend to be mostly dissimilar statistically. The mean cosine similarity between them is only 0.22. Interestingly, the amounts of online references neither increase nor decrease similarity between the online information sources. In contrast, vulnerabilities primarily associated with GitHub and programming language ecosystems tend to have more dissimilar online references than other vulnerabilities. Also vulnerability reports, whether in bug tracking systems or bug bounty platforms, tend to increase dissimilarity. With these empirical results, the paper contributes to the efforts to narrow an enduring knowledge gap related to the online tracking and associated traceability of software vulnerabilities.

Sources

Related papers