Comparison of OpenAlex and Scopus coverage of German institutions' publications in top-tier journals

arXiv:2605.01337 · cs.DL · Submitted 2026-05-02 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Comparison of OpenAlex and Scopus coverage of German institutions' publications in top-tier journals".

Jane: Here's the gist of the paper. The study assesses the quality of institutional affiliation data in OpenAlex using German research institutions by systematically comparing its publication counts with those in Scopus to determine its suitability for institutional evaluation contexts.

Tom: Journal-Level Comparison OpenAlex generally contains more publications at the journal level, reflecting its broader coverage.

Jane: The mean raw difference was-365.21 (95% CI

-458.32, -272.11: ), and the mean relative difference was-0.082 (95% CI

-0.091, -0.073: ).

Tom: Discrepancies in counts are illustrated by journals like Cancer Research and Journal of Clinical Oncology, where OpenAlex had a difference of-38 and-49 respectively.

Paper summary: Tom: So we’re looking at this paper today called "Comparison of OpenAlex and Scopus coverage of German institutions' publications in top-tier journals." Basically, they wanted to see how well OpenAlex stacks up against Scopus when it comes to tracking publications from German research institutions in those top-tier journals.

Jane: Right. The main thing they’re testing is the quality of that institutional affiliation data within OpenAlex because that stuff is super crucial if you want to evaluate organizations or track productivity at a high level.

Lu: What they are really investigating is whether OpenAlex actually has the right affiliations, or if a lot of it's missing or wrong compared to what Scopus has recorded.

Meng: From my side, I’m thinking about the practical use here; if we’re trying to deploy this for some kind of performance monitoring, you need to know if the counts are reliable.

Lalam: The paper claims that OpenAlex generally has more publications at the journal level overall, which they say reflects broader coverage across journals >

Tom: But then it gets interesting when you zoom in on specific institutions; they found that while the relationship between those institutional publication counts in both databases is very high, almost identical in ranking, it’s not perfect.

Jane: They mean that even though the rankings are similar, the absolute numbers don't line up cleanly for every single institution. The results show OpenAlex has generally lower institutional publication counts than Scopus does >

Lu: They found the mean raw difference was one hundred ninety-one point eleven and they noted a mean relative difference of just about zero point zero seven, which is pretty close on the surface >

Meng: But that close correlation isn't enough for some applications. The paper points out that if you look at specific journals with similar total publication counts in both databases, nearly all institutions had higher publication counts in Scopus than in OpenAlex >

Lalam: They also found that the relationship between those institutional outputs is very strong, using Spearman’s correlation of zero point nine eight and a Lin’s concordance correlation coefficient of zero point nine five five > <ref:2605.01337#pg1>

Tom: So, the core thesis here isn't just about raw numbers; it’s about how much trust you can put in OpenAlex when you need to make decisions based on those counts for specific entities >

Jane: They are saying that if institutional metadata is incomplete or inaccurate, the results of entire organizations or regions can be misrepresented, which makes understanding the extent of that issue crucial >

Lu: The study specifically aims to address a gap because most prior evaluations treated completeness and accuracy as aggregate properties of the database rather than looking at how specific institutions are affected >

Meng: That’s fair. I mean, if I’m trying to set up a production pipeline, knowing that some of the data points might be unreliable is a major blocker for scaling things up >

Lalam: The paper looked into potential causes for these differences, and they pointed to publishers not sharing affiliation metadata via Crossref as one reason >

Tom: And another key difference they highlighted is how publication years are assigned; OpenAlex usually records the earliest available year, while Scopus assigns the year corresponding to the formal issue publication >

Jane: They also mentioned that OpenAlex indexes certain document types, like book reviews or obituaries, that Scopus doesn't consistently include in its indexing >

Lu: And they noted a limitation they found concerning larger papers, specifically citing a "one hundred-plus author problem" where some papers were affected by OpenAlex filtering limitations >

Meng: That’s something I’ve seen in practice. Dealing with those massive author lists can really break simple filters or lead to missing data points at scale >

Tom: So, they concluded that while OpenAlex is a strong foundation for comparison, it needs improvement in affiliation metadata before it can be used for evaluation contexts that rely on absolute publication counts >

Jane: They suggest that relative comparisons might still yield similar conclusions, but the underrepresentation observed at the institutional level raises concerns for performance-based funding schemes or institutional monitoring where absolute numbers matter >

Lu: The study ultimately concludes that OpenAlex is already a promising starting point, but continued effort on completeness and precision is essential for realizing its full potential as an open alternative >

Meng: So, the practical implication is that if you’re doing serious evaluations based on these counts, you need to supplement your analysis with checks on metadata completeness >

Tom: Exactly. It’s a solid tool for seeing patterns, but if you need absolute precision for institutional accountability, you have to be careful with how much weight you put on the raw numbers from OpenAlex versus Scopus >

Jane: So, the paper shows us that while the structure of the data is there and correlations are strong, the underlying fidelity of those affiliations needs work before we can fully trust it for granular institutional evaluation >

Conclusion: Tom: So we're wrapping up this look at OpenAlex versus Scopus for German research institutions, focusing on their publications in top journals >

Jane: Right, the authors are comparing how much data each database captures about those specific German research groups >

Lu: Basically they’re testing if OpenAlex is a good stand-in for Scopus when you need to track institutional output >

Meng: So what they found is that while the general idea of publication volume is similar, the actual counts for individual institutions differ quite a bit >

Lalam: They measured the raw difference at nineteen one point eleven and the relative difference was about zero point seven >

Tom: That means even when you look at journals with similar total publication counts in both databases, nearly every institution had more publications recorded in Scopus than in OpenAlex >

Jane: So the big question is, if we use OpenAlex for performance monitoring, are we getting a complete picture of what those institutions are actually doing >

Lu: The conclusion they draw is that OpenAlex is suitable for general comparison but requires better affiliation data before it can be used for tracking absolute publication numbers >

Tom: They found that relative comparisons might work, but for things like performance-based funding, where the exact count matters, this missing data in OpenAlex is a problem >

Jane: It suggests that while the structure of the data is promising, we still need to improve how they collect and verify those institutional links >

Lu: They point out several reasons why the counts differ: things like publishers not sharing metadata or different ways the databases assign publication years >

Meng: So it’s not just one database being "wrong," but there are a few technical reasons, like OpenAlex following a different rule set than Scopus >

Tom: It means if you’re building a model on this data, you have to be careful about how much weight you put on the absolute numbers coming from OpenAlex versus what Scopus shows >

Jane: It's a good reminder that even with strong correlations, precision in affiliation metadata is what makes an evaluation truly useful >

Lu: And that leads us into how this affects future work, specifically needing better data alignment to make these institutional comparisons really reliable >

Andrey Lovakov, Ivan Sterligov

German Centre for Higher Education Research and Science Studies (DZHW) · E-Quadrat Science and Education

cs.DL

Submitted: 2026-05-02

Updated: 2026-11-16

Comments: revised and expanded. earlier version was presented at STI 2026 and published in: De Parade, I., Vandewalle, E., & Engels, T. C. E. (2026, August 10). STI-ENID 2026 Conference Proceedings. 30th Annual International Conference on Science and Technology Indicators (STI-ENID 2026), University of Antwerp, Antwerp, Belgium

Project page: https://www.origins-cluster.de

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 77/100

The gist: The study assesses the quality of institutional affiliation data in OpenAlex using German research institutions by systematically comparing its publication counts with those in Scopus to determine…

Key concepts

Journal-Level Comparison
This section compares how many publications each database lists for specific journals. OpenAlex generally shows more journal entries than Scopus, suggesting broader coverage. Differences are caused by factors like how each database handles conference papers and publication years.
Institution-Level Discrepancies
When looking at institutions, Scopus often reports higher publication counts than OpenAlex for the same journals. However, the relationship between institutional rankings is very strong (Spearman's r = 0.98), meaning the relative ranking of institutions remains nearly identical in both sources.
Affiliation Metadata Issues
Several problems exist in how affiliation data is recorded, including publishers not sharing metadata and differences in how publication years are assigned. OpenAlex sometimes includes document types like book reviews that Scopus might exclude, affecting the final count.

Terminology

Summary

The study assesses the quality of institutional affiliation data in OpenAlex using German research institutions by systematically comparing its publication counts with those in Scopus to determine its suitability for institutional evaluation contexts. p. 0

Where the paper says it

This study assesses the quality of affiliation data in OpenAlex using German research institutions. Publications from top-tier journals were analyzed and institutional publication counts in OpenAlex were systematically compared with counts in Scopus.

Journal-Level Comparison

OpenAlex generally contains more publications at the journal level, reflecting its broader coverage. The mean raw difference was-365. p. 7

Where the paper says it

On average, OpenAlex contained more publications per journal than Scopus. The mean raw difference

21 (95% CI [-458. p. 8
Where the paper says it

21 (95% CI [-458. 11]), and the mean relative difference was -0.

32, -272.11]), and the mean relative difference was-0. p. 8
Where the paper says it

21 (95% CI [-458. 11]), and the mean relative difference was -0.

082 (95% CI [-0. p. 8
Where the paper says it

11]), and the mean relative difference was -0. 082 (95% CI [-0.

091, -0.073]). Discrepancies in counts are illustrated by journals like Cancer Research and Journal of Clinical Oncology, where OpenAlex had a difference of-38 and-49 respectively. p. 9
Where the paper says it

Journals with the largest differences in number of publications between Scopus and OpenAlex, 2020–2024, more in OpenAlex Title Openalex Scopus diff Cancer Research 40199 2161 -38038 Journal of Clinical Oncology 38136 3185 -34951 Blood 35229 4442 -30787 Journal of the American College of Cardiology 28149 3760 -24389 Circulation 25474 3330 -22144 American Journal of Gastroenterology 23919 2586 -21333 Gastroenterology 22462 3363 -19099 Value in Health 20049 1100 -18949 Annals of Oncology 19949 1042 -18907 Journal of the American Society of Nephrology 19897 1384 -18513 European Heart Journal 22615 4682 -17933 Annals of the Rheumatic Diseases 18239 2086 -16153 Chest 19078 3640 -15438 The FASEB Journal 19399 4180 -15219 Neurology 19819 6208 -13611 Journal of Urology 17085 4852 -12233 Medicine & Science in Sports & Exercise 13440 1452 -11988 British Journal of Surgery 13051 2484 -10567 Neuro-Oncology 11782 1330 -10452 Diabetes 11074 1201 -9873 Journal of the Acoustical Society of America 12066 3588 -8478 Journal of Hepatology 10108 2236 -7872 Journal of Immunology 10204 2465 -7739 Journal of Investigative Dermatology 9750 2044 -7706 European psychiatry 8167 462 -7705 Journal of Thoracic Oncology 8593 1332 -7261 Gastrointestinal Endoscopy 9659 2522 -7137 The Journal of Heart and Lung Transplantation 8276 1208 -7068 Journal of Animal Science 8884 2082 -6802 American Journal of Obstetrics and Gynecology 9447 2858 -6589 Table 2. Journals with the largest differences in number of publications between Scopus and OpenAlex, 2020–2024, more in Scopus Title Openalex Scopus diff Cochrane Database of Systematic Reviews 9 3335 3326 New England Journal of Medicine 5642 7269 1627 Science of the Total Environment 40474 41662 1188 Journal of machine learning research 326 1293 967 Synthese 2323 3149 826 Journal of Cleaner Production 23306 24116 810 Zeitschrift für Papyrologie und Epigraphik 15 802 787 Distributed Computing Systems 0 726 726 International mathematics research notices 1760 2385 625 Catalysis Today 2287 2824 537 ACM Transactions on Information Systems 0 486 486

These differences can be attributed to several factors, including OpenAlex indexing supplementary issues with conferences’ papers, differences in publication year assignment between the two databases, and the inclusion of document types not consistently present in Scopus. p. 8
Where the paper says it

First, OpenAlex indexes supplementary issues with conferences’ papers, abstracts and posters for some journals that are not covered in Scopus, leading to higher publication counts. Second, differences in the assignment of publication year contribute to mismatches within the analyzed time window.

Institution-Level Discrepancies

When focusing on journals with similar total publication counts in both databases, nearly all institutions had higher publication counts in Scopus than in OpenAlex. The mean raw difference was 191. p. 7

Where the paper says it

On average, OpenAlex contained more publications per journal than Scopus. The mean raw difference

11 (95% CI [143.57, 238.66]), and the mean relative difference was 0. p. 8
Where the paper says it

21 (95% CI [-458. 11]), and the mean relative difference was -0.

07 (95% CI [0.052, 0.088]). Despite these absolute differences, the relationship between institutional publication counts in the two databases was very strong (Spearman’s r = 0. p. 8
Where the paper says it

The distribution of relative differences was skewed (Figure 3), with several journals showing particularly large discrepancies. Table 1 lists the 30 journals with the largest absolute differences in publication counts between the two databases for the years 2020–2024.

98), showing that institutional rankings by publication volume are nearly identical in both sources. p. 0
Where the paper says it

However, institutional publication counts in OpenAlex are consistently lower, indicating missing or incorrectly assigned affiliations. Nevertheless, the correlations between institutional outputs in both databases are very high, suggesting that relative institutional rankings remain stable.

Lin’s concordance correlation coefficient is 0. p. 7
Where the paper says it

Following Scheidsteger et al. (2025), we calculated two types of correlation coefficients to assess the similarity between two databases: 1) Spearman’s rank correlation coefficient, and 2) Lin’s concordance correlation coefficient, which combines measures of precision and accuracy to determine how far the observed data points deviate from the line of perfect concordance (the 45-degree line on a square scatter plot).

955 (95% CI [0.946, 0.962], which could be interpreted as almost complete agreement. p. 1
Where the paper says it

Collectively, this growing body of work highlights that while OpenAlex performs well across many dimensions, the quality of specific metadata fields varies substantially. In particular, several studies have noted that institutional affiliation data is less complete and less accurate (Alonso-Álvarez & van Eck, 2025; Zhang et al.

Temporal and Field Variations

Discrepancies between Scopus and OpenAlex have grown in the most recent year, with the mean relative difference reaching 0.128 in 2024. p. 0

Where the paper says it

Recent studies have shown that OpenAlex surpasses many bibliometric databases in terms of document and journal coverage, enabling more inclusive and representative bibliometric analyses (Alonso-Álvarez & van Eck, 2025; Alperin et al. , 2024; Culbert et al.

This suggests that discrepancies in institutional affiliation data are dynamic. p. 8
Where the paper says it

These cases illustrate that the highest discrepancies are within the medical/health sciences as almost all top journals with the highest differences are from this area. A closer inspection suggests several underlying causes.

Field-level analyses show a clear variation in the level of discrepancy across different fields of research, with the greatest discrepancies occurring in fields spanning multiple domains, including Health Sciences, Natural Sciences and Engineering, and Social Sciences. p. 3
Where the paper says it

Sample of journals We start with 2074 journals from the top tier of the 2023 edition of Norwegian list. This list covers all research fields and includes 603 journals in Natural Sciences and Engineering, 536 in Humanities, 461 in Health Sciences, 444 in Social Sciences.

Possible Causes Identified

Several potential causes for the observed differences have been explored. These include:

  1. Publishers not sharing affiliation metadata via Crossref, although OpenAlex produces GROBID-extracted metadata from OA PDFs. p. 3

    Where the paper says it

    The data was extracted via the API in March 2025. To be able to isolate the issues of the data quality related with indexing the journals and the issues of the data quality related with affiliation data for analysis at the level of institutions we selected only those journals which have the same or very similar number of publications (ratio of Scopus/OpenAlex in the range 0.

  2. Differences in indexing or assignment of publication years, as OpenAlex generally records the earliest available publication year while Scopus assigns the year corresponding to the formal issue publication. p. 8

    Where the paper says it

    OpenAlex generally records the earliest available publication year (e. , online first), whereas Scopus assigns the year corresponding to the formal issue publication.

  3. Certain document types indexed in OpenAlex but not consistently included in Scopus, such as book reviews and obituaries. p. 8

    Where the paper says it

    As a result, some articles fall within the 2020–2024 range in one database but outside it in the other. Third, certain document types, such as book reviews, biographies, obituaries, tributes, and interviews are indexed in OpenAlex but are not consistently included in Scopus, further increasing the divergence in counts.

  4. Isolated metadata errors in OpenAlex, such as incorrect publication years or sources. p. 8

    Where the paper says it

    Third, certain document types, such as book reviews, biographies, obituaries, tributes, and interviews are indexed in OpenAlex but are not consistently included in Scopus, further increasing the divergence in counts. Finally, discrepancies are also caused by isolated metadata errors in OpenAlex, such as incorrect publication years or sources.

  5. Discrepancies arising from different parent/child/related configurations between the two databases, as Scopus relies on internal rules and organization input while OpenAlex blindly follows ROR. p. 4

    Where the paper says it

    Both databases instead display lineage counts, combining the counts of papers of a parent ID with those of all its children, but they differ in their inner workings. OpenAlex directly relies on ROR and its hierarchy of parent/child/related organisations and offers a convenient way to retrieve the full paper set of the parent and its children (but not ‘related’) via its authorships.

Conclusion on Suitability

The findings suggest that OpenAlex is suitable for comparative institutional analyses in academic research but requires further improvement in affiliation metadata before it can be used for evaluation contexts that rely on absolute publication counts. While relative comparisons may yield similar conclusions, the underrepresentation observed at the institutional level raises concerns for performance-based funding schemes or institutional monitoring where absolute publication counts matter. p. 0

Where the paper says it

Nevertheless, the correlations between institutional outputs in both databases are very high, suggesting that relative institutional rankings remain stable. These findings suggest that OpenAlex is suitable for comparative institutional analyses in academic research but requires further improvement in affiliation metadata before it can be used for evaluation contexts that rely on absolute publication counts.

It is vital to supplement analysis with checks on metadata completeness and exploring ROR children/relationships data to align with analysis aims. p. 1
Where the paper says it

Consequently, assessing the completeness and precision of institutional metadata in OpenAlex is essential, especially when the database is used for evaluation or policy purposes. In the case of OpenAlex, correct document-institution links are vital at the country level also, as the countries are assigned to documents via resolved institutions.

The study concludes that OpenAlex is already a strong and promising foundation but lags behind Scopus in terms of complete coverage for applied/administrative workflows. p. 3
Where the paper says it

The findings will help clarify whether OpenAlex can already serve as a robust foundation for institutional-level research evaluations, or whether further improvement of its metadata is needed before it can replace or complement commercial databases. Data and methods 2.

Supplementary Data Insights

Manual checks on specific sources revealed issues such as the 100-plus author problem, where papers with many authors can be affected by limitations in OpenAlex filtering, and discrepancies arising from how ROR structures parent/child relationships versus Scopus's internal rules. For instance, for Leipzig University and Nature Communications, 30 of the 91 missing papers had 100+ authors, which is clearly due to the aforementioned OpenAlex limitation. p. 6

Where the paper says it

To ensure better data quality, instead of relying on ISSNs we used internal source IDs for Scopus and OpenAlex that were matched with the journals. An important limitation of OpenAlex - both API and web UI - concerns the handling of papers with >100 authors: according to the docs, “if you filter works using an author ID or ROR, you will not receive works where that author is listed further than 100 places down on the list of authors.

Furthermore, some papers were missing raw affiliations entirely or had incorrect raw affiliations despite clear metadata. p. 2
Where the paper says it

First, it ensures that the analyzed publications are among the most visible and influential outputs of German research institutions. Second, it serves as a proxy of how missing or incorrect affiliations could affect evaluation outcomes, since these journals tend to carry more weight in performance-based assessments.

These issues highlight the distinction between completeness and precision in affiliation data. p. 2
Where the paper says it

However, the institution-level precision of affiliation data in OpenAlex remains largely unexplored. Most prior evaluations have treated completeness and accuracy as aggregate properties of the database, rather than examining how specific institutions are affected.

Final Assessment of Data Quality

The study highlights that OpenAlex performs well in terms of coverage but its current affiliation data appear less precise than those in Scopus when evaluated at the level of individual institutions. Analyses relying on absolute publication numbers or fine-grained attribution may be more sensitive to the incompleteness of affiliation data in OpenAlex. p. 0

Where the paper says it

Nevertheless, the correlations between institutional outputs in both databases are very high, suggesting that relative institutional rankings remain stable. These findings suggest that OpenAlex is suitable for comparative institutional analyses in academic research but requires further improvement in affiliation metadata before it can be used for evaluation contexts that rely on absolute publication counts.

Continued efforts to improve completeness and precision of institutional affiliations will be essential for realizing OpenAlex’s full potential as an open alternative for research evaluation. p. 1
Where the paper says it

Inaccurate or incomplete affiliations can distort the representation of institutional productivity and any metrics based on publications. Consequently, assessing the completeness and precision of institutional metadata in OpenAlex is essential, especially when the database is used for evaluation or policy purposes.

The paper is 2605. p. 2
Where the paper says it

Based on expert review, it includes journals considered to be of high academic quality, offering a balanced representation across disciplines and avoiding overreliance on citation-based metrics such as impact factors. The list is divided into several levels, with the paper published in the top-tier awarding a university three times more points than a paper in the second tier venue.

01337.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Implement a cross-database metadata reconciliation engine that uses ROR hierarchy data from OpenAlex to programmatically resolve parent/child relationships before calculating institution-level publication counts, addressing differences in parent/child hierarchies as a cause of discrepancies.

  2. Develop an automated flag system for papers with more than 100 authors or those lacking any institution profile ID, specifically targeting the 100-plus author problem and missing raw affiliations, to mitigate their impact on institutional publication counts.

  3. Create a dynamic discrepancy monitoring tool that tracks the yearly percentage of OpenAlex works not linked to any institution profile ID, allowing users to identify when affiliation quality deteriorates over time in specific journals or for specific institutions.

  4. Implement a field-level analysis module that automatically filters research fields based on journal counts and institutional publication density, ensuring robust comparisons across disciplines as suggested by the paper's findings regarding clear variation in the level of discrepancy across different fields of research.

  5. Integrate publisher metadata parsing capabilities to extract affiliation information from open access PDFs, aiming to close gaps where publishers do not share affiliation metadata via Crossref, thereby improving OpenAlex's completeness.

Abstract

OpenAlex has recently emerged as a leading alternative to proprietary bibliometric sources. However, concerns remain regarding the quality of its metadata, especially the institutional profiles which are crucial for evaluating organizations. This study assesses the quality of affiliation data in OpenAlex using German research institutions. Publications from top-tier journals were analyzed and institutional publication counts in OpenAlex were systematically compared with counts in Scopus. The results show that OpenAlex generally contains more publications at the journal level, reflecting its broader coverage. However, institutional publication counts in OpenAlex are consistently lower; it contained 6.8% fewer institution-paper links in the analyzed sample. Aggregate institutional counts nevertheless showed very high rank correlation and concordance with Scopus. Discrepancies varied substantially across fields, publishers and journals. Detailed investigation identified missing affiliation information, differences in institutional boundaries, and divergent year and journal assignments, while errors present in both databases. These findings suggest that OpenAlex is suitable for comparative institutional analyses in academic research but requires further improvement in affiliation metadata before it can be used for evaluation contexts that rely on absolute publication counts.

Sources

Related papers