Comparison of OpenAlex and Scopus coverage of German institutions' publications in top-tier journals
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "Comparison of OpenAlex and Scopus coverage of German institutions' publications in top-tier journals".
Jane: Here's the gist of the paper. The study assesses the quality of institutional affiliation data in OpenAlex using German research institutions by systematically comparing its publication counts with those in Scopus to determine its suitability for institutional evaluation contexts.
Tom: Journal-Level Comparison OpenAlex generally contains more publications at the journal level, reflecting its broader coverage.
Jane: The mean raw difference was-365.21 (95% CI
-458.32, -272.11: ), and the mean relative difference was-0.082 (95% CI
-0.091, -0.073: ).
Tom: Discrepancies in counts are illustrated by journals like Cancer Research and Journal of Clinical Oncology, where OpenAlex had a difference of-38 and-49 respectively.
Paper summary: Tom: So we’re looking at this paper today called "Comparison of OpenAlex and Scopus coverage of German institutions' publications in top-tier journals." Basically, they wanted to see how well OpenAlex stacks up against Scopus when it comes to tracking publications from German research institutions in those top-tier journals.
Jane: Right. The main thing they’re testing is the quality of that institutional affiliation data within OpenAlex because that stuff is super crucial if you want to evaluate organizations or track productivity at a high level.
Lu: What they are really investigating is whether OpenAlex actually has the right affiliations, or if a lot of it's missing or wrong compared to what Scopus has recorded.
Meng: From my side, I’m thinking about the practical use here; if we’re trying to deploy this for some kind of performance monitoring, you need to know if the counts are reliable.
Lalam: The paper claims that OpenAlex generally has more publications at the journal level overall, which they say reflects broader coverage across journals >
Tom: But then it gets interesting when you zoom in on specific institutions; they found that while the relationship between those institutional publication counts in both databases is very high, almost identical in ranking, it’s not perfect.
Jane: They mean that even though the rankings are similar, the absolute numbers don't line up cleanly for every single institution. The results show OpenAlex has generally lower institutional publication counts than Scopus does >
Lu: They found the mean raw difference was one hundred ninety-one point eleven and they noted a mean relative difference of just about zero point zero seven, which is pretty close on the surface >
Meng: But that close correlation isn't enough for some applications. The paper points out that if you look at specific journals with similar total publication counts in both databases, nearly all institutions had higher publication counts in Scopus than in OpenAlex >
Lalam: They also found that the relationship between those institutional outputs is very strong, using Spearman’s correlation of zero point nine eight and a Lin’s concordance correlation coefficient of zero point nine five five > <ref:2605.01337#pg1>
Tom: So, the core thesis here isn't just about raw numbers; it’s about how much trust you can put in OpenAlex when you need to make decisions based on those counts for specific entities >
Jane: They are saying that if institutional metadata is incomplete or inaccurate, the results of entire organizations or regions can be misrepresented, which makes understanding the extent of that issue crucial >
Lu: The study specifically aims to address a gap because most prior evaluations treated completeness and accuracy as aggregate properties of the database rather than looking at how specific institutions are affected >
Meng: That’s fair. I mean, if I’m trying to set up a production pipeline, knowing that some of the data points might be unreliable is a major blocker for scaling things up >
Lalam: The paper looked into potential causes for these differences, and they pointed to publishers not sharing affiliation metadata via Crossref as one reason >
Tom: And another key difference they highlighted is how publication years are assigned; OpenAlex usually records the earliest available year, while Scopus assigns the year corresponding to the formal issue publication >
Jane: They also mentioned that OpenAlex indexes certain document types, like book reviews or obituaries, that Scopus doesn't consistently include in its indexing >
Lu: And they noted a limitation they found concerning larger papers, specifically citing a "one hundred-plus author problem" where some papers were affected by OpenAlex filtering limitations >
Meng: That’s something I’ve seen in practice. Dealing with those massive author lists can really break simple filters or lead to missing data points at scale >
Tom: So, they concluded that while OpenAlex is a strong foundation for comparison, it needs improvement in affiliation metadata before it can be used for evaluation contexts that rely on absolute publication counts >
Jane: They suggest that relative comparisons might still yield similar conclusions, but the underrepresentation observed at the institutional level raises concerns for performance-based funding schemes or institutional monitoring where absolute numbers matter >
Lu: The study ultimately concludes that OpenAlex is already a promising starting point, but continued effort on completeness and precision is essential for realizing its full potential as an open alternative >
Meng: So, the practical implication is that if you’re doing serious evaluations based on these counts, you need to supplement your analysis with checks on metadata completeness >
Tom: Exactly. It’s a solid tool for seeing patterns, but if you need absolute precision for institutional accountability, you have to be careful with how much weight you put on the raw numbers from OpenAlex versus Scopus >
Jane: So, the paper shows us that while the structure of the data is there and correlations are strong, the underlying fidelity of those affiliations needs work before we can fully trust it for granular institutional evaluation >
Conclusion: Tom: So we're wrapping up this look at OpenAlex versus Scopus for German research institutions, focusing on their publications in top journals >
Jane: Right, the authors are comparing how much data each database captures about those specific German research groups >
Lu: Basically they’re testing if OpenAlex is a good stand-in for Scopus when you need to track institutional output >
Meng: So what they found is that while the general idea of publication volume is similar, the actual counts for individual institutions differ quite a bit >
Lalam: They measured the raw difference at nineteen one point eleven and the relative difference was about zero point seven >
Tom: That means even when you look at journals with similar total publication counts in both databases, nearly every institution had more publications recorded in Scopus than in OpenAlex >
Jane: So the big question is, if we use OpenAlex for performance monitoring, are we getting a complete picture of what those institutions are actually doing >
Lu: The conclusion they draw is that OpenAlex is suitable for general comparison but requires better affiliation data before it can be used for tracking absolute publication numbers >
Tom: They found that relative comparisons might work, but for things like performance-based funding, where the exact count matters, this missing data in OpenAlex is a problem >
Jane: It suggests that while the structure of the data is promising, we still need to improve how they collect and verify those institutional links >
Lu: They point out several reasons why the counts differ: things like publishers not sharing metadata or different ways the databases assign publication years >
Meng: So it’s not just one database being "wrong," but there are a few technical reasons, like OpenAlex following a different rule set than Scopus >
Tom: It means if you’re building a model on this data, you have to be careful about how much weight you put on the absolute numbers coming from OpenAlex versus what Scopus shows >
Jane: It's a good reminder that even with strong correlations, precision in affiliation metadata is what makes an evaluation truly useful >
Lu: And that leads us into how this affects future work, specifically needing better data alignment to make these institutional comparisons really reliable >
Andrey Lovakov, Ivan Sterligov
German Centre for Higher Education Research and Science Studies (DZHW) · E-Quadrat Science and Education
cs.DL
Submitted: 2026-05-02
Updated: 2026-11-16
Comments: revised and expanded. earlier version was presented at STI 2026 and published in: De Parade, I., Vandewalle, E., & Engels, T. C. E. (2026, August 10). STI-ENID 2026 Conference Proceedings. 30th Annual International Conference on Science and Technology Indicators (STI-ENID 2026), University of Antwerp, Antwerp, Belgium
Project page: https://www.origins-cluster.de
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 77/100
The gist: The study assesses the quality of institutional affiliation data in OpenAlex using German research institutions by systematically comparing its publication counts with those in Scopus to determine…
Key concepts
- Journal-Level Comparison
- This section compares how many publications each database lists for specific journals. OpenAlex generally shows more journal entries than Scopus, suggesting broader coverage. Differences are caused by factors like how each database handles conference papers and publication years.
- Institution-Level Discrepancies
- When looking at institutions, Scopus often reports higher publication counts than OpenAlex for the same journals. However, the relationship between institutional rankings is very strong (Spearman's r = 0.98), meaning the relative ranking of institutions remains nearly identical in both sources.
- Affiliation Metadata Issues
- Several problems exist in how affiliation data is recorded, including publishers not sharing metadata and differences in how publication years are assigned. OpenAlex sometimes includes document types like book reviews that Scopus might exclude, affecting the final count.
Terminology
Summary
The study assesses the quality of institutional affiliation data in OpenAlex using German research institutions by systematically comparing its publication counts with those in Scopus to determine its suitability for institutional evaluation contexts. p. 0 This study assesses the quality of affiliation data in
OpenAlex using German research institutions. Publications from top-tier journals were analyzed
and institutional publication counts in OpenAlex were systematically compared with counts in
Scopus.Where the paper says it
Journal-Level Comparison
OpenAlex generally contains more publications at the journal level, reflecting its broader coverage. The mean raw difference was-365. p. 7 On average,
OpenAlex contained more publications per journal than Scopus. The mean raw difference 21 (95% CI [-458. 11]), and the mean relative difference was -0. 21 (95% CI [-458. 11]), and the mean relative difference was -0. 11]), and the mean relative difference was -0. 082
(95% CI [-0. Journals with the largest differences in number of publications between Scopus
and OpenAlex, 2020–2024, more in OpenAlex
Title Openalex Scopus diff
Cancer Research 40199 2161 -38038
Journal of Clinical Oncology 38136 3185 -34951
Blood 35229 4442 -30787
Journal of the American College of Cardiology 28149 3760 -24389
Circulation 25474 3330 -22144
American Journal of Gastroenterology 23919 2586 -21333
Gastroenterology 22462 3363 -19099
Value in Health 20049 1100 -18949
Annals of Oncology 19949 1042 -18907
Journal of the American Society of Nephrology 19897 1384 -18513
European Heart Journal 22615 4682 -17933
Annals of the Rheumatic Diseases 18239 2086 -16153
Chest 19078 3640 -15438
The FASEB Journal 19399 4180 -15219
Neurology 19819 6208 -13611
Journal of Urology 17085 4852 -12233
Medicine & Science in Sports & Exercise 13440 1452 -11988
British Journal of Surgery 13051 2484 -10567
Neuro-Oncology 11782 1330 -10452
Diabetes 11074 1201 -9873
Journal of the Acoustical Society of America 12066 3588 -8478
Journal of Hepatology 10108 2236 -7872
Journal of Immunology 10204 2465 -7739
Journal of Investigative Dermatology 9750 2044 -7706
European psychiatry 8167 462 -7705
Journal of Thoracic Oncology 8593 1332 -7261
Gastrointestinal Endoscopy 9659 2522 -7137
The Journal of Heart and Lung Transplantation 8276 1208 -7068
Journal of Animal Science 8884 2082 -6802
American Journal of Obstetrics and Gynecology 9447 2858 -6589
Table 2. Journals with the largest differences in number of publications between Scopus
and OpenAlex, 2020–2024, more in Scopus
Title Openalex Scopus diff
Cochrane Database of Systematic Reviews 9 3335 3326
New England Journal of Medicine 5642 7269 1627
Science of the Total Environment 40474 41662 1188
Journal of machine learning research 326 1293 967
Synthese 2323 3149 826
Journal of Cleaner Production 23306 24116 810
Zeitschrift für Papyrologie und Epigraphik 15 802 787
Distributed Computing Systems 0 726 726
International mathematics research notices 1760 2385 625
Catalysis Today 2287 2824 537
ACM Transactions on Information Systems 0 486 486 First, OpenAlex indexes supplementary issues with conferences’
papers, abstracts and posters for some journals that are not covered in Scopus, leading
to higher publication counts. Second, differences in the assignment of publication year
contribute to mismatches within the analyzed time window.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Institution-Level Discrepancies
When focusing on journals with similar total publication counts in both databases, nearly all institutions had higher publication counts in Scopus than in OpenAlex. The mean raw difference was 191. p. 7 On average,
OpenAlex contained more publications per journal than Scopus. The mean raw difference 21 (95% CI [-458. 11]), and the mean relative difference was -0. The
distribution of relative differences was skewed (Figure 3), with several journals showing
particularly large discrepancies. Table 1 lists the 30 journals with the largest absolute differences in publication counts
between the two databases for the years 2020–2024. However, institutional publication counts in OpenAlex are
consistently lower, indicating missing or incorrectly assigned affiliations. Nevertheless, the
correlations between institutional outputs in both databases are very high, suggesting that relative
institutional rankings remain stable. Following Scheidsteger et al. (2025), we calculated two types of correlation coefficients to
assess the similarity between two databases: 1) Spearman’s rank correlation coefficient,
and 2) Lin’s concordance correlation coefficient, which combines measures of precision
and accuracy to determine how far the observed data points deviate from the line of
perfect concordance (the 45-degree line on a square scatter plot). Collectively, this growing body
of work highlights that while OpenAlex performs well across many dimensions, the quality
of specific metadata fields varies substantially. In particular, several studies have noted
that institutional affiliation data is less complete and less accurate (Alonso-Álvarez & van
Eck, 2025; Zhang et al.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Temporal and Field Variations
Discrepancies between Scopus and OpenAlex have grown in the most recent year, with the mean relative difference reaching 0.128 in 2024. p. 0 Recent studies have shown that OpenAlex surpasses many
bibliometric databases in terms of document and journal coverage, enabling more
inclusive and representative bibliometric analyses (Alonso-Álvarez & van Eck, 2025;
Alperin et al. , 2024; Culbert et al. These cases illustrate that the
highest discrepancies are within the medical/health sciences as almost all top journals
with the highest differences are from this area. A closer inspection suggests several
underlying causes. Sample of journals
We start with 2074 journals from the top tier of the 2023 edition of Norwegian list. This list
covers all research fields and includes 603 journals in Natural Sciences and Engineering,
536 in Humanities, 461 in Health Sciences, 444 in Social Sciences.Where the paper says it
Where the paper says it
Where the paper says it
Possible Causes Identified
Several potential causes for the observed differences have been explored. These include:
-
Publishers not sharing affiliation metadata via Crossref, although OpenAlex produces GROBID-extracted metadata from OA PDFs. p. 3
Where the paper says it
The data was extracted via the API in March 2025. To be able to isolate the issues of the data quality related with indexing the journals and the issues of the data quality related with affiliation data for analysis at the level of institutions we selected only those journals which have the same or very similar number of publications (ratio of Scopus/OpenAlex in the range 0.
-
Differences in indexing or assignment of publication years, as OpenAlex generally records the earliest available publication year while Scopus assigns the year corresponding to the formal issue publication. p. 8
Where the paper says it
OpenAlex generally records the earliest available publication year (e. , online first), whereas Scopus assigns the year corresponding to the formal issue publication.
-
Certain document types indexed in OpenAlex but not consistently included in Scopus, such as book reviews and obituaries. p. 8
Where the paper says it
As a result, some articles fall within the 2020–2024 range in one database but outside it in the other. Third, certain document types, such as book reviews, biographies, obituaries, tributes, and interviews are indexed in OpenAlex but are not consistently included in Scopus, further increasing the divergence in counts.
-
Isolated metadata errors in OpenAlex, such as incorrect publication years or sources. p. 8
Where the paper says it
Third, certain document types, such as book reviews, biographies, obituaries, tributes, and interviews are indexed in OpenAlex but are not consistently included in Scopus, further increasing the divergence in counts. Finally, discrepancies are also caused by isolated metadata errors in OpenAlex, such as incorrect publication years or sources.
-
Discrepancies arising from different parent/child/related configurations between the two databases, as Scopus relies on internal rules and organization input while OpenAlex blindly follows ROR. p. 4
Where the paper says it
Both databases instead display lineage counts, combining the counts of papers of a parent ID with those of all its children, but they differ in their inner workings. OpenAlex directly relies on ROR and its hierarchy of parent/child/related organisations and offers a convenient way to retrieve the full paper set of the parent and its children (but not ‘related’) via its authorships.
Conclusion on Suitability
The findings suggest that OpenAlex is suitable for comparative institutional analyses in academic research but requires further improvement in affiliation metadata before it can be used for evaluation contexts that rely on absolute publication counts. While relative comparisons may yield similar conclusions, the underrepresentation observed at the institutional level raises concerns for performance-based funding schemes or institutional monitoring where absolute publication counts matter. p. 0 Nevertheless, the
correlations between institutional outputs in both databases are very high, suggesting that relative
institutional rankings remain stable. These findings suggest that OpenAlex is suitable for
comparative institutional analyses in academic research but requires further improvement in
affiliation metadata before it can be used for evaluation contexts that rely on absolute publication
counts. Consequently, assessing the
completeness and precision of institutional metadata in OpenAlex is essential, especially
when the database is used for evaluation or policy purposes. In the case of OpenAlex,
correct document-institution links are vital at the country level also, as the countries are
assigned to documents via resolved institutions. The findings will help clarify whether OpenAlex can already serve as a robust foundation
for institutional-level research evaluations, or whether further improvement of its metadata
is needed before it can replace or complement commercial databases. Data and methods
2.Where the paper says it
Where the paper says it
Where the paper says it
Supplementary Data Insights
Manual checks on specific sources revealed issues such as the To ensure better data quality,
instead of relying on ISSNs we used internal source IDs for Scopus and OpenAlex that
were matched with the journals. An important limitation of OpenAlex - both API and web UI - concerns the handling of
papers with >100 authors: according to the docs, “if you filter works using an author ID or
ROR, you will not receive works where that author is listed further than 100 places down
on the list of authors. First, it ensures that the
analyzed publications are among the most visible and influential outputs of German
research institutions. Second, it serves as a proxy of how missing or incorrect affiliations
could affect evaluation outcomes, since these journals tend to carry more weight in
performance-based assessments. However, the institution-level precision of affiliation data in OpenAlex remains largely
unexplored. Most prior evaluations have treated completeness and accuracy as
aggregate properties of the database, rather than examining how specific institutions are
affected.100-plus author problem,
where papers with many authors can be affected by limitations in OpenAlex filtering, and discrepancies arising from how ROR structures parent/child relationships versus Scopus's internal rules. For instance, for Leipzig University and Nature Communications, 30 of the 91 missing papers had 100+ authors, which is clearly due to the aforementioned OpenAlex limitation. p. 6Where the paper says it
Where the paper says it
Where the paper says it
Final Assessment of Data Quality
The study highlights that OpenAlex performs well in terms of coverage but its current affiliation data appear less precise than those in Scopus when evaluated at the level of individual institutions. Analyses relying on absolute publication numbers or fine-grained attribution may be more sensitive to the incompleteness of affiliation data in OpenAlex. p. 0 Nevertheless, the
correlations between institutional outputs in both databases are very high, suggesting that relative
institutional rankings remain stable. These findings suggest that OpenAlex is suitable for
comparative institutional analyses in academic research but requires further improvement in
affiliation metadata before it can be used for evaluation contexts that rely on absolute publication
counts. Inaccurate or incomplete affiliations can distort the representation of institutional
productivity and any metrics based on publications. Consequently, assessing the
completeness and precision of institutional metadata in OpenAlex is essential, especially
when the database is used for evaluation or policy purposes. Based on expert review, it includes journals considered to be of high academic quality,
offering a balanced representation across disciplines and avoiding overreliance on
citation-based metrics such as impact factors. The list is divided into several levels, with
the paper published in the top-tier awarding a university three times more points than a
paper in the second tier venue.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Implement a cross-database metadata reconciliation engine that uses ROR hierarchy data from OpenAlex to programmatically resolve parent/child relationships before calculating institution-level publication counts, addressing
differences in parent/child hierarchies
as a cause of discrepancies. -
Develop an automated flag system for papers with more than 100 authors or those lacking any institution profile ID, specifically targeting the
100-plus author problem
and missing raw affiliations, to mitigate their impact on institutional publication counts. -
Create a dynamic discrepancy monitoring tool that tracks the yearly percentage of OpenAlex works not linked to any institution profile ID, allowing users to identify when affiliation quality deteriorates over time in specific journals or for specific institutions.
-
Implement a field-level analysis module that automatically filters research fields based on journal counts and institutional publication density, ensuring robust comparisons across disciplines as suggested by the paper's findings regarding
clear variation in the level of discrepancy across different fields of research.
-
Integrate publisher metadata parsing capabilities to extract affiliation information from open access PDFs, aiming to close gaps where publishers
do not share affiliation metadata via Crossref,
thereby improving OpenAlex's completeness.
Abstract
OpenAlex has recently emerged as a leading alternative to proprietary bibliometric sources. However, concerns remain regarding the quality of its metadata, especially the institutional profiles which are crucial for evaluating organizations. This study assesses the quality of affiliation data in OpenAlex using German research institutions. Publications from top-tier journals were analyzed and institutional publication counts in OpenAlex were systematically compared with counts in Scopus. The results show that OpenAlex generally contains more publications at the journal level, reflecting its broader coverage. However, institutional publication counts in OpenAlex are consistently lower; it contained 6.8% fewer institution-paper links in the analyzed sample. Aggregate institutional counts nevertheless showed very high rank correlation and concordance with Scopus. Discrepancies varied substantially across fields, publishers and journals. Detailed investigation identified missing affiliation information, differences in institutional boundaries, and divergent year and journal assignments, while errors present in both databases. These findings suggest that OpenAlex is suitable for comparative institutional analyses in academic research but requires further improvement in affiliation metadata before it can be used for evaluation contexts that rely on absolute publication counts.
Sources
- An analysis of the suitability of OpenAlex for bibliometric analyses
- OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts
- The open access coverage of OpenAlex, Scopus and Web of Science
Related papers
- Persistence Paradox in Dynamic Science: Evidence from the Deep Learning Revolution
- How Deep Do Large Language Models Internalize Scientific Literature and Citation Practices?
- Have Large Language Models Enhanced the Way Civil & Environmental Engineers Write? A Quantitative Analysis of Scholarly Communication over 25 Years
- MTRACE: Multilingual Retrieval-Augmented Generation for Temporally Diverse Text Corpora
- Ending the NIH embargo accelerated public access to funded research, but substantial delays remain
- Entering the Elite, But Remaining Outside Its Core? The Temporal Structure of Gender Inequality in the Global Citation