Comparison of OpenAlex and Scopus coverage of German institutions' publications in top-tier journals

summary

The gist

The study assesses the quality of institutional affiliation data in OpenAlex using German research institutions by systematically comparing its publication counts with those in Scopus to determine…

In short

The study compared OpenAlex and Scopus data for German research institutions to check if OpenAlex is suitable for institutional evaluation. Findings show discrepancies, particularly at the institution level, where Scopus often has higher counts. While institutional rankings are similar, OpenAlex's affiliation data needs improvement in completeness and precision before it can reliably support absolute count-based evaluations.

Key concepts

Journal-Level Comparison
This section compares how many publications each database lists for specific journals. OpenAlex generally shows more journal entries than Scopus, suggesting broader coverage. Differences are caused by factors like how each database handles conference papers and publication years.
Institution-Level Discrepancies
When looking at institutions, Scopus often reports higher publication counts than OpenAlex for the same journals. However, the relationship between institutional rankings is very strong (Spearman's r = 0.98), meaning the relative ranking of institutions remains nearly identical in both sources.
Affiliation Metadata Issues
Several problems exist in how affiliation data is recorded, including publishers not sharing metadata and differences in how publication years are assigned. OpenAlex sometimes includes document types like book reviews that Scopus might exclude, affecting the final count.

Terminology used across episodes

This episode discusses

The paper

Comparison of OpenAlex and Scopus coverage of German institutions' publications in top-tier journals · Read on arXiv

Andrey Lovakov, Ivan Sterligov

German Centre for Higher Education Research and Science Studies (DZHW) · E-Quadrat Science and Education

OpenAlex has recently emerged as a leading alternative to proprietary bibliometric sources. However, concerns remain regarding the quality of its metadata, especially the institutional profiles which are crucial for evaluating organizations. This study assesses the quality of affiliation data in OpenAlex using German research institutions. Publications from top-tier journals were analyzed and institutional publication counts in OpenAlex were systematically compared with counts in Scopus. The results show that OpenAlex generally contains more publications at the journal level, reflecting its broader coverage. However, institutional publication counts in OpenAlex are consistently lower; it contained 6.8% fewer institution-paper links in the analyzed sample. Aggregate institutional counts nevertheless showed very high rank correlation and concordance with Scopus. Discrepancies varied substantially across fields, publishers and journals. Detailed investigation identified missing affiliation information, differences in institutional boundaries, and divergent year and journal assignments, while errors present in both databases. These findings suggest that OpenAlex is suitable for comparative institutional analyses in academic research but requires further improvement in affiliation metadata before it can be used for evaluation contexts that rely on absolute publication counts.

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Comparison of OpenAlex and Scopus coverage of German institutions' publications in top-tier journals".

Jane: Here's the gist of the paper. The study assesses the quality of institutional affiliation data in OpenAlex using German research institutions by systematically comparing its publication counts with those in Scopus to determine its suitability for institutional evaluation contexts.

Tom: Journal-Level Comparison OpenAlex generally contains more publications at the journal level, reflecting its broader coverage.

Jane: The mean raw difference was-365.21 (95% CI

-458.32, -272.11: ), and the mean relative difference was-0.082 (95% CI

-0.091, -0.073: ).

Tom: Discrepancies in counts are illustrated by journals like Cancer Research and Journal of Clinical Oncology, where OpenAlex had a difference of-38 and-49 respectively.

Paper summary: Tom: So we’re looking at this paper today called "Comparison of OpenAlex and Scopus coverage of German institutions' publications in top-tier journals." Basically, they wanted to see how well OpenAlex stacks up against Scopus when it comes to tracking publications from German research institutions in those top-tier journals.

Jane: Right. The main thing they’re testing is the quality of that institutional affiliation data within OpenAlex because that stuff is super crucial if you want to evaluate organizations or track productivity at a high level.

Lu: What they are really investigating is whether OpenAlex actually has the right affiliations, or if a lot of it's missing or wrong compared to what Scopus has recorded.

Meng: From my side, I’m thinking about the practical use here; if we’re trying to deploy this for some kind of performance monitoring, you need to know if the counts are reliable.

Lalam: The paper claims that OpenAlex generally has more publications at the journal level overall, which they say reflects broader coverage across journals >

Tom: But then it gets interesting when you zoom in on specific institutions; they found that while the relationship between those institutional publication counts in both databases is very high, almost identical in ranking, it’s not perfect.

Jane: They mean that even though the rankings are similar, the absolute numbers don't line up cleanly for every single institution. The results show OpenAlex has generally lower institutional publication counts than Scopus does >

Lu: They found the mean raw difference was one hundred ninety-one point eleven and they noted a mean relative difference of just about zero point zero seven, which is pretty close on the surface >

Meng: But that close correlation isn't enough for some applications. The paper points out that if you look at specific journals with similar total publication counts in both databases, nearly all institutions had higher publication counts in Scopus than in OpenAlex >

Lalam: They also found that the relationship between those institutional outputs is very strong, using Spearman’s correlation of zero point nine eight and a Lin’s concordance correlation coefficient of zero point nine five five > <ref:2605.01337#pg1>

Tom: So, the core thesis here isn't just about raw numbers; it’s about how much trust you can put in OpenAlex when you need to make decisions based on those counts for specific entities >

Jane: They are saying that if institutional metadata is incomplete or inaccurate, the results of entire organizations or regions can be misrepresented, which makes understanding the extent of that issue crucial >

Lu: The study specifically aims to address a gap because most prior evaluations treated completeness and accuracy as aggregate properties of the database rather than looking at how specific institutions are affected >

Meng: That’s fair. I mean, if I’m trying to set up a production pipeline, knowing that some of the data points might be unreliable is a major blocker for scaling things up >

Lalam: The paper looked into potential causes for these differences, and they pointed to publishers not sharing affiliation metadata via Crossref as one reason >

Tom: And another key difference they highlighted is how publication years are assigned; OpenAlex usually records the earliest available year, while Scopus assigns the year corresponding to the formal issue publication >

Jane: They also mentioned that OpenAlex indexes certain document types, like book reviews or obituaries, that Scopus doesn't consistently include in its indexing >

Lu: And they noted a limitation they found concerning larger papers, specifically citing a "one hundred-plus author problem" where some papers were affected by OpenAlex filtering limitations >

Meng: That’s something I’ve seen in practice. Dealing with those massive author lists can really break simple filters or lead to missing data points at scale >

Tom: So, they concluded that while OpenAlex is a strong foundation for comparison, it needs improvement in affiliation metadata before it can be used for evaluation contexts that rely on absolute publication counts >

Jane: They suggest that relative comparisons might still yield similar conclusions, but the underrepresentation observed at the institutional level raises concerns for performance-based funding schemes or institutional monitoring where absolute numbers matter >

Lu: The study ultimately concludes that OpenAlex is already a promising starting point, but continued effort on completeness and precision is essential for realizing its full potential as an open alternative >

Meng: So, the practical implication is that if you’re doing serious evaluations based on these counts, you need to supplement your analysis with checks on metadata completeness >

Tom: Exactly. It’s a solid tool for seeing patterns, but if you need absolute precision for institutional accountability, you have to be careful with how much weight you put on the raw numbers from OpenAlex versus Scopus >

Jane: So, the paper shows us that while the structure of the data is there and correlations are strong, the underlying fidelity of those affiliations needs work before we can fully trust it for granular institutional evaluation >

Conclusion: Tom: So we're wrapping up this look at OpenAlex versus Scopus for German research institutions, focusing on their publications in top journals >

Jane: Right, the authors are comparing how much data each database captures about those specific German research groups >

Lu: Basically they’re testing if OpenAlex is a good stand-in for Scopus when you need to track institutional output >

Meng: So what they found is that while the general idea of publication volume is similar, the actual counts for individual institutions differ quite a bit >

Lalam: They measured the raw difference at nineteen one point eleven and the relative difference was about zero point seven >

Tom: That means even when you look at journals with similar total publication counts in both databases, nearly every institution had more publications recorded in Scopus than in OpenAlex >

Jane: So the big question is, if we use OpenAlex for performance monitoring, are we getting a complete picture of what those institutions are actually doing >

Lu: The conclusion they draw is that OpenAlex is suitable for general comparison but requires better affiliation data before it can be used for tracking absolute publication numbers >

Tom: They found that relative comparisons might work, but for things like performance-based funding, where the exact count matters, this missing data in OpenAlex is a problem >

Jane: It suggests that while the structure of the data is promising, we still need to improve how they collect and verify those institutional links >

Lu: They point out several reasons why the counts differ: things like publishers not sharing metadata or different ways the databases assign publication years >

Meng: So it’s not just one database being "wrong," but there are a few technical reasons, like OpenAlex following a different rule set than Scopus >

Tom: It means if you’re building a model on this data, you have to be careful about how much weight you put on the absolute numbers coming from OpenAlex versus what Scopus shows >

Jane: It's a good reminder that even with strong correlations, precision in affiliation metadata is what makes an evaluation truly useful >

Lu: And that leads us into how this affects future work, specifically needing better data alignment to make these institutional comparisons really reliable >

More episodes

← Home