A survey of diversity quantification in natural language processing: The why, what, where and how

summary

Video file (mp4)

The gist

The gist The concept of diversity has received increasing attention in natural language processing (NLP) in recent years, and many measures are used to quantify it, but it is often addressed in an ad

In short

This survey examines how diversity is quantified in Natural Language Processing (NLP), analyzing over 300 papers using a 'why, what, where, and how' framework. It reveals that diversity motivations are often unclear, measures are inconsistent, and the choice of metrics lacks justification. The paper suggests new research must explicitly address these gaps.

Key concepts

Cross-domain perspective on diversity
This looks at how diverse fields like ecology or biology justify diversity as a goal. It shows that while the concepts differ, they often share underlying assumptions about what makes a system rich and stable, which helps frame how NLP diversity should be viewed.
Stirling’s cross-domain framework
This provides a structural way to categorize diversity across different fields. It posits that diversity is a property of a system where elements can be grouped into categories. This allows researchers to measure variety (number of categories), balance (evenness), and disparity (differences between categories).
NLP-specific framework
This framework organizes the study of NLP diversity around four questions: why it matters, what is measured, where it's measured, and how it's measured. This structure helps make diverse research comparable and highlights recurring issues like inconsistent measurement methods.
Diversity families
Based on Stirling’s framework, the survey identifies seven families of diversity measures (like dRichness or dAvePairwise). These categories group different mathematical approaches to quantifying diversity, showing which types of metrics are most frequently used in NLP research.

Terminology used across episodes

This episode discusses

The paper

A survey of diversity quantification in natural language processing: The why, what, where and how · Read on arXiv

Louis Estève, Marie-Catherine de Marneffe, Nurit Melnik, Agata Savary, Olha Kanishcheva

Université Paris-Saclay, CNRS, LISN · FNRS - UCLouvain · The Open University of Israel

The concept of diversity received increasing attention in natural language processing (NLP) in recent years. It became an advocated property of datasets and systems, with many measures used to quantify it, but also many cross-paper inconsistencies. To address this fragmentation, we take inspiration from other scientific fields where diversity has been more thoroughly conceptualized. We build upon (Sarkar, 2010) and (Stirling, 2007), two unified cross-domain frameworks inspired from ecology, economy and other domains. The first one advocates explicit justification of diversity endorsement. The latter distinguishes three dimensions of diversity: variety, balance, and disparity. Surveying over 300 recent diversity-related papers from ACL Anthology, we propose an NLP-specific framework with 4 perspectives: why diversity is important, what diversity is measured on, where it is measured, and how. We find that in NLP richness (the simplest variety) and distance averaging (the simplest disparity) are the most prevalent measure families, and that pure balance measures are nearly absent. Moreover, half of the measures used in NLP do not conform to Stirling's framework. Our analysis increases comparability of approaches to diversity in NLP, allowing us to formulate recommendations for the field. We recommend that the field systematically use a unified terminology, and that it engage in debates about the desirable properties of diversity measures as well as about the normative assumptions behind their use.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "A survey of diversity quantification in natural language processing".

Jane: Here's the gist of the paper. The gist The concept of diversity has received increasing attention in natural language processing (NLP) in recent years, and many measures are used to quantify it, but it is often addressed in an ad hoc manner with few explicit justifications and many cross-paper inconsistencies <ref:2507.20858#pg14>.

Tom: Cross-domain perspective on diversity The paper examines how other fields approached diversity, such as ecology, biology, linguistics, cultural studies, economics, and policy studies.

Jane: Sarkar observes that what all studied fields have in common is richness (the number of categories in a system), which is a “rather trivial concept of diversity” according to Sarkar (2010, p.

Tom: 136).

Title and authors: Jane: So the paper starts by setting the stage, explaining that the concept of diversity has become very important in NLP lately. It notes that it's now an advocated property for both datasets and systems.

Tom: And they immediately point out the issue: many people use measures for diversity in an ad hoc way, without really justifying why they chose them or what they are trying to achieve.

Jane: They found that this has led to a lot of conceptual fragmentation concerning diversity, suggesting NLP belongs in fields where diversity is discussed but often left undefined or analytically neglected.

Lu: The authors want to address this fragmentation by taking inspiration from studies outside of NLP where diversity has been systematically analyzed, like ecology and economics.

Meng: So they’re not just looking at existing papers; they are trying to build a unified conceptual framework for how we quantify things in NLP.

Tom: They propose using Stirling's unified framework—variety, balance, and disparity—as the foundation for this new NLP-specific approach.

Jane: It seems like the core contribution is proposing this four-perspective framework as a way to make sense of all those different ways diversity is currently being talked about.

Lalam: They apply this framework to over three hundred papers and show that it helps increase the comparability between them, which helps us see global tendencies <ref:2507.20858#pg1>.

Tom: So it’s less about finding one perfect measure and more about understanding the landscape of what’s being measured.

Jane: It really shifts the focus from just picking a number to understanding the underlying motivations and context behind those numbers.

The paper's summary: Lu: They break down what they found into three classes of objects that are measured: in-text categories, meta-linguistic categories, and processing categories.

Tom: So, in-text categories relate to linguistic properties assumed to be inherent in the text itself, like lexical diversity based on words as elements and unique words as categories.

Jane: Then there are meta-linguistic categories which look at how texts are classified within datasets, where languages might be the categories themselves.

Meng: And finally, processing categories which are external to the text, looking at the processes applied to the data, like annotations or different types of tasks performed.

Tom: The paper also looks at where diversity is measured across a standard machine learning pipeline—from data collection all the way through evaluation.

Jane: They found that the "Output data" stage is most often quantified, which makes sense because that’s where we see the final results of our models.

Lu: When they looked at how diversity was measured, they found over one hundred different names for diversity measures and applied Stirling's framework to them <ref:2507.20858#pg3>.

Tom: And seven families of measures emerged from this analysis: dRichness for variety, dBalance for balance, dEntropy which is a hybrid of variety and balance, plus three others targeting variety-balance combinations like dVBOth and dAvePairwise.

Jane: The most frequently used measures they found were those from the dRichness family, the AvePairwise family, and dTTR variants.

Lalam: This gives us a concrete set of tools to look at what’s out there and how different these tools are from each other.

The paper's improvements: Tom: They recommend that new papers addressing this topic position themselves clearly on the why, what, where and how perspectives instead of just getting lost in technical details.

Jane: This means authors need to answer questions about their underlying normative assumptions—the reasons behind wanting diversity in the first place.

Lu: They also suggest a lot of debate around which objects are being measured and which pipeline stage is relevant for that specific diversity measure.

Meng: And they want more in-depth discussions about the desirable properties of these diversity measures themselves, not just reporting a single number without context.

Tom: Specifically, they question the status of any measures that don't fit Stirling’s framework because if they don't fit that structure, it probably means we haven't understood what we are actually measuring.

Jane: They want papers to be more explicit about their choices regarding the measure and why that choice matters for their research goals.

Conclusion: Tom: The main implication is that future work needs to be very deliberate about its conceptual grounding before it starts just piling up more ad hoc metrics.

Jane: It suggests that authors should be very clear about their normative assumptions—whether they are aiming for ethical inclusion or practical performance gains—because those assumptions dictate the kind of diversity we choose to measure.

Lu: The survey itself is valuable because it shows us what’s been done and where the gaps are, especially regarding how different fields like ecology inform our understanding here.

Meng: From a deployment standpoint, it tells us that if we want robust systems, we need to think about diversity across the whole pipeline, not just the final output stage.

Lalam: I'm confident this framework will help future researchers move past just counting things and start actually building meaningful systems around diversity in NLP.

More episodes

← Home