Versteasch du mi? Computational and Socio-Linguistic Perspectives on GenAI, LLMs, and Non-Standard Language

summary

The gist

The design of Large Language Models (LLMs) and generative artificial intelligence (GenAI) has been shown to be “unfair” to less-spoken languages (Petrov et al…

In short

LLMs are unfair to less-spoken languages because their design reflects historical biases toward standardized European languages. Technical issues like tokenization and limited context windows create a 'tokenization tax' for non-standard speakers. The paper argues that solving this requires decolonial strategies centered on community agency and recognizing linguistic variation as fundamental, not an error.

Key concepts

Tokenization
This is how AI breaks down text into pieces (tokens) to process it. The paper explains that the method used (BPE) favors common words, meaning speakers of less common or non-standard languages have to use more tokens for the same idea, creating a disadvantage.
Tokenization Tax
This refers to the computational inefficiency caused by tokenization. Because non-standard language varieties require more tokens, speakers effectively pay a 'tax' in terms of processing power and complexity when using AI systems compared to those using standard languages like English.
Linguistic Agency
This concept means giving speakers control over how their language is used and processed by technology. The paper calls for policies that center linguistic agency, ensuring communities have the right to report 'dialect gaps' and act as auditors of AI systems.

Terminology used across episodes

This episode discusses

The paper

Versteasch du mi? Computational and Socio-Linguistic Perspectives on GenAI, LLMs, and Non-Standard Language · Read on arXiv

Verena Platzgummer, John McCrae, Sina Ahmadi

University of Galway · University of Zurich

The design of Large Language Models (LLMs) and generative artificial intelligence (GenAI) has been shown to be "unfair" to less-spoken languages (Petrov et al., 2023) and to deepen the digital language divide (Bella et al., 2023). Critical sociolinguistic work has also argued that these technologies are not only made possible by prior sociohistorical processes of linguistic standardisation, often grounded in European nationalist and colonial projects (Migge and Schneider, 2025), but also exacerbate epistemologies of language as "monolithic, monolingual, syntactically standardized systems of meaning" (Schneider, 2024, p. 5). In our paper, we draw on earlier work on the intersections of technology and language policy (Kelly-Holmes, 2019) and bring our respective expertise in critical sociolinguistics and computational linguistics to bear on an interrogation of these arguments. We take two different complexes of non-standard linguistic varieties in our respective repertoires-South Tyrolean dialects, which are widely used in informal communication in South Tyrol, Italy (Alber et al., 2024), as well as varieties of Kurdish-as starting points to an interdisciplinary exploration of the intersections between GenAI and linguistic variation and standardisation. We discuss both how LLMs can be made to deal with non-standard language from a technical perspective, and whether, when or how this can contribute to "democratic and decolonial digital and machine learning strategies" (Migge and Schneider, 2025, p. 12), which has direct policy implications.

DOI: 10.1007/s10993-026-09811-8

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Versteasch du mi? Computational and Socio-Linguistic Perspectives on GenAI, LLMs, and Non-Standard Language".

Jane: Here's the gist of the paper. The design of Large Language Models (LLMs) and generative artificial intelligence (GenAI) has been shown to be “unfair” to less-spoken languages (Petrov et al., 2023) and to deepen the digital language divide (Bella et al., 2023).

Tom: Systemic Inequalities in Language Technology Critical sociolinguistic work argues that these technologies are not only made possible by prior sociohistorical processes of linguistic standardisation, often grounded in European nationalist and colonial projects (Migge and Schneider, 2025), but also exacerbate epistemologies of language as “monolithic, monolingual, syntactically standardized systems of meaning” (Schneider, 2024, p.

Jane: 5).

Tom: When AI systems fail to process non-standard varieties, including historically minoritized languages, they exclude speakers from full access to digital spaces (Birnie, 2026).

Paper summary: Tom: So looking at the whole paper, 'Versteasch du mi? Computational and Socio-Linguistic Perspectives on GenAI, LLMs, and Non-Standard Language', the core idea is that excluding non-standard varieties isn't just a technical flaw in how we code things >

Jane: It’s a complex interaction between the technology itself, how we evaluate it, our underlying language ideologies about what makes language valuable, and those big political histories of standardisation >

Lu: They argue that treating linguistic variation as just noise to be eliminated is wrong; it should be seen as a fundamental characteristic of human communication that needs respect >

Meng: From the production side, this means if we don't change how we train these models, we’re going to keep shipping tools that are inherently biased toward one specific way of speaking >

Lalam: And I’m sure of it because the paper shows that standard varieties receive a different set of forces shaping their digital marginalization than the less spoken ones >

Tom: So, for a listener just tuning in, what does this all boil down to in simple terms? It means AI design isn't just about making things run faster; it’s about who gets to use the tools and how they have to speak to get access >

Conclusion: Tom: So we’ve talked about how these new models are built on language that mostly sounds like English, and how that leaves other languages behind.

Jane: Right, so the paper is titled "Versteasch du mi? Computational and Socio-Linguistic Perspectives on GenAI, LLMs, and Non-Standard Language," and it looks at how this isn't just a tech problem but a bigger issue with history.

Lu: It really digs into how language standardization—those old systems based on European or colonial projects—is baked into the very structure of these AI tools.

Meng: I mean, from a deployment standpoint, if the evaluation sets are only built around standard varieties, then what we ship is inherently biased toward that standard.

Lalam: I’m looking at this from the model side, and it confirms what we’ve seen: when the tokenizer uses byte pair encoding to break words based on how often they appear statistically, it creates a tokenization tax for under-resourced languages.

Tom: A tokenization tax. So speakers of those languages literally pay more in computing power just to get the same amount of information processed by the AI.

Jane: And it’s not just about speed; it’s about context windows too, where dialects need more tokens to say the same thing that a standard speaker can say in fewer words.

Lu: The paper points out three main areas where we need to improve things: the way the LLM architecture handles tokenization, how it deals with spelling variations and grammar complexity, and creating actual benchmarks for these other languages.

Meng: Benchmarks are tough when you don't have enough data to test them properly. We need real traffic, not just clean datasets that look perfect on paper.

Tom: So what does this mean for the big picture? It suggests we can’t just treat linguistic variation as some kind of noise to be cleaned out of the system.

Jane: Exactly. The authors argue we have to move toward strategies that center linguistic agency, meaning the communities actually speaking these languages need a say in how these tools are built and tested.

Lu: They suggest things like requiring reporting on dialect gaps so that community data sovereignty is respected, and having civil society groups act as auditors to check where the models fail local norms.

Meng: That sounds like a massive operational overhead for shipping software at scale, but if it prevents catastrophic failure in niche markets, I see the value.

Tom: It really boils down to recognizing that the exclusion isn't just technical; it’s rooted in how we’ve historically viewed language itself.

Jane: And the authors conclude that treating linguistic variation as something separate from communication is a fundamental misunderstanding of what human interaction actually is.

Lu: The big picture here is that these digital infrastructures are deeply tied to political and historical processes, not just math.

Tom: So for listeners who want to get into this stuff, the takeaway is that next time you use a generative tool, remember that the way it’s built and tested has real-world consequences for speakers of less common languages.

Jane: And that opens up a whole new conversation about how we design these systems democratically.

More episodes

← Home