Versteasch du mi? Computational and Socio-Linguistic Perspectives on GenAI, LLMs, and Non-Standard Language
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Versteasch du mi? Computational and Socio-Linguistic Perspectives on GenAI, LLMs, and Non-Standard Language".
Jane: Here's the gist of the paper. The design of Large Language Models (LLMs) and generative artificial intelligence (GenAI) has been shown to be “unfair” to less-spoken languages (Petrov et al., 2023) and to deepen the digital language divide (Bella et al., 2023).
Tom: Systemic Inequalities in Language Technology Critical sociolinguistic work argues that these technologies are not only made possible by prior sociohistorical processes of linguistic standardisation, often grounded in European nationalist and colonial projects (Migge and Schneider, 2025), but also exacerbate epistemologies of language as “monolithic, monolingual, syntactically standardized systems of meaning” (Schneider, 2024, p.
Jane: 5).
Tom: When AI systems fail to process non-standard varieties, including historically minoritized languages, they exclude speakers from full access to digital spaces (Birnie, 2026).
Paper summary: Tom: So looking at the whole paper, 'Versteasch du mi? Computational and Socio-Linguistic Perspectives on GenAI, LLMs, and Non-Standard Language', the core idea is that excluding non-standard varieties isn't just a technical flaw in how we code things >
Jane: It’s a complex interaction between the technology itself, how we evaluate it, our underlying language ideologies about what makes language valuable, and those big political histories of standardisation >
Lu: They argue that treating linguistic variation as just noise to be eliminated is wrong; it should be seen as a fundamental characteristic of human communication that needs respect >
Meng: From the production side, this means if we don't change how we train these models, we’re going to keep shipping tools that are inherently biased toward one specific way of speaking >
Lalam: And I’m sure of it because the paper shows that standard varieties receive a different set of forces shaping their digital marginalization than the less spoken ones >
Tom: So, for a listener just tuning in, what does this all boil down to in simple terms? It means AI design isn't just about making things run faster; it’s about who gets to use the tools and how they have to speak to get access >
Conclusion: Tom: So we’ve talked about how these new models are built on language that mostly sounds like English, and how that leaves other languages behind.
Jane: Right, so the paper is titled "Versteasch du mi? Computational and Socio-Linguistic Perspectives on GenAI, LLMs, and Non-Standard Language," and it looks at how this isn't just a tech problem but a bigger issue with history.
Lu: It really digs into how language standardization—those old systems based on European or colonial projects—is baked into the very structure of these AI tools.
Meng: I mean, from a deployment standpoint, if the evaluation sets are only built around standard varieties, then what we ship is inherently biased toward that standard.
Lalam: I’m looking at this from the model side, and it confirms what we’ve seen: when the tokenizer uses byte pair encoding to break words based on how often they appear statistically, it creates a tokenization tax for under-resourced languages.
Tom: A tokenization tax. So speakers of those languages literally pay more in computing power just to get the same amount of information processed by the AI.
Jane: And it’s not just about speed; it’s about context windows too, where dialects need more tokens to say the same thing that a standard speaker can say in fewer words.
Lu: The paper points out three main areas where we need to improve things: the way the LLM architecture handles tokenization, how it deals with spelling variations and grammar complexity, and creating actual benchmarks for these other languages.
Meng: Benchmarks are tough when you don't have enough data to test them properly. We need real traffic, not just clean datasets that look perfect on paper.
Tom: So what does this mean for the big picture? It suggests we can’t just treat linguistic variation as some kind of noise to be cleaned out of the system.
Jane: Exactly. The authors argue we have to move toward strategies that center linguistic agency, meaning the communities actually speaking these languages need a say in how these tools are built and tested.
Lu: They suggest things like requiring reporting on dialect gaps so that community data sovereignty is respected, and having civil society groups act as auditors to check where the models fail local norms.
Meng: That sounds like a massive operational overhead for shipping software at scale, but if it prevents catastrophic failure in niche markets, I see the value.
Tom: It really boils down to recognizing that the exclusion isn't just technical; it’s rooted in how we’ve historically viewed language itself.
Jane: And the authors conclude that treating linguistic variation as something separate from communication is a fundamental misunderstanding of what human interaction actually is.
Lu: The big picture here is that these digital infrastructures are deeply tied to political and historical processes, not just math.
Tom: So for listeners who want to get into this stuff, the takeaway is that next time you use a generative tool, remember that the way it’s built and tested has real-world consequences for speakers of less common languages.
Jane: And that opens up a whole new conversation about how we design these systems democratically.
Verena Platzgummer, John McCrae, Sina Ahmadi
University of Galway · University of Zurich
cs.CL
Submitted: 2026-03-30
Updated: 2026-10-30
Comments: Published at the Language Policy journal
DOI: 10.1007/s10993-026-09811-8
Code: https://github.com/mainlp/germanic-lrl-corpora
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 79/100
The gist: The design of Large Language Models (LLMs) and generative artificial intelligence (GenAI) has been shown to be “unfair” to less-spoken languages (Petrov et al…
Key concepts
- Tokenization
- This is how AI breaks down text into pieces (tokens) to process it. The paper explains that the method used (BPE) favors common words, meaning speakers of less common or non-standard languages have to use more tokens for the same idea, creating a disadvantage.
- Tokenization Tax
- This refers to the computational inefficiency caused by tokenization. Because non-standard language varieties require more tokens, speakers effectively pay a 'tax' in terms of processing power and complexity when using AI systems compared to those using standard languages like English.
- Linguistic Agency
- This concept means giving speakers control over how their language is used and processed by technology. The paper calls for policies that center linguistic agency, ensuring communities have the right to report 'dialect gaps' and act as auditors of AI systems.
Terminology
Summary
The design of Large Language Models (LLMs) and generative artificial intelligence (GenAI) has been shown to be “unfair” to less-spoken languages (Petrov et al. p. 0 ahmadi@uzh. ch
Abstract
The design of Large Language Models
(LLMs) and generative artificial intelligence
(GenAI) has been shown to be “unfair” to
less-spoken languages (Petrov et al. ch
Abstract
The design of Large Language Models
(LLMs) and generative artificial intelligence
(GenAI) has been shown to be “unfair” to
less-spoken languages (Petrov et al. , 2023)
and to deepen the digital language divide
(Bella et al.Where the paper says it
Where the paper says it
Systemic Inequalities in Language Technology
Critical sociolinguistic work argues that these technologies are not only made possible by prior sociohistorical processes of linguistic standardisation, often grounded in European nationalist and colonial projects (Migge and Schneider, 2025), but also exacerbate epistemologies of language as “monolithic, monolingual, syntactically standardized systems of meaning” (Schneider, 2024, p. 5). When AI systems fail to process non-standard varieties, including historically minoritized languages, they exclude speakers from full access to digital spaces (Birnie, 2026). p. 0 Critical
sociolinguistic work argues that these technologies not only emerge from historical processes of
linguistic standardization rooted in colonial and
nationalist projects (Migge and Schneider, 2025),
but also reinforce epistemologies of language as
“monolithic, monolingual, syntactically standardized systems of meaning” (Schneider, 2024, p. When AI systems fail to process non-standard varieties, including historically minoritized languages,
they exclude speakers from full access to digital
spaces (Birnie, 2026).Where the paper says it
Computational Barriers to Non-Standard Language Processing
Underlying those generative AI systems that generate text, LLMs have demonstrated remarkable capabilities in what computational linguistics refers to as high-resource linguistic environments (Phan et al., 2025). However, their deployment across the global linguistic landscape remains deeply asymmetrical The paper identifies three main areas where current research on LLMs can be improved for nonstandard languages:
-
The architecture of LLMs, especially around the tokenization of the input text
-
The handling of morphological complexity and orthographic variation
-
The creation of relevant benchmarks
The Impact of Tokenization and Evaluation
The paper details how the tokenizer, using Byte Pair Encoding (BPE), breaks words based on statistical frequency, leading to computational inefficiency for under-resourced languages. This results in a , 2020), which is a hard limit
on how many tokens it can consider at one time. Because dialects require more tokens to express
the same idea, they fill up the model’s memory
faster, leading to poorer reasoning and shorter possible conversations.tokenization tax
where speakers of such varieties literally pay more when they prompt generative AI in these varieties to process the same amount of information given in English. Furthermore, most AI models have a context window, which is a hard limit on how many tokens it can consider at one time, and dialects require more tokens to express the same idea. p. 4Where the paper says it
Case Studies Illustrating Marginalization
The paper uses two case studies to illustrate situated implications of language policies in language technology: South Tyrolean German and Kurdish varieties. For South Tyrolean German, the issue is limited visibility within computational infrastructures, where it is largely subsumed under broader linguistic categories. p. 1 It can be estimated that the complex
of South Tyrolean German varieties is the primary
medium of everyday interaction for about 300,000
of the province’s inhabitants (ASTAT, 2025), yet
it has been largely absent from LLM training data. Its relative marginalization is therefore less a consequence of political exclusion than of limited visibility within computational infrastructures, where
it is largely subsumed under broader linguistic categories and rarely evaluated in its own right. Kurdish, by contrast, represents a dialect continuum
spoken by over 40 million people across multiple
nation-states, characterized by orthographic diversity, systematic political suppression, and acute
digital underrepresentation. While Central Kurdish
(Sorani) and Northern Kurdish (Kurmanji) have
achieved modest computational resources, varieties
such as Southern Kurdish, Hawrami, and Zazaki
remain almost entirely invisible to language technology.Where the paper says it
Where the paper says it
Policy Implications for Democratic Strategies
The paper argues that addressing LLM marginalization requires a reorientation toward “democratic and decolonial digital and machine learning strategies” (Migge and Schneider, 2025) that center linguistic agency. This includes requiring “dialect gap” reporting to ensure community data sovereignty. p. 1 Building on earlier work examining technology
and language policy intersections (Kelly-Holmes,
2019), we argue that addressing LLM marginalization of non-standard varieties requires more than
technical fixes; it demands a reorientation toward
“democratic and decolonial digital and machine
learning strategies” (Migge and Schneider, 2025)
that center linguistic agency and treat variation as
constitutive of human communication rather than
noise to eliminate. This has direct policy implications, from requiring “dialect gap” reporting to
ensuring community data sovereignty. As such, state-led AI initiatives should
avoid replicating the discriminatory practices of the
past, e. by adopting a policy framework of linguistic agency, centered on the principle of “nothing about us without us”, as similarly suggested by
(Liu et al. , 2020)) to
collect authentic speech and text with explicit community consent. Furthermore, CSOs can serve as
algorithmic auditors, performing socio-pragmatic
red-teaming to identify where models fail to respect
local norms or inadvertently enforce standardization.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion on Linguistic Variation
The paper concludes that the exclusion of non-standard varieties cannot be understood solely as a technical problem, but rather emerges from the interaction of technical infrastructures, evaluation regimes, language ideologies, and broader political and historical processes. Future work should treat linguistic variation as a fundamental characteristic of human communication rather than as noise to be eliminated. p. 1 While Central Kurdish
(Sorani) and Northern Kurdish (Kurmanji) have
achieved modest computational resources, varieties
such as Southern Kurdish, Hawrami, and Zazaki
remain almost entirely invisible to language technology. Building on earlier work examining technology
and language policy intersections (Kelly-Holmes,
2019), we argue that addressing LLM marginalization of non-standard varieties requires more than
technical fixes; it demands a reorientation toward
“democratic and decolonial digital and machine
learning strategies” (Migge and Schneider, 2025)
that center linguistic agency and treat variation as
constitutive of human communication rather than
noise to eliminate.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improve tokenization to account for morphological complexity in non-standard languages by developing language-specific tokenizers that accommodate fusional languages like Kurdish, which currently result in
excessive subword splitting—where meaningful linguistic representations are fractured.
-
Implement a dialect gap reporting mechanism that explicitly measures
the reduction in performance in dialects versus English and by requiring this to be explicitly reported or even directly supported by incentives in government policy
to shift the paradigm towards more explicit support for non-standardized languages. -
Develop culturally and linguistically tailored benchmarks instead of relying on translated English benchmarks, such as creating a benchmark that measures
the moral of a specific proverb, where success requires an understanding of the cultural metaphors that do not exist in the model’s English-centric training data.
-
Integrate speech foundation models into evaluation frameworks to assess how well AI understands regional accents and phonetic variations by using tools like Voxlect, which will help determine
whether models can handle inconsistent orthography and non-standard spelling without crashing or defaulting to English or standard languages.
-
Design policy frameworks that mandate community data sovereignty for non-standard varieties, ensuring that speakers of these varieties
retain data sovereignty over their linguistic repertoires
to prevent the reproduction of intra-linguistic hierarchies.
Abstract
The design of Large Language Models (LLMs) and generative artificial intelligence (GenAI) has been shown to be "unfair" to less-spoken languages (Petrov et al., 2023) and to deepen the digital language divide (Bella et al., 2023). Critical sociolinguistic work has also argued that these technologies are not only made possible by prior sociohistorical processes of linguistic standardisation, often grounded in European nationalist and colonial projects (Migge and Schneider, 2025), but also exacerbate epistemologies of language as "monolithic, monolingual, syntactically standardized systems of meaning" (Schneider, 2024, p. 5). In our paper, we draw on earlier work on the intersections of technology and language policy (Kelly-Holmes, 2019) and bring our respective expertise in critical sociolinguistics and computational linguistics to bear on an interrogation of these arguments. We take two different complexes of non-standard linguistic varieties in our respective repertoires-South Tyrolean dialects, which are widely used in informal communication in South Tyrol, Italy (Alber et al., 2024), as well as varieties of Kurdish-as starting points to an interdisciplinary exploration of the intersections between GenAI and linguistic variation and standardisation. We discuss both how LLMs can be made to deal with non-standard language from a technical perspective, and whether, when or how this can contribute to "democratic and decolonial digital and machine learning strategies" (Migge and Schneider, 2025, p. 12), which has direct policy implications.
Sources
- Towards Bridging the Digital Language Divide
- Longformer: The Long-Document Transformer
- Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures
- Voxlect: A Speech Foundation Model Benchmark for Modeling Dialects and Regional Languages Around the Globe
- TranslateGemma Technical Report
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- Steering LLMs toward Korean Local Speech: Iterative Refinement Framework for Faithful Dialect Translation
- Humanity's Last Exam
- MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data