Inherited Weights: Measuring the Tokenization Premium and the Sovereignty Cost of Language Models for the Global South
cs.CL, cs.AI, cs.CY
Submitted: 2026-11-10
Updated: 2026-11-10
Comments: 6 pages, 2 tables. Data and measurement code: https://doi.org/10.5281/zenodo.22899767
License: http://creativecommons.org/licenses/by/4.0/
The gist: Most language models deployed outside the United States and China are fine-tuned descendants of a small number of pretrained families.
Terminology
Abstract
Most language models deployed outside the United States and China are fine-tuned descendants of a small number of pretrained families. We argue that fine-tuning leaves the decisive properties of a model inherited from its base: vocabulary, knowledge, cultural alignment, memorized data and license. We contribute (i) an updated, reproducible measurement of the tokenization premium across 31 languages and 7 tokenizers released in 2023-2024, on 27 articles of the Universal Declaration of Human Rights, with explicit correction for unknown-token collapse. Premiums reach 10x for Amharic in older tokenizers and remain at 6.8x in the newest commercial one; Indigenous South American languages pay between 1.6x and 2.7x under global tokenizers. A vocabulary built from Portuguese lowers the premium for 11 of 13 Latin-script Global South languages relative to every global tokenizer tested, by about 24% for Guarani and Quechua. (ii) An eight-test black-box sovereignty audit that a buyer can run without access to weights. (iii) A regime argument: under energy and liability constraints, accuracy per joule, calibration and selective risk should replace raw accuracy as design objectives, with metrics that abstention cannot inflate. We close with preliminary internal measurements from a proprietary architecture built around these objectives, reported with their limitations. Data and measurement code are included.
Sources
- LIMA: Less Is More for Alignment
- Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
- Whose Opinions Do Language Models Reflect?
- Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
- Language Model Tokenizers Introduce Unfairness Between Languages
- Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models
- Extracting Training Data from Large Language Models
- Poisoning Web-Scale Training Datasets is Practical
- Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
- Tucano: Advancing Neural Text Generation for Portuguese
- How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models
- On Calibration of Modern Neural Networks
- Language Models (Mostly) Know What They Know
- Why Language Models Hallucinate
- Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
- PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models
- Effective Prompt Extraction from Language Models
- Selective Classification for Deep Neural Networks
- Reducing Tokenization Premiums for Low-Resource Languages
- Measuring the Tokenization Premium: A Cost Audit for Underserved Language Communities
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data