The Indic Token Tax in 2026: Measuring Tokenizer. Disparity and Its Cost for the Languages of India
cs.CL, cs.AI
Submitted: 2026-11-09
Updated: 2026-11-15
Code: https://github.com/HavonAI/indic-token-tax
Terminology
Sources
- No Language Left Behind: Scaling Human-Centered Machine Translation
- My Boli: Code-mixed Marathi-English Corpora, Pretrained Language Models and Evaluation Benchmarks
- Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages
- IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages
- MorphTok: Morphologically Grounded Tokenization for Indian Languages
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data