A Multi-Word Tokenizer for 204 Languages That Needs Fewer Tokens Than GPT-4o's, Including on English
cs.CL, cs.AI
Submitted: 2026-11-09
Updated: 2026-11-15
Code: https://github.com/HavonAI/indic-token-tax
Terminology
Sources
- SuperBPE: Space Travel for Language Models
- Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages
- MorphTok: Morphologically Grounded Tokenization for Indian Languages
- My Boli: Code-mixed Marathi-English Corpora, Pretrained Language Models and Evaluation Benchmarks
- No Language Left Behind: Scaling Human-Centered Machine Translation
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data