A Multi-Word Tokenizer for 204 Languages That Needs Fewer Tokens Than GPT-4o's, Including on English

arXiv:2611.13059 · cs.CL, cs.AI · Submitted 2026-11-09 · Read on arXiv

cs.CL, cs.AI

Submitted: 2026-11-09

Updated: 2026-11-15

Code: https://github.com/HavonAI/indic-token-tax

Terminology

Sources

Related papers