MoTo: Mixture of Tokenizers Towards Fair Multilingual Language Modeling
cs.CL, cs.LG
Submitted: 2026-11-08
Updated: 2026-11-08
Code: https://github.com/ahmetaa/zemberek-nlp
Terminology
Sources
- MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies
- Tokens with Meaning: A Hybrid Tokenization Approach for Turkish
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- You Can Learn Tokenization End-to-End with Reinforcement Learning
- DeepSeek-V3 Technical Report
- Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization
- Training-Free Tokenizer Transplantation via Orthogonal Matching Pursuit
- The Llama 3 Herd of Models
- Dynamic Chunking for End-to-End Hierarchical Sequence Modeling
- MorphPiece : A Linguistic Tokenizer for Large Language Models
- FastText.zip: Compressing text classification models
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- Language-Specific Effects of Tokenizer Choice in Multilingual Language Models
- What Language is This? Ask Your Tokenizer
- Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP
- Bolmo: Byteifying the Next Generation of Language Models
- GPT-4 Technical Report
- INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge
- StochasTok: Improving Fine-Grained Subword Understanding in LLMs
- Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data