What Do Tokenizer Comparisons Measure? Disentangling Granularity, Root Assignment, and Word-Level Readout across Amharic and Hausa
cs.CL, cs.LG
Submitted: 2026-11-10
Updated: 2026-11-10
Terminology
Sources
- MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies
- When Tokenizers Fail: Byte-Level Chunking for Zero-Shot Transfer to Low-Resource Languages
- Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP
- No Language Left Behind: Scaling Human-Centered Machine Translation
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data