When does text encoding matter for language modeling?
cs.CL
Submitted: 2026-11-10
Updated: 2026-11-10
Code: https://github.com/genesith/AetherModels
Terminology
Sources
- Byte Pair Encoding is Suboptimal for Language Model Pretraining
- Bridging the Gap for Tokenizer-Free Language Models
- Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance
- Dynamic Chunking for End-to-End Hierarchical Sequence Modeling
- IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages
- BBPE16: UTF-16-based byte-level byte-pair encoding for improved multilingual speech recognition
- BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization
- Compute Optimal Tokenization
- Character Distributions of Classical Chinese Literary Texts: Zipf's Law, Genres, and Epochs
- Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models
- Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP
- Bolmo: Byteifying the Next Generation of Language Models
- Bit-level BPE: Below the byte boundary
- Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models
- grapheme-kit: Grapheme-Level Metrics and Text Processing for Multilingual NLP
- FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
- The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?
- Egalitarian Language Representation in Language Models: It All Begins with Tokenizers
- Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization
- OpenCSG Chinese Corpus: A Series of High-quality Chinese Datasets for LLM Training
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data