Tokenizer Myths and Where To Find Them
cs.CL, cs.LG
Submitted: 2026-11-03
Updated: 2026-11-03
Comments: 35 pages
Code: https://github.com/karpathy/autoresearch
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Sources
- Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models
- Character-Level Language Modeling with Deeper Self-Attention
- Tokenizer Choice For LLM Training: Negligible or Crucial?
- TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior
- Program Synthesis with Large Language Models
- Establishing Task Scaling Laws via Compute-Efficient Model Ladders
- Lessons from the Trenches on Reproducible Evaluation of Language Models
- u-$\mu$P: The Unit-Scaled Maximal Update Parametrization
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit
- You should evaluate your language model on marginal likelihood over tokenisations
- Evaluating Large Language Models Trained on Code
- Should you marginalize over possible tokenizations?
- Bridging the Gap for Tokenizer-Free Language Models
- Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Training Verifiers to Solve Math Word Problems
- Two Counterexamples to Tokenization and the Noiseless Channel
- Are All Languages Equally Hard to Language-Model?
- Getting the most out of your tokenizer for pre-training and domain adaptation
- You Can Learn Tokenization End-to-End with Reinforcement Learning
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data