Theoretical Analysis of Byte-Pair Encoding
cs.DS, cs.CL
Submitted: 2024-11-13
Updated: 2026-10-30
Code: https://github.com/google/sentencepiece
Terminology
Sources
- Tokenizer Choice For LLM Training: Negligible or Crucial?
- Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP
- Neural Machine Translation of Rare Words with Subword Units
- Tokenization Is More Than Compression
- LLaMA: Open and Efficient Foundation Language Models
- A Formal Perspective on Byte-Pair Encoding
Related papers
- PHastR: Faster Minimal Perfect Hashing with Rotated Layouts
- Fine-Grained Complexity of Approximating Vector Knapsack: A Faster Algorithm and Equivalence with k-SUM in 2D
- Polynomial-time computation of p-contraction fixed points for even p
- Separations Between Submodular-Like Graph Widths
- Bounded Integer Programming in Almost N 2d Time
- Fast Odd-Permutation Sums in Characteristic Two and Shortest Even Directed Cycles