Greedy Byte-Pair Encoding Achieves 0.391 of the Optimum, Past the One-Third Barrier for Packing-Based Analyses
cs.DS, cs.CL
Submitted: 2026-11-05
Updated: 2026-11-09
Code: https://github.com/rsennrich/subword-nmt
Terminology
Sources
- Tokenisation over Bounded Alphabets is Hard
- Theoretical Analysis of Byte-Pair Encoding
- A Partition Cover Approach to Tokenization
Related papers
- Exploiting Low Scanwidth to Resolve Soft Polytomies
- PHastR: Faster Minimal Perfect Hashing with Rotated Layouts
- Fine-Grained Complexity of Approximating Vector Knapsack: A Faster Algorithm and Equivalence with k-SUM in 2D
- Polynomial-time computation of p-contraction fixed points for even p
- Separations Between Submodular-Like Graph Widths
- Bounded Integer Programming in Almost N 2d Time