Tokenization: Hardness and Bicriteria Approximation Algorithms
cs.DS, cs.AI, cs.CC
Submitted: 2026-11-11
Updated: 2026-11-11
Terminology
Sources
- A Partition Cover Approach to Tokenization
- Tokenisation over Bounded Alphabets is Hard
- Tokenisation via Convex Relaxations
- Getting the most out of your tokenizer for pre-training and domain adaptation
Related papers
- Exploiting Low Scanwidth to Resolve Soft Polytomies
- PHastR: Faster Minimal Perfect Hashing with Rotated Layouts
- Fine-Grained Complexity of Approximating Vector Knapsack: A Faster Algorithm and Equivalence with k-SUM in 2D
- Polynomial-time computation of p-contraction fixed points for even p
- Separations Between Submodular-Like Graph Widths
- Bounded Integer Programming in Almost N 2d Time