Pair-Merge Tokenisation Is Hard to Approximate on Binary Strings and Strongly NP-Hard on Unary Datasets
cs.DS, cs.CC, cs.CL
Submitted: 2026-11-10
Updated: 2026-11-10
Terminology
Sources
- Exact Algorithms and Complexity of the Smallest Grammar Problem
- Tokenisation over Bounded Alphabets is Hard
- Theoretical Analysis of Byte-Pair Encoding
- A Partition Cover Approach to Tokenization
- A Birthday Repetition Theorem and Complexity of Approximating Dense CSPs
- Computational Complexity of Determining the Assembly Index
Related papers
- Exploiting Low Scanwidth to Resolve Soft Polytomies
- PHastR: Faster Minimal Perfect Hashing with Rotated Layouts
- Fine-Grained Complexity of Approximating Vector Knapsack: A Faster Algorithm and Equivalence with k-SUM in 2D
- Polynomial-time computation of p-contraction fixed points for even p
- Separations Between Submodular-Like Graph Widths
- Bounded Integer Programming in Almost N 2d Time