GreedTok Is a 0.3287-Approximation for Tokenization, Within 3% of the Best Possible Constant
cs.DS
Submitted: 2026-11-05
Updated: 2026-11-09
Comments: 57 pages, 5 figures. Code, checkers and supplement: https://doi.org/10.5281/zenodo.24073514
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- A Partition Cover Approach to Tokenization
- Tokenisation is NP-Complete
- Tokenisation over Bounded Alphabets is Hard
- Theoretical Analysis of Byte-Pair Encoding
- Tokenisation via Convex Relaxations
- Global Cardinality Constraints Make Approximating Some Max-2-CSPs Harder
- A Note on Max $k$-Vertex Cover: Faster FPT-AS, Smaller Approximate Kernel and Improved Approximation
Related papers
- Exploiting Low Scanwidth to Resolve Soft Polytomies
- PHastR: Faster Minimal Perfect Hashing with Rotated Layouts
- Fine-Grained Complexity of Approximating Vector Knapsack: A Faster Algorithm and Equivalence with k-SUM in 2D
- Polynomial-time computation of p-contraction fixed points for even p
- Separations Between Submodular-Like Graph Widths
- Bounded Integer Programming in Almost N 2d Time