LEMON-ZEST: Evolution-Informed Tokenization for Efficient Protein Language Modeling
cs.LG, q-bio.BM
Submitted: 2026-09-29
Updated: 2026-09-29
Comments: Accepted to NeurIPS 2026. 9 pages, 4 figures, 3 tables
License: http://creativecommons.org/licenses/by/4.0/
The gist: Protein Language Models (PLMs) have made remarkable progress following scaling laws established in natural language processing across sequence- and structure-based tasks, yet the potential of
Terminology
Abstract
Protein Language Models (PLMs) have made remarkable progress following scaling laws established in natural language processing across sequence- and structure-based tasks, yet the potential of tokenization remains underexploited. Unlike human language, proteins preserve structure despite extensive sequence variation a property standard tokenization strategies fundamentally fail to capture. We introduce ZEST (Zoned Encoding of Sequence Traits), an evolution-informed vocabulary derived from conserved regions of multiple sequence alignments. ZEST allows embedding domain-level biological priors directly at the tokenization stage rather than learning them implicitly through scale. ZEST natively compresses sequences to an average token length of 4 residues, enabling our model to process 4,000 residues within a standard 1024-token context window. Building on this, we present LEMON (Layered Extraction of Molecular Ordering from Nature), a compact 200M-parameter sequence-based model for detection of remote homology between protein sequences trained on a single H100 GPU for one week. Despite its modest size, LEMON outperforms state-of-the-art models ranging from 600M to 3B parameters. Our results demonstrate that evolution-informed tokenization can substitute for massive parameter scaling, opening a new direction for efficient, biologically-grounded protein representation learning. All code, model weights, and results are publicly available under the MIT license.
Sources
- Training Compute-Optimal Large Language Models
- Scaling Laws for Neural Language Models
- Protein Structure Tokenization via Geometric Byte Pair Encoding
- PUMA: Learning a Mutation-Aware Vocabulary of Protein Units
- GLU Variants Improve Transformer
- Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- Ankh: Optimized Protein Language Model Unlocks General-Purpose Modelling
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks