RAGenome: Scaling Retrieval-Based Genomic Language Models to Long Contexts

The gist The first retrieval-based gLM that scales pretraining to longer contexts (100× longer than existing MSA-based gLMs) allows it to capture both across-species evolutionary relationships and within-species longer-range interactions.

RAGenome Architecture and Retrieval Mechanism

RAGenome is introduced as the first retrieval-based gLM that scales pretraining to long contexts without sacrificing the explicit modeling of evolutionary constraints (Page 4). The retrieval mechanism involves taking a sequence from the query species and retrieving homologous sequences from a Whole-Genome Alignment (WGA) of 100 vertebrates (Page 4). A key inefficiency in current MSA-based gLMs is that they naively process every input token, while WGAs are highly dominated by uninformative gap tokens that correspond to unaligned segments (Page 4). To make retrieval-based pretraining over the whole genome efficient, RAGenome discards gap tokens entirely and keeps only aligned nucleotides (Page 4). It preserves the original alignment column of each token and uses it as its position in rotary position embeddings (RoPE) (Page 4). Inspired by a similar retrieval technique in pLMs, RAGenome concatenates all the sequences in a single array and distinguishes between them by adding taxonomy embeddings for each species (Page 4).

Taxonomy Embeddings

RAGenome includes species information by adding taxonomy embeddings to each token representation (Page 4). This is achieved by mapping each species k to its NCBI taxonomic lineage, spanning superkingdom, kingdom, phylum, class, order, family, genus, and species (Page 4). A single shared embedding for each species is obtained by concatenating the nominal embeddings of each taxonomy level it belongs to (Page 4). This design enables phylogenetically related species to share common information through their taxonomy embeddings (Page 13). A UMAP projection of the learned taxonomy embeddings confirms that phylogenetically related species cluster together in the embedding space (Page 13).

Transformer Architecture

RAGenome is a multi-head Transformer where standard self-attention layers are replaced with two types of attention layers tailored to capture within-species and cross-species dependencies (Page 4). Intra-species local attention replaces full self-attention with local (sliding window) attention, where the cost scales linearly instead of quadratically with sequence length (Page 4). To enable the model to directly incorporate evolutionary information from the retrieved homologous sequences, inter-species cross-attention layers are inserted in the middle of the network (Page 4). Cross-attention at position i is computed as: CrossAttn(h0i, h1:N)i = softmax q0i(K1:N)⊤√d V1:N (Page 4).

Pretraining Strategy and Scaling

During pretraining of RAGenome, Homo sapiens are set as the query species and homologous sequences are retrieved from a WGA of 100 vertebrates, excluding species in the same clade as the query (Page 4). The model is optimized using a masked language modeling objective where short contiguous spans of nucleotides (3 to 6 nucleotides) are masked together (Page 4). The target segment length L is set to 13,312 nucleotides, selected as a multiple of the local attention window size w = 512 that is long enough to capture the gene structures and splice-site interactions (Page 4). To account for the retrieval mechanism as well, RAGenome applies a context extension strategy by gradually increasing the input context length to improve stability and reduce training time (Page 4). This involves four stages: Short conserved regions, Coverage extension, Context extension, and Full-length context (Page 14).

Performance on Downstream Tasks

RAGenome is evaluated on two complementary downstream tasks: Gene finding and Variant effect prediction (Page 5). For gene finding from BEND, RAGenome achieves an MCC of 0.60, substantially outperforming GPN-Star (MCC 0.45) and narrowing the remaining gap to large-scale gLMs (Page 8). For variant effect prediction from ClinVar, RAGenome achieves an AUROC of 0.87, surpassed only by the MSA-based models GPN-MSA (0.90) and GPN-Star (0.92) (Page 15). The results indicate that RAGenome does not trade one capability for the other: the same model achieves strong long-range understanding without sacrificing the short-range, evolutionary-driven performance that makes MSA-based gLMs effective (Page 8).

Scaling and Ablation Results

The performance of RAGenome on gene finding improves substantially after each pretraining stage, with the largest gains occurring immediately after each transition (Page 8). Variant effect prediction remains flat across all pretraining stages, indicating that its reliance on local evolutionary conservation signals is preserved (Page 8). In inference, performance improves monotonically with inference context length across all checkpoints, and the fully-scaled model achieves the best performance (Page 14). Furthermore, ablation on retrieval budget B shows that variant effect prediction saturates by B = 40,000 (Page 15). RAGenome is presented as the first gLM to unify explicit evolutionary modeling with long-range genomic understanding within a single, efficient architecture (Page 10).

Limitations

RAGenome requires a WGA at inference and cannot currently be applied to sequences or species for which no alignment of any sort is available (Page 9). While the architecture supports any query species and alignment, applying it to a new one would likely require additional pretraining (Page 9). The authors trained a single 168M-parameter RAGenome model and leave scaling with model size and data to future work (Page 9).

Conclusion

In this work, RAGenome is the first retrieval-based gLM that scales to input context up to 13,312 nucleotides (100× longer than MSA-based methods) by using an efficient retrieval mechanism that removes gap tokens from the MSA (Page 10). RAGenome surpasses GPN-Star, the strongest MSA-based gLM, on gene finding, demonstrating strong long-range capabilities (Page 14). At the same time, it outperforms much larger gLMs such as NT-MS and Evo2 on variant effect prediction by explicitly modeling evolutionary signals through an MSA (Page 15). Together, these results suggest that RAGenome is the first gLM to unify explicit evolutionary modeling with long-range genomic understanding within a single, efficient architecture (Page 10). The authors expect that further scaling of RAGenome in model size, training data and context length will continue to close the remaining gap to large-scale gLMs, while preserving the efficiency benefits of a much smaller, retrieval-based approach (Page 10).

AI Use Statement

In this work, we used generative AI tools for drafting and editing text throughout the manuscript and suggesting paper structure, titles, and keywords (Page 10). The authors have not used generative AI tools to develop theoretical models or conceptual frameworks, propose or refine hypotheses, design or give feedback on research methodology or experiments, implement methods, or support qualitative or thematic data analysis (Page 10). All experimental results, tables, and figures were produced independently by the authors (Page 10).

Ethics Statement

This work uses only publicly available genomic data: the human reference genome, a whole-genome alignment (Ye et al., 2026), and the public BEND (Marin et al., 2024) and ClinVar (Landrum et al., 2014) benchmarks, none of which contain individually identifiable data (Page 10).

Reproducibility Statement

In Section 4, the authors present in detail the retrieval mechanism, architecture, and pretraining procedure of RAGenome, with full hyperparameters in Appendix A.4 (Page 10). All datasets for training and evaluating the model are publicly available (Page 10). Code is available at https://github.com/PanosAntoniadis/RAGenome and the trained model weights at https://huggingface.co/pantoniadis/RAGenome (Page 10).

Acknowledgments

Model training and inference was made possible on Gefion supercomputer by a pilot project from DCAI and a voucher grant awarded by the Novo Nordisk Foundation (NNF25OC0105165) (Page 10). Further support was provided from the Novo Nordisk Foundation through the MLSS Center (Basic Machine Learning Research in Life Science, NNF20OC0062606), CAZAI (DDB and OW, NNF22OC0077058), NovoSTAR programme (RD), and the Pioneer Centre for AI (DNRF grant number P1) (Page 13).

References

The references listed include works such as DNABERT: pre-trained bidirectional encoder representations from transformers model for DNA-language in genome (Page 13), GPNStar: Predicting genome-wide functional constraints with GPN-Star (Page 15), and Evo2: Genome modelling and design across all domains of life with Evo 2 (Page 13).

**Appendix A.

1. Bold header: Long-Context Gene Finding Capability Enhancement

RAGenome can now perform nucleotide-level 9-class classification for gene finding tasks, achieving an MCC of 0.60, which substantially outperforms both MSA-based methods (GPN-Star) and mid-scale gLMs on this capability because it captures long-range interactions by utilizing a context window up to 13,312 nucleotides.

2. Bold header: Unified Evolutionary Modeling and Long-Range Understanding

The model can simultaneously model evolutionary relationships and long-range genomic structures; this is achieved by the architecture's ability to alternate between intra-species local attention and inter-species cross-attention layers, allowing it to capture both dependencies at once.

3. Bold header: Efficient Scaling for Long Genomic Contexts

RAGenome can scale its pretraining context length up to 13,312 nucleotides using an efficient retrieval mechanism that discards gap tokens entirely and keeps only aligned nucleotides, effectively scaling the input context 100× longer than MSA-based methods while remaining more computationally efficient than large-scale models.

4. Bold header: Robust Variant Effect Prediction

The system maintains strong performance on prioritizing pathogenic variants, achieving an AUROC of 0.87, demonstrating that it efficiently models evolutionary signals despite the long input context, successfully unifying evolutionary modeling with long-range capabilities for this task.

The genome holds the blueprint that governs the biological properties of the cell. Consequently, advancing our knowledge of genomic function is crucial both for a broader understanding of biology and for continued biomedical advances. The success of large language models on natural language and protein sequences has motivated similar efforts on genomic data. However, standard genomic language models (gLMs) often require extremely large computational resources and still fall behind traditional methods on some downstream tasks. Recently, MSA-based pretraining has been proposed as an efficient alternative, but existing models are limited to short input contexts, restricting their use to short-range tasks, such as variant effect prediction. In this work, we present RAGenome, the first retrieval-based gLM that scales pretraining to longer contexts (100 times longer than existing MSA-based gLMs), allowing it to capture both across-species evolutionary relationships and within-species longer-range interactions. Trained on whole-genome alignments from 100 vertebrates, RAGenome substantially improves the long-range capabilities of MSA-based gLMs, raising gene finding performance from 0.45 to 0.60, while remaining competitive on purely evolutionary-based tasks like prioritizing pathogenic variants. RAGenome provides competitive gLM performance at a fraction of the training cost, unifying evolutionary modeling and long-range capabilities within a single, flexible, scalable framework. Code is available at https://github.com/PanosAntoniadis/RAGenome.