ViroGym: Realistic Large-Scale Benchmarks for Evaluating Viral Proteins
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.
Ines: We're looking at the paper: "ViroGym: Realistic Large-Scale Benchmarks for Evaluating Viral Proteins".
Marcus: Here's the gist of the paper. Protein language models (pLMs) have shown strong potential for zero-shot prediction of missense variant effects, yet systematic benchmarking on viral proteins remains limited, a critical gap given the need for proactive tools that can anticipate emerging mutations ahead of experimental validation Protein language models (pLMs) have shown strong potential for zero-shot prediction of missense variant effects, yet systematic benchmarking on viral proteins remains limited, a critical gap given the need for proactive tools that can anticipate emerging mutations ahead of experimental validation.
Ines: ViroGym Benchmark Scope ViroGym introduces a comprehensive benchmark evaluating pLMs across three tasks: Mutational effect prediction, Antigenic diversity prediction, and Pandemic prediction.
Marcus: The benchmark consists of 79 deep mutational scanning (DMS) assays covering eukaryotic viruses with 552,065 mutated sequences across 7 phenotypic readouts, 21 influenza neutralisation tasks, and a real-world pandemic prediction task for SARS-CoV-2.
Ines: ViroGym includes over 552,065 mutated sequence readouts, 2,691 viral sequence–titer pairs, and 24,187 naturally occurring single-mutation frequency measurements.
Paper summary: Ines: So, to unpack this ViroGym benchmark, the authors are setting up these assays using seventy-nine deep mutational scanning or DMS assays across eukaryotic viruses <ref:2603.06740#pg1>.
Marcus: These DMS assays are really high-throughput experimental techniques that systematically test millions of single amino acid variants and map each mutation to a measured functional property.
Ines: And they’ve done this across five hundred fifty-two thousand sixty-five mutated sequences for these viruses, with seven different types of outcomes being measured <ref:2603.06740#pg1>.
Marcus: On top of that, they’re including twenty-one tasks for influenza neutralization and even a real-world pandemic prediction task for SARS-CoV two.
Yuki: That scale is pretty big. Having over five hundred fifty-two thousand sixty-five mutated sequence readouts and twenty-four thousand one hundred eighty-seven naturally occurring single mutation frequency measurements gives you a lot of data to work with on the evolution side.
Ines: They also built in some other data sources for their process, including neutralisation assays and genomic surveillance from the GISAID database for SARS-CoV two.
Marcus: The paper claims that this benchmark helps us see how well pLMs perform on fitness landscapes, antigenic diversity, and pandemic forecasting.
Yuki: It’s interesting because they are focusing on single-sequence pLMs that can be applied without needing a multiple sequence alignment from a database like GISAID.
Ines: That’s one of the key points they make about their focus, contrasting it with other models that rely on those alignments to predict immune escape.
Conclusion: Marcus: So, looking at the ViroGym paper, the main finding is that the ProGen two family of pLMs consistently performed the best across all three tasks they set up.
Ines: Specifically, ProGen two-XL ranked first on both aggregate metrics and it significantly outperformed twenty-one other competing models on recall in that mutational effect prediction task.
Yuki: That performance on the pandemic prediction task is also quite telling; ProGen two-XL achieved a mean Top ten percent recall of zero point three nine and a mean AUROC of zero point seven four across five different variant eras.
Marcus: The authors draw a central conclusion that in vitro and in silico signals, meaning those lab tests and the computer models, are complementary rather than redundant when it comes to predicting real-world emergence.
Ines: They found that performance on the DMS and neutralisation benchmarks predicts which models can generalize to real-world emergence even though the mutation sets they see barely overlap.
Yuki: That low overlap between those different sources of signal is actually a good thing because each one captures a distinct facet of viral fitness that the others miss.
Marcus: They also point out that their work suggests this finding about DMS-selected pLMs transferring well to real-world mutation prediction should be interpreted as being specific to SARS-CoV two.
Ines: And they flag a limitation, saying that cross-viral extension of the pandemic prediction task is currently infeasible because it’s tied specifically to SARS-CoV two Spike protein mutations.
Yuki: Plus, they acknowledge that handling insertions and deletions, or indels, which often disrupt protein function, is still a key challenge for these pLMs.
Marcus: Overall, the paper suggests that combining fitness signals with antigenic and evolutionary signals is a principled strategy for prioritizing mutations and forecasting emerging variants.
Yichen Zhou, Jonathan Golob, Amir Karimi, *Stefan Bauer*, *Patrick Schwab*
GlaxoSmithKline · Technical University of Munich · University of Washington
q-bio.QM, cs.AI
Submitted: 2026-03-06
Updated: 2026-11-13
Code: https://github.com/GSK-AI/viroGym
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 100/100
The gist: Protein language models (pLMs) have shown strong potential for zero-shot prediction of missense variant effects, yet systematic benchmarking on viral proteins remains limited…
Key concepts
- Deep Mutational Scanning (DMS)
- DMS is an experimental technique used to test how changing a single amino acid in a protein affects its function. Scientists systematically test millions of variants and measure the resulting functional changes, providing concrete data on mutation effects.
- Antigenic Diversity Prediction
- This task evaluates how well pLMs can predict the variety of viral strains that might emerge based on their sequences. It tests a model's ability to understand the structural features that determine how different viruses interact with the immune system.
- Pandemic Prediction
- This involves using data, including genomic surveillance from databases like GISAID, to predict potential global outbreaks. The benchmark measures a model's ability to forecast when a virus might become a pandemic threat based on its mutation patterns.
Terminology
Summary
Protein language models (pLMs) have shown strong potential for zero-shot prediction of missense variant effects, yet systematic benchmarking on viral proteins remains limited, a critical gap given the need for proactive tools that can anticipate emerging mutations ahead of experimental validation Protein language models (pLMs) have shown strong potential for zero-shot prediction of missense variant effects, yet systematic benchmarking on viral proteins remains limited, a critical gap given the need for proactive tools that can anticipate emerging mutations ahead of experimental validation. p. 0 ViroGym: Realistic Large-Scale Benchmarks for
Evaluating Viral Proteins
Yichen Zhou
GlaxoSmithKline; Technical University of Munich
yichen. com
Jonathan Golob
GlaxoSmithKline; University of Washington
Amir Karimi
KTH Royal Institute of Technology
Stefan Bauer∗
Technical University of Munich
Patrick Schwab∗
GlaxoSmithKline
Abstract
Protein language models (pLMs) have shown strong potential for zero-shot prediction of missense variant effects, yet systematic benchmarking on viral proteins
remains limited, a critical gap given the need for proactive tools that can anticipate emerging mutations ahead of experimental validation.Where the paper says it
ViroGym Benchmark Scope
ViroGym introduces a comprehensive benchmark evaluating pLMs across three tasks: Mutational effect prediction, Antigenic diversity prediction, and Pandemic prediction. The benchmark consists of 79 deep mutational scanning (DMS) assays covering eukaryotic viruses with 552,065 mutated sequences across 7 phenotypic readouts, 21 influenza neutralisation tasks, and a real-world pandemic prediction task for SARS-CoV-2. p. 0 com
Jonathan Golob
GlaxoSmithKline; University of Washington
Amir Karimi
KTH Royal Institute of Technology
Stefan Bauer∗
Technical University of Munich
Patrick Schwab∗
GlaxoSmithKline
Abstract
Protein language models (pLMs) have shown strong potential for zero-shot prediction of missense variant effects, yet systematic benchmarking on viral proteins
remains limited, a critical gap given the need for proactive tools that can anticipate emerging mutations ahead of experimental validation. Here we introduce
ViroGym, a comprehensive benchmark evaluating pLMs across three tasks: 79
deep mutational scanning (DMS) assays covering eukaryotic viruses with 552,065
mutated sequences across 7 phenotypic readouts, 21 influenza neutralisation tasks,
and a real-world pandemic prediction task for SARS-CoV-2. • Pandemic prediction, which identifies models with strong zero-shot generalization suitable
for modelling mutations observed in natural viral evolution. ViroGym includes over 552,065 mutated sequence readouts, 2,691 viral sequence–titer pairs, and
24,187 naturally occurring single-mutation frequency measurements. Of the DMS entries, 501,984
are single amino acid substitutions and 3,188 are deletions or indels (see Table 7 in Appendix A. It
spans 13 virus types and 7 phenotypic categories, providing broad coverage across viral families and
functional properties (see Table 8 in Appendix A.Where the paper says it
Where the paper says it
Where the paper says it
Data Construction Details
The dataset process involves several curated components, including DMS assays, neutralisation assays, and GISAID database data.
-
DMS is a high-throughput experimental technique that characterizes a protein’s functional landscape by systematically evaluating millions of its single–amino-acid variants and mapping each mutation (genotype) to a measured functional property (phenotype). p. 2
Where the paper says it
1 Dataset sources DMS. DMS is a high-throughput experimental technique that characterizes a protein’s functional landscape by systematically evaluating millions of its single–amino-acid variants and mapping each mutation (genotype) to a measured functional property (phenotype) Fowler and Fields [2014].
-
The neutralisation assay quantifies serum antibody using neutralisation titers across all relevant viral strains within a single experiment, enabling models to jointly learn over viral sequence variation and antigenic response. p. 2
Where the paper says it
Neutralisation assay. In contrast to traditional serological assays, which assess antibody neutralisation against a single viral strain per serum sample, sequence-based high-throughput neutralisation assays quantify serum antibody using neutralisation titers across all relevant viral strains within a single experiment Loes et al.
-
The GISAID database provides genomic surveillance data for SARS-CoV-2, where mutations are labelled emergent if their frequency rose by at least 10−4 globally or if they achieved a ≥10× fold-increase from a non-zero baseline. p. 2
Where the paper says it
ViroGym builds on these insights by evaluating whether pLMs can recapitulate and extend this predictive signal without experimental data at inference time, and by systematically comparing pLM-based predictions against DMS assays and GISAID surveillance data. 3 ViroGym The benchmark comprises 79 DMS assays, 21 sequencing-based neutralisation assays for influenza A, and a real-world prediction task derived from the Global Initiative on Sharing All Influenza Data (GISAID), which provides genomic surveillance data for SARS-CoV-2.
Model Performance Findings
The ProGen2 family consistently achieves the strongest performance across all three tasks in ViroGym. Specifically, ProGen2-XL ranks first on both aggregate metrics and significantly outperforms 21 of 24 competing models on recall in the mutational effect prediction task. p. 4 1 Table 14). Under this analysis,
ProGen2-XL ranks first on both aggregate metrics and significantly outperforms 21 of 24 competing
models on recall; the three non-significant comparisons involve the closest ProGen2 variants (Base,
BFD90) and Tranception L, where differences are small. To further examine how this relationship evolves across variant eras, we compute for each era the
top-10% Recall of era-matched DMS binding assays and of ProGen2-XL against the is_emergent,
alongside the Jaccard similarity between the two top-10% prediction sets (Figure 3). ProGen2-XL
maintains recall between 30% and 75% throughout the pandemic, while DMS recall decays from 30%
in 2021 to 0% by 2025 KP. Figure 2 shows Top-10% Recall,
and AUROC for the best-performing variant from each model family across all five windows. ProGen2 and Tranception consistently rank above ESM-family and ProtGPT2 models across all
time windows, with ProGen2 XL achieving a mean Top-10% Recall of 0. A complementary question is whether the same ordering holds as the
dominant strain shifts from Wuhan-Hu-1 through successive variant eras. To assess this, we perform a
pseudo-temporal validation across five variant eras (2021 Wuhan-Hu-1, 2022 Alpha, 2023 BA.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Complementary Signals
The central finding from ViroGym is that in vitro and in silico signals are complementary, not redundant, as performance on DMS and neutralisation benchmarks predicts which models generalise to real-world emergence even though the mutation sets they surface barely overlap. The low overlap among DMS, neutralisation, pLM, and GISAID signals is a feature because each source captures a distinct facet of viral fitness that the others miss. p. 8 The central finding from ViroGym is that in vitro and in silico signals are complementary, not
redundant: performance on DMS and neutralisation benchmarks predicts which models generalise to
real-world emergence, even though the mutation sets they surface barely overlap. The low overlap
among DMS, neutralisation, pLM, and GISAID signals is not a limitation but a feature, as each
source captures a distinct facet of viral fitness that the others miss.Where the paper says it
Future Directions
Key challenges for pLMs include handling insertions and deletions (indels), which often disrupt protein function. Additionally, viral proteins frequently exceed the typical length of proteins, suggesting that pLMs could achieve improved performance by expanding their token representations and increasing context length. p. 8 Lastly, viral proteins frequently exceed the typical length of proteins,
whereas most pLMs are limited to context windows of fewer than 1024 residues. These observations
suggest that pLMs could achieve improved performance by expanding their token representations,
incorporating larger and more diverse training datasets, and increasing context length to better capture
long-range dependencies in viral proteins. Reliable, large-scale mutation frequency data
for Influenza A and other viruses are substantially more difficult to obtain from GISAID, making
cross-viral extension of this task currently infeasible. As a result, the finding that DMS-selected
pLMs transfer well to real-world mutation prediction should be interpreted as SARS-CoV-2-specific;
9Where the paper says it
Where the paper says it
Data Availability
The data and code supporting this study are publicly available on GitHub at https://github.com/GSK-AI/viroGym. p. 9 Data and Code Availability
The data and code supporting this study are publicly available on GitHub at https://github. com/
GSK-AI/viroGym. whether this complementarity between DMS and pLM predictions generalizes to other viral families
remains an important direction for future work. Similarly, pseudo-temporal validation across SARSCoV-2 variant eras (2020–2024) provides encouraging evidence of temporal generalization, but proper
lineage-level temporal forecasting, which must account for epistatic context and lineage emergence
dynamics, remains a complex challenge that we leave to future work. The low overlap
among DMS, neutralisation, pLM, and GISAID signals is not a limitation but a feature, as each
source captures a distinct facet of viral fitness that the others miss. Combining fitness, antigenic, and
evolutionary signals is therefore a principled strategy for mutation prioritisation, antigen selection,
and early warning of emerging variants. Therefore, given the time efficiency of early vaccine development and data
leakage risks, we mainly focus on single sequence-based pLMs in ViroGym. To minimise the risk
of training data contamination, we deliberately include only well-established pLM versions with
documented pre-training cutoffs prior to 2021 (e. 5) through the 2025
KP. 3 era, demonstrating that the predictive signal does not degrade sharply as the reference sequence
drifts from Wuhan-Hu-1. ESM-family and ProtGPT2 models trail consistently, with mean recalls
below 0. 25, suggesting that models pre-trained on large and diverse protein databases (ProGen2,
Tranception) are better calibrated to emergent mutation frequency than those trained on curated or
narrower sequence sets. 2 Figure 10). The low DMS–GISAID overlap reflects the mechanistic gap between
a single controlled experimental condition and the composite in vivo fitness that natural selection
acts on.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Limitations
A limitation of our work is that the pandemic prediction task and its associated real-world validation are specific to SARS-CoV-2 Spike protein mutations, making cross-viral extension of this task currently infeasible. Proper lineage-level temporal forecasting, which must account for epistatic context and lineage emergence dynamics, remains a complex challenge that is left to future work. p. 9 whether this complementarity between DMS and pLM predictions generalizes to other viral families
remains an important direction for future work. Similarly, pseudo-temporal validation across SARSCoV-2 variant eras (2020–2024) provides encouraging evidence of temporal generalization, but proper
lineage-level temporal forecasting, which must account for epistatic context and lineage emergence
dynamics, remains a complex challenge that we leave to future work.Where the paper says it
References
The references listed support the claims made throughout the paper. The references include works on Deep mutational scanning, MSA-based evolutionary models, DMS correlation studies, and DMS-guided viral forecasting. p. 2 ViroGym therefore focuses on single-sequence pLMs that can be applied without any
MSA, while acknowledging these MSA-based approaches as complementary tools. DMS-guided viral forecasting. DMS Correlation Studies. Livesey and Marsh Livesey and Marsh [2025] recently collected 13 new
DMS datasets from ProteinGym and evaluated 97 variant effect predictors (VEPs) across 36 human
proteins. EVE Frazer et al. [2021] fits a variational
autoencoder to a protein family’s MSA to assign unsupervised fitness scores, while EVEscape Thadani
et al. Beyond benchmarking, recent work has explored whether DMS measurements can directly forecast real-world viral evolution. Dadonaite et al. ViroGym therefore focuses on single-sequence pLMs that can be applied without any
MSA, while acknowledging these MSA-based approaches as complementary tools. DMS-guided viral forecasting. com
Jonathan Golob
GlaxoSmithKline; University of Washington
Amir Karimi
KTH Royal Institute of Technology
Stefan Bauer∗
Technical University of Munich
Patrick Schwab∗
GlaxoSmithKline
Abstract
Protein language models (pLMs) have shown strong potential for zero-shot prediction of missense variant effects, yet systematic benchmarking on viral proteins
remains limited, a critical gap given the need for proactive tools that can anticipate emerging mutations ahead of experimental validation. Here we introduce
ViroGym, a comprehensive benchmark evaluating pLMs across three tasks: 79
deep mutational scanning (DMS) assays covering eukaryotic viruses with 552,065
mutated sequences across 7 phenotypic readouts, 21 influenza neutralisation tasks,
and a real-world pandemic prediction task for SARS-CoV-2. Therefore, given the time efficiency of early vaccine development and data
leakage risks, we mainly focus on single sequence-based pLMs in ViroGym. To minimise the risk
of training data contamination, we deliberately include only well-established pLM versions with
documented pre-training cutoffs prior to 2021 (e. References
A. Aditham, C. Second,
lineage-level forecasting is complicated by the non-linear phylodynamics of SARS-CoV-2: successful
variants do not always descend from the currently dominant strain Markov et al. [2023], Carabelli et al. A familiar example is the current vaccine development system for SARS-CoV-2 and influenza, which
involves a semi-annual strain selection process recommended by the World Health Organization
(WHO). This production system, especially for seasonal influenza vaccines, has remained largely
unchanged for over 40 years Wei et al. 2098
ESM2 0. 2 Antigenic diversity prediction
The current influenza vaccine strains are selected based on the degree to which circulating viruses
have drifted from previously dominant strains, with this distance assessed by integrating both genetic
and antigenic evolution Smith et al. ESM-family and ProtGPT2 models trail consistently, with mean recalls
below 0. 25, suggesting that models pre-trained on large and diverse protein databases (ProGen2,
Tranception) are better calibrated to emergent mutation frequency than those trained on curated or
narrower sequence sets.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Appendix Details
The benchmark construction details are provided in Appendix A.1 for DMS datasets and Table 8 for virus/phenotype distribution. p. 2 Beyond the functional categories
considered in ProteinGym, ViroGym introduces an additional function type: immune escape, which
represents a critical phenotype for viral proteins and is directly relevant to vaccine and therapeutic
development. Of the 79 DMS assays, 23 overlap with ProteinGym while 56 are newly curated,
including all immune escape assays and all datasets published in recent years. Across all three evaluation metrics
(Table 4), ProGen2-XL achieves the strongest performance, consistent with its dominance on the DMS
benchmark (Table 2) and competitive results on neutralisation prediction (Table 3). Tranception is the
consistently second-best family across all three tasks, leading the neutralisation benchmark (Spearman
0. Metrics reported for all baselines include Top
10% Recall, absolute Spearman’s rank correlation, and Precision@3 between model scores and
mutation frequencies from GISAID. Evaluated on 24,187 substitution mutations (deletion mutations
excluded for fair cross-model comparison, as VESPA/VESPAl cannot score deletions; results on all
25,460 mutations excluding VESPA/VESPAl are in Appendix Table 12). The
selection of DMS assays in ViroGym follows the guidelines established by ProteinGym. As a result,
ViroGym includes DMS assays covering SARS-CoV-2, Influenza A, HIV and 10 other viruses
(Detailed reference can be found in Table 5 and 6 of Appendix A. Influenza
neutralisation titers are residualised against serum effects before export. Full processing and QC
details are provided in Appendix C. The
selection of DMS assays in ViroGym follows the guidelines established by ProteinGym. As a result,
ViroGym includes DMS assays covering SARS-CoV-2, Influenza A, HIV and 10 other viruses
(Detailed reference can be found in Table 5 and 6 of Appendix A. 4 Evaluation metrics
We use two ranking-based metrics: absolute Spearman rank correlation to measure agreement
between predicted and experimental rankings Notin et al. [2023], and top-10% recall to focus
evaluation on the highest-impact mutations (following ProteinGym convention).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Evaluate protein language models on fitness landscapes by scoring amino acid sequences under suitable strategies, as demonstrated by ProGen2 family achieving
the strongest performance across all three tasks
in ViroGym. This allows pLMs to be benchmarked against real-world emergence and evolutionary constraints for vaccine and antiviral development. -
Enable proactive vaccine design by using pLMs to estimate the functional impact and fitness consequences of candidate mutations without requiring prior evolutionary or epidemiological information, as noted by
LLMs trained to predict amino acid sequences, known as pLMs, have had success with estimating the functional impact and fitness consequences of candidate mutations.
-
Improve variant prioritization by combining in vitro and in silico signals; the central finding is that "in vitro and in silico signals are complementary, not redundant: performance on DMS and neutralisation benchmarks predicts which models generalise to real-world emergence, even though the mutation sets they surface barely overlap."
Abstract
Protein language models (pLMs) have shown strong potential for zero-shot prediction of missense variant effects, yet systematic benchmarking on viral proteins remains limited, a critical gap given the need for proactive tools that can anticipate emerging mutations ahead of experimental validation. Here we introduce ViroGym, a comprehensive benchmark evaluating pLMs across three tasks: 79 deep mutational scanning (DMS) assays covering eukaryotic viruses with 552,065 mutated sequences across 7 phenotypic readouts, 21 influenza neutralisation tasks, and a real-world pandemic prediction task for SARS-CoV-2. We benchmark well-established pLMs on fitness landscapes, antigenic diversity, and pandemic forecasting, and find that the ProGen2 family consistently achieves the strongest performance across all three tasks. Crucially, DMS and neutralisation performance reliably identifies models that generalise to real-world emergence, even though the mutation sets they surface barely overlap, revealing that complementary in vitro benchmarks capture the evolutionary constraints needed for real-world mutation forecasting. The data and code supporting this study are publicly available on GitHub at https://github.com/GSK-AI/viroGym
Sources
Related papers
- CellCanon: An Agentic Cellular World Model for Prediction, Explanation, and Observation-Driven Updating
- Probing and steering biology across Boltz-1s trunk-diffusion boundary
- Trajectory Inference of Human Aging from Cross-Sectional DNA Methylation Data
- Bridging the Modality Bottleneck in Pathology MIL through Molecularly Guided Feature Adaptation
- Molecular notation can act as a shortcut in AI benchmark evaluation
- Towards Efficient Auto-Interpretability of Boltz-1 Sparse Autoencoders