ViroGym: Realistic Large-Scale Benchmarks for Evaluating Viral Proteins
summary
The gist
Protein language models (pLMs) have shown strong potential for zero-shot prediction of missense variant effects, yet systematic benchmarking on viral proteins remains limited…
In short
ViroGym is a new benchmark testing protein language models (pLMs) on viral proteins using three tasks: predicting missense effects, antigenic diversity, and pandemic risk. The study found that ProGen2-XL performs best across all tasks. It shows that in vitro and in silico signals complement each other for predicting real-world viral emergence.
Key concepts
- Deep Mutational Scanning (DMS)
- DMS is an experimental technique used to test how changing a single amino acid in a protein affects its function. Scientists systematically test millions of variants and measure the resulting functional changes, providing concrete data on mutation effects.
- Antigenic Diversity Prediction
- This task evaluates how well pLMs can predict the variety of viral strains that might emerge based on their sequences. It tests a model's ability to understand the structural features that determine how different viruses interact with the immune system.
- Pandemic Prediction
- This involves using data, including genomic surveillance from databases like GISAID, to predict potential global outbreaks. The benchmark measures a model's ability to forecast when a virus might become a pandemic threat based on its mutation patterns.
Terminology used across episodes
This episode discusses
- ViroGym: Realistic Large-Scale Benchmarks for Evaluating Viral Proteins · Paper Radio
- Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval
The paper
ViroGym: Realistic Large-Scale Benchmarks for Evaluating Viral Proteins · Read on arXiv
Yichen Zhou, Jonathan Golob, Amir Karimi, *Stefan Bauer*, *Patrick Schwab*
GlaxoSmithKline · Technical University of Munich · University of Washington
Protein language models (pLMs) have shown strong potential for zero-shot prediction of missense variant effects, yet systematic benchmarking on viral proteins remains limited, a critical gap given the need for proactive tools that can anticipate emerging mutations ahead of experimental validation. Here we introduce ViroGym, a comprehensive benchmark evaluating pLMs across three tasks: 79 deep mutational scanning (DMS) assays covering eukaryotic viruses with 552,065 mutated sequences across 7 phenotypic readouts, 21 influenza neutralisation tasks, and a real-world pandemic prediction task for SARS-CoV-2. We benchmark well-established pLMs on fitness landscapes, antigenic diversity, and pandemic forecasting, and find that the ProGen2 family consistently achieves the strongest performance across all three tasks. Crucially, DMS and neutralisation performance reliably identifies models that generalise to real-world emergence, even though the mutation sets they surface barely overlap, revealing that complementary in vitro benchmarks capture the evolutionary constraints needed for real-world mutation forecasting. The data and code supporting this study are publicly available on GitHub at https://github.com/GSK-AI/viroGym
Transcript
Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.
Ines: We're looking at the paper: "ViroGym: Realistic Large-Scale Benchmarks for Evaluating Viral Proteins".
Marcus: Here's the gist of the paper. Protein language models (pLMs) have shown strong potential for zero-shot prediction of missense variant effects, yet systematic benchmarking on viral proteins remains limited, a critical gap given the need for proactive tools that can anticipate emerging mutations ahead of experimental validation Protein language models (pLMs) have shown strong potential for zero-shot prediction of missense variant effects, yet systematic benchmarking on viral proteins remains limited, a critical gap given the need for proactive tools that can anticipate emerging mutations ahead of experimental validation.
Ines: ViroGym Benchmark Scope ViroGym introduces a comprehensive benchmark evaluating pLMs across three tasks: Mutational effect prediction, Antigenic diversity prediction, and Pandemic prediction.
Marcus: The benchmark consists of 79 deep mutational scanning (DMS) assays covering eukaryotic viruses with 552,065 mutated sequences across 7 phenotypic readouts, 21 influenza neutralisation tasks, and a real-world pandemic prediction task for SARS-CoV-2.
Ines: ViroGym includes over 552,065 mutated sequence readouts, 2,691 viral sequence–titer pairs, and 24,187 naturally occurring single-mutation frequency measurements.
Paper summary: Ines: So, to unpack this ViroGym benchmark, the authors are setting up these assays using seventy-nine deep mutational scanning or DMS assays across eukaryotic viruses <ref:2603.06740#pg1>.
Marcus: These DMS assays are really high-throughput experimental techniques that systematically test millions of single amino acid variants and map each mutation to a measured functional property.
Ines: And they’ve done this across five hundred fifty-two thousand sixty-five mutated sequences for these viruses, with seven different types of outcomes being measured <ref:2603.06740#pg1>.
Marcus: On top of that, they’re including twenty-one tasks for influenza neutralization and even a real-world pandemic prediction task for SARS-CoV two.
Yuki: That scale is pretty big. Having over five hundred fifty-two thousand sixty-five mutated sequence readouts and twenty-four thousand one hundred eighty-seven naturally occurring single mutation frequency measurements gives you a lot of data to work with on the evolution side.
Ines: They also built in some other data sources for their process, including neutralisation assays and genomic surveillance from the GISAID database for SARS-CoV two.
Marcus: The paper claims that this benchmark helps us see how well pLMs perform on fitness landscapes, antigenic diversity, and pandemic forecasting.
Yuki: It’s interesting because they are focusing on single-sequence pLMs that can be applied without needing a multiple sequence alignment from a database like GISAID.
Ines: That’s one of the key points they make about their focus, contrasting it with other models that rely on those alignments to predict immune escape.
Conclusion: Marcus: So, looking at the ViroGym paper, the main finding is that the ProGen two family of pLMs consistently performed the best across all three tasks they set up.
Ines: Specifically, ProGen two-XL ranked first on both aggregate metrics and it significantly outperformed twenty-one other competing models on recall in that mutational effect prediction task.
Yuki: That performance on the pandemic prediction task is also quite telling; ProGen two-XL achieved a mean Top ten percent recall of zero point three nine and a mean AUROC of zero point seven four across five different variant eras.
Marcus: The authors draw a central conclusion that in vitro and in silico signals, meaning those lab tests and the computer models, are complementary rather than redundant when it comes to predicting real-world emergence.
Ines: They found that performance on the DMS and neutralisation benchmarks predicts which models can generalize to real-world emergence even though the mutation sets they see barely overlap.
Yuki: That low overlap between those different sources of signal is actually a good thing because each one captures a distinct facet of viral fitness that the others miss.
Marcus: They also point out that their work suggests this finding about DMS-selected pLMs transferring well to real-world mutation prediction should be interpreted as being specific to SARS-CoV two.
Ines: And they flag a limitation, saying that cross-viral extension of the pandemic prediction task is currently infeasible because it’s tied specifically to SARS-CoV two Spike protein mutations.
Yuki: Plus, they acknowledge that handling insertions and deletions, or indels, which often disrupt protein function, is still a key challenge for these pLMs.
Marcus: Overall, the paper suggests that combining fitness signals with antigenic and evolutionary signals is a principled strategy for prioritizing mutations and forecasting emerging variants.
More episodes
- 2610.33881-Sensorimotor behavior from an untrained brain-and-nerve-cord connectome
- 2610.40344-Morphological Features of Murine Pulmonary Arterial Networks for Diagnosing Vascular Disease
- Daily Summary for 2026-11-19
- 2609.10193-Are You Learning Biological Signal or Shortcuts? Auditing and Mitigating Bias in Protein-Protein Interaction Datasets
- 2610.38236-Generative discovery of chemical reaction networks
- 2507.11176-An Interpretable AI framework Quantifying Traditional Chinese Medicine Principles Towards Enhancing and Integrating with Modern Biomedicine
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2504.03590-Towards a Unified Framework for Determining Conformational Ensembles of Disordered Proteins
- 2603.15217-A multiscale discrete-to-continuum framework for structured population models