VOICE: A Vision-Omics Foundation Model Integrating Direct and Retrieval-Based Prediction of In-situ Single-Cell Gene Expression
summary
The gist
Spatial transcriptomics can resolve gene expression at single-cell resolution, but it is costly, limited to targeted panels of a few hundred to thousand genes…
In short
VOICE is a multimodal foundation model that predicts single-cell gene expression from H&E images using paired Xenium data. It achieves this by aligning cell morphology with expression embeddings and fusing direct prediction (from morphology) and retrieval prediction (from similar cells) via per-gene weighting. This allows for robust, single-cell resolution gene expression prediction.
Key concepts
- Contrastive Learning
- This is a training method where the model learns to map similar inputs to similar outputs. In VOICE, it aligns cell morphology with its known gene expression profile by learning what visual features correspond to specific gene activity across millions of cells.
- Direct and Retrieval Branches
- The model has two ways to predict expression: one directly regresses expression from the cell's H&E morphology. The other retrieves measured expression from similar reference cells. Fusing these two predictions allows VOICE to select the most reliable prediction for each gene.
- Per-Gene Weighted Fusion
- Instead of using a single average, VOICE assigns a unique weight ($eta_g$) to the direct and retrieval predictions for every individual gene. This weighting is optimized per tissue, enabling the model to intelligently combine the two prediction methods based on which one performs better for that specific gene.
- SE(2)-Equivariant Transformer
- This component uses a specialized neural network architecture designed to understand spatial relationships. It aggregates information from neighboring cells by considering both how similar their shapes are and how far apart they are spatially, improving the model's awareness of the local tissue environment.
Terminology used across episodes
This episode discusses
- VOICE: A Vision-Omics Foundation Model Integrating Direct and Retrieval-Based Prediction of In-situ Single-Cell Gene Expression · Paper Radio
- Masked Generative Transformer Is What You Need for Image Editing
- Dynamic Interaction-Aware and Causality-Disentangled Framework for Multimodal Sentiment Analysis
- Deformba: Vision State Space Model with Adaptive State Fusion
- Is Your Driving World Model an All-Around Player?
- A Multimodal Foundation Model of Spatial Transcriptomics and Histology for Biological Discovery and Clinical Prediction
The paper
VOICE: A Vision-Omics Foundation Model Integrating Direct and Retrieval-Based Prediction of In-situ Single-Cell Gene Expression · Read on arXiv
Xin Luo, Yicheng Tao, Haoxuan Zeng, Suyuan Wang, Chenzi Ouyang, Meiqi Zhu, Kai Liu, †Shuibing Chen†, Jie Liu†
University of Michigan · Weill Cornell Medicine
Spatial transcriptomics can resolve gene expression at single-cell resolution, but it is costly, limited to targeted panels of a few hundred to a few thousand genes, and applicable to only a small number of samples. H&E imaging, by contrast, is cheap and collected routinely at scale. This makes predicting single-cell expression directly from morphology a practical way to bring molecular analysis to large tissue archives. We therefore present VOICE, a multimodal foundation model that predicts single-cell gene expression from H&E images using paired Xenium data. VOICE first aligns cell centered H&E morphology from a pathology foundation model with single-cell expression embeddings from a transcriptome foundation model, trained using contrastive learning over 23 million cells. Next it predicts expression through two branches. One branch directly regresses expression from morphology. The other branch retrieves measured expression from similar reference cells, recovering genes that do not have morphological signal. Because genes vary in morphological predictability, VOICE fuses the two branches with a per-gene weight. After training, VOICE generalizes to heldout patients, slides, and partially overlapping gene panels from Xenium, and it consistently outperforms prior single-cell expression prediction methods on seven metrics.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "VOICE: A Vision-Omics Foundation Model Integrating Direct and Retrieval-Based Prediction of In-situ Single-Cell Gene Expression".
Jane: Here's the gist of the paper. Spatial transcriptomics can resolve gene expression at single-cell resolution, but it is costly, limited to targeted panels of a few hundred to thousand genes, and applicable to only a small number of samples <ref:2608.08366#pg2> The authors present VOICE, a multimodal foundation model that predicts single-cell gene expression from H&E images using paired Xenium data <ref:2608.08366#pg2>.
Tom: Model Architecture and Training Stages VOICE first aligns cell-centered H&E morphology with single-cell expression embeddings from a transcriptome foundation model through contrastive learning over 23 million cells <ref:2608.08366#pg3>.
Jane: Next, it predicts expression through two branches: one directly regresses expression from morphology, and the other retrieves measured expression from similar reference cells <ref:2608.08366#pg2>.
Tom: Because genes vary in morphological predictability, VOICE fuses the two branches with a per-gene weight <ref:2608.08366#pg2>.
Title and authors: Jane: So, what exactly is this paper saying about this VOICE model? What are its main contributions?
Tom: They're focusing on a few key improvements. First, they introduce panel-independent contrastive alignment, which means the way they link morphology to expression works even if you change the set of genes you're looking at.
Lu: That panel independence is important because it lets them align cell level morphology from UNI2-h with a frozen single cell foundation model expression embeddings using LoRA adaptation and a symmetric InfoNCE loss over twenty-three million cells from fifteen tissues and seventy-five slides <ref:2608.08366#pg1>.
Meng: twenty-three million cells sounds like a lot of data to train on, but that's what they need to make sure the alignment is robust across different types of tissue samples.
Jane: And they also have this two-branch prediction setup where one branch regresses gene expression directly from morphology, and the other retrieves expression from reference cells whose embeddings are closest to the query cell's.
Tom: That retrieval branch lets them recover genes even when there isn't a strong visual signal linking shape to that specific gene, which is smart because not all genes are predictable based on what they look like.
Lu: The model learns in three stages: first aligning morphology with expression through contrastive learning, then fine-tuning the image encoder against measured gene counts jointly with an SE two spatial decoder, and finally fusing the direct and retrieval predictions per gene.
Meng: Stage two sounds like where they really test the relationship between the cell's neighbors and its actual gene counts using that spatial decoder.
The paper's summary: Tom: So, to put it simply, VOICE is a model that predicts single-cell gene expression from H andE images. It does this by learning how cell shape relates to gene expression through two different prediction methods and then blending the results based on which method seems more reliable for a specific gene.
Jane: That sounds like they're trying to combine the strengths of both direct prediction and reference-based retrieval, depending on what the data suggests.
Lu: The fusion mechanism is where it gets interesting; they use a per-gene gate, which is a weight between the two branches that tells the model whether to trust the direct regression from morphology or pull information from similar reference cells.
Tom: They estimate these gene-specific weights by searching over a grid of values between zero and one for each tissue to maximize how well the final fused prediction matches the actual ground truth expression across all cells on that slide.
Jane: That per-gene weighting is crucial because genes aren't all predictable based on shape; some are, and some require context from other cells.
Meng: So, if we think about what this means for deploying it, it suggests they aren't just relying on one single prediction strategy for every gene across the whole slide.
The paper's improvements: Tom: The paper points out a few specific ways they improved things. They showed that combining the direct and retrieval predictions in Stage two gives them the highest value across every single metric on their test slides, which is a big win for them.
Jane: That suggests that just having one branch isn't enough; integrating both prediction types during the fine-tuning phase is what actually boosts performance.
Lu: They also showed that increasing the training scale from seven million to twenty-three million cells further improved performance, which supports the idea that large scale and heterogeneous cell level supervision is really valuable.
Meng: But there's a caveat, and they mention something about generalization; they can predict genes measured in a target tissue's test set that were never seen in any training slides from the same tissue.
Jane: That ability to predict genes from other tissues during testing, even if it wasn't explicitly trained on them, is what the authors call cross-tissue knowledge transfer.
Tom: They can transfer morphological knowledge of genes from one tissue to a gene measured in another tissue when that gene is only measured in the target tissue.
Conclusion: Jane: So, wrapping this up, VOICE is this multimodal foundation model that predicts single-cell gene expression from H andE images by using both direct regression and retrieval methods fused with a per-gene gate weighted average.
Tom: It achieves top performance across breast, lung, ovary, and pancreas slides when compared against the baselines they tested.
Lu: The key innovation is that it learns to align morphology from UNI2-h with the transcriptomic latent space through contrastive learning without needing specific gene panels for alignment.
Meng: From a production standpoint, running this model needs to handle that per-gene weighting estimation, which involves searching over a grid of values for each tissue, so you'd need some upfront computation to set up the final prediction weights.
Lalam: My internal assessment is that VOICE is quite confident in its ability to transfer knowledge across tissues because the mechanism allows it to learn these dependencies directly from the data structure, and I am very sure of this result.
Jane: It seems like a really practical tool for researchers who have cheap H andE images but don't have access to full single-cell sequencing data for every single gene in every sample.
Tom: Exactly. It shifts the bottleneck from expensive sequencing to affordable imaging, and that’s a significant move for spatial biology research.
More episodes
- 2603.28113-The multiplicative Lipschitz bound as an inductive bias
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound