VOICE: A Vision-Omics Foundation Model Integrating Direct and Retrieval-Based Prediction of In-situ Single-Cell Gene Expression

arXiv:2608.08366 · cs.CV, q-bio.GN · Submitted 2026-08-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "VOICE: A Vision-Omics Foundation Model Integrating Direct and Retrieval-Based Prediction of In-situ Single-Cell Gene Expression".

Jane: Here's the gist of the paper. Spatial transcriptomics can resolve gene expression at single-cell resolution, but it is costly, limited to targeted panels of a few hundred to thousand genes, and applicable to only a small number of samples <ref:2608.08366#pg2> The authors present VOICE, a multimodal foundation model that predicts single-cell gene expression from H&E images using paired Xenium data <ref:2608.08366#pg2>.

Tom: Model Architecture and Training Stages VOICE first aligns cell-centered H&E morphology with single-cell expression embeddings from a transcriptome foundation model through contrastive learning over 23 million cells <ref:2608.08366#pg3>.

Jane: Next, it predicts expression through two branches: one directly regresses expression from morphology, and the other retrieves measured expression from similar reference cells <ref:2608.08366#pg2>.

Tom: Because genes vary in morphological predictability, VOICE fuses the two branches with a per-gene weight <ref:2608.08366#pg2>.

Title and authors: Jane: So, what exactly is this paper saying about this VOICE model? What are its main contributions?

Tom: They're focusing on a few key improvements. First, they introduce panel-independent contrastive alignment, which means the way they link morphology to expression works even if you change the set of genes you're looking at.

Lu: That panel independence is important because it lets them align cell level morphology from UNI2-h with a frozen single cell foundation model expression embeddings using LoRA adaptation and a symmetric InfoNCE loss over twenty-three million cells from fifteen tissues and seventy-five slides <ref:2608.08366#pg1>.

Meng: twenty-three million cells sounds like a lot of data to train on, but that's what they need to make sure the alignment is robust across different types of tissue samples.

Jane: And they also have this two-branch prediction setup where one branch regresses gene expression directly from morphology, and the other retrieves expression from reference cells whose embeddings are closest to the query cell's.

Tom: That retrieval branch lets them recover genes even when there isn't a strong visual signal linking shape to that specific gene, which is smart because not all genes are predictable based on what they look like.

Lu: The model learns in three stages: first aligning morphology with expression through contrastive learning, then fine-tuning the image encoder against measured gene counts jointly with an SE two spatial decoder, and finally fusing the direct and retrieval predictions per gene.

Meng: Stage two sounds like where they really test the relationship between the cell's neighbors and its actual gene counts using that spatial decoder.

The paper's summary: Tom: So, to put it simply, VOICE is a model that predicts single-cell gene expression from H andE images. It does this by learning how cell shape relates to gene expression through two different prediction methods and then blending the results based on which method seems more reliable for a specific gene.

Jane: That sounds like they're trying to combine the strengths of both direct prediction and reference-based retrieval, depending on what the data suggests.

Lu: The fusion mechanism is where it gets interesting; they use a per-gene gate, which is a weight between the two branches that tells the model whether to trust the direct regression from morphology or pull information from similar reference cells.

Tom: They estimate these gene-specific weights by searching over a grid of values between zero and one for each tissue to maximize how well the final fused prediction matches the actual ground truth expression across all cells on that slide.

Jane: That per-gene weighting is crucial because genes aren't all predictable based on shape; some are, and some require context from other cells.

Meng: So, if we think about what this means for deploying it, it suggests they aren't just relying on one single prediction strategy for every gene across the whole slide.

The paper's improvements: Tom: The paper points out a few specific ways they improved things. They showed that combining the direct and retrieval predictions in Stage two gives them the highest value across every single metric on their test slides, which is a big win for them.

Jane: That suggests that just having one branch isn't enough; integrating both prediction types during the fine-tuning phase is what actually boosts performance.

Lu: They also showed that increasing the training scale from seven million to twenty-three million cells further improved performance, which supports the idea that large scale and heterogeneous cell level supervision is really valuable.

Meng: But there's a caveat, and they mention something about generalization; they can predict genes measured in a target tissue's test set that were never seen in any training slides from the same tissue.

Jane: That ability to predict genes from other tissues during testing, even if it wasn't explicitly trained on them, is what the authors call cross-tissue knowledge transfer.

Tom: They can transfer morphological knowledge of genes from one tissue to a gene measured in another tissue when that gene is only measured in the target tissue.

Conclusion: Jane: So, wrapping this up, VOICE is this multimodal foundation model that predicts single-cell gene expression from H andE images by using both direct regression and retrieval methods fused with a per-gene gate weighted average.

Tom: It achieves top performance across breast, lung, ovary, and pancreas slides when compared against the baselines they tested.

Lu: The key innovation is that it learns to align morphology from UNI2-h with the transcriptomic latent space through contrastive learning without needing specific gene panels for alignment.

Meng: From a production standpoint, running this model needs to handle that per-gene weighting estimation, which involves searching over a grid of values for each tissue, so you'd need some upfront computation to set up the final prediction weights.

Lalam: My internal assessment is that VOICE is quite confident in its ability to transfer knowledge across tissues because the mechanism allows it to learn these dependencies directly from the data structure, and I am very sure of this result.

Jane: It seems like a really practical tool for researchers who have cheap H andE images but don't have access to full single-cell sequencing data for every single gene in every sample.

Tom: Exactly. It shifts the bottleneck from expensive sequencing to affordable imaging, and that’s a significant move for spatial biology research.

Xin Luo, Yicheng Tao, Haoxuan Zeng, Suyuan Wang, Chenzi Ouyang, Meiqi Zhu, Kai Liu, †Shuibing Chen†, Jie Liu†

University of Michigan · Weill Cornell Medicine

cs.CV, q-bio.GN

Submitted: 2026-08-08

Updated: 2026-10-31

Code: https://github.com/mahmoodlab/HEST

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 82/100

The gist: Spatial transcriptomics can resolve gene expression at single-cell resolution, but it is costly, limited to targeted panels of a few hundred to thousand genes…

Key concepts

Contrastive Learning
This is a training method where the model learns to map similar inputs to similar outputs. In VOICE, it aligns cell morphology with its known gene expression profile by learning what visual features correspond to specific gene activity across millions of cells.
Direct and Retrieval Branches
The model has two ways to predict expression: one directly regresses expression from the cell's H&E morphology. The other retrieves measured expression from similar reference cells. Fusing these two predictions allows VOICE to select the most reliable prediction for each gene.
Per-Gene Weighted Fusion
Instead of using a single average, VOICE assigns a unique weight ($eta_g$) to the direct and retrieval predictions for every individual gene. This weighting is optimized per tissue, enabling the model to intelligently combine the two prediction methods based on which one performs better for that specific gene.
SE(2)-Equivariant Transformer
This component uses a specialized neural network architecture designed to understand spatial relationships. It aggregates information from neighboring cells by considering both how similar their shapes are and how far apart they are spatially, improving the model's awareness of the local tissue environment.

Terminology

Summary

Spatial transcriptomics can resolve gene expression at single-cell resolution, but it is costly, limited to targeted panels of a few hundred to thousand genes, and applicable to only a small number of samples <ref:2608. p. 0

Where the paper says it

edu Jie Liu† University of Michigan Ann Arbor, Michigan, USA drjieliu@umich. edu Abstract Spatial transcriptomics can resolve gene expression at single-cell resolution, but it is costly, limited to targeted panels of a few hundred to thousand genes, and applicable to only a small number of samples.

08366#pg2> The authors present VOICE, a multimodal foundation model that predicts single-cell gene expression from H&E images using paired Xenium data <ref:2608. p. 0
Where the paper says it

This makes predicting single-cell expression directly from morphology a practical way to bring molecular analysis to large tissue archives. We therefore present VOICE, a multimodal foundation model that predicts single-cell gene expression from H&E images using paired Xenium data.

08366#pg2>.

Model Architecture and Training Stages

VOICE first aligns cell-centered H&E morphology with single-cell expression embeddings from a transcriptome foundation model through contrastive learning over 23 million cells <ref:2608.08366#pg3>. Next, it predicts expression through two branches: one directly regresses expression from morphology, and the other retrieves measured expression from similar reference cells <ref:2608. p. 1

Where the paper says it

Then it predicts gene expressions of every query cell by two complementary branches from this space. One branch directly regresses gene expression from cell morphology, and the other branch retrieves expression from reference cells whose aligned embeddings are closest to the query’s, recovering genes without morphological signal.

08366#pg2>. Because genes vary in morphological predictability, VOICE fuses the two branches with a per-gene weight <ref:2608. p. 0
Where the paper says it

The other branch retrieves measured expression from similar reference cells, recovering genes that do not have morphological signal. Because genes vary in morphological predictability, VOICE fuses the two branches with a per-gene weight.

08366#pg2>. The model learns in three stages: Stage 1 aligns cell morphology with its gene-expression profile through contrastive learning <ref:2608. p. 1
Where the paper says it

VOICE learns the prediction of genes in two training stages and fuses the prediction results at per gene level from two branches (Fig. Stage 1 aligns cell morphology with its gene-expression profile through contrastive learning.

08366#pg2>, Stage 2 continues finetuning the image encoder against measured gene counts jointly with an SE(2) spatial decoder <ref:2608. p. 2
Where the paper says it

fine-tune the UNI2-h image encoder, producing an image adapter that maps morphological embeddings into the transcriptomic latent space defined by scFoundation [2, 11]. Stage 2 continues finetuning the same image adapter against measured gene counts jointly with an SE(2) spatial decoder learning the relationship between each cell and its neighboring cells.

08366#pg2>, and Stage 3 fuses the direct and retrieval predictions using a per-gene weighted average <ref:2608. p. 4
Where the paper says it

The retrieval branch uses the same feature ℎ 𝜃 𝑖 to identify similar reference cells and obtains 𝑅ˆ 𝑖𝑔 through the neighbor average of Eq. Because the relative usefulness of morphology-based inference and reference-cell retrieval varies across genes, Stage 3 combines the two predictions using a gene-specific weighted average: 𝑦ˆ𝑖𝑔 = 𝛽𝑔 𝐴ˆ 𝑖𝑔 + (1 − 𝛽𝑔) 𝑅ˆ 𝑖𝑔, 𝛽𝑔 ∈ [0, 1], (7) The fusion weights are estimated separately for each tissue.

08366#pg2>.

Feature Engineering for Morphology

To encode cell morphology from H&E images, a 55 μm H&E field of view centered at the cell's coordinate is extracted and resampled to 224×224 pixels <ref:2608.08366#pg2>. This crop is then passed through UNI2-h, which converts it into a 16 × 16 grid of 1536-dimensional patch tokens that interact to learn spatial relationships <ref:2608. p. 2

Where the paper says it

This crop includes the surrounding region of the cell so that passing it through UNI2-h attends it to its neighborhood. UNI2-h converts the crop into a 16 × 16 grid of 1536-dimensional patch tokens which interacts with other tokens to learn spatial relationship.

08366#pg2>. The cell morphology representation is calculated as an overlap-weighted average of these patch token embeddings <ref:2608. p. 2
Where the paper says it

Therefore, we rasterize the cell boundary Ω𝑖 within the image crop and compute the fractional overlap 𝑚𝑖𝑡 ∈ [0, 1] between cell 𝑖 and patch token v𝑖𝑡 . The cell morphology representation is calculated as an overlap-weighted average of the patch token embeddings (Fig.

08366#pg2>.

Per-Gene Fusion Mechanism

The final prediction is achieved by fusing the two branches using a gene-specific weighted average: yˆig = βg Aˆig + (1 − βg) Rˆig, where βg ∈ [0, 1] <ref:2608.08366#pg2>. The fusion weights are estimated separately for each tissue by searching over a grid of values in [0, 1] to maximize the correlation between the fused prediction and ground-truth expression across all cells within that slide <ref:2608. p. 4

Where the paper says it

Because the relative usefulness of morphology-based inference and reference-cell retrieval varies across genes, Stage 3 combines the two predictions using a gene-specific weighted average: 𝑦ˆ𝑖𝑔 = 𝛽𝑔 𝐴ˆ 𝑖𝑔 + (1 − 𝛽𝑔) 𝑅ˆ 𝑖𝑔, 𝛽𝑔 ∈ [0, 1], (7) The fusion weights are estimated separately for each tissue. Specifically, for each gene 𝑔, we first estimate a slide-specific weight for every training slide by searching over a grid of values in [0, 1] and selecting the value that maximizes the correlation between the fused prediction and ground-truth expression across all cells within that slide.

08366#pg2>.

Generalization and Cross-Tissue Transfer

VOICE generalizes to held-out patients and slides, predicting genes measured in the target tissue's test set but never seen in any training slides from the same tissue, thus transferring knowledge of these genes from other tissues <ref:2608.08366#pg2>. Specifically, for genes measured only in other tissues, VOICE can transfer cross-tissue morphological knowledge to them <ref:2608. p. 6

Where the paper says it

Because the two groups contain different genes, their absolute PCC values should not be interpreted as a direct measure of which group is intrinsically easier to predict. Instead, these results show that VOICE can transfer cross-tissue morphological knowledge to genes that were never measured in the target tissue during training.

08366#pg2>.

Performance and Ablation Results

VOICE-23M achieves the highest PCC among baselines for all seven metrics on the breast, lung, ovary, and pancreas slides <ref:2608.08366#pg2>. The ablation study in Table 7 demonstrates that combining the Stage 2 direct and retrieval predictions gives the highest value in every column <ref:2608. p. 5

Where the paper says it

Across breast and lung, VOICE also gives the highest median per-gene PCC in every displayed SVG and HVG subset. The ablation study in Table 6 demonstrates the effectiveness of the proposed fusion module.

08366#pg2>. Increasing the training scale from 7 million to 23 million cells further improved performance, supporting the value of large-scale and heterogeneous cell-level supervision <ref:2608. p. 6
Where the paper says it

VOICE consistently outperformed existing single-cell prediction methods in both within-slide and cross-slide evaluation. Increasing the training scale from 7 million to 23 million cells further improved performance, supporting the value of large-scale and heterogeneous cell-level supervision.

08366#pg2>.

Future Directions

Future work will extend VOICE to CosMx and other cell-resolved spatial transcriptomics and evaluate its utility in downstream tasks <ref:2608.08366#pg2>. The study uses publicly available human H&E and Xenium datasets obtained from published studies and databases <ref:2608. p. 7

Where the paper says it

Future work will extend VOICE to CosMx and other cell-resolved spatial transcriptomics and evaluate its utility in downstream tasks. This study uses publicly available human H&E and Xenium datasets obtained from published studies and databases.

08366#pg2>. VOICE is intended for research purposes only and should not be used for clinical diagnosis or treatment decisions without further validation <ref:2608. p. 7
Where the paper says it

This study uses publicly available human H&E and Xenium datasets obtained from published studies and databases. VOICE is intended for research purposes only and should not be used for clinical diagnosis or treatment decisions without further validation.

08366#pg2>.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Better prediction of genes absent from same-tissue training data by learning from other tissues through cross-tissue knowledge transfer. The model can transfer knowledge of these genes from other tissues because it learns to predict other-tissue-only genes even when they are not measured in the target tissue during training.

  2. More robust prediction across different Xenium assays by utilizing a shared gene vocabulary and panel-specific loss masking. This allows the model to learn jointly from datasets with heterogeneous gene panels, meaning it can perform well even when comparing slides with different gene sets.

  3. Superior performance in complex biological scenarios using per-gene gated fusion of direct and retrieval predictions. VOICE fuses the two branches with a per-gene weight that is fit for each gene, allowing it to select the more reliable branch to yield the final prediction based on whether morphology predicts a gene well or if retrieval is necessary.

  4. Enhanced spatial context awareness in predictions using an SE(2)-equivariant transformer. The model uses an SE(2)-equivariant transformer to aggregate neighboring cell embeddings according to both morphological similarity and spatial separation, making predictions more sensitive to local tissue microenvironments.

  5. Improved generalization across unseen tissue types by performing zero-shot transfer from Xenium data to CosMx data using a nucleus-level matching procedure. VOICE can extend its capabilities beyond Xenium by establishing a nucleus-level procedure that establishes the missing correspondence directly between H&E and DAPI images for CosMx, even without any CosMx expression data during training.

Abstract

Spatial transcriptomics can resolve gene expression at single-cell resolution, but it is costly, limited to targeted panels of a few hundred to a few thousand genes, and applicable to only a small number of samples. H&E imaging, by contrast, is cheap and collected routinely at scale. This makes predicting single-cell expression directly from morphology a practical way to bring molecular analysis to large tissue archives. We therefore present VOICE, a multimodal foundation model that predicts single-cell gene expression from H&E images using paired Xenium data. VOICE first aligns cell centered H&E morphology from a pathology foundation model with single-cell expression embeddings from a transcriptome foundation model, trained using contrastive learning over 23 million cells. Next it predicts expression through two branches. One branch directly regresses expression from morphology. The other branch retrieves measured expression from similar reference cells, recovering genes that do not have morphological signal. Because genes vary in morphological predictability, VOICE fuses the two branches with a per-gene weight. After training, VOICE generalizes to heldout patients, slides, and partially overlapping gene panels from Xenium, and it consistently outperforms prior single-cell expression prediction methods on seven metrics.

Sources

Related papers