Bridging the Modality Bottleneck in Pathology MIL through Molecularly Guided Feature Adaptation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.
Ines: We're looking at the paper: "Bridging the Modality Bottleneck in Pathology MIL through Molecularly Guided Feature Adaptation".
Marcus: Here's the gist of the paper. Multiple instance learning (MIL) is the dominant framework for whole-slide image analysis in computational pathology, typically combining a frozen patch encoder, a projection layer, and a slide-level aggregator The authors introduce Molecularly Informed Staining Transform (MIST), a plug-in replacement for the MIL projection layer that uses paired spatial transcriptomics only during training to construct virtual molecular stains MIST clusters gene expression profiles into cross-modal prototypes, anchors them in the frozen foundation model feature space, and uses them to reorganize H&E patch features along molecularly guided axes The Modality Bottleneck and MIST Intervention The paper addresses a fundamental mismatch where frozen pathology foundation models are pretrained mainly from H&E morphology, yet clinical endpoints like biomarker status are governed by molecular state The authors identify the projection layer as a "largely morphology-only bottleneck" that limits endpoints governed by molecular state MIST is introduced as a mechanism to bridge this gap by converting paired spatial transcriptomics (ST) into virtual molecular stains This approach allows for cross-modal adaptation without requiring transcriptomics at inference and does not modify the frozen pathology foundation model How the Training Signal is Built MIST realizes feature-level virtual molecular staining in three distinct steps First, it derives molecular prototypes by clustering gene-expression profiles and anchoring the resulting prototypes in the pathology FM feature space Second, each patch is softly associated with multiple prototypes through affinities, reflecting that local tissue regions often express overlapping molecular programs Finally, prototype-conditioned low-rank residual transforms generate one stained view per molecular concept which are combined according to the affinity weights and added back to the original patch embedding The prototypes are constructed offline from paired ST and H&E data using a process involving extracting morphology embeddings and clustering gene vectors into Ko groups Where the Gains Come From The empirical evaluation demonstrates that MIST improves 240 out of 256 configurations over the standard projection layer, achieving an average gain of +3.5% across 23 downstream tasks and 8 MIL aggregators The gains are primarily driven by gene-derived prototypes, as ablations confirm that replacing them with random or morphology-derived alternatives removes much of the improvement Specifically, MIST improves survival prediction by +5.2%, tissue subtyping by +3.3%, and biomarker prediction by +2.6% on average The theoretical grounding shows that the output difference decomposes into a term capturing molecular routing induced by differences in prototype affinity, which can exceed the morphology-only baseline Interpretability and Efficiency Interpretability analyses show that learned prototype affinities capture spatially coherent molecular programs directly from H&E images without explicit tissue annotations during training For instance, prototype signatures align with recognizable tissue programs such as surfactant-related genes for alveolar epithelium Furthermore, MIST is computationally efficient; with K=8 prototypes and rank r=16, the entire MIST module contains only 0.87M parameters, and it reduces the parameter count of ILRA by 9.0% through shared feature compression This design allows MIST to remain a reusable molecularly informed adaptation module without modifying the frozen encoder MIST successfully introduces spatial transcriptomic priors into frozen pathology foundation model pipelines while requiring only H&E images at inference The method supports the hypothesis that MIST and capacity-focused MIL redesigns address complementary limitations, as their gains combine almost additively While MIST relies on paired ST and H&E data, its ability to infer biologically meaningful molecular distributions from morphology suggests it captures clinically meaningful tissue heterogeneity MIST aims to improve whole-slide image analysis by introducing molecular priors into pathology foundation model pipelines without requiring transcriptomic measurements at inference The authors also note that MIST relies on molecular supervision derived from existing ST datasets, which remain limited in scale and tissue diversity The paper concludes by highlighting the potential of MIST to broaden access to molecularly-informed computational pathology systems where transcriptomics is unavailable Page 1 --- Bridging the Modality Bottleneck in Pathology MIL through Virtual Molecular Staining Yucheng Xing, Pei Liu, Jingying Ma, Ruping Hong, Jiangdong Qiu, Tianyu Liu, Kai He, Ling Huang† Mengling Feng 1National University of Singapore; 2Hunan University; 3Peking Union Medical College Hospital (PUMCH), China; 4Imperial College London Abstract Multiple instance learning (MIL) is the dominant framework for whole-slide image analysis in computational pathology, typically combining a frozen patch encoder, a projection layer, and a slide-level aggregator While encoders and aggregators have been extensively studied, the projection layer remains a largely morphology-only bottleneck This limits endpoints such as biomarker status and survival, which are governed by a molecular state that is not fully captured by H&E morphology We introduce Molecularly Informed Staining Transform (MIST), a plug-in replacement for the MIL projection layer that uses paired spatial transcriptomics only during training to construct virtual molecular stains MIST clusters gene expression profiles into cross-modal prototypes, anchors them in the frozen foundation model feature space, and uses them to reorganize H&E patch features along molecularly guided axes It requires no transcriptomics at inference and can be inserted before standard MIL aggregators We evaluate MIST across 23 downstream tasks and 8 MIL aggregators MIST improves 240 of 256 configurations over the standard projection layer, with an average gain of +3.5% Ablations confirm that gene-derived prototypes are the primary source of the gains Page 2 --- We address this bottleneck at the projection layer A standard MIL pipeline contains a frozen patch encoder, a per-patch projection layer, and a slide-level aggregator Existing projection-layer redesigns still operate inside the FM’s morphological feature space They can better use morphology, but they cannot introduce molecular axes that the encoder never observed during pretraining Inspired by this role of staining, we introduce Molecularly Informed Staining Transform (MIST) for MIL MIST converts paired spatial transcriptomics (ST), available only during training, into virtual molecular stains that reorganize H&E patch features along gene-defined axes Paired ST is a natural source of molecular supervision because it co-registers H&E morphology with spot-level gene expression MIST instead moves molecular supervision one stage later, providing a reusable molecularly informed adaptation module without modifying the frozen encoder Page 3 --- By operating at the projection layer, it provides a reusable molecularly informed adaptation module without modifying the frozen encoder MIST is a form of cross-modal adaptation for frozen foundation-model pipelines Instead of requiring the auxiliary modality at inference, MIST uses it during training to define a molecularly grounded coordinate system MIST realizes feature-level virtual molecular staining in three steps First, it derives molecular prototypes by clustering gene-expression profiles and anchoring the resulting prototypes in the pathology FM feature space, allowing each prototype to encode the morphological signature of a distinct molecular concept Second, each patch is softly associated with multiple prototypes through affinities, reflecting the fact that local tissue regions often express overlapping molecular programs Finally, prototype-conditioned low-rank residual transforms generate one stained view per molecular concept, which are combined according to the affinity weights and added back to the original patch embedding Page 4 --- We evaluate MIST across 23 downstream tasks and 8 MIL aggregators using a frozen encoder MIST improves 240 of 256 configurations over the standard projection layer, with consistent gains across endpoint types, including survival prediction (+5.2%), tissue subtyping (+3.3%), and biomarker prediction (+2.6%) Ablation studies confirm that gene-derived prototypes are the primary source of the gains Our contributions are as follows: • A modality-bottleneck view of MIL projection layers.
Ines: We identify the projection layer as a modular but modality-limited stage in frozen-FM pathology MIL, and show that molecular priors provide an effective intervention beyond morphology-only projection capacity • Virtual molecular staining through cross-modal prototypes.
Marcus: We introduce a projection-layer mechanism that injects spatial transcriptomic priors during training while requiring only H&E at inference, enabling molecularly informed adaptation without retraining the pathology FM Page 5 --- MIST realizes feature-level virtual molecular staining in three steps First, it derives molecular prototypes by clustering gene-expression profiles and anchoring the resulting prototypes in the pathology FM feature space, allowing each prototype to encode the morphological signature of a distinct molecular concept Second, each patch is softly associated with multiple prototypes through affinities, reflecting the fact that local tissue regions often express overlapping molecular programs Finally, prototype-conditioned low-rank residual transforms generate one stained view per molecular concept, which are combined according to the affinity weights and added back to the original patch embedding Page 6 --- We evaluate MIST across 23 downstream tasks and 8 MIL aggregators using a frozen encoder MIST improves 240 of 256 configurations over the standard projection layer, with consistent gains across endpoint types, including survival prediction (+5.2%), tissue subtyping (+3.3%), and biomarker prediction (+2.6%) Ablation studies confirm that gene-derived prototypes are the primary source of the gains Page 7 --- Table 1 Survival prediction Concordance index across four datasets using eight MIL aggregators ABMIL CLAM TransMIL Trans.
Ines: ILRA Mean Max DSMIL Avg.
Paper summary: Ines: We’re moving on to a more detailed look at this study, "Bridging the Modality Bottleneck in Pathology MIL through Molecularly Guided Feature Adaptation." The central idea is that standard MIL frameworks for pathology often get stuck because the projection layer only looks at H andE morphology. This limits their ability to predict molecular states, which are crucial for things like survival or disease status.
Marcus: Specifically, the authors argue that this projection layer acts as a bottleneck because it can’t introduce molecular axes that the foundation model never saw during its initial training on just H andE images. They propose MIST as a solution to bridge this gap.
Yuki: So what exactly is MIST doing? The summary says it uses paired spatial transcriptomics only during training to construct virtual molecular stains, which are then clustered into cross-modal prototypes. These prototypes are anchored in the feature space of the frozen foundation model.
Ines: That’s right. The authors describe a three-step process: first, they derive these molecular prototypes by clustering gene expression profiles and anchoring them in the pathology foundation model feature space so each prototype encodes a distinct molecular concept.
Marcus: Second, each patch is associated with multiple prototypes through affinities that reflect how often local tissue regions express overlapping molecular programs. This step allows patches to be linked to several different concepts simultaneously.
Yuki: And finally, they use these prototype conditions to generate one stained view per molecular concept, which are combined according to those affinity weights and added back into the original patch embedding. It’s like creating a new set of features that are guided by molecular structure.
Ines: The paper claims that this MIST mechanism allows them to reorganize H andE patch features along these molecularly guided axes, effectively injecting spatial transcriptomic priors into the pipeline without needing transcriptomics at inference time.
Marcus: And the key takeaway is that they evaluate this across twenty-three downstream tasks and eight MIL aggregators, showing it improves two hundred forty out of two hundred fifty-six configurations over the standard projection layer. They report an average gain of plus three point five percent across those twenty-three tasks and eight aggregators.
Yuki: The summary highlights that the empirical evidence shows these cross-modal prototype affinities capture spatially coherent and pathologically meaningful molecular programs, which is what makes this approach biologically interesting.
Ines: It’s about using paired spatial transcriptomics only during training to define a molecularly grounded coordinate system, which is then used for adaptation without retraining the foundation model. That keeps the encoder frozen while adding powerful new information.
Marcus: And they also point out that MIST can increase separation between patches that look morphologically similar but have different underlying molecular states beyond what morphology alone could allow. That’s a crucial distinction.
Yuki: So we're looking at a system where the goal is to use existing spatial transcriptomics data to define a coordinate system for the feature space, which then informs how we interpret H andE images for molecular tasks.
Conclusion: Ines: So wrapping up this discussion on "Bridging the Modality Bottleneck in Pathology MIL through Molecularly Guided Feature Adaptation," the authors are essentially proposing a method to overcome the limitation where H andE morphology alone isn't enough for molecular prediction.
Marcus: They did this by developing MIST, which is a plug-in replacement for the standard projection layer that uses spatial transcriptomics during training to build virtual molecular stains and reorganize patch features along these learned axes. It’s about making the foundation model more versatile.
Yuki: What does this mean for the biology we actually see? It suggests that tissue heterogeneity isn't just a random variation in shape; it has specific, genetically defined patterns that we can now potentially map onto the image data.
Ines: Exactly. It means we can start predicting things like survival or subtyping by accounting for those underlying genetic drivers that might be hidden beneath the surface morphology. The authors are showing that molecular supervision, even if it’s derived from a limited source like spatial transcriptomics, can provide significant predictive lift when integrated intelligently.
Marcus: The implication is that for future computational pathology systems, we move towards pipelines where molecular priors are used to guide feature adaptation during training instead of just relying on morphology-only transformations.
Yuki: It opens the door for systems that can interpret tissue not just structurally, but functionally too, giving us a richer data source from the same whole-slide image.
National University of Singapore · Hunan University, China · §Peking Union Medical College Hospital (PUMCH), China · Imperial College London, United Kingdom
q-bio.QM, cs.CV, cs.LG
Submitted: 2026-05-12
Updated: 2026-11-04
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 98/100
The gist: Multiple instance learning (MIL) is the dominant framework for whole-slide image analysis in computational pathology, typically combining a frozen patch encoder, a projection layer…
Key concepts
- Modality Bottleneck
- Frozen pathology foundation models are primarily trained on H&E morphology. This means they are excellent at recognizing tissue structure but struggle to capture clinical outcomes dependent on the underlying molecular state, creating a limitation when predicting things like biomarker status or survival.
- Molecularly Informed Staining Transform (MIST)
- MIST is a replacement for the standard projection layer in MIL. It uses paired spatial transcriptomics during training to build virtual 'molecular stains' from gene expression profiles. These stains are then used to guide and reorganize H&E patch features, introducing molecular information into the model.
- Cross-modal Prototypes
- MIST clusters gene expression profiles into prototypes that are anchored in the pathology foundation model's feature space. Each prototype represents a distinct molecular concept. Patches are then associated with these prototypes via affinities, allowing the model to understand how different tissue regions express overlapping molecular programs.
- Molecular Routing
- The paper shows that MIST can separate morphologically similar patches based on their underlying molecular differences. This separation is captured by a term in the output difference decomposition, indicating that prototype affinity drives a specific type of feature reorganization beyond what morphology alone can achieve.
Terminology
Summary
Multiple instance learning (MIL) is the dominant framework for whole-slide image analysis in computational pathology, typically combining a frozen patch encoder, a projection layer, and a slide-level aggregator The authors introduce Molecularly Informed Staining Transform (MIST), a plug-in replacement for the MIL projection layer that uses paired spatial transcriptomics only during training to construct virtual molecular stains MIST clusters gene expression profiles into cross-modal prototypes, anchors them in the frozen foundation model feature space, and uses them to reorganize H&E patch features along molecularly guided axes
The Modality Bottleneck and MIST Intervention
The paper addresses a fundamental mismatch where frozen pathology foundation models are pretrained mainly from H&E morphology, yet clinical endpoints like biomarker status are governed by molecular state The authors identify the projection layer as a largely morphology-only bottleneck
that limits endpoints governed by molecular state MIST is introduced as a mechanism to bridge this gap by converting paired spatial transcriptomics (ST) into virtual molecular stains This approach allows for cross-modal adaptation without requiring transcriptomics at inference and does not modify the frozen pathology foundation model
How the Training Signal is Built
MIST realizes feature-level virtual molecular staining in three distinct steps First, it derives molecular prototypes by clustering gene-expression profiles and anchoring the resulting prototypes in the pathology FM feature space Second, each patch is softly associated with multiple prototypes through affinities, reflecting that local tissue regions often express overlapping molecular programs Finally, prototype-conditioned low-rank residual transforms generate one stained view per molecular concept which are combined according to the affinity weights and added back to the original patch embedding The prototypes are constructed offline from paired ST and H&E data using a process involving extracting morphology embeddings and clustering gene vectors into Ko groups
Where the Gains Come From
The empirical evaluation demonstrates that MIST improves 240 out of 256 configurations over the standard projection layer, achieving an average gain of +3.5% across 23 downstream tasks and 8 MIL aggregators The gains are primarily driven by gene-derived prototypes, as ablations confirm that replacing them with random or morphology-derived alternatives removes much of the improvement Specifically, MIST improves survival prediction by +5. p. 1 Ablation studies further show
that replacing gene-derived prototypes with random or morphology-derived alternatives removes
much of the gain, suggesting that the improvement primarily arises from cross-modal molecular
priors rather than additional morphology-only modeling capacity. Our contributions are as follows:
• A modality-bottleneck view of MIL projection layers. 2% on survival prediction, +3. 3% on tissue subtyping, and
+2. 3% on tissue subtyping, and
+2. 6% on biomarker prediction.Where the paper says it
Where the paper says it
Where the paper says it
Interpretability and Efficiency
Interpretability analyses show that learned prototype affinities capture spatially coherent molecular programs directly from H&E images without explicit tissue annotations during training For instance, prototype signatures align with recognizable tissue programs such as surfactant-related genes for alveolar epithelium Furthermore, MIST is computationally efficient; with K=8 prototypes and rank r=16, the entire MIST module contains only 0.87M parameters, and it reduces the parameter count of ILRA by 9. p. 2 Prototype-based methods learn a compact
set of representative concepts to summarize complex inputs [27, 28, 29]. In CPath, prior work models
patches as Gaussian mixtures and aggregates through the mixture parameters [30], clusters patches
into histological types and represents slides as type-frequency distributions [31], extends prototypebased representations to multimodal survival prediction with bulk transcriptomics at both training
and inference [32], and couples morphological prototypes with class anchors under an evidential
fusion framework for uncertainty-aware survival [33].Where the paper says it
Conclusion and Future Directions
MIST successfully introduces spatial transcriptomic priors into frozen pathology foundation model pipelines while requiring only H&E images at inference The method supports the hypothesis that MIST and capacity-focused MIL redesigns address complementary limitations, as their gains combine almost additively While MIST relies on paired ST and H&E data, its ability to infer biologically meaningful molecular distributions from morphology suggests it captures clinically meaningful tissue heterogeneity MIST aims to improve whole-slide image analysis by introducing molecular priors into pathology foundation model pipelines without requiring transcriptomic measurements at inference The authors also note that MIST relies on molecular supervision derived from existing ST datasets, which remain limited in scale and tissue diversity The paper concludes by highlighting the potential of MIST to broaden access to molecularly-informed computational pathology systems where transcriptomics is unavailable
--- Page 1 ---
Bridging the Modality Bottleneck in Pathology MIL through Virtual Molecular Staining Yucheng Xing, Pei Liu, Jingying Ma, Ruping Hong, Jiangdong Qiu, Tianyu Liu, Kai He, Ling Huang† Mengling Feng 1National University of Singapore; 2Hunan University; 3Peking Union Medical College Hospital (PUMCH), China; 4Imperial College London Abstract Multiple instance learning (MIL) is the dominant framework for whole-slide image analysis in computational pathology, typically combining a frozen patch encoder, a projection layer, and a slide-level aggregator While encoders and aggregators have been extensively studied, the projection layer remains a largely morphology-only bottleneck This limits endpoints such as biomarker status and survival, which are governed by a molecular state that is not fully captured by H&E morphology We introduce Molecularly Informed Staining Transform (MIST), a plug-in replacement for the MIL projection layer that uses paired spatial transcriptomics only during training to construct virtual molecular stains MIST clusters gene expression profiles into cross-modal prototypes, anchors them in the frozen foundation model feature space, and uses them to reorganize H&E patch features along molecularly guided axes It requires no transcriptomics at inference and can be inserted before standard MIL aggregators We evaluate MIST across 23 downstream tasks and 8 MIL aggregators MIST improves 240 of 256 configurations over the standard projection layer, with an average gain of +3.5% Ablations confirm that gene-derived prototypes are the primary source of the gains
--- Page 2 ---
We address this bottleneck at the projection layer A standard MIL pipeline contains a frozen patch encoder, a per-patch projection layer, and a slide-level aggregator Existing projection-layer redesigns still operate inside the FM’s morphological feature space They can better use morphology, but they cannot introduce molecular axes that the encoder never observed during pretraining Inspired by this role of staining, we introduce Molecularly Informed Staining Transform (MIST) for MIL MIST converts paired spatial transcriptomics (ST), available only during training, into virtual molecular stains that reorganize H&E patch features along gene-defined axes Paired ST is a natural source of molecular supervision because it co-registers H&E morphology with spot-level gene expression MIST instead moves molecular supervision one stage later, providing a reusable molecularly informed adaptation module without modifying the frozen encoder
--- Page 3 ---
By operating at the projection layer, it provides a reusable molecularly informed adaptation module without modifying the frozen encoder MIST is a form of cross-modal adaptation for frozen foundation-model pipelines Instead of requiring the auxiliary modality at inference, MIST uses it during training to define a molecularly grounded coordinate system MIST realizes feature-level virtual molecular staining in three steps First, it derives molecular prototypes by clustering gene-expression profiles and anchoring the resulting prototypes in the pathology FM feature space, allowing each prototype to encode the morphological signature of a distinct molecular concept Second, each patch is softly associated with multiple prototypes through affinities, reflecting the fact that local tissue regions often express overlapping molecular programs Finally, prototype-conditioned low-rank residual transforms generate one stained view per molecular concept, which are combined according to the affinity weights and added back to the original patch embedding
--- Page 4 ---
We evaluate MIST across 23 downstream tasks and 8 MIL aggregators using a frozen encoder MIST improves 240 of 256 configurations over the standard projection layer, with consistent gains across endpoint types, including survival prediction (+5.2%), tissue subtyping (+3. p. 0 2% on survival prediction, +3. 3% on tissue subtyping, and
+2. 2% on survival prediction, +3. 3% on tissue subtyping, and
+2. 3% on tissue subtyping, and
+2. 6% on biomarker prediction. 3% on tissue subtyping, and
+2. 6% on biomarker prediction. Ablation studies further show
that replacing gene-derived prototypes with random or morphology-derived alternatives removes
much of the gain, suggesting that the improvement primarily arises from cross-modal molecular
priors rather than additional morphology-only modeling capacity. Our contributions are as follows:
• A modality-bottleneck view of MIL projection layers. Our contributions are as follows:
• A modality-bottleneck view of MIL projection layers. We identify the projection layer as a
modular but modality-limited stage in frozen-FM pathology MIL, and show that molecular priors
provide an effective intervention beyond morphology-only projection capacity.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 5 ---
MIST realizes feature-level virtual molecular staining in three steps First, it derives molecular prototypes by clustering gene-expression profiles and anchoring the resulting prototypes in the pathology FM feature space, allowing each prototype to encode the morphological signature of a distinct molecular concept Second, each patch is softly associated with multiple prototypes through affinities, reflecting the fact that local tissue regions often express overlapping molecular programs Finally, prototype-conditioned low-rank residual transforms generate one stained view per molecular concept, which are combined according to the affinity weights and added back to the original patch embedding
--- Page 6 ---
We evaluate MIST across 23 downstream tasks and 8 MIL aggregators using a frozen encoder MIST improves 240 of 256 configurations over the standard projection layer, with consistent gains across endpoint types, including survival prediction (+5.2%), tissue subtyping (+3.3%), and biomarker prediction (+2.6%) Ablation studies confirm that gene-derived prototypes are the primary source of the gains
--- Page 7 ---
Table 1 Survival prediction Concordance index across four datasets using eight MIL aggregators ABMIL CLAM TransMIL Trans. ILRA Mean Max DSMIL Avg. p. 6 Cohort Status ABMIL CLAM TransMIL Trans. ILRA Mean Max DSMIL Avg. BRCA
Base 60. 14)
+Ours 69.Where the paper says it
Where the paper says it
Table 2 Molecular biomarker prediction Mean performance across biomarker tasks within each dataset, with GBMLGG additionally averaged across TCGA and EBRAINS
--- Page 8 ---
We compare MIST against the standard linear layer across eight MIL aggregators We further combine MIST with MAMMOTH to examine whether modality bottlenecks and morphology-focused capacity bottlenecks are complementary
--- Page 9 ---
MIST can increase separation between morphologically similar but molecularly distinct patches beyond what morphology alone permits The first term corresponds to the morphology-only transformation, while the second term captures molecular routing induced by differences in prototype affinity
--- Page 10 ---
The output difference decomposes as h˜i − h˜j = (2I + W¯)ϵv + X k ∆α k Uk Vhbar, where the second term captures molecular routing induced by differences in prototype affinity
--- Page 11 ---
The first term corresponds to the morphology-only transformation, while the second term captures molecular routing induced by differences in prototype affinity
--- Page 12 ---
MIST can separate morphologically similar but molecularly distinct patches beyond what morphology alone permits
--- Page 13 ---
We derive the prototypes from paired ST and H&E data in HEST-1k, which provides gene expression and morphology co-localized at Visium-spot resolution across 1,276 paired slides and 26 organ types with millions of spots
--- Page 14 ---
Extracting Paired Morphology and Gene Expression at Each Spot. We encode each patch with the frozen pathology foundation model UNI2-h [7] to obtain its morphology embedding xj ∈ R D For the corresponding Visium spots, we follow SEAL’s processing pipeline [15] to obtain the gene expression vector gj: empty spots and genes detected in less than 10% of spots are removed; raw counts are normalized to 104 per spot and log(1+x)-transformed
--- Page 15 ---
Clustering Within Each Organ to Form Prototypes. We construct prototypes per organ since molecular concepts differ substantially across organs and downstream tasks are typically organspecific This construction is central to MIST: clustering in gene space discovers molecular concepts, while the prototypes themselves live in morphological feature space so they can condition H&E patch embeddings directly
--- Page 16 ---
This construction is central to MIST: clustering in gene space discovers molecular concepts, while the prototypes themselves live in morphological feature space so they can condition H&E patch embeddings directly
--- Page 17 ---
We first derive the prototypes from paired ST and H&E data in HEST-1k, which provides gene expression and morphology co-localized at Visium-spot resolution across 1,276 paired slides and 26 organ types with millions of spots
--- Page 18 ---
Extracting Paired Morphology and Gene Expression at Each Spot.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Cross-modal adaptation for frozen foundation-model pipelines: MIST allows molecular supervision to be injected during training via paired spatial transcriptomics without modifying or retraining the frozen pathology foundation model, making it reusable across evolving models.
-
Feature reorganization along molecularly guided axes: MIST reorganizes H&E patch features along gene-defined axes using prototype-conditioned low-rank residual transforms, which allows the model to separate
morphologically similar but molecularly distinct patches
by capturingmolecular routing induced by differences in prototype affinity.
-
Enhanced prediction of survival and biomarker status: By leveraging cross-modal priors, MIST improves survival prediction by an average gain of +5.2% and biomarker prediction by +2.6%, specifically benefiting
weaker aggregators that rely on coarse pooling operations
becausesurvival outcomes are primarily driven by molecular state and microenvironmental composition.
-
Molecular interpretation from morphology alone: The system enables inference without transcriptomics, as the learned prototype affinities capture "spatially coherent molecular programs from H&E alone," allowing researchers to infer biologically meaningful molecular distributions directly from morphology.
-
Organ-specific prototype construction: MIST constructs prototypes
per organ since molecular concepts differ substantially across organs and downstream tasks are typically organspecific,
ensuring that the virtual stains applied to patches are relevant to the specific tissue context being analyzed.
Abstract
Multiple instance learning (MIL) is the dominant framework for whole-slide image analysis in computational pathology, typically combining a frozen patch encoder, a projection layer, and a slide-level aggregator. While encoders and aggregators have been extensively studied, the projection layer remains a largely morphology-only bottleneck. This limits endpoints such as biomarker status and survival, which are governed by a molecular state that is not fully captured by H&E morphology. We introduce Molecularly Informed Staining Transform (MIST), a plug-in replacement for the MIL projection layer that uses paired spatial transcriptomics only during training to construct virtual molecular stains. MIST clusters gene expression profiles into cross-modal prototypes, anchors them in the frozen foundation model feature space, and uses them to reorganize H&E patch features along molecularly guided axes. It requires no transcriptomics at inference and can be inserted before standard MIL aggregators. We evaluate MIST across 23 downstream tasks and 8 MIL aggregators. MIST improves 240 of 256 configurations over the standard projection layer, with an average gain of +3.5%, observed consistently across endpoint types: +5.2% on survival prediction, +3.3% on tissue subtyping, and +2.6% on biomarker prediction. Ablations confirm that gene-derived prototypes are the primary source of the gains, while spatial, biological, and pathological analyses show that cross-modal prototype affinities capture spatially coherent molecular programs from H&E alone.
Sources
- Towards Spatial Transcriptomics-driven Pathology Foundation Models
- A Multimodal Foundation Model of Spatial Transcriptomics and Histology for Biological Discovery and Clinical Prediction
- Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
- MINT: Molecularly Informed Training with Spatial Transcriptomics Supervision for Pathology Foundation Models
- SPADE: Spatial Transcriptomics and Pathology Alignment Using a Mixture of Data Experts for an Expressive Latent Space
- Multimodal Prototyping for cancer survival prediction
- DPsurv: Dual-Prototype Evidential Fusion for Uncertainty-Aware and Interpretable Whole-Slide Image Survival Prediction
Related papers
- ViroGym: Realistic Large-Scale Benchmarks for Evaluating Viral Proteins
- CellCanon: An Agentic Cellular World Model for Prediction, Explanation, and Observation-Driven Updating
- Probing and steering biology across Boltz-1s trunk-diffusion boundary
- Trajectory Inference of Human Aging from Cross-Sectional DNA Methylation Data
- Molecular notation can act as a shortcut in AI benchmark evaluation
- Towards Efficient Auto-Interpretability of Boltz-1 Sparse Autoencoders