Do Pathology Foundation Models Encode Disease Progression? A Pseudotime Analysis of Visual Representations

summary

The gist

Vision foundation models can implicitly learn to represent continuous processes from independent static observations…

In short

The study used diffusion pseudotime to analyze how pathology foundation models organize disease states in their latent space. The findings show that these models learn coherent progression directions, and trajectory fidelity—measuring agreement between inferred progression order and true labels—is a strong predictor of generalization performance across different diseases.

Key concepts

Diffusion Pseudotime (DPT)
This method reconstructs developmental trajectories from single-cell data by calculating transition probabilities between nearby points in feature space. For foundation models, it builds a transition matrix based on patch embeddings to map out how disease states might progress sequentially in the model's internal representation.
Trajectory Fidelity ($ au$)
This metric measures how well the inferred ordering of disease progression (pseudotime) matches the actual ground-truth ordinal class labels. Higher fidelity indicates that the model's learned representations capture a meaningful, coherent biological progression direction.
Geometric Structure
The analysis reveals that embeddings do not just separate disease states; they arrange them in a specific geometry reflecting biological progression. This geometric constraint provides a robust signal for generalization, suggesting the models learn structure aligned with real-world biological changes.

Terminology used across episodes

This episode discusses

The paper

Do Pathology Foundation Models Encode Disease Progression? A Pseudotime Analysis of Visual Representations · Read on arXiv

Massachusetts Institute of Technology · Dana-Farber Cancer Institute · Brigham and Women’s Hospital · Harvard Medical School

Vision foundation models trained on discretely sampled images achieve strong performance on classification benchmarks, yet whether their representations encode the continuous processes underlying their training data remains unclear. This question is especially pertinent in computational pathology, where we posit that models whose latent representations implicitly capture ordinal disease progression may better reflect underlying biology, support more robust generalization, and enable quantitative analyses of features associated with disease transitions. Using diffusion pseudotime, a method developed to infer developmental trajectories from single-cell transcriptomics, we probe whether foundation models organize disease states along coherent progression directions in representation space. Across four cancer progressions and six models, we find that all pathology-specific models recover trajectory orderings significantly exceeding null baselines, with pathology vision-only models achieving the highest average fidelities. Model rankings by trajectory fidelity on reference diseases predict few-shot classification performance on held-out diseases (ρ= 0.91), and exploratory analysis shows cell-type composition varies along inferred trajectories in patterns consistent with known stromal remodeling. Together, these results suggest that vision foundation models can implicitly learn to represent continuous processes from independent static observations, and that trajectory fidelity provides a complementary measure of representation quality beyond downstream performance. While demonstrated in pathology, this framework could be applied to other domains where continuous processes are observed through static snapshots.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Do Pathology Foundation Models Encode Disease Progression? A Pseudotime Analysis of Visual Representations".

Jane: Here's the gist of the paper. Vision foundation models can implicitly learn to represent continuous processes from independent static observations, and that trajectory fidelity provides a complementary measure of representation quality beyond downstream performance Pruning redundant reasoning tokens The paper investigates whether vision foundation models encode disease progression by applying diffusion pseudotime to their latent representations This analysis probes whether these models organize disease states along coherent progression directions in representation space The core finding is that pathology-specific models recover trajectory orderings significantly exceeding null baselines, with vision-only models achieving the highest fidelities How the training signal is built The method utilizes diffusion pseudotime (DPT), which reconstructs developmental trajectories from single-cell transcriptomics by computing transition probabilities between nearby points in feature space For foundation models, a transition matrix T is constructed using a weighted k-nearest-neighbor graph over patch embeddings to account for varying sampling densities The transition probability between patches xi and xj is defined by the formula exp −∥xi − xj∥2 cosine 2σiσj (1) The DPT distance dpt(i, j) is the Euclidean distance between accumulated transition probabilities M(i, ·) and M(j, ·) Where the gains come from The trajectory fidelity metric used is Kendall’s Rank Correlation Coefficient (τ), which quantifies agreement between inferred pseudotime ordering and ground-truth ordinal class labels Models with higher trajectory fidelity correlate strongly with few-shot generalization performance, with mean ρ = 0.92 Furthermore, model rankings by trajectory fidelity on reference diseases strongly predict few-shot classification performance on held-out diseases (ρ = 0.92) Geometric structure and biological grounding The analysis demonstrates that embeddings do not merely separate disease states but arrange them in a geometry that specifically reflects the biological progression direction Trajectory fidelity complements classification benchmarks by offering insights into continuous progressions rather than discrete labels Biological grounding analysis shows that the observed variation in fibroblast and neutrophil abundance across and within disease states aligns with known patterns of stromal remodeling during CRC progression The geometric constraint imposed by trajectory fidelity offers a robust signal of generalization, as model rankings by reference τ on held-out diseases match rankings by few-shot F1 (mean ρ = 0.92) Emergence across network depth Trajectory fidelity emerges progressively across network depth, with vision-only models showing the strongest gains between 37.5–87.5% of relative depth Early layers encode some progression-relevant structure, while the diffusion manifold dimensionality remains stable at ≈4–5 dimensions throughout This suggests that deeper layers build richer representational capacity while the underlying biological progression signal remains intrinsically low-dimensional The approach provides a framework to probe whether learned representations capture the dynamics underlying their training data Trajectory exploration and interpretation Pseudotime enables analysis of transitional disease states, offering a more granular ordering of patches than discrete labels This capability allows for the reconstruction of a ”morphological movie” from non-temporal data By observing patches in the pseudotime ranges where classifications have boundaries or overlap, researchers can identify specific morphological features that precede distinct clinical endpoints This visualization tool provides a powerful, grounded framework for exploring the mechanisms of disease progression in an unprecedented way The approach may extend to any vision domain where continuous processes are observed through static snapshots Limitations and future utility Methodological constraints include the assumption of ergodicity, which means diffusion-based methods can recover relative topological ordering but cannot identify absolute temporal rates Disease states are defined by coarse ordinal labels, and the analysis reflects correlational structure rather than causal mechanisms The approach holds significant theoretical utility for computational pathology by providing a grounded framework for exploring the mechanisms of disease progression in an unprecedented way This work is methodological, probing the geometric structure of pathology foundation model representations rather than proposing clinical tools The approach may extend to any vision domain where continuous processes are observed through static snapshots, offering a framework to probe whether learned representations capture the dynamics underlying their training data The code to produce this analysis is available at https://github.

Title and authors: Tom: So we're looking at this paper titled "Do Pathology Foundation Models Encode Disease Progression? A Pseudotime Analysis of Visual Representations," and it’s about whether these massive vision models are just memorizing static pictures, or if they actually learn the continuous process of a disease unfolding.

Jane: That’s the big question, isn't it? The authors are taking diffusion pseudotime, which is a method used for single-cell transcriptomics to figure out developmental trajectories from gene expression data, and applying that idea to these foundation models' latent representations.

Lu: What they’re essentially trying to test is if these models organize disease states in their internal space along a coherent direction of progression. If they do, it means the representation isn't just separating discrete disease types; it’s capturing the continuous journey between them.

Meng: From an engineering side, I wonder how this translates to deployment. If we can map out this progression structure, does that help us build better diagnostic tools? Or is it just academic curiosity right now?

Lalam: Based on the summary, they found that pathology-specific models do recover trajectory orderings significantly better than a null baseline; vision-only models actually achieved the highest fidelity scores in this analysis.

Tom: That’s interesting because it suggests that the structure learned by these specific models—the ones trained on pathology images—is inherently more organized along a progression path than a general model would be.

Jane: But how do they prove that ordering is meaningful? They use Kendall’s rank correlation coefficient, which is this metric for measuring agreement between the pseudotime ordering they inferred and the actual ground-truth class labels.

Lu: That fidelity score, τ, quantifies how well the inferred continuous process matches what we already know about disease stages. They found that models with higher trajectory fidelity correlate strongly with better performance when they have very little data for a new task, which is a few-shot generalization metric.

Meng: So this means if we can make these foundation models learn that progression structure, it might translate into faster adaptation in real-world scenarios where you only have a handful of examples for a specific disease. That’s something I could actually use in production testing.

Title and authors: Lalam: It also showed that model rankings based on trajectory fidelity on reference diseases predicted few-shot classification performance on held out diseases with a correlation of zero point nine two.

Tom: A correlation of zero point nine two is substantial, but we gotta remember the caveat they mentioned: this method assumes ergodicity, which means diffusion-based methods can find the relative topological ordering but they cannot identify absolute temporal rates.

Jane: Exactly, so it tells us *that* there is a progression direction, not *how fast* that progression happens in real time. This is a huge distinction when we think about modeling dynamic biological systems.

Lu: The analysis also showed that trajectory fidelity complements the classification benchmarks because it offers insights into continuous progressions instead of just discrete labels, which aligns with the geometric structure they observed in the embeddings.

Meng: I like that part about the geometry; if we can see this underlying geometric constraint, it gives us a robust signal for generalization, even when we haven't seen those specific diseases before.

Lalam: And they found that this ranking by trajectory fidelity on reference diseases matched rankings by few-shot F1 scores on held out diseases with the same zero point nine two correlation.

Tom: It’s really neat that the biological grounding analysis also showed that the variation in cell types, like fibroblasts and neutrophils, aligns with known patterns of stromal remodeling during colorectal cancer progression.

Jane: That connects the abstract mathematical ordering to actual biology; it shows that what the model is learning geometrically corresponds to real physical changes happening in tissue.

Lu: The way they found this emerges progressively across network depth; vision-only models showed the strongest gains between thirty seven point five and eighty seven point five percent of relative depth, suggesting deeper layers build richer capacity while the underlying progression signal stays low dimensional, around four or five dimensions.

Meng: So we don't need to train the whole gigantic model just to capture this; some structure is already being built in those early layers that’s relevant to the biology.

Lalam: That suggests that the underlying biological signal itself is intrinsically low dimensional, even if the model learns a very complex mapping.

Title and authors: Tom: So what does this mean for us practically? It gives us a framework to probe whether learned representations truly capture the dynamics of the training data, not just surface-level correlations.

Jane: The exploration part is really powerful; pseudotime lets them look at transitional disease states, which gives them a more granular ordering than just looking at discrete labels.

Lu: It allows for reconstructing what they call a "morphological movie" from static images by observing patches in the pseudotime ranges where classifications have boundaries or overlap, which helps identify specific morphological features that come before certain clinical endpoints.

Meng: That visualization tool sounds incredibly useful for feature engineering; instead of just looking at a final classification, you can see the path the tissue took to get there.

Lalam: It’s a grounded framework for exploring progression mechanisms in any vision domain where continuous processes are observed through static snapshots.

Tom: So to wrap this up, this work on "Do Pathology Foundation Models Encode Disease Progression? A Pseudotime Analysis of Visual Representations" suggests that foundation models can implicitly learn these continuous disease progressions if they are trained on the right data.

Jane: And the key is using diffusion pseudotime to measure trajectory fidelity, which acts as a way to check representation quality beyond just looking at how well the model classifies known diseases.

Lu: The main implication is that we can use this geometric structure—this progression direction in representation space—as a robust signal for generalization, and it shows how deep layers build on top of that inherent biological signal.

Meng: For deployment, it’s a way to test if our models are learning the right kind of continuous information, not just spurious correlations.

Lalam: It’s important to remember the limitation they stated: diffusion-based methods can only recover relative topological ordering because they assume ergodicity and don't capture absolute temporal rates.

Tom: So we have a good picture of the structure, but not the speed of life in that structure.

Jane: Right, and it’s still a big step forward for understanding how these models actually see and organize complex biological processes like disease progression.

The paper's summary: Tom: So, we're looking at this paper that asks if these massive vision models actually learn how diseases progress by looking at their internal representations using something called diffusion pseudotime.

Jane: Right, so the main idea is they are taking a method from single-cell biology—diffusion pseudotime—and applying it to the hidden layers of these foundation models to see if those hidden layers organize disease states along a path.

Lu: What they're testing is whether these models just separate discrete disease types or if they actually capture the continuous journey between them in their internal space.

Meng: From my side, I wonder if this means we can finally map out the progression of a pathology without needing thousands of labeled examples for every single stage.

Tom: Exactly. The core finding is that pathology-specific models recover this ordering way better than a baseline where they don't know anything about disease structure at all.

Jane: They use this trajectory fidelity metric, which is Kendall’s rank correlation coefficient, to see how well their inferred progression matches the actual known stages of the disease.

Lu: And that metric isn't just a vanity score; they found models with high trajectory fidelity correlate strongly with better few-shot generalization performance on new diseases.

Meng: So it’s not just about memorizing pictures; if a model learns this continuous structure, it should be much better at handling new stuff quickly.

Tom: They also showed that the embeddings aren't just separating states; they are arranged in a geometry that actually matches the biological progression direction, like how cells change during tissue remodeling.

Jane: That’s the big biological grounding part—it shows the math isn't arbitrary; it lines up with known patterns of things like fibroblast and neutrophil abundance changing during colorectal cancer.

Lu: And they found this fidelity ranking on reference diseases actually predicted few-shot performance on held out diseases with a zero point nine two correlation, which is pretty strong evidence.

Meng: A zero point nine two correlation is solid, but we have to remember the authors are careful; this method assumes ergodicity, which means it tells us the relative order but not the absolute speed of progression.

Tom: So what this actually changes for us at scale is that if we train models on pathology data, they might implicitly learn a dynamic system rather than just a static lookup table.

Jane: It gives researchers a new way to visualize transitional states; you can look at where classifications overlap in the pseudotime space and see which specific morphological features appear right before a clinical endpoint.

Lu: This approach offers us a framework to probe whether the learned representations are capturing the actual dynamics underlying their training data, even deep within those network layers.

Meng: For deployment, it means we could use this trajectory fidelity score as an extra quality check on our models before we send them out to diagnose patients.

Tom: It’s a very grounded way to test if the AI is learning the right kind of continuous information about disease progression.

Jane: And that opens up possibilities for understanding how complex biological systems actually unfold over time, even when we only have static snapshots of those systems.

The paper's improvements: Tom: So, beyond just seeing if they *can* learn progression, the paper suggests how we can actually improve these systems using this trajectory fidelity score to guide training.

Jane: Right, so instead of just accepting a model’s raw performance on a classification task—like its accuracy—we can use this metric to steer the learning process toward representations that have better underlying biological structure.

Lu: They found that by optimizing for higher trajectory fidelity, we get models that generalize much better when we only give them a few examples for a new disease.

Meng: That means if I’m building something like an AI diagnostic tool, I can use this fidelity score to pick the best foundation model baseline, instead of just picking the one with the highest raw accuracy number.

Tom: They also show that there's this geometric constraint imposed by trajectory fidelity that helps us understand *why* a model is good; it gives us insight into how well it captures biological progression direction.

Jane: And this geometric structure is really powerful because it links the representation space directly to real-world biological patterns, like those changes we see in stromal remodeling during cancer progression.

Lu: They found that when we look at model rankings based on trajectory fidelity across different reference diseases, those rankings accurately predict how well the model will perform on held out diseases.

Meng: That’s a huge validation because it means the internal structure they are measuring isn't just an artifact of training data; it’s a reliable predictor of future performance.

Tom: They also found that this fidelity emerges progressively across different layers of the network; early layers capture some progression-relevant structure, and deeper layers build on top of that capacity.

Jane: So, we can stop thinking about the whole model as one black box and start understanding which parts are responsible for encoding that specific continuous biological ordering.

Lu: They suggest a path forward where we use this trajectory analysis not just to study what models learned, but to actively design training signals that encourage the representation learning to follow those coherent progression directions.

Meng: If we can bake this structure into the training objective, it might make our deployed models much more robust when they encounter novel tissue types or disease variations in the real world.

Tom: It moves us from just measuring static accuracy to measuring dynamic organization, which is a big step for how we evaluate complex AI systems.

Jane: So this work gives us a method not just to see what's inside the model, but also how we can guide it toward learning the continuous processes that biology actually follows.

Conclusion: Tom: So to wrap up, this paper "Do Pathology Foundation Models Encode Disease Progression? A Pseudotime Analysis of Visual Representations" suggests that foundation models can implicitly learn these continuous disease progressions if they are trained on the right data.

Jane: And the key is using diffusion pseudotime to measure trajectory fidelity, which acts as a way to check representation quality beyond just looking at how well the model classifies known diseases.

Lu: The main implication is that we can use this geometric structure—this progression direction in representation space—as a robust signal for generalization, and it shows how deep layers build on top of that inherent biological signal.

Meng: It’s important to remember they are flagging a limitation: the method assumes ergodicity, meaning it recovers relative ordering but doesn't tell us the absolute speed of progression.

Tom: So we have a good picture of the structure and direction, but not the actual timeline of how fast things move in that biological process.

Jane: That’s true; it gives us a roadmap for understanding disease mechanisms without needing absolute temporal data right away, which is useful for many real-world scenarios.

Lu: They also found that this analysis is really grounded biologically, showing that the mathematical structure aligns with known changes in cell type abundance during tissue remodeling.

Meng: That connection to actual biology makes this approach much more trustworthy than just looking at raw accuracy numbers on a test set.

Lalam: I’m quite sure of their conclusion here; they've established that trajectory fidelity is a strong predictor of few-shot performance on held out diseases, correlating with zero point nine two.

Tom: So the whole point is that if we want AI models to understand continuous change, we should be looking for this kind of geometric structure in their representations.

Jane: It's a framework for probing whether learned representations are capturing the dynamics underlying their training data rather than just surface-level correlations.

Lu: This opens the door for us to design training signals that actively encourage those models to follow coherent progression directions when they see pathology images.

Meng: From an engineering standpoint, it means we can use this fidelity score as a quality check before we deploy any model into a clinical pipeline.

Tom: It's a very grounded way to test if the AI is learning the right kind of continuous information about disease progression in that specific paper.

Jane: It’s important to remember though, this work is methodological; it's probing the geometry of pathology foundation model representations rather than proposing a clinical tool directly.

Lu: Yes, and they're open to extending this approach to any vision domain where continuous processes are observed through static snapshots.

More episodes

← Home