P3CA: Encoder-Agnostic Interpretation of Vision Foundation Model Embeddings via Spatial Probing

The paper proposes position-prompted PCA (P3 CA), an encoder-agnostic method for local probing of channel-rich spatial tensors. Given a user-selected spatial prompt, P3 CA estimates the feature normalization and dominant covariance directions within that region, then applies the resulting projection to the full tensor to visualize where locally informative directions are expressed. This produces a region-conditioned representation lens without modifying the encoder, retraining, or requiring task-specific labels. The method is implemented in EmbedVision, an interactive 3D Slicer-based workflow, and evaluated across natural images, colorectal pathology foundation-model embeddings, and spatial transcriptomic tensors. Across these settings, prompted projections reveal local structure suppressed by global PCA, improve prompt-matched pathology discrimination from frozen three-dimensional projections, and support comparison between learned and measured spatial representations.

The method is formulated as follows: Given an input image, it is patchified into spatial units, mapped by a vision encoder to feature vectors, and reassembled on their spatial grid to form X ∈ RH×W×C. Each spatial location p is associated with a feature vector xp ∈ RC. The goal of P3 CA is to generate a three-channel representation map whose contrast is optimized for a user-selected spatial context. Let G denote the full spatial grid and let omega ⊂ G denote the spatial prompt. P3 CA first fits a feature-wise preprocessing transform Tomega using only vectors inside omega and applies it to all spatial locations. Local contrast is defined as the variance of projected feature values within the prompted region, leading to maximizing wT Σomega w where Σomega is the region-conditioned covariance matrix. The solution is given by the top three eigenvectors of Σomega, showing that the local-contrast objective leads to PCA fitted within the prompted region. The locally estimated basis is then applied to every location in the full tensor, producing Zomega ∈ RH×W×3. Global PCA is recovered when omega = G; otherwise, P3 CA visualizes where the directions that maximize local contrast in the prompt are expressed across the image. For quantitative comparison, regional contrast gain is defined as CGomega(R) = Vomega(R)/VG(R), where values above one indicate that P3 CA preserves more three-dimensional feature variation in R than the global projection.

The interactive workflow is implemented in EmbedVision, built on 3D Slicer and MassVision platforms. EmbedVision loads a spatial tensor and optional reference image, supports rectangular or irregular prompts, computes global PCA or P3 CA, and displays the resulting representation map with image overlays. The interface is designed for iterative inspection, allowing users to move or redraw prompts, compare local projections with the global projection, and inspect whether prompt-relevant structure appears elsewhere in the image. EmbedVision stores the fitted preprocessing and projection transforms together with the prompt mask and display parameters, enabling cross-image or cross-sample comparison without refitting the projection. The same workflow supports learned embedding tensors and measured spatial tensors, allowing foundation-model features, molecular measurements, and gene-expression data to be explored within a common spatial interface.

The paper evaluates P3 CA in three settings. First, a natural-image example encoded with DINOv3 was used to assess whether spatial prompts expose local anatomical structure not emphasized by global PCA. Global PCA captures coarse separation between puppies and background, but facial subregions remain mixed. Prompted projections selectively increase variation associated with selected local structure: ear, muzzle/nose, and broader face prompts achieve contrast gains of 8.39, 11.26, and 13.56 over global PCA, respectively. Applying the same projections to labeled patches shows that ear and muzzle/nose prompts better separate their corresponding facial structures, while the broader face prompt gives the clearest differentiation among facial regions. This establishes that DINOv3 embeddings contain fine-grained local structure suppressed by a global projection but exposed by changing only the spatial region used to fit the projection, with an expected trade-off that optimizing contrast for a local prompt can compress unrelated regions.

Second, colorectal pathology embeddings from GigaPath and H0-mini were evaluated. Global PCA captures broad tissue-level variation but merges several diagnostically relevant regions. P3 CA projections associated with prompts 1 and 2 produce stronger local contrast in their selected regions for both encoders, with contrast gains of 2.48 and 2.24 for GigaPath, and 2.31 and 2.01 for H0-mini. The same frozen three-dimensional projections used for visualization were tested for downstream discrimination using LDA classification with leave-one-patient-out cross-validation. Prompt 1 significantly improved adenocarcinoma classification to 85.71% for GigaPath and 85.41% for H0-mini, compared with 69.07% and 59.01% for global PCA (Wilcoxon signed-rank test, p < 0.01). Prompt 2 significantly improved smooth-muscle classification to 84.04% and 79.50%, compared with 49.19% and 53.63% for global PCA (p < 0.01). Non-matching prompts did not provide comparable gains, showing that prompt-conditioned projections that improve visual contrast also define fixed 3D representations that remain discriminative for tissue classification.

Third, a glioblastoma sample with co-registered H&E and 10x Visium spatial transcriptomics was used to compare morphology-derived CONCH embeddings with native gene-expression tensors. Global PCA of the CONCH embedding captures broad morphology-associated variation but does not clearly recover the microenvironment organization shown by the module reference map. Global PCA of the transcriptomic tensor better reflects the major molecular compartments. P3 CA further refines this view by emphasizing variation within the prompted region, revealing local heterogeneity around the malignant region, including a boundary between malignant transcriptional programs and the surrounding non-malignant tumour microenvironment that is less apparent in the global projection. This extends P3 CA beyond learned image embeddings to measured molecular tensors in a shared spatial coordinate system.

The paper notes limitations: P3 CA is intended as an exploratory representation-probing tool rather than a standalone validation metric. Its output depends on the selected prompt, and optimizing contrast for one region can compress unrelated structures elsewhere in the image. The projection is linear, so nonlinear relationships in the embedding space may not be fully captured by three components. RGB colors should be interpreted as coordinates in a fitted projection space rather than fixed semantic labels. Larger studies are needed to evaluate prompt stability, inter-user variability, and the relationship between local projection structure and downstream clinical performance.

Improvements to AI systems:

1. Region-Conditioned Representation Probing: AI systems can now probe their internal feature spaces conditioned on user-selected spatial regions, rather than relying solely on global statistics. This allows the system to expose fine-grained, locally discriminative structures that global PCA suppresses, enabling more precise interpretation of what the model encodes in specific areas (e.g., facial subregions, tissue types).

2. Encoder-Agnostic Visualization without Retraining: The system can generate task-specific, interpretable 3-channel visualizations of any frozen vision encoder's embeddings (e.g., DINOv3, GigaPath, H0-mini) by fitting PCA only on a prompted region and applying it globally. This requires no fine-tuning, no labels, and no architecture changes, making it immediately deployable for model introspection.

3. Prompt-Conditioned Downstream Classification: The same frozen 3D projections produced by P3 CA can be reused as fixed feature representations for downstream tasks. The improved AI system can achieve significantly higher classification accuracy (e.g., +16.6% for adenocarcinoma, +34.9% for smooth muscle) compared to global PCA projections, simply by choosing a spatial prompt that matches the target class—enabling label-efficient, interpretable feature extraction.

4. Cross-Modal Spatial Comparison: The system can now compare learned embeddings (e.g., CONCH from H&E) and measured molecular tensors (e.g., spatial transcriptomics) in a shared spatial coordinate system. By applying the same prompt-conditioned projection to both, it can reveal whether morphology-derived features align with or diverge from gene-expression patterns, enabling joint analysis of pathology and molecular data.

5. Interactive Iterative Hypothesis Testing: The AI system can support an interactive workflow (via EmbedVision) where users can move, resize, or redraw prompts in real time and immediately see how the projected representation changes across the entire image. This allows for rapid, iterative exploration of where does this local structure appear elsewhere?—facilitating discovery of non-obvious spatial correlations.

6. Transferable Projection Transforms: The system can store the fitted preprocessing and projection transforms along with the prompt mask, enabling cross-image or cross-sample comparison without refitting. This allows the AI to apply a previously learned local contrast lens to new data, supporting consistent, reproducible analysis across datasets.

7. Quantitative Contrast Gain Metric: The system can compute a regional contrast gain (CGomega(R) = Vomega(R)/VG(R)) to quantitatively measure how much more feature variation is preserved in a target region compared to global PCA. This provides a scalar, interpretable metric for evaluating whether a prompt improves local discriminability, useful for automated prompt selection or benchmarking.

8. Local Heterogeneity Discovery in Molecular Data: The system can apply P3 CA to non-image tensors (e.g., gene-expression matrices) to reveal local heterogeneity that global projections miss—such as boundaries between malignant and non-malignant transcriptional programs in tumors—enabling finer-grained spatial biology analysis.

9. Prompt-Stability and Variability Analysis: The improved system can be extended to systematically evaluate prompt stability and inter-user variability, providing confidence intervals for projection outputs and helping users understand how sensitive their interpretations are to prompt choice—leading to more robust and trustworthy AI-assisted analysis.

10. Nonlinear Extension Readiness: While P3 CA is linear, the framework can be adapted to incorporate nonlinear dimensionality reduction (e.g., kernel PCA or autoencoders) within the prompted region, allowing the AI to capture nonlinear relationships in the embedding space that three linear components miss, without losing the region-conditioning benefit.