ImmuVis: Hyperconvolutional Foundation Model for Imaging Mass Cytometry

summary

The gist

ImmuVis presents a family of efficient foundation models for imaging mass cytometry that unifies variable-input learning, efficiency, and uncertainty in a single architecture…

In short

ImmuVis introduces a unified foundation model for imaging mass cytometry that uses marker-adaptive hyperconvolutions. This allows a single model to handle any subset of measured markers without retraining, offering flexibility and efficiency. The framework provides reliable predictions by modeling uncertainty through modified Gaussian heteroscedastic regression, making it practical for real-world clinical applications.

Key concepts

Marker-adaptive hyperconvolutions
This is the core innovation where the model generates convolutional kernels based on learned marker embeddings. This means one model can adapt to different sets of measured markers just by changing these learned embeddings, eliminating the need to retrain the entire architecture for new panels.
Heteroscedastic regression (β-NLL)
This technique is used in the loss function to model predictive uncertainty. It produces estimates of how uncertain a prediction is, which correlates strongly with how much error is expected during reconstruction. This makes the model reliable by flagging predictions where evidence from measured markers is weak.
Pan-marker latent representation
The encoder maps the entire multiplex image into a single, comprehensive latent space that captures information about all possible markers. The decoder then uses this unified representation to make point-wise predictions for any specific marker subset requested, ensuring flexibility across different analyses.

Terminology used across episodes

This episode discusses

The paper

ImmuVis: Hyperconvolutional Foundation Model for Imaging Mass Cytometry · Read on arXiv

Dawid Uchal, Marcin Mozejko, Krzysztof Gogolewski, Piotr Kupidura, Fabian Ozga, Szymon Łukasik, Jakub Giezgała, Tomasz Noconłk, Kacper Pietrzyk, Robert Pieniuta, Mateusz Sulimowicz, Michal Zmysłowski, Michal Orzyłowski, Tomasz Siłkowski, Karol Zagrłdka, Eike Staub, Ewa Szczurek

Faculty of Mathematics, Informatics and Mechanics, University of Warsaw · Merck Healthcare KGaA Darmstadt, Germany · Institute of AI for Health Helmholtz Munich

We present ImmuVis, a family of efficient foundation models for imaging mass cytometry (IMC), a high-throughput multiplex imaging technology that handles molecular marker measurements as image channels and enables large-scale spatial tissue profiling. Unlike natural images, multiplex imaging lacks a fixed channel space, as real-world marker sets vary across studies, violating a core assumption of standard vision backbones. To address this, ImmuVis introduces marker-adaptive hyperconvolutions that generate convolutional kernels from learned marker embeddings, enabling a single model to operate on arbitrary measured marker subsets without retraining. We pretrain ImmuVis on the largest dataset to date, IMC17M (28 cohorts, 24,405 images, 265 markers, over 17M patches), using self-supervised masked reconstruction. ImmuVis outperforms state-of-the-art baselines and ablations in virtual staining and downstream classification tasks at substantially lower compute cost than transformer-based alternatives, and is the sole model that provides calibrated uncertainty via a heteroscedastic likelihood objective. These results position ImmuVis as a practical framework for real-world IMC modeling.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "ImmuVis: Hyperconvolutional Foundation Model for Imaging Mass Cytometry".

Jane: Here's the gist of the paper. ImmuVis presents a family of efficient foundation models for imaging mass cytometry that unifies variable-input learning, efficiency, and uncertainty in a single architecture; this framework is practical for real-world IMC modeling Core Innovation The central innovation of ImmuVis is the introduction of marker-adaptive hyperconvolutions that generate convolutional kernels from learned marker embeddings, allowing a single model to operate on arbitrary measured marker subsets without retraining This mechanism enables a single model to operate on arbitrary measured marker subsets without retraining The architecture instantiates a marker-specific encoder-decoder pair, where the encoder maps the multiplex image to a pan-marker latent representation, and the decoder maps this representation to image-level point-wise predictions for the requested markers This design allows for panel flexibility by conditioning the feature extractor on learned marker embeddings Model Family and Efficiency ImmuVis introduces a family of foundation models spanning different computer vision backbones, including fully convolutional ImmuVisConv based on ConvNeXt v2, Swin Transformer-based ImmuVisSwin, and token-based ImmuVisViT This entire family offers substantially lower inference cost than token-concatenation-based alternatives The model family is trained on the largest IMC dataset to date, IMC17M (28 cohorts, 24,405 images, 265 markers) using self-supervised masked reconstruction Uncertainty and Reliability ImmuVis models model predictive uncertainty via modified Gaussian heteroscedastic regression to produce uncertainty estimates that highly correlate with reconstruction error The loss function used is the modified Gaussian negative loglikelihood, β-NLL, which is designed to be reliability-aware rather than purely point-estimated Uncertainty estimation fidelity is assessed by Pearson correlation between log variance and reconstruction error, which shows that all ImmuVis variants produce uncertainty estimates that closely track reconstruction error with Pearson correlations above.92 Performance Across Tasks The family of models achieves consistent improvements over state-of-the-art baselines in virtual staining and downstream classification tasks at substantially lower compute cost than transformer-based alternatives In virtual staining, ImmuVis variants outperform competitors on MSE, with the strongest results obtained by ImmuVisSwin (.061) and ImmuVisConv (.068), compared with.101 for VirTuesIMC17M+OP For representation learning for cell typing, ImmuVisConv achieves the best average Macro-F1 (.709), outperforming the strongest VirTues variant (.669) and EVAOP (.601) Computational Trade-offs ImmuVis variants operate in a substantially lower-compute time regime than prior foundation models for IMC, with ImmuVisConv requiring 89.9 GFLOPs and 13.4±0.2 ms per forward pass ImmuVisConv provides the most efficient overall trade-off, with the fewest parameters among ImmuVis variants (44.6M), the lowest GFLOPs, and lower activation memory than ImmuVisViT The results indicate that marker-adaptive ImmuVis architectures improve accuracy without relying on the high computational cost of token-based foundation models Limitations and Future Directions Several limitations remain, including the training distribution being constrained by marker availability and dataset bias, and the likelihood model assuming conditional independence across pixels and channels Future work could extend ImmuVis to additional multiplex modalities or incorporate richer priors or structured uncertainty The results suggest that operator-level channel adaptivity is an effective recipe for building practical foundation models in multiplex imaging, combining panel flexibility, efficiency, and reliability-aware predictions within a single architecture Extended Analysis The appendix provides detailed analyses of the methodology, including sampling and masking procedures for the Masked Modelling task It also details preprocessing steps such as arcsinh transformation and Butterworth low-pass filtering, which are shown to preserve meaningful marker variation while limiting technology-specific noise Furthermore, extended uncertainty calibration analysis shows that all ImmuVis variants produce uncertainty estimates that closely track reconstruction error with Pearson correlations above.92 The locality index analysis reveals that ImmuVisConvNext achieved the lowest average locality index across images and markers (26.04), outperforming VirTues (36.48) The results indicate that operator-level channel adaptivity is an effective recipe for building practical foundation models in multiplex imaging, combining panel flexibility, efficiency, and reliability-aware predictions within a single architecture Clinical Prediction In representation learning for clinical predictions, ImmuVisSwin achieves the best average Macro-F1 (.641), followed by ImmuVisConv (.638), both outperforming the strongest VirTues variant (.627) and EVAOP (.585) These results indicate that marker-adaptive pretraining yields representations that remain informative for clinical predictions under minimal downstream supervision The results suggest that operator-level channel adaptivity is an effective recipe for building practical foundation models in multiplex imaging, combining panel flexibility, efficiency, and reliability-aware predictions within a single architecture Overall, the results suggest that operator-level channel adaptivity is an effective recipe for building practical foundation models in multiplex imaging, combining panel flexibility, efficiency, and reliability-aware predictions within a single architecture The paper concludes that ImmuVis provides a practical, panel-flexible IMC foundation models family that is deployable at cohort scale and enables reliability-aware virtual staining The code is available at https://anonymous.4open.science/r/ImmuVis-9173 The results suggest that operator-level channel adaptivity is an effective recipe for building practical foundation models in multiplex imaging, combining panel flexibility, efficiency, and reliability-aware predictions within a single architecture The code is available at https://anonymous.4open.science/r/ImmuVis-9173 The results suggest that operator-level channel adaptivity is an effective recipe for building practical foundation models in multiplex imaging, combining panel flexibility, efficiency, and reliability-aware predictions within a single architecture The code is available at https://anonymous.4open.science/r/ImmuVis-9173 The results suggest that operator-level channel adaptivity is an effective recipe for building practical foundation models in multiplex imaging, combining panel flexibility, efficiency, and reliability-aware predictions within a single architecture References

1: Charlotte Giesen, Hao A O Wang, Denis Schapiro, Nevena Zivanovic, Andrea Jacobs, Bodo Hattendorf, Peter J Schüffler, Daniel Grolimund, Joachim M Buhmann, Simone Brandt, Zsuzsanna Varga, Peter J Wild, Detlef Günther and Bernd Bodenmiller.

Tom: Highly multiplexed imaging of tumor tissues with subcellular resolution by mass cytometry

2: Qing Chang, Olga I.

Jane: Ornatsky, Iram Siddiqui, Alexander Loboda, Vladimir I.

Tom: Baranov and David W.

Title and authors: Tom: So, we’ve seen how ImmuVis uses these marker-adaptive kernels to handle different marker sets without retraining the whole model before, and now let’s talk about what this whole family of models actually achieves when they put it all together on that big dataset IMC17M.

Jane: What I get from the summary is that ImmuVis isn't just a clever trick for handling variable input; it’s a whole system designed for real-world mass cytometry modeling. It says this framework unifies variable input learning, efficiency, and uncertainty estimation all inside one architecture.

Lu: Exactly. The core takeaway is that they built something practical. They are addressing the headache of fixed channel spaces in imaging by conditioning the feature extractor on those learned marker embeddings. This means one foundation model can process any measured marker subset without needing a whole new architecture or retraining from scratch.

Meng: I’m interested in how much this efficiency actually translates to deployment, because that’s where most AI research stalls. The summary points out they are comparing their family—ImmuVisConv, ImmuVisSwin, and ImmuVisViT—against token concatenation methods. And they show these new ones have substantially lower inference cost than those alternatives.

Tom: That cost reduction is significant because it makes virtual staining feasible at cohort scale, which is what the paper really wants to point out for practical use in clinical settings. It’s not just about theoretical accuracy; it’s about being able to run this efficiently on production hardware.

Jane: And they tie that efficiency right into reliability through uncertainty estimation. The summary explains that they use a modified Gaussian heteroscedastic regression loss function, called the beta negative loglikelihood, to produce uncertainty estimates that actually correlate with reconstruction error.

Lu: That correlation is what makes it reliable; it’s not just a point estimate of an answer; it’s a measure of how confident the model is based on the actual data you provided.

Meng: If that correlation stays above zero point nine two, that gives us a concrete signal for trusting the output when we’re dealing with weak evidence, which is exactly what we need when you’re trying to validate a prediction for a patient. But I still have my eye on the compute numbers.

Tom: Right, so they show that ImmuVisConv, which uses the convolutional backbone, actually offers the best overall trade-off. It requires eighty-nine point nine GFLOPs and takes thirteen point four milliseconds per forward pass on a single NVIDIA RTX five thousand Ada Generation GPU. That’s a big speed difference compared to some of the token based models they tested.

Jane: So, in short, the paper says ImmuVis gives us a panel-flexible family of foundation models that are deployable at cohort scale and enable reliability aware virtual staining. It turns the problem of marker variability into an advantage by making it easier to build robust representations.

Lu: And they lay out some limitations too; they mention that their training data distribution is constrained by marker availability, and also that their likelihood model assumes conditional independence across pixels and channels. That’s where future work needs to come in if we want this framework to be truly general.

Tom: They suggest extending ImmuVis to additional multiplex modalities, or incorporating richer priors or structured uncertainty into the model architecture. That shows they know where the gaps are and have a path forward.

The paper's summary: Tom: So we’ve seen how ImmuVis uses these marker-adaptive kernels to handle different marker sets without retraining, and now we’re looking at what they suggest is actually better about their architecture itself.

Jane: What I see here is that the way they designed the whole thing allows it to be much more flexible than previous models because it conditions the feature extractor on those learned embeddings. It means one model can handle arbitrary marker combinations without needing a whole new structure.

Lu: Yeah, and they emphasize how this architectural choice directly leads to better performance for representation learning. They show that the convolutional variant, ImmuVisConv, achieves the best average macro F1 score for cell typing on dataset thirty-eight.

Meng: Better accuracy is good, but I need to know if that comes at a steep price in terms of computational resources compared to what we’re currently running in production. The summary points out that their models offer substantially lower inference cost than those token concatenation methods.

Tom: That efficiency is a big deal because it makes the whole thing viable for real-world modeling, not just lab experiments where you can afford huge compute budgets. It’s about making this usable at cohort scale.

Jane: And they are also pushing the uncertainty aspect further. They are showing that their models produce calibrated uncertainty estimates that track reconstruction error very closely, with Pearson correlations above zero point nine two in the virtual staining task setup.

Lu: That means when you get a prediction, you actually get a reliable signal on how much you should trust it based on what the input data tells us. It moves it from just being an answer to being a trustworthy prediction.

Meng: A reliable uncertainty signal is something I need to see before I can even think about deploying this for anything clinical or high-stakes, because bad predictions with high confidence are worse than uncertain ones that are correctly calibrated.

Tom: Exactly. They’re showing that this operator level channel adaptivity—that ability to change based on the panel—is an effective recipe for building models that combine flexibility, efficiency, and those reliability signals all in one package.

Jane: It really suggests that for complex imaging like mass cytometry, we shouldn't just keep using fixed architectures; we need models that can adapt their structure or feature extraction based on the specific data they are looking at.

Lu: The implication is that this approach might become a standard way to build foundation models for any field where the input data structure varies widely, not just in imaging, but anywhere you have variable inputs.

Meng: So what does this mean for us? It means we can start thinking about building these kinds of adaptive systems into our own pipelines instead of always starting from scratch with a massive fixed model.

The paper's improvements: Tom: So to wrap up, ImmuVis presents a family of efficient foundation models that are deployable at cohort scale and enable reliability aware virtual staining for imaging mass cytometry using this hyperconvolutional architecture paper.

Jane: It’s pretty clear that this work shows operator level channel adaptivity is a really effective recipe for building practical foundation models in multiplex imaging, combining panel flexibility with efficiency and those reliability signals.

Lu: I think the big implication here is that we can start thinking about building these kinds of adaptive systems into our own pipelines instead of always starting from scratch with a massive fixed model.

Meng: It’s a practical framework for real-world modeling, and the efficiency trade-offs they show—like ImmuVisConv requiring eighty-nine point nine GFLOPs—are what I need to see to think about deployment.

Lalam: I'm confident in saying this is a very solid direction because it tackles the variable input problem head on while providing a concrete path toward more trustworthy predictions.

Tom: It’s definitely something worth keeping on our radar, especially since they showed that representation learning for clinical predictions remains informative even with minimal downstream supervision.

Jane: Exactly, and I think it sets a really high bar for how we should approach building models that need to handle complex biological data where the input isn't always perfectly standardized.

Lu: The limitations they flag about the training distribution being constrained by marker availability are important, though; they’re telling us exactly where the next research effort needs to go.

Meng: So if we want to move forward, maybe our focus should be on testing these adaptive kernels on even more diverse and noisy real-world data sets.

Tom: Yeah, that makes sense. We need to see how robust this whole architecture is when the marker availability shifts unexpectedly in a new study.

Jane: It’s exciting because it proves we don't have to sacrifice flexibility just for efficiency, which is a huge win for the whole field of medical imaging AI.

Conclusion: Tom: So to wrap up, ImmuVis presents its family of efficient foundation models for imaging mass cytometry, showing how operator level channel adaptivity is a recipe for practical modeling that combines flexibility with efficiency and reliability signals.

Jane: It’s clear that this work shows we can build systems that handle variable input without needing to completely retrain the entire model structure.

Lu: I think the big implication here is that we can start thinking about building these kinds of adaptive systems into our own pipelines instead of always starting from scratch with a massive fixed model.

Meng: It’s a practical framework for real-world modeling, and the efficiency trade-offs they show—like ImmuVisConv requiring eighty-nine point nine GFLOPs—are what I need to see to think about deployment.

Lalam: I'm confident in saying this is a very solid direction because it tackles the variable input problem head on while providing a concrete path toward more trustworthy predictions.

Tom: It’s definitely something worth keeping on our radar, especially since they showed that representation learning for clinical predictions remains informative even with minimal downstream supervision.

Jane: Exactly, and I think it sets a really high bar for how we should approach building models that need to handle complex biological data where the input isn't always perfectly standardized.

Lu: The limitations they flag about the training distribution being constrained by marker availability are important, though; they’re telling us exactly where the next research effort needs to go.

Meng: So if we want to move forward, maybe our focus should be on testing these adaptive kernels on even more diverse and noisy real-world data sets.

Tom: Yeah, that makes sense. We need to see how robust this whole architecture is when the marker availability shifts unexpectedly in a new study.

Jane: It’s exciting because it proves we don't have to sacrifice flexibility just for efficiency, which is a huge win for the whole field of medical imaging AI.

More episodes

← Home