ImmuVis: Hyperconvolutional Foundation Model for Imaging Mass Cytometry
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "ImmuVis: Hyperconvolutional Foundation Model for Imaging Mass Cytometry".
Jane: Here's the gist of the paper. ImmuVis presents a family of efficient foundation models for imaging mass cytometry that unifies variable-input learning, efficiency, and uncertainty in a single architecture; this framework is practical for real-world IMC modeling Core Innovation The central innovation of ImmuVis is the introduction of marker-adaptive hyperconvolutions that generate convolutional kernels from learned marker embeddings, allowing a single model to operate on arbitrary measured marker subsets without retraining This mechanism enables a single model to operate on arbitrary measured marker subsets without retraining The architecture instantiates a marker-specific encoder-decoder pair, where the encoder maps the multiplex image to a pan-marker latent representation, and the decoder maps this representation to image-level point-wise predictions for the requested markers This design allows for panel flexibility by conditioning the feature extractor on learned marker embeddings Model Family and Efficiency ImmuVis introduces a family of foundation models spanning different computer vision backbones, including fully convolutional ImmuVisConv based on ConvNeXt v2, Swin Transformer-based ImmuVisSwin, and token-based ImmuVisViT This entire family offers substantially lower inference cost than token-concatenation-based alternatives The model family is trained on the largest IMC dataset to date, IMC17M (28 cohorts, 24,405 images, 265 markers) using self-supervised masked reconstruction Uncertainty and Reliability ImmuVis models model predictive uncertainty via modified Gaussian heteroscedastic regression to produce uncertainty estimates that highly correlate with reconstruction error The loss function used is the modified Gaussian negative loglikelihood, β-NLL, which is designed to be reliability-aware rather than purely point-estimated Uncertainty estimation fidelity is assessed by Pearson correlation between log variance and reconstruction error, which shows that all ImmuVis variants produce uncertainty estimates that closely track reconstruction error with Pearson correlations above.92 Performance Across Tasks The family of models achieves consistent improvements over state-of-the-art baselines in virtual staining and downstream classification tasks at substantially lower compute cost than transformer-based alternatives In virtual staining, ImmuVis variants outperform competitors on MSE, with the strongest results obtained by ImmuVisSwin (.061) and ImmuVisConv (.068), compared with.101 for VirTuesIMC17M+OP For representation learning for cell typing, ImmuVisConv achieves the best average Macro-F1 (.709), outperforming the strongest VirTues variant (.669) and EVAOP (.601) Computational Trade-offs ImmuVis variants operate in a substantially lower-compute time regime than prior foundation models for IMC, with ImmuVisConv requiring 89.9 GFLOPs and 13.4±0.2 ms per forward pass ImmuVisConv provides the most efficient overall trade-off, with the fewest parameters among ImmuVis variants (44.6M), the lowest GFLOPs, and lower activation memory than ImmuVisViT The results indicate that marker-adaptive ImmuVis architectures improve accuracy without relying on the high computational cost of token-based foundation models Limitations and Future Directions Several limitations remain, including the training distribution being constrained by marker availability and dataset bias, and the likelihood model assuming conditional independence across pixels and channels Future work could extend ImmuVis to additional multiplex modalities or incorporate richer priors or structured uncertainty The results suggest that operator-level channel adaptivity is an effective recipe for building practical foundation models in multiplex imaging, combining panel flexibility, efficiency, and reliability-aware predictions within a single architecture Extended Analysis The appendix provides detailed analyses of the methodology, including sampling and masking procedures for the Masked Modelling task It also details preprocessing steps such as arcsinh transformation and Butterworth low-pass filtering, which are shown to preserve meaningful marker variation while limiting technology-specific noise Furthermore, extended uncertainty calibration analysis shows that all ImmuVis variants produce uncertainty estimates that closely track reconstruction error with Pearson correlations above.92 The locality index analysis reveals that ImmuVisConvNext achieved the lowest average locality index across images and markers (26.04), outperforming VirTues (36.48) The results indicate that operator-level channel adaptivity is an effective recipe for building practical foundation models in multiplex imaging, combining panel flexibility, efficiency, and reliability-aware predictions within a single architecture Clinical Prediction In representation learning for clinical predictions, ImmuVisSwin achieves the best average Macro-F1 (.641), followed by ImmuVisConv (.638), both outperforming the strongest VirTues variant (.627) and EVAOP (.585) These results indicate that marker-adaptive pretraining yields representations that remain informative for clinical predictions under minimal downstream supervision The results suggest that operator-level channel adaptivity is an effective recipe for building practical foundation models in multiplex imaging, combining panel flexibility, efficiency, and reliability-aware predictions within a single architecture Overall, the results suggest that operator-level channel adaptivity is an effective recipe for building practical foundation models in multiplex imaging, combining panel flexibility, efficiency, and reliability-aware predictions within a single architecture The paper concludes that ImmuVis provides a practical, panel-flexible IMC foundation models family that is deployable at cohort scale and enables reliability-aware virtual staining The code is available at https://anonymous.4open.science/r/ImmuVis-9173 The results suggest that operator-level channel adaptivity is an effective recipe for building practical foundation models in multiplex imaging, combining panel flexibility, efficiency, and reliability-aware predictions within a single architecture The code is available at https://anonymous.4open.science/r/ImmuVis-9173 The results suggest that operator-level channel adaptivity is an effective recipe for building practical foundation models in multiplex imaging, combining panel flexibility, efficiency, and reliability-aware predictions within a single architecture The code is available at https://anonymous.4open.science/r/ImmuVis-9173 The results suggest that operator-level channel adaptivity is an effective recipe for building practical foundation models in multiplex imaging, combining panel flexibility, efficiency, and reliability-aware predictions within a single architecture References
1: Charlotte Giesen, Hao A O Wang, Denis Schapiro, Nevena Zivanovic, Andrea Jacobs, Bodo Hattendorf, Peter J Schüffler, Daniel Grolimund, Joachim M Buhmann, Simone Brandt, Zsuzsanna Varga, Peter J Wild, Detlef Günther and Bernd Bodenmiller.
Tom: Highly multiplexed imaging of tumor tissues with subcellular resolution by mass cytometry
2: Qing Chang, Olga I.
Jane: Ornatsky, Iram Siddiqui, Alexander Loboda, Vladimir I.
Tom: Baranov and David W.
Title and authors: Tom: So, we’ve seen how ImmuVis uses these marker-adaptive kernels to handle different marker sets without retraining the whole model before, and now let’s talk about what this whole family of models actually achieves when they put it all together on that big dataset IMC17M.
Jane: What I get from the summary is that ImmuVis isn't just a clever trick for handling variable input; it’s a whole system designed for real-world mass cytometry modeling. It says this framework unifies variable input learning, efficiency, and uncertainty estimation all inside one architecture.
Lu: Exactly. The core takeaway is that they built something practical. They are addressing the headache of fixed channel spaces in imaging by conditioning the feature extractor on those learned marker embeddings. This means one foundation model can process any measured marker subset without needing a whole new architecture or retraining from scratch.
Meng: I’m interested in how much this efficiency actually translates to deployment, because that’s where most AI research stalls. The summary points out they are comparing their family—ImmuVisConv, ImmuVisSwin, and ImmuVisViT—against token concatenation methods. And they show these new ones have substantially lower inference cost than those alternatives.
Tom: That cost reduction is significant because it makes virtual staining feasible at cohort scale, which is what the paper really wants to point out for practical use in clinical settings. It’s not just about theoretical accuracy; it’s about being able to run this efficiently on production hardware.
Jane: And they tie that efficiency right into reliability through uncertainty estimation. The summary explains that they use a modified Gaussian heteroscedastic regression loss function, called the beta negative loglikelihood, to produce uncertainty estimates that actually correlate with reconstruction error.
Lu: That correlation is what makes it reliable; it’s not just a point estimate of an answer; it’s a measure of how confident the model is based on the actual data you provided.
Meng: If that correlation stays above zero point nine two, that gives us a concrete signal for trusting the output when we’re dealing with weak evidence, which is exactly what we need when you’re trying to validate a prediction for a patient. But I still have my eye on the compute numbers.
Tom: Right, so they show that ImmuVisConv, which uses the convolutional backbone, actually offers the best overall trade-off. It requires eighty-nine point nine GFLOPs and takes thirteen point four milliseconds per forward pass on a single NVIDIA RTX five thousand Ada Generation GPU. That’s a big speed difference compared to some of the token based models they tested.
Jane: So, in short, the paper says ImmuVis gives us a panel-flexible family of foundation models that are deployable at cohort scale and enable reliability aware virtual staining. It turns the problem of marker variability into an advantage by making it easier to build robust representations.
Lu: And they lay out some limitations too; they mention that their training data distribution is constrained by marker availability, and also that their likelihood model assumes conditional independence across pixels and channels. That’s where future work needs to come in if we want this framework to be truly general.
Tom: They suggest extending ImmuVis to additional multiplex modalities, or incorporating richer priors or structured uncertainty into the model architecture. That shows they know where the gaps are and have a path forward.
The paper's summary: Tom: So we’ve seen how ImmuVis uses these marker-adaptive kernels to handle different marker sets without retraining, and now we’re looking at what they suggest is actually better about their architecture itself.
Jane: What I see here is that the way they designed the whole thing allows it to be much more flexible than previous models because it conditions the feature extractor on those learned embeddings. It means one model can handle arbitrary marker combinations without needing a whole new structure.
Lu: Yeah, and they emphasize how this architectural choice directly leads to better performance for representation learning. They show that the convolutional variant, ImmuVisConv, achieves the best average macro F1 score for cell typing on dataset thirty-eight.
Meng: Better accuracy is good, but I need to know if that comes at a steep price in terms of computational resources compared to what we’re currently running in production. The summary points out that their models offer substantially lower inference cost than those token concatenation methods.
Tom: That efficiency is a big deal because it makes the whole thing viable for real-world modeling, not just lab experiments where you can afford huge compute budgets. It’s about making this usable at cohort scale.
Jane: And they are also pushing the uncertainty aspect further. They are showing that their models produce calibrated uncertainty estimates that track reconstruction error very closely, with Pearson correlations above zero point nine two in the virtual staining task setup.
Lu: That means when you get a prediction, you actually get a reliable signal on how much you should trust it based on what the input data tells us. It moves it from just being an answer to being a trustworthy prediction.
Meng: A reliable uncertainty signal is something I need to see before I can even think about deploying this for anything clinical or high-stakes, because bad predictions with high confidence are worse than uncertain ones that are correctly calibrated.
Tom: Exactly. They’re showing that this operator level channel adaptivity—that ability to change based on the panel—is an effective recipe for building models that combine flexibility, efficiency, and those reliability signals all in one package.
Jane: It really suggests that for complex imaging like mass cytometry, we shouldn't just keep using fixed architectures; we need models that can adapt their structure or feature extraction based on the specific data they are looking at.
Lu: The implication is that this approach might become a standard way to build foundation models for any field where the input data structure varies widely, not just in imaging, but anywhere you have variable inputs.
Meng: So what does this mean for us? It means we can start thinking about building these kinds of adaptive systems into our own pipelines instead of always starting from scratch with a massive fixed model.
The paper's improvements: Tom: So to wrap up, ImmuVis presents a family of efficient foundation models that are deployable at cohort scale and enable reliability aware virtual staining for imaging mass cytometry using this hyperconvolutional architecture paper.
Jane: It’s pretty clear that this work shows operator level channel adaptivity is a really effective recipe for building practical foundation models in multiplex imaging, combining panel flexibility with efficiency and those reliability signals.
Lu: I think the big implication here is that we can start thinking about building these kinds of adaptive systems into our own pipelines instead of always starting from scratch with a massive fixed model.
Meng: It’s a practical framework for real-world modeling, and the efficiency trade-offs they show—like ImmuVisConv requiring eighty-nine point nine GFLOPs—are what I need to see to think about deployment.
Lalam: I'm confident in saying this is a very solid direction because it tackles the variable input problem head on while providing a concrete path toward more trustworthy predictions.
Tom: It’s definitely something worth keeping on our radar, especially since they showed that representation learning for clinical predictions remains informative even with minimal downstream supervision.
Jane: Exactly, and I think it sets a really high bar for how we should approach building models that need to handle complex biological data where the input isn't always perfectly standardized.
Lu: The limitations they flag about the training distribution being constrained by marker availability are important, though; they’re telling us exactly where the next research effort needs to go.
Meng: So if we want to move forward, maybe our focus should be on testing these adaptive kernels on even more diverse and noisy real-world data sets.
Tom: Yeah, that makes sense. We need to see how robust this whole architecture is when the marker availability shifts unexpectedly in a new study.
Jane: It’s exciting because it proves we don't have to sacrifice flexibility just for efficiency, which is a huge win for the whole field of medical imaging AI.
Conclusion: Tom: So to wrap up, ImmuVis presents its family of efficient foundation models for imaging mass cytometry, showing how operator level channel adaptivity is a recipe for practical modeling that combines flexibility with efficiency and reliability signals.
Jane: It’s clear that this work shows we can build systems that handle variable input without needing to completely retrain the entire model structure.
Lu: I think the big implication here is that we can start thinking about building these kinds of adaptive systems into our own pipelines instead of always starting from scratch with a massive fixed model.
Meng: It’s a practical framework for real-world modeling, and the efficiency trade-offs they show—like ImmuVisConv requiring eighty-nine point nine GFLOPs—are what I need to see to think about deployment.
Lalam: I'm confident in saying this is a very solid direction because it tackles the variable input problem head on while providing a concrete path toward more trustworthy predictions.
Tom: It’s definitely something worth keeping on our radar, especially since they showed that representation learning for clinical predictions remains informative even with minimal downstream supervision.
Jane: Exactly, and I think it sets a really high bar for how we should approach building models that need to handle complex biological data where the input isn't always perfectly standardized.
Lu: The limitations they flag about the training distribution being constrained by marker availability are important, though; they’re telling us exactly where the next research effort needs to go.
Meng: So if we want to move forward, maybe our focus should be on testing these adaptive kernels on even more diverse and noisy real-world data sets.
Tom: Yeah, that makes sense. We need to see how robust this whole architecture is when the marker availability shifts unexpectedly in a new study.
Jane: It’s exciting because it proves we don't have to sacrifice flexibility just for efficiency, which is a huge win for the whole field of medical imaging AI.
Dawid Uchal, Marcin Mozejko, Krzysztof Gogolewski, Piotr Kupidura, Fabian Ozga, Szymon Łukasik, Jakub Giezgała, Tomasz Noconłk, Kacper Pietrzyk, Robert Pieniuta, Mateusz Sulimowicz, Michal Zmysłowski, Michal Orzyłowski, Tomasz Siłkowski, Karol Zagrłdka, Eike Staub, Ewa Szczurek
Faculty of Mathematics, Informatics and Mechanics, University of Warsaw · Merck Healthcare KGaA Darmstadt, Germany · Institute of AI for Health Helmholtz Munich
cs.CV
Submitted: 2026-02-04
Updated: 2026-11-15
Comments: 38 pages, 19 figures
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 90/100
The gist: ImmuVis presents a family of efficient foundation models for imaging mass cytometry that unifies variable-input learning, efficiency, and uncertainty in a single architecture…
Key concepts
- Marker-adaptive hyperconvolutions
- This is the core innovation where the model generates convolutional kernels based on learned marker embeddings. This means one model can adapt to different sets of measured markers just by changing these learned embeddings, eliminating the need to retrain the entire architecture for new panels.
- Heteroscedastic regression (β-NLL)
- This technique is used in the loss function to model predictive uncertainty. It produces estimates of how uncertain a prediction is, which correlates strongly with how much error is expected during reconstruction. This makes the model reliable by flagging predictions where evidence from measured markers is weak.
- Pan-marker latent representation
- The encoder maps the entire multiplex image into a single, comprehensive latent space that captures information about all possible markers. The decoder then uses this unified representation to make point-wise predictions for any specific marker subset requested, ensuring flexibility across different analyses.
Terminology
Summary
ImmuVis presents a family of efficient foundation models for imaging mass cytometry that unifies variable-input learning, efficiency, and uncertainty in a single architecture; this framework is practical for real-world IMC modeling
Core Innovation
The central innovation of ImmuVis is the introduction of marker-adaptive hyperconvolutions that generate convolutional kernels from learned marker embeddings, allowing a single model to operate on arbitrary measured marker subsets without retraining This mechanism enables a single model to operate on arbitrary measured marker subsets without retraining The architecture instantiates a marker-specific encoder-decoder pair, where the encoder maps the multiplex image to a pan-marker latent representation, and the decoder maps this representation to image-level point-wise predictions for the requested markers This design allows for panel flexibility by conditioning the feature extractor on learned marker embeddings
Model Family and Efficiency
ImmuVis introduces a family of foundation models spanning different computer vision backbones, including fully convolutional ImmuVisConv based on ConvNeXt v2, Swin Transformer-based ImmuVisSwin, and token-based ImmuVisViT This entire family offers substantially lower inference cost than token-concatenation-based alternatives The model family is trained on the largest IMC dataset to date, IMC17M (28 cohorts, 24,405 images, 265 markers) using self-supervised masked reconstruction
Uncertainty and Reliability
ImmuVis models model predictive uncertainty via modified Gaussian heteroscedastic regression to produce uncertainty estimates that highly correlate with reconstruction error The loss function used is the modified Gaussian negative loglikelihood, β-NLL, which is designed to be reliability-aware rather than purely point-estimated Uncertainty estimation fidelity is assessed by Pearson correlation between log variance and reconstruction error, which shows that all ImmuVis variants produce uncertainty estimates that closely track reconstruction error with Pearson correlations above.92
Performance Across Tasks
The family of models achieves consistent improvements over state-of-the-art baselines in virtual staining and downstream classification tasks at substantially lower compute cost than transformer-based alternatives In virtual staining, ImmuVis variants outperform competitors on MSE, with the strongest results obtained by ImmuVisSwin (.061) and ImmuVisConv (. p. 6 As shown in Table 1, ImmuVis variants outperform competitors on MSE, with
the strongest results obtained by ImmuVisSwin
(. 061) and ImmuVisConv (. 061) and ImmuVisConv (. 068), compared with
. Results Table 3 shows that ImmuVis variants provide the strongest representations for
cell typing. ImmuVisConv achieves the best
average Macro-F1 (. ImmuVisConv achieves the best
average Macro-F1 (. 709), outperforming the
strongest VirTues variant (. 709), outperforming the
strongest VirTues variant (. 669) and EVAOP
(.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Computational Trade-offs
ImmuVis variants operate in a substantially lower-compute time regime than prior foundation models for IMC, with ImmuVisConv requiring 89.9 GFLOPs and 13. p. 7 We compare the inference-time computational cost of ImmuVis variants, VirTues, and
EVAOP on a single forward pass for an IMC image of shape [40, 128, 128]. We report active parameter
count, GFLOPs, mean±std inference time in milliseconds, and peak activation memory, all obtained
on a single NVIDIA RTX 5000 Ada Generation GPU aggregated over 100 random input runs. We compare the inference-time computational cost of ImmuVis variants, VirTues, and
EVAOP on a single forward pass for an IMC image of shape [40, 128, 128]. We report active parameter
count, GFLOPs, mean±std inference time in milliseconds, and peak activation memory, all obtained
on a single NVIDIA RTX 5000 Ada Generation GPU aggregated over 100 random input runs. 2M
parameters, 1. ImmuVisConv provides the most efficient overall trade-off, with the fewest
parameters among ImmuVis variants (44.Where the paper says it
Where the paper says it
Where the paper says it
Limitations and Future Directions
Several limitations remain, including the training distribution being constrained by marker availability and dataset bias, and the likelihood model assuming conditional independence across pixels and channels Future work could extend ImmuVis to additional multiplex modalities or incorporate richer priors or structured uncertainty The results suggest that operator-level channel adaptivity is an effective recipe for building practical foundation models in multiplex imaging, combining panel flexibility, efficiency, and reliability-aware predictions within a single architecture
Extended Analysis
The appendix provides detailed analyses of the methodology, including sampling and masking procedures for the Masked Modelling task It also details preprocessing steps such as arcsinh transformation and Butterworth low-pass filtering, which are shown to preserve meaningful marker variation while limiting technology-specific noise Furthermore, extended uncertainty calibration analysis shows that all ImmuVis variants produce uncertainty estimates that closely track reconstruction error with Pearson correlations above.92 The locality index analysis reveals that ImmuVisConvNext achieved the lowest average locality index across images and markers (26. p. 9 [9] Yuval Bussi and Leeat Keren. Multiplexed image analysis: what have we achieved and where
are we headed? ImmuVisConv achieves the best
average Macro-F1 (. 709), outperforming the
strongest VirTues variant (.Where the paper says it
Where the paper says it
Clinical Prediction
In representation learning for clinical predictions, ImmuVisSwin achieves the best average Macro-F1 (.641), followed by ImmuVisConv (. p. 7 ImmuVisSwin achieves the best average Macro-F1 (. 641), followed by ImmuVisConv (. 641), followed by ImmuVisConv (. 638), both
outperforming the strongest VirTues variant (. 638), both
outperforming the strongest VirTues variant (. 627) and EVAOP (.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
Overall, the results suggest that operator-level channel adaptivity is an effective recipe for building practical foundation models in multiplex imaging, combining panel flexibility, efficiency, and reliability-aware predictions within a single architecture The paper concludes that ImmuVis provides a practical, panel-flexible IMC foundation models family that is deployable at cohort scale and enables reliability-aware virtual staining The code is available at https://anonymous.4open.science/r/ImmuVis-9173 The results suggest that operator-level channel adaptivity is an effective recipe for building practical foundation models in multiplex imaging, combining panel flexibility, efficiency, and reliability-aware predictions within a single architecture The code is available at https://anonymous. p. 9 If IMC adoption
reaches clinical practice, ImmuVis could support diagnostic pipelines through virtual staining of
missing markers and calibrated uncertainty flagging low-confidence predictions. Overall, our results
suggest that operator-level channel adaptivity is an effective recipe for building practical foundation
models in multiplex imaging, combining panel flexibility, efficiency, and reliability-aware predictions
within a single architecture. If IMC adoption
reaches clinical practice, ImmuVis could support diagnostic pipelines through virtual staining of
missing markers and calibrated uncertainty flagging low-confidence predictions. Overall, our results
suggest that operator-level channel adaptivity is an effective recipe for building practical foundation
models in multiplex imaging, combining panel flexibility, efficiency, and reliability-aware predictions
within a single architecture.Where the paper says it
Where the paper says it
References
[1] Charlotte Giesen, Hao A O Wang, Denis Schapiro, Nevena Zivanovic, Andrea Jacobs, Bodo Hattendorf, Peter J Schüffler, Daniel Grolimund, Joachim M Buhmann, Simone Brandt, Zsuzsanna Varga, Peter J Wild, Detlef Günther and Bernd Bodenmiller. Highly multiplexed imaging of tumor tissues with subcellular resolution by mass cytometry
[2] Qing Chang, Olga I. Ornatsky, Iram Siddiqui, Alexander Loboda, Vladimir I. p. 9 [2] Qing Chang, Olga I. Ornatsky, Iram Siddiqui, Alexander Loboda, Vladimir I. Ornatsky, Iram Siddiqui, Alexander Loboda, Vladimir I. Baranov, and
David W.Where the paper says it
Where the paper says it
[3] Vladan Milosevic. Different approaches to imaging mass cytometry data analysis
[4] Nat. Methods Editorial. p. 9 Bioinformatics Advances, 3(1):vbad046, 04 2023. Methods Editorial.Where the paper says it
[5] Alina Bollhagen and Bernd Bodenmiller. Highly multiplexed tissue imaging in precision oncology and translational cancer research
[6] Natalie de Souza, Shan Zhao, and Bernd Bodenmiller. Multiplex protein imaging in tumour biology
[7] Sean C. Bendall, Garry P. p. 9 [7] Sean C. Bendall, Garry P. Bendall, Garry P. Nolan, Mario Roederer, and Pratip K. Nolan, Mario Roederer, and Pratip K. Chattopadhyay.Where the paper says it
Where the paper says it
Where the paper says it
[8] Stéphane Chevrier, Helena L. Crowell, Vito R. p. 9 [8] Stéphane Chevrier, Helena L. Crowell, Vito R. Crowell, Vito R. Zanotelli, Stefanie Engler, Mark D. Zanotelli, Stefanie Engler, Mark D. Robinson,
and Bernd Bodenmiller.Where the paper says it
Where the paper says it
Where the paper says it
[9] Yuval Bussi and Leeat Keren. Multiplexed image analysis: what have we achieved and where are we headed? p. 9 [9] Yuval Bussi and Leeat Keren. Multiplexed image analysis: what have we achieved and where
are we headed?Where the paper says it
[10] Leena Latonen, Sonja Koivukoski, Umair Khan, and Pekka Ruusuvuori. Virtual staining for histology by deep learning
[11] Yu-Chen Lo, Timothy J. Keyes, Astraea Jager, Jolanda Sarno, Pablo Domizi, Ravindra Majeti, Kathleen M. Sakamoto, Norman Lacayo, Charles G. Mullighan et al. Cytofin enables integrated analysis of public mass cytometry datasets using generalized anchors
[12] Muhammad Shaban et al. Deep learning model imputes missing stains in multiplex images
[13] Johann Wenckstern, Eeshaan Jain, Kiril Vasilev, Matteo Pariset, Andreas Wicki, Gabriele Gut and Charlotte Bunne. Ai-powered virtual tissues from spatial proteomics for clinical diagnostics and biomedical discovery
[14] Yufan Liu et al. Modeling patient tissues at molecular resolution with eva
[15] David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks
[16] Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan and Zicheng Liu. Dynamic convolution: Attention over convolution kernels
[17] James M. Dolezal et al. Uncertainty-informed deep learning models enable high-confidence predictions for digital histopathology
[18] Lorenzo Papa et al. A survey on efficient vision transformers: Algorithms, techniques, and performance benchmarking
[19] Sanghyun Woo et al. Convnext v2: Co-designing and scaling convnets with masked autoencoders
[20] Ze Liu et al. Swin transformer: Hierarchical vision transformer using shifted windows
[21] Alexey Dosovitskiy et al. An image is worth 16x16 words: Transformers for image recognition at scale
[22] Mohammed Zidane et al. A review on deep learning applications in highly multiplexed tissue imaging data analysis
[23] Luke Ternes et al. Computational multiplex panel reduction to maximize information retention in breast cancer tissue microarrays
[24] Oren Kraus et al. Masked autoencoders for microscopy are scalable learners of cellular biology
[25] Yujia Bao, Srinivasan Sivanandan, and Theofanis Karaletsos. Channel vision transformers: An image is worth 1 x 16 x 16 words
[26] Rishi Bommasani et al. On the opportunities and risks of foundation models
[27] Maxime Oquab et al. DINOv2: Learning robust visual features without supervision
[28] Muhammad Shaban et al. A foundation model for spatial proteomics
[30] Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan and Zicheng Liu. Dynamic convolution: Attention over convolution kernels
[31] Kaiming He et al. Masked autoencoders are scalable vision learners
[32] Andrew Stirn et al. Faithful heteroscedastic regression with neural networks
[33] Maximilian Seitzer et al. On the pitfalls of heteroscedastic uncertainty estimation with probabilistic neural networks
[34] Zeming Lin et al. Evolutionary-scale prediction of atomiclevel protein structure with a language model
[35] Yiqun Chen and James Zou. Genept: A simple but effective foundation model for genes and cells built from chatgpt
[36] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization
[37] S. Butterworth. p. 5 Training procedure We trained all models for 200 epochs with AdamW [36] optimizer using
weight decay 10−4
and a cosine annealing learning-rate schedule with linear warm-up (for full details
see Appendix A. Preprocessing Overview For training ImmuVis model family, we preprocess raw IMC images
with variance-stabilizing arcsinh(x/5) transformation, frequency-based Butterworth [37] denoising, and intensity normalization, and then train/evaluate on fixed-size crops with standard spatial
augmentations; full procedural details and all preprocessing parameters are given in Appendix A.Where the paper says it
[38] Nils Eling et al. Multi-modal image analysis for large-scale cancer tissue studies within immucan
[39] Esther Danenberg et al. Breast tumor microenvironment structures are associated with genomic features and clinical outcome
[40] Lena Cords et al. Cancer-associated fibroblast phenotypes are associated with patient outcome in non-small cell lung cancer
[41] Lena Cords, Sandra Tietscher, Tobias Anzeneder, Claus Langwieder, Martin Rees, Natalie de Souza and Bernd Bodenmiller. Cancer-associated fibroblast classification in single-cell and spatial proteomics data
[42] Maximilian Ilse et al. Attention-based deep multiple instance learning
[43] IMMUcan Consortium. Immucan – an integrated european immuno-oncology profiling platform
[50] Jakob Einhaus et al. Spatial subsetting enables integrative modeling of oral squamous cell carcinoma multiplex imaging data
[51] Michael J. Haley et al. Hypoxia coordinates the spatial landscape of myeloid cells within glioblastoma to affect survival
[52] Tobias Hoch et al. Multiplexed imaging mass cytometry of the chemokine milieus in melanoma characterizes features of the response to immunotherapy
[53] Hartland W. Jackson, Jana R. Fischer, Vito R. p. 9 [8] Stéphane Chevrier, Helena L. Crowell, Vito R. ImmuVis: Hyperconvolutional Foundation Models for
Imaging Mass Cytometry
Dawid Uchal1∗ Marcin Mozejko ˙
1∗ Krzysztof Gogolewski1∗ Piotr Kupidura1
Fabian Ozga1 Szymon Łukasik1 Jakub Giezgała1 Tomasz Nocon´
1 Kacper Pietrzyk1
Robert Pieniuta1 Mateusz Sulimowicz1 Michał Zmysłowski1 Michał Orzyłowski1
Tomasz Siłkowski1 Karol Zagródka1 Eike Staub2 Ewa Szczurek1,3
1Faculty of Mathematics, Informatics and Mechanics, University of Warsaw, Warsaw, Poland
2Merck Healthcare KGaA, Darmstadt, Germany
3
Institute of AI for Health, Helmholtz Munich, Neuherberg, Germany
∗
shared first author order determined by coin flips
em. szczurek@mimuw.Where the paper says it
Where the paper says it
[60] Xu Xiao et al. Multiplexed imaging mass cytometry reveals distinct tumor-immune microenvironments linked to immunotherapy responses in melanoma
[61] Mayar Allam et al. Spatially variant immune infiltration scoring in human cancer tissues
[62] Thomas Hu et al. Single-cell spatial metabolomics with cell-type specific protein profiling for tissue systems biology
[63] Dan Moldoveanu et al. Spatially mapping the immune landscape of melanoma using imaging mass cytometry
[64] Daniel Schulz, Vito Riccardo Tomaso Zanotelli, Jana Raja Fischer, Denis Schapiro, Stefanie Engler, Xiao-Kang Lun, Hartland Warren Jackson and Bernd Bodenmiller. Simultaneous multiplexed imaging of mrna and proteins with subcellular resolution in breast cancer tissue samples by mass cytometry
[65] Xiao Qian Wang et al. Spatial predictors of immunotherapy response in triple-negative breast cancer
[67] Lasse Meyer et al. A stratification system for breast cancer based on basoluminal tumor cells and spatial tumor architecture
[68] Stéfan van der Walt, Johannes L. Schönberger, Juan Nunez-Iglesias, François Boulogne, Joshua D. Warner and the scikit-image contributors. p. 0 szczurek@mimuw. pl
Abstract
We present ImmuVis, a family of efficient foundation models for imaging mass
cytometry (IMC), a high-throughput multiplex imaging technology that handles
molecular marker measurements as image channels and enables large-scale spatial
tissue profiling.Where the paper says it
[69] Alina Bollhagen et al. High-resolution imaging mass cytometry to map subcellular structures
[70] Jonas Windhager et al. An end-to-end workflow for multiplexed image processing and analysis
[71] Tsuyoshi Hosogane, Ruben Casanova, and Bernd Bodenmiller. Dna-barcoded signal amplification for imaging mass cytometry enables sensitive and highly multiplexed tissue imaging
[72] Sean C. Bendall et al. p. 9 [7] Sean C. Bendall, Garry P.Where the paper says it
Table 9
Model [38] [39] [40] Avg. Macro-F1 scores are reported for 10-fold cross-validation; higher values indicate better performance The best method per endpoint / method is shown in bold
Table 10
Model [38] [39] [40] Avg. Macro-F1 scores are reported for 10-fold cross-validation; higher values indicate better performance The best score for each endpoint is shown in bold
Table 12
Model Probe Danenberg et al. [39] Cords et al. p. 7 4 Representation Learning: Cell Typing
Setup To evaluate the quality of learned single-cell representations, we use three datasets: IMMUcan Phenotyping [38], with expert-curated manual cell-type annotations (kept as a hold-out validation
set), and Danenberg et al. [39] and Cords et al. 4 Representation Learning: Cell Typing
Setup To evaluate the quality of learned single-cell representations, we use three datasets: IMMUcan Phenotyping [38], with expert-curated manual cell-type annotations (kept as a hold-out validation
set), and Danenberg et al. [39] and Cords et al. [39] Cords et al. [40]
PAM50 Grade ER ERBB2 Subtype Relapse Grade Avg. [39] Cords et al. [40]
PAM50 Grade ER ERBB2 Subtype Relapse Grade Avg.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Table 5
Model Param. GFLOPs Time [ms] Mem. p. 8 Model Param. GFLOPs Time [ms] Mem. 641
Table 5: Computational efficiency. Inference cost measured
on 100 random inputs of shape [40, 128, 128] on a single
NVIDIA RTX 5000 Ada Generation GPU.Where the paper says it
Where the paper says it
Table 4
Model Danenberg et al. [39] Cords et al. [40] PAM50 Grade ER ERBB2 Subtype Relapse Grade Avg. Macro-F1 scores are reported for 10-fold cross-validation; higher values indicate better performance ImmuVisConv achieves the best ER prediction
Table 8
Model MSE (↓) Pearson (↑) Uncertainty calibration measured by Pearson correlation (prediction vs reconstruction error), expected calibration error (ECE), and area under sparsification error (AUSE) ImmuVisConv leads on Pearson (.941), while ImmuVisSwin achieves the lowest AUSE (. p. 6 ImmuVisConv leads on Pearson (. 941),
while ImmuVisSwin achieves the lowest AUSE (.Where the paper says it
Figure 9
Each point is one LOO reconstruction (5 960 per model). Strong log–log scaling between mean predicted variance and MAE holds for all three ImmuVis variants The relationship confirms that the heteroscedastic head captures local ambiguity in the reconstruction
Figure 10
Each boxplot summarizes the distribution of marker-level Pearson correlation computed across all images ImmuVisSwin also achieves the highest Pearson correlation (.778), closely followed by ImmuVisConv (. p. 6 ImmuVisSwin also achieves the highest Pearson correlation (. 778), closely followed by
ImmuVisConv (.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Marker-adaptive hyperconvolutions enable a single model to operate on arbitrary measured marker subsets without retraining, addressing the lack of fixed channel space in multiplex imaging. This allows a single foundation model to process arbitrary marker combinations without architectural modification or retraining.
-
The family of efficient foundation models based on the hyperconvolutional architecture spans fully convolutional ImmuVisConv, Swin Transformer-based ImmuVisSwin, and ViT-based ImmuVisViT, offering substantially lower inference cost than token-concatenation-based alternatives. This provides a practical framework for real-world IMC modeling.
-
The model incorporates a heteroscedastic likelihood objective that produces calibrated uncertainty via a modified Gaussian heteroscedastic regression, highlighting unconfident predictions when measured marker sets provide weak evidence for target markers. This makes virtual staining reliability-aware rather than purely point-estimated.
-
The models can perform virtual staining and representation learning tasks with strong zero-shot performance on unseen panels, as demonstrated by the comparison against state-of-the-art baselines like VirTues and EVA. This capability is attributed to pan-marker representations that preserve local spatial contexts and capture short-range dependencies crucial to tissue architecture.
-
The models provide calibrated uncertainty estimates that closely track reconstruction error, with Pearson correlations above 0.92 across all variants in the virtual staining task setup. This offers a practical reliability signal for downstream applications, ensuring predictions are trustworthy when evidence is weak.
-
The ImmuVisConv variant achieves the best average Macro-F1 score for cell typing (0.790 on dataset [38]) and the best representation learning performance across clinical endpoints (Table 12). This indicates that marker-adaptive embeddings, particularly the convolutional variant, yield stronger single-cell representations for linear-probe cell typing than prior IMC foundation models.
-
The ImmuVisConv variant demonstrates superior computational efficiency, requiring only 89.9 GFLOPs and 13.4 milliseconds per forward pass, which is more than an order of magnitude faster than VirTues (1049.3 GFLOPs and 28.52 milliseconds). This reduces compute by more than an order of magnitude relative to VirTues and nearly 40-fold relative to EVA.
Abstract
We present ImmuVis, a family of efficient foundation models for imaging mass cytometry (IMC), a high-throughput multiplex imaging technology that handles molecular marker measurements as image channels and enables large-scale spatial tissue profiling. Unlike natural images, multiplex imaging lacks a fixed channel space, as real-world marker sets vary across studies, violating a core assumption of standard vision backbones. To address this, ImmuVis introduces marker-adaptive hyperconvolutions that generate convolutional kernels from learned marker embeddings, enabling a single model to operate on arbitrary measured marker subsets without retraining. We pretrain ImmuVis on the largest dataset to date, IMC17M (28 cohorts, 24,405 images, 265 markers, over 17M patches), using self-supervised masked reconstruction. ImmuVis outperforms state-of-the-art baselines and ablations in virtual staining and downstream classification tasks at substantially lower compute cost than transformer-based alternatives, and is the sole model that provides calibrated uncertainty via a heteroscedastic likelihood objective. These results position ImmuVis as a practical framework for real-world IMC modeling.
Related papers
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs
- From Formulas to Protocols: A Compositional View of Binary Target Segmentation Metrics
- Traffic Sign Recognition in Autonomous Driving: Dataset, Benchmark, and Field Experiment
- Efficient All-in-One Weather Restoration using Spectral Harmonization
- Peak-End-Net: A Peak-End Rule Inspired Framework for Generalizable Video Aesthetic Assessment