CD-RCM: Generalizable Continuous-Depth Novel View Synthesis for Reflectance Confocal Microscopy
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "CD-RCM: Generalizable Continuous-Depth Novel View Synthesis for Reflectance Confocal Microscopy".
Jane: Here's the gist of the paper. Reflectance confocal microscopy (RCM) provides noninvasive, cellular-resolution “optical biopsies” of human skin in vivo by acquiring en-face images at successive depths, forming a sparse z-stack Our goal is to provide continuous-depth visualization by interpolating intermediate sections and making the 3D volume isotropic This novel approach introduces CD-RCM, a feedforward model that predicts realistic, unseen depths from sparsely sampled RCM stacks Experiments demonstrate that CD-RCM achieves high-fidelity novel-view synthesis with sub-second inference time.
Tom: Modeling the Imaging Physics The work models RCM acquisition as a virtual camera undergoing axial translation to leverage geometric priors The authors approximate the imaging process as a virtual pinhole camera with fixed orientation translating along z This abstraction enables them to exploit Plücker ray embeddings and transformer architectures designed for multi-view geometry.
Jane: They further define virtual camera parameters, approximating intrinsics using a fixed pinhole model where focal lengths are set proportional to image resolution Stack canonicalization is employed to standardize poses via normalization, embedding each stack within a shared normalized coordinate system while preserving relative depth relations.
Tom: CD-RCM Architecture and Training CD-RCM builds upon transformer-based novel-view synthesis architectures by mapping input slices and their Plücker ray embeddings into a unified token space The process involves tokenizing input views using ViT patches and then mapping them to d-dimensional input patch tokens via a linear layer Target views are similarly tokenized, and the entire sequence is processed by a decoder-only transformer comprising 24 self-attention blocks and MLPs.
Title and authors: Tom: So we're looking at the paper "CD-RCM: Generalizable Continuous-Depth Novel View Synthesis for Reflectance Confocal Microscopy". This work is about taking those sparse stacks from reflectance confocal microscopy, which are like snapshots taken at different depths in human skin, and turning them into something continuous.
Jane: That’s right, Tom. So instead of just having these separate slices stacked up, this paper proposes a way to create intermediate sections you haven't actually imaged yet in one go without having to optimize things for every single stack.
Lu: The core idea is using a feedforward model called CD-RCM that models the depth variation itself as something the AI can query. It leverages the structured way RCM stacks are acquired, treating that axial process as a dimension it can work with.
Meng: But what does this actually mean for us in terms of deployment? Does this just give us a prettier picture, or is it something that actually helps when we're running these models on real tissue data?
Lalam: The paper explains that CD-RCM predicts anatomically consistent slices in a single forward pass, which is the main shift from previous methods that needed per-stack optimization.
Tom: Exactly. It moves the complexity from being something you do for every stack to being handled by this single model structure. We're talking about building isotropic three dee volumes from sparse data, which is really useful for getting those cross-sectional views we need in pathology <ref:2606.12635#pg1>.
Jane: And they address a big problem there: they don't claim to increase the actual optical resolution of the microscope, which is important because we want the synthesized image to look like it came from real tissue.
Lu: They build on transformer-based view synthesis architectures, mapping those input slices and their ray embeddings into a unified token space. That tokenization process, using ViT patches and then linear layers for d dimensional input patch tokens, is how the model processes the data before decoding it into the final slice.
Meng: How big is this model to run? Because I'm looking at production constraints, if this model needs to be optimized for every single patient stack, that’s a huge cost and deployment headache.
Tom: The paper claims sub-second inference time for novel-view synthesis, which is promising because it suggests we could generate these new views quickly once the model is trained.
Jane: They also introduced a skin-specific perceptual loss called LSPF to make sure the reconstructed slices actually look right at a microtextural level, not just photometrically correct.
Title and authors: Lu: That LSPF part is key; it's based on a domain-adapted feature extractor trained specifically on RCM mosaics, which helps the model capture those fine details that standard image loss functions often miss.
Meng: So you’re saying it’s not just about getting the overall shape right, but also capturing the tiny patterns in the skin structure? That adds a layer of complexity to training.
Tom: Right. The authors trained this CD-RCM using a weighted combination of photometric reconstruction loss, LPIPS, and that new LSPF loss to balance structural accuracy with those microtextural patterns we just talked about.
Jane: So the main takeaway is that this feedforward model can generate continuous depth visualization from sparse stacks in one go, and it’s designed to be generalizable across different patients and imaging sites.
Lu: And they mention that they are building on recent work like Zhang et al., two thousand twenty-four Jin et al <ref:2606.12635#pg2>., two thousand twenty-five Hong et al <ref:2606.12635#pg2>., two thousand twenty-four and Imtiaz et al <ref:2606.12635#pg2>., two thousand twenty-five to make this approach robust for internal geometry reconstruction <ref:2606.12635#pg2>.
Meng: I gotta ask about the limitations they mentioned. They talk about using SIFT-based registration to reduce inter-slice motion, but they admit there’s still residual misalignment in some stacks that can degrade the reconstruction quality.
Tom: That residual misalignment is a known issue when dealing with real experimental data, so acknowledging it shows a realistic view of the pipeline's current state.
Jane: And they also have to be careful because those synthesized slices are model predictions, not actual observed tissue features, which is a necessary disclaimer for any synthetic data.
Lu: They also noted that while generalizable frameworks exist, they are primarily designed for surface-level reconstruction of opaque objects under known camera poses, which limits their direct application to internal geometry reconstruction like RCM does.
Meng: So it’s good for the surface, but maybe less so for what we really need in depth visualization? That’s a fair critique from a production standpoint.
Tom: Still, the goal here is continuous-depth visualization, which opens up possibilities for things like better analysis of skin microstructure and stratum delineation.
Jane: This work suggests that CD-RCM could significantly improve how we visualize RCM data by reducing the acquisition burden while giving us more consistent cross-sectional examinations.
Lu: And the authors suggest future work could involve integrating this into interactive clinical visualization tools to test if densified stacks actually help downstream tasks like delineating the dermal-epidermal junction.
Title and authors: Meng: From a deployment angle, if they can reduce the number of required input stacks while maintaining fidelity, that could seriously cut down on our data acquisition time in a real clinic setting.
Lalam: I'm confident in the paper's direction because it shows a clear path toward synthesizing novel views without needing to re-optimize for every input stack.
Tom: It’s definitely an interesting piece of work, especially how they structured the modeling physics to leverage those geometric priors from the axial acquisition process.
Jane: So, we’ve seen how CD-RCM uses a transformer architecture and a novel loss function to bridge the gap between sparse stacks and continuous three dee visualization <ref:2606.12635#pg1>.
Lu: The ambition here is that this method moves us closer to having generalizable models for internal geometry reconstruction from microscopy data rather than just surface views.
Meng: It sounds like a solid foundation, but we'll need more data diversity before we can really trust it in a complex clinical environment where skin types vary so much.
Tom: Exactly, the authors caution that curating a larger and more diverse dataset spanning a wider range of skin types will definitely be needed to improve robustness and performance.
Jane: So, looking ahead, CD-RCM is pointing toward accelerating stack acquisition by reconstructing intermediate slices with this model when we only sparsely sample tissue sections.
Lu: That idea of reconstructing those intermediate slices on the fly using a feedforward model rather than pre-calculating everything is a significant direction for representation learning in imaging.
Meng: It’s just a prediction, remember, so any downstream analysis needs to be handled with the understanding that these synthesized slices are model predictions and not directly observed tissue features.
Tom: That's the necessary caution we need to keep in mind when we talk about using these CD-RCM outputs for clinical decisions.
Jane: And that’s where we come to the end of our discussion on this paper, "CD-RCM: Generalizable Continuous-Depth Novel View Synthesis for Reflectance Confocal Microscopy". It provides a strong framework for generating continuous depth visualization from sparse RCM stacks.
Lu: We think the approach of modeling depth variation as a queryable dimension is really interesting because it makes the model inherently capable of handling varying depths without extra work.
Meng: And I just hope that when we start putting this into production, the computational cost stays manageable and that those residual misalignments don't become a real issue under heavy use.
Lalam: It’s a promising direction for imaging science, showing how transformer architectures can be adapted to solve specific geometric reconstruction problems in microscopy.
The paper's summary: Tom: So, basically, CD-RCM takes those sparse depth images from reflectance confocal microscopy and uses an AI model to predict all the missing slices in between them in one go without having to do any per-stack math.
Jane: Right, so instead of taking ten separate snapshots and having a computer guess what’s in the middle of each one, this model learns the pattern of how depth changes across all those snapshots at once.
Tom: Exactly. The model is trained to create a continuous depth volume that looks anatomically plausible even for depths it never actually saw during training. It builds an isotropic three dee volume, which means it’s perfectly uniform in all directions, not just flat slices stacked on top of each other.
Jane: That’s the big win there. Because they are doing this as a single feedforward pass, it's way faster than old methods that had to optimize something for every single stack individually. It’s about generating those cross-sectional views we need in pathology much more efficiently.
Tom: And they didn't just rely on standard image reconstruction losses; they added something called an LSPF loss, which is skin specific. It’s designed to capture those tiny microtextural patterns that are important for diagnosis, not just the general color or shape.
Jane: That’s smart because it means the synthesized slices aren't going to look blurry or just generally wrong; they have to look like real tissue at a microscopic level. That focus on texture is what makes this approach more than just a nice-looking reconstruction.
Tom: The performance numbers show that CD-RCM beats classical interpolation methods across the board in both how accurately it reconstructs the light and how perceptually similar the final image looks to a human eye. It’s consistent across all their metrics, which is good for building trust in the model's output.
Jane: So what this means for us on a daily basis is that we could potentially use these models to speed up our analysis of skin microstructure significantly, reducing the time spent waiting for high-resolution stacks or trying to manually interpolate data.
Tom: Yeah, and they’re talking about using this framework not just for visualization but also for actually densifying those sparse stacks. They can take a collection of thin slices and reconstruct them into a solid three dee volume that matches the actual size of the sample you started with.
Jane: That capability is huge because it gets us closer to getting that clear view of things like the dermal-epidermal junction, which is something we really need to study accurately.
Tom: The authors are still being cautious about using these synthesized slices because they are just model predictions and not actual tissue features, so you have to treat them with some care in your downstream analysis.
Jane: That’s the necessary caveat because it keeps us grounded; we know it’s an AI prediction, not a direct observation of what the microscope saw. But this framework is still pointing toward a future where we can reconstruct internal geometry from much sparser input data than before.
The paper's improvements: Tom: So, the paper is suggesting three major upgrades to how we use this CD-RCM thing for actual work. It’s not just about making one better; it’s about adding capabilities that make it much more versatile for real-world imaging scenarios.
Jane: Right, so they are looking at how they can make the system more flexible and capable of handling real lab data better than just a perfect demo on clean data.
Tom: First, they want to enable post-hoc axial densification. That means taking those sparse stacks we have and using the model to reconstruct them into a solid volume that matches the actual physical dimensions of the sample you started with.
Jane: That’s important because it solves that staircase artifact problem you see when you try to stack up thin slices, giving us a proper isotropic three dee volume for analysis.
Tom: And second, they focused on domain-specific perceptual supervision. Instead of just using general image losses, they trained a skin specific feature loss called LSPF that captures those fine microtextural patterns we talked about earlier.
Jane: That’s a big deal because it means the synthesized slices will look diagnostically relevant—they won't just be photometrically correct but they’ll actually have the skin texture detail needed for accurate analysis.
Tom: And the third point is about making it more generalizable, meaning it should work across different types of skin or even different imaging sites without needing a whole new training run for every single one.
Jane: That moves it from being a specialized tool for one kind of scan to something that could be used widely in various clinical settings, which is where the real impact is going to be.
Tom: The authors are also suggesting integrating this into interactive tools. They want to see if we can use this model on actual patient data and then test if densifying the stacks helps with downstream tasks like localizing that dermal-epidermal junction.
Jane: That’s what they want—to move past just generating pretty pictures and actually using the model to help us make better medical decisions about skin structure.
Tom: It’s a solid plan for moving from a novel view synthesis concept to a practical, usable tool that helps researchers and clinicians work with their existing sparse data more effectively.
Conclusion: Tom: So we’re wrapping up on CD-RCM: Generalizable Continuous-Depth Novel View Synthesis for Reflectance Confocal Microscopy. This paper shows that we can build a single feedforward model to generate continuous depth visualizations from sparse microscopy stacks without needing per-stack optimization.
Jane: Right, so the main point is that this AI can predict all the missing slices in between them in one go, creating an isotropic volume and giving us cross-sections we couldn't get before.
Tom: And they achieved high fidelity with sub-second inference time across both photometric and perceptual metrics, which means it’s actually fast enough to be useful in some workflows.
Jane: It does mean that for someone who just listens, this suggests a way to dramatically reduce the time spent acquiring those dense stacks when you only have sparse data available.
Lu: From my side, the idea of modeling the depth variation itself as something the AI can query is powerful; it changes how we think about representation learning in three dee reconstruction.
Meng: It’s ambitious, and I gotta ask—how much does running this model cost when you have a whole batch of patient stacks to process? That inference time needs to be really low for us to deploy this at scale.
Lalam: I’m very sure that CD-RCM is the right direction because it handles the complexity of depth variation by incorporating those geometric priors directly into the transformer architecture.
Tom: Exactly, and they did a lot of work on training it using a combination of reconstruction loss and that skin specific feature loss to make sure the texture looks right.
Jane: So we’ve seen how this framework uses a transformer architecture and a novel loss function to bridge the gap between sparse stacks and continuous three dee visualization.
Lu: It moves us closer to having generalizable models for internal geometry reconstruction from microscopy data rather than just surface views.
Meng: I still have my reservations about those residual misalignments they mentioned; if the input stacks are a little messy, the output volume might get messy too.
Lalam: The authors flag that synthesized slices are model predictions, not directly observed tissue features, so we need to use them with clear communication about that limitation.
Tom: So it’s a promising direction for imaging science, showing how transformer architectures can be adapted to solve specific geometric reconstruction problems in microscopy.
Jane: This work suggests that CD-RCM could significantly improve how we visualize RCM data by reducing the acquisition burden while giving us more consistent cross-sectional examinations.
Lu: And they suggest future work should focus on integrating this into interactive clinical tools to test if densified stacks actually help with tasks like delineating the dermal-epidermal junction.
Northeastern University · Memorial Sloan Kettering Cancer Center
cs.CV
Submitted: 2026-06-10
Updated: 2026-11-10
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 83/100
The gist: Reflectance confocal microscopy (RCM) provides noninvasive, cellular-resolution “optical biopsies” of human skin in vivo by acquiring en-face images at successive depths…
Key concepts
- CD-RCM
- A novel feedforward model that predicts realistic, unseen depths from sparsely sampled RCM stacks. It works by mapping input slices and their geometric embeddings into a unified token space to synthesize continuous 3D volumes.
- Modeling the Imaging Physics
- This involves abstracting the RCM process as a virtual camera moving along the z-axis. This abstraction allows researchers to use established geometric tools like Plücker ray embeddings and transformer architectures designed for multi-view geometry.
- Skin-Specific Perceptual Feature Loss (LSPF)
- A specialized loss function used during training that focuses on capturing microtextural patterns specific to skin. By using this loss, the model ensures synthesized slices retain diagnostically relevant morphological details beyond simple photometric accuracy.
Terminology
Summary
Reflectance confocal microscopy (RCM) provides noninvasive, cellular-resolution “optical biopsies” of human skin in vivo by acquiring en-face images at successive depths, forming a sparse z-stack Our goal is to provide continuous-depth visualization by interpolating intermediate sections and making the 3D volume isotropic This novel approach introduces CD-RCM, a feedforward model that predicts realistic, unseen depths from sparsely sampled RCM stacks Experiments demonstrate that CD-RCM achieves high-fidelity novel-view synthesis with sub-second inference time. p. 0 org
∗ denotes equal contribution
Abstract
Reflectance confocal microscopy (RCM) provides noninvasive, cellular-resolution
“optical biopsies” of human skin in vivo by acquiring en-face images at successive
depths, forming a sparse z-stack. Due to optical limitations, these stacks are
anisotropic 3D volumes with lateral resolution (0.Where the paper says it
Modeling the Imaging Physics
The work models RCM acquisition as a virtual camera undergoing axial translation to leverage geometric priors The authors approximate the imaging process as a virtual pinhole camera with fixed orientation translating along z This abstraction enables them to exploit Plücker ray embeddings and transformer architectures designed for multi-view geometry. They further define virtual camera parameters, approximating intrinsics using a fixed pinhole model where focal lengths are set proportional to image resolution Stack canonicalization is employed to standardize poses via normalization, embedding each stack within a shared normalized coordinate system while preserving relative depth relations. p. 3 ,N}
∥ci∥∞ to standardize all poses via ci ← (ci − c¯)/c∞. This embeds each stack
within a shared normalized coordinate system while preserving relative depth relations.Where the paper says it
CD-RCM Architecture and Training
CD-RCM builds upon transformer-based novel-view synthesis architectures by mapping input slices and their Plücker ray embeddings into a unified token space The process involves tokenizing input views using ViT patches and then mapping them to d-dimensional input patch tokens via a linear layer Target views are similarly tokenized, and the entire sequence is processed by a decoder-only transformer comprising 24 self-attention blocks and MLPs. The training objective combines photometric reconstruction loss, LPIPS, and a skin-specific perceptual feature loss LSPF The authors train CD-RCM using a weighted combination of these losses to enforce structural accuracy and capture microtextural patterns. p. 5 5 Training Objective
We train CD-RCM using a weighted combination of complementary photometric and perceptual
losses:
L = LMSE(ˆIt, It)
| {z }
photometric loss
+ λ · LLPIPS(ˆIt, It) + γ · LSPF
| {z }
perceptual loss
, (8)
where λ and γ are scalar weights. LMSE computes pixel-wise mean squared error between ˆIt and It,
enforcing structural accuracy.Where the paper says it
Performance and Perceptual Fidelity
Quantitative results show that CD-RCM consistently outperforms classical interpolation methods across both photometric and perceptual metrics Table 1 indicates that CD-RCM achieves the best performance across all metrics, improving both photometric fidelity (PSNR, SSIM) and perceptual similarity (LPIPS, LSPF). Specifically, CD-RCM produces more accurate and perceptually consistent reconstructions than Gaussian interpolation which introduces noticeable blur The ablation study confirms that the proposed skin-specific perceptual feature loss LSPF yields the best overall performance across all metrics. p. 6 While Gaussian interpolation achieves relatively strong PSNR
and SSIM among the baselines, all interpolation approaches exhibit higher perceptual errors due to
their inability to model depth-dependent anatomical structure. In contrast, CD-RCM produces more
accurate and perceptually consistent reconstructions, achieving the best results across all metrics.Where the paper says it
Clinical and Future Implications
This work has potential positive societal impact in biomedical imaging by enabling continuous-depth reconstruction from sparse RCM stacks CD-RCM may improve visualization, reduce acquisition burden, and support more consistent analysis of skin microstructure Future work aims to integrate CD-RCM into interactive clinical visualization tools to evaluate whether densified stacks improve downstream analyses like DEJ localization. The framework also suggests a promising direction toward accelerating stack acquisition by sparsely sampling tissue sections and reconstructing intermediate slices with CD-RCM. p. 8 More broadly, our work illustrates how adapting modern neural rendering architectures to domainspecific imaging physics can open new possibilities for computational microscopy. A promising
direction is to accelerate stack acquisition by sparsely sampling tissue sections and reconstructing
intermediate slices with CD-RCM.Where the paper says it
Limitations and Mitigation
The pipeline uses SIFT-based registration to reduce inter-slice motion, but residual misalignment persists in some stacks which can degrade reconstruction quality Moreover, curating a larger and more diverse dataset spanning a wider range of skin types may further improve robustness and performance. The authors caution that synthesized slices are model predictions and should be used with clear communication that they are not directly observed tissue features. p. 6 As highlighted
in the insets, conventional interpolation methods struggle to accurately reconstruct layered tissue
structures and fine cellular features due to the lack of anatomical priors. In contrast, CD-RCM
generates anatomically plausible intermediate slices that preserve the structural continuity of the
anatomical morphology, closely matching the ground truth.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Continuous-Depth Visualization for RCM Stacks: The CD-RCM model can synthesize anatomically plausible intermediate slices at arbitrary unseen depths from sparsely sampled RCM stacks in a single feedforward pass without per-stack optimization. This enables the generation of isotropic 3D volumes and histopathology-like cross-sectional examinations, which is superior to classical interpolation methods.
-
Incorporating Domain-Specific Perceptual Supervision: The system can capture microtextural patterns not well represented by natural image losses by using a skin specific perceptual feature loss (LSPF) trained via self supervised domain adaptation on RCM mosaics. This ensures that the synthesized slices preserve diagnostically relevant morphological details, leading to higher fidelity reconstructions.
-
Enabling Post-Hoc Axial Densification: The CD-RCM framework allows for densification of sparse RCM stacks to match actual sample dimensions, enabling isotropic 3D visualization and reducing staircase artifacts in cross sections. This capability supports downstream tasks like DEJ localization and strata delineation more effectively than current methods.
Abstract
Reflectance confocal microscopy (RCM) provides noninvasive, cellular-resolution "optical biopsies" of human skin in vivo by acquiring en-face images at successive depths, forming a sparse z-stack. Due to optical limitations, these stacks are anisotropic 3D volumes with lateral resolution (0.5 μ m) about 6 times higher compared to axial resolution, which is defined by the optical sectioning (3 μ m), limiting the interpretation of tissue. Our goal is to provide continuous-depth visualization by interpolating intermediate sections and making the 3D volume isotropic. Such a representation permits arbitrary-direction sectioning, including histopathology-like cross-sectional examination, without requiring per-patient optimization. To that end, we introduce the first RCM-specific novel-view synthesis (NVS) approach, CD-RCM, a feedforward model that predicts realistic, unseen depths from sparsely sampled RCM stacks. Classical neural rendering methods focus on reconstruction from surface-level multi-view observations. In contrast to surface-level camera views, RCM can acquire optically sectioned en-face images of tissue beyond the surface up to 200 μ m. However, during visualization of the RCM stacks, observations of the shallower sections (towards the surface) obscure the deeper ones. This unique axial imaging geometry and layer-dependent anatomical organization motivated our development of a tailored architectural and training framework that explicitly accounts for RCM's depth-resolved, occlusive imaging physics. Experiments demonstrate that CD-RCM achieves high-fidelity novel-view synthesis with sub-second inference time.
Sources
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs