Continuous 3-D Latent Diffusion for Medical Image Generation and Reconstruction
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Physics. The day in general physics: instruments, methods and results across the field.
Kai: I'm Kai.
Mira: And I'm Mira, here with Lev.
Kai: We're looking at the paper: "Continuous 3-D Latent Diffusion for Medical Image Generation and Reconstruction".
Mira: Here's the gist of the paper. Continuous 3-D Latent Diffusion for Medical Image Generation and Reconstruction proposes a continuous 3-D latent diffusion model framework built on a coordinaterendering autoencoder (CoRe-AE) to handle high-resolution medical image generation and measurement-guided reconstruction efficiently.
Kai: This design avoids overlapping sub-volumes and remains differentiable with respect to the latent, allowing the decoder to be used inside measurement guided reconstruction.
Mira: Framework Overview The framework addresses two tasks under constrained memory: generation of new realistic volumes from a learned data distribution p(x) and recovery of x from incomplete measurements y through the forward model A(x).
Kai: The problem is reformulated in the latent space Z by solving find z ∼ p(z) s.t.
Paper summary: Kai: So, wrapping up this talk on "Continuous three-D Latent Diffusion for Medical Image Generation and Reconstruction," we see a framework that uses a coordinaterendering autoencoder to generate images and reconstruct them from measurements in a way that avoids repeated dense sub-volume processing <ref:2607.16491#pg1,Continuous 3-D Latent Diffusion for Medical Image Generation and Reconstruction>. It achieves speed gains by factoring out the decoder passes rather than by making the model inherently smaller.
Mira: The authors are proposing this continuous latent diffusion model on top of their CoRe-AE, which allows them to use this same latent prior for both unconditional generation and reconstruction guided by measurements through a differentiable setup. This means the latent space is used as a shared representation across different tasks.
Lev: If you look at the title and the authors, it’s about moving towards continuous representations for volumetric medical data, which is where things are heading in this field. It’s about making these models practical for real-world clinical use, even if they still have that fidelity gap compared to pixel-domain solvers.
Kai: The implication is that we can now use these diffusion priors more directly inside reconstruction pipelines without having to constantly worry about the computational cost of decoding every single voxel in every step. It’s a structural improvement in how the model interacts with the measurement process.
Mira: Essentially, they are showing a path where you can generate realistic three dee medical volumes and then reconstruct them from incomplete data using this continuous latent diffusion method, provided you accept that iterative refinement still needs to close that gap to the highest possible fidelity <ref:2607.16491#pg1>.
Conclusion: Kai: So, we’ve been looking at this paper on continuous three dimensional latent diffusion for medical image generation and reconstruction. It’s about using a coordinaterendering autoencoder to make these tasks work without having to process huge chunks of data repeatedly.
Mira: Right, it frames the problem in the latent space and shows how you can use that same learned data distribution for both creating new images and fixing bad measurements. It's a shared representation across different problems.
Lev: From an engineering standpoint, avoiding those repeated dense sub volume passes is huge because each iteration in reconstruction has to do a full decoder pass with gradients, so cutting that down makes the whole thing much faster on real hardware.
Kai: The authors built this framework to handle generating realistic volumes and recovering them from incomplete measurements. They trained their autoencoder first, then used those learned latents for both generation and reconstruction guided by those measurements through a diffusion process.
Mira: It really shows how you can use the latent space as a bridge between creating something new from scratch and trying to fix data that’s already been collected but is missing pieces. That's a powerful connection for medical imaging.
Lev: The performance numbers they show, like the reconstruction being two point four to four point nine decibels below pixel domain solvers, tell us it’s not perfect yet, but it’s getting close enough for clinical consideration in some scenarios.
Kai: So the main thing here is making this continuous latent diffusion model practical enough to use directly inside measurement guided reconstruction pipelines without having those massive computational overheads we usually deal with.
Mira: It suggests a future where the diffusion model itself is baked into how we reconstruct medical images, not just used as a separate step after the initial image has been formed.
Lev: We’re still looking at how to push those results onto actual hardware and see if that speedup translates into something useful for real-time clinical applications.
Youness Mellak, Antoine De Paepe, Dimitris Visvikis, Alexandre Bousse
University of Brest · LaTIM · Inserm
physics.med-ph
Submitted: 2026-07-17
Updated: 2026-11-06
Comments: 21 pages, 9 figures, 13 tables
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 84/100
The gist: Continuous 3-D Latent Diffusion for Medical Image Generation and Reconstruction proposes a continuous 3-D latent diffusion model framework built on a coordinaterendering autoencoder (CoRe-AE) to…
Key concepts
- Continuous 3-D Latent Diffusion Model
- This is a diffusion model designed to generate high-resolution medical images by learning the underlying data distribution in a compact latent space. It works by iteratively refining random noise into meaningful image representations, enabling both new image creation and recovery from measurements.
- Coordinatorendering Autoencoder (CoRe-AE)
- This is a specialized autoencoder used to compress 3D medical images into a lower-dimensional latent grid. The encoder reduces spatial resolution first, followed by standard convolutional blocks. This allows the model to operate efficiently in the latent space for generation and reconstruction.
- Measurement Guided Reconstruction
- This process uses incomplete data (measurements) to recover the full 3D image volume. The framework reformulates this problem in a latent space where it can be solved by modeling how measurements relate to the latent representation, enabling guided recovery.
Terminology
Summary
Continuous 3-D Latent Diffusion for Medical Image Generation and Reconstruction proposes a continuous 3-D latent diffusion model framework built on a coordinaterendering autoencoder (CoRe-AE) to handle high-resolution medical image generation and measurement-guided reconstruction efficiently. p. 0 1
Continuous 3-D Latent Diffusion for Medical Image
Generation and Reconstruction
Youness Mellak, Antoine De Paepe, Dimitris Visvikis, Alexandre Bousse
Abstract—High-resolution three-dimensional (3-D) medical latent diffusion remains limited by the cost of decoding full volumes, which existing latent autoencoders (AEs) handle through
overlapping sub-volume decoding. We propose a continuous 3-
D latent diffusion model framework for computed tomography (CT) and magnetic resonance imaging (MRI) generation
and measurement-guided reconstruction, built on a coordinaterendering autoencoder (CoRe-AE). Its decoder applies its convolutional stage once on the compact latent grid and renders voxel
intensities from spatial coordinates through a lightweight implicit
head, evaluated in memory-bounded chunks. This design avoids
overlapping sub-volumes and remains differentiable with respect
to the latent, so that the decoder can be used inside measurementguided reconstruction.Where the paper says it
Where the paper says it
Framework Overview
The framework addresses two tasks under constrained memory: generation of new realistic volumes from a learned data distribution p(x) and recovery of x from incomplete measurements y through the forward model A(x). The problem is reformulated in the latent space Z by solving find z ∼ p(z) s. p. 1 (3)
The inverse problem (2) can then be reformulated in the
latent space by replacing the image prior p(x) with a prior
p(z) over latent representations. Specifically, we solve
find z ∼ p(z) s. To make both tasks tractable at clinical resolution on a
single GPU, x is represented by a compact latent variable
z ∈ Z ≜ R
Cz×d×h×w, with latent channels Cz and latent
spatial dimensions (d, h, w) = (D/r, H/r, W/r) obtained
from a spatial downsampling factor r; the values of Cz and
r are specified in Section II-B1. The latent is produced and
decoded by a jointly trained encoder–decoder pair (Eϕ, Dψ)
with trainable parameters ϕ and ψ. The framework follows a two-stage design. The AE pair
(Eϕ, Dψ) is trained first and then frozen (Section II-B); the
diffusion prior is trained on the resulting latents (Section II-C)
and reused, without modification, as a prior that regularizes
reconstruction from y via decoded data consistency A ◦ Dψ
(Section II-D). The framework follows a two-stage design. The AE pair
(Eϕ, Dψ) is trained first and then frozen (Section II-B); the
diffusion prior is trained on the resulting latents (Section II-C)
and reused, without modification, as a prior that regularizes
reconstruction from y via decoded data consistency A ◦ Dψ
(Section II-D).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Latent Autoencoder Design
The encoder Eϕ has two stages: Eϕ = Eenc ϕ2 ◦ Edown ϕ1. The first stage, Edown ϕ1, reduces the spatial resolution by a factor of r = 4 along each axis using a small stack of lightweight, depthwise-separable convolutional blocks interleaved with two strided convolutions. p. 2 ,
Eϕ = E
enc
ϕ2
◦ E
down
ϕ1
, ϕ = (ϕ1, ϕ2) . (5)
The first stage, E
down
ϕ1
, reduces the spatial resolution by
a factor of r = 4 along each axis using a small stack of
lightweight, depthwise-separable convolutional blocks interleaved with two strided convolutions:
h = E
down
ϕ1
(x) ∈ R
Ch×d×h×w , (6)
where Ch is the hidden channel width. This lets
E
down
ϕ1
stay cheap in activation memory even though it operates
on the largest, full-resolution feature maps. The second stage, E
enc
ϕ2
, applies standard convolutional
residual blocks at full channel width on the already-reduced
grid h, which contains 64 times fewer voxels than x. In this view, decoding is no longer the construction of a
dense D × H × W tensor, which is what makes conventional
decoders expensive, but the evaluation of the continuous image
at any queried c, and it decomposes into two operations:
describing the local image content at each cell of G˜, and
combining the corresponding predictions at the query point. We factor the decoder into a feature-decoding stage and a
coordinate-decoding stage,
Dψ = Dhead
ψ2
◦ Dfeat
ψ1
, ψ = (ψ1, ψ2). a) LIIF head: This head uses the LIIF local ensemble. The vector-valued decoder Dhead
ψ2
is defined using a shared
real-valued MLP, D
mlp
ψ2
. b) CLIF head: The LIIF head predicts each voxel independently, so it cannot use the local context of neighboring
voxels. The CLIF head, introduced for 2-D images in [24],
replaces the MLP by a small convolutional network applied
on the output grid.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Training Objectives
The AE is trained with a regularized reconstruction objective L = 1/N Σ xˆ(ci) − x(ci) + λKLDKL(qϕ(· x)∥ N (0Z, IZ). The Latent Diffusion Model is trained as an unconditional denoising diffusion probabilistic model (DDPM) directly on latents z = µϕ(x) from the training set. p. 3 (12)
where xˆ(ci) is given by (10) and qϕ(· | x) by (7). Latent Diffusion Model
With Eϕ and Dψ fixed, a 3-D U-Net noise predictor ϵθ
parametrized by θ is trained as an unconditional denoising
diffusion probabilistic model (DDPM) directly on latents z =
µϕ(x) from the training set,
LDDPM = Ez,ϵ,t h
∥ϵ − ϵθ(zt, t)∥
2
2
i
, (13)
where ϵ ∼ N (0Z , IZ ), zt is a degraded version of z at
uniformly sampled time step t = 1, .Where the paper says it
Where the gains come from
The primary computational gain comes from avoiding repeated dense sub-volume processing; instead, the decoder runs its convolutions only once on the latent grid and reuses these features for all output voxels. This factorization establishes that the gain comes from avoiding repeated decoder passes, not from a smaller model. p. 9 The gain therefore comes from
avoiding repeated decoder passes, not from a smaller model
(Supplementary Fig. This factorization also establishes the main limitation of
the method. 694
3D MedDiffusion 8× 25. 202
clear trade-off: the LIIF head favours voxel-wise fidelity, and
the CLIF head favours perceptual quality. At
5123
, the decoder was 4–18 times faster than the MAISI and
3D MedDiffusion AEs and required the lowest peak graphics
processing unit (GPU) memory, including when its gradient was
computed. CoRe-AE also achieved the highest peak signal-tonoise ratio (PSNR) and structural similarity index (SSIM) among
the evaluated AEs on both modalities.Where the paper says it
Where the paper says it
Where the paper says it
Reconstruction Performance
The framework is used for reconstruction via the hard-data-consistency procedure of ReSample. The latent prior reconstructs good-quality volumes from severely undersampled measurements (Tables IV and V). p. 7 0
CoRe-AE (CLIF) 38. 0
The latent prior reconstructs good-quality volumes from
severely undersampled measurements (Tables IV and V). 968), so the fine details that are recovered are at the
right place. However, our reconstructions keep only 90–92%
of the high-frequency energy of the ground truth (PHF), and
their texture contrast error is higher (8. The resulting frozen 3-
D latent prior generated full volumes without patch seams and,
without retraining, reconstructed sparse-view CT and accelerated
MRI through hard data consistency, within 2. 9 dB PSNR
of the pixel-domain decomposed diffusion sampler (DDS). The latent-space stage,
however, needs one decoder pass with gradient per iteration,
and with the full schedule it makes the reconstruction slower
than DDS (about 1500 s). The full cost analysis is given in
Supplementary Section S5.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Continuous 3-D Latent Diffusion for Medical Image Generation and Reconstruction can generate full volumes without patch seams by using a single volumetric latent prior that supports both generation and reconstruction on one GPU.
-
The system can perform measurement guided reconstruction by being differentiable with respect to the latent, allowing it to be used inside measurement guided reconstruction procedures.
-
The model can reconstruct sparse-view CT and accelerated MRI through hard data consistency, achieving 2.4–4.9 dB PSNR of the pixel-domain decomposed diffusion sampler DDS.
-
The system can handle high-resolution medical volumes on a single GPU by using a memory efficient encoder that applies dense, wide-channel layers only after spatial downsampling, ensuring full 5123 volumes fit on one GPU.
-
The system can render voxel intensities from spatial coordinates through a lightweight implicit head, evaluated in memory bounded chunks to avoid overlapping sub-volumes and remain differentiable with respect to the latent.
Abstract
High-resolution three-dimensional (3-D) medical latent diffusion remains limited by the cost of decoding full volumes, which existing latent autoencoders (AEs) handle through overlapping sub-volume decoding. We propose a continuous 3-D latent diffusion model framework for computed tomography (CT) and magnetic resonance imaging (MRI) generation and measurement-guided reconstruction, built on a coordinate-rendering autoencoder (CoRe-AE). Its decoder applies its convolutional stage once on the compact latent grid and renders voxel intensities from spatial coordinates through a lightweight implicit head, evaluated in memory-bounded chunks. This design avoids overlapping sub-volumes and remains differentiable with respect to the latent, so that the decoder can be used inside measurement-guided reconstruction. The framework was evaluated on CT volumes of 512 voxels and MRI volumes of 256 voxels. At 512, the decoder was 4--18 times faster than the MAISI and 3D MedDiffusion AEs and required the lowest peak graphics processing unit (GPU) memory, including when its gradient was computed. CoRe-AE also achieved the highest peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) among the evaluated AEs on both modalities. The resulting frozen 3-D latent prior generated full volumes without patch seams and, without retraining, reconstructed sparse-view CT and accelerated MRI through hard data consistency, within 2.4--4.9 dB PSNR of the pixel-domain decomposed diffusion sampler (DDS). A single volumetric latent prior thus supports both generation and reconstruction of high-resolution medical volumes on one GPU. Our code and pretrained weights are available at https://github.com/Mellak/continuous-3d-latent-diffusion.
Sources
- MAISI-v2: Accelerated 3D High-Resolution Medical Image Synthesis with Rectified Flow and Region-specific Contrastive Loss
- Bilevel Optimized Implicit Neural Representation for Scan-Specific Accelerated MRI Reconstruction
- LoFi: Neural Local Fields for Scalable Image Reconstruction
- Diff-INR: Generative Regularization for Electrical Impedance Tomography
- MedIL: Implicit Latent Spaces for Generating Heterogeneous Medical Images at Arbitrary Resolutions
- Diffusion Posterior Sampling for General Noisy Inverse Problems
- Prompt-tuning latent diffusion models for inverse problems
- TorchRadon: Fast Differentiable Routines for Computed Tomography
- Improving 2D Diffusion Models for 3D Medical Imaging with Inter-Slice Consistent Stochasticity
- Investigating Data Memorization in 3D Latent Diffusion Models for Medical Image Synthesis
- SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers
- SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer
Related papers
- BART Online Open-Source Sequence Toolbox for Computational MRI
- Constrained estimation of rotational invariants of the cumulant expansion (RICE) for rapid tensor-valued diffusion MRI
- Improved Myelin Water Fraction Estimation and Uncertainty Mapping in Multicomponent T2 Relaxometry via NNLS Bootstrap Aggregation
- Radiation damage to normal mammalian tissue in vivo with laser-driven protons at ultra-high instantaneous dose rate
- BEAM3R: Beam's-eye-view architecture with Mamba-3 for implicit dose reconstruction
- MORSE-PI -- Flexible and artefact-free image reconstruction for structural and functional QSM and other phase-critical imaging applications