Continuous 3-D Latent Diffusion for Medical Image Generation and Reconstruction

arXiv:2607.16491 · physics.med-ph · Submitted 2026-07-17 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Physics. The day in general physics: instruments, methods and results across the field.

Kai: I'm Kai.

Mira: And I'm Mira, here with Lev.

Kai: We're looking at the paper: "Continuous 3-D Latent Diffusion for Medical Image Generation and Reconstruction".

Mira: Here's the gist of the paper. Continuous 3-D Latent Diffusion for Medical Image Generation and Reconstruction proposes a continuous 3-D latent diffusion model framework built on a coordinaterendering autoencoder (CoRe-AE) to handle high-resolution medical image generation and measurement-guided reconstruction efficiently.

Kai: This design avoids overlapping sub-volumes and remains differentiable with respect to the latent, allowing the decoder to be used inside measurement guided reconstruction.

Mira: Framework Overview The framework addresses two tasks under constrained memory: generation of new realistic volumes from a learned data distribution p(x) and recovery of x from incomplete measurements y through the forward model A(x).

Kai: The problem is reformulated in the latent space Z by solving find z ∼ p(z) s.t.

Paper summary: Kai: So, wrapping up this talk on "Continuous three-D Latent Diffusion for Medical Image Generation and Reconstruction," we see a framework that uses a coordinaterendering autoencoder to generate images and reconstruct them from measurements in a way that avoids repeated dense sub-volume processing <ref:2607.16491#pg1,Continuous 3-D Latent Diffusion for Medical Image Generation and Reconstruction>. It achieves speed gains by factoring out the decoder passes rather than by making the model inherently smaller.

Mira: The authors are proposing this continuous latent diffusion model on top of their CoRe-AE, which allows them to use this same latent prior for both unconditional generation and reconstruction guided by measurements through a differentiable setup. This means the latent space is used as a shared representation across different tasks.

Lev: If you look at the title and the authors, it’s about moving towards continuous representations for volumetric medical data, which is where things are heading in this field. It’s about making these models practical for real-world clinical use, even if they still have that fidelity gap compared to pixel-domain solvers.

Kai: The implication is that we can now use these diffusion priors more directly inside reconstruction pipelines without having to constantly worry about the computational cost of decoding every single voxel in every step. It’s a structural improvement in how the model interacts with the measurement process.

Mira: Essentially, they are showing a path where you can generate realistic three dee medical volumes and then reconstruct them from incomplete data using this continuous latent diffusion method, provided you accept that iterative refinement still needs to close that gap to the highest possible fidelity <ref:2607.16491#pg1>.

Conclusion: Kai: So, we’ve been looking at this paper on continuous three dimensional latent diffusion for medical image generation and reconstruction. It’s about using a coordinaterendering autoencoder to make these tasks work without having to process huge chunks of data repeatedly.

Mira: Right, it frames the problem in the latent space and shows how you can use that same learned data distribution for both creating new images and fixing bad measurements. It's a shared representation across different problems.

Lev: From an engineering standpoint, avoiding those repeated dense sub volume passes is huge because each iteration in reconstruction has to do a full decoder pass with gradients, so cutting that down makes the whole thing much faster on real hardware.

Kai: The authors built this framework to handle generating realistic volumes and recovering them from incomplete measurements. They trained their autoencoder first, then used those learned latents for both generation and reconstruction guided by those measurements through a diffusion process.

Mira: It really shows how you can use the latent space as a bridge between creating something new from scratch and trying to fix data that’s already been collected but is missing pieces. That's a powerful connection for medical imaging.

Lev: The performance numbers they show, like the reconstruction being two point four to four point nine decibels below pixel domain solvers, tell us it’s not perfect yet, but it’s getting close enough for clinical consideration in some scenarios.

Kai: So the main thing here is making this continuous latent diffusion model practical enough to use directly inside measurement guided reconstruction pipelines without having those massive computational overheads we usually deal with.

Mira: It suggests a future where the diffusion model itself is baked into how we reconstruct medical images, not just used as a separate step after the initial image has been formed.

Lev: We’re still looking at how to push those results onto actual hardware and see if that speedup translates into something useful for real-time clinical applications.

Youness Mellak, Antoine De Paepe, Dimitris Visvikis, Alexandre Bousse

University of Brest · LaTIM · Inserm

physics.med-ph

Submitted: 2026-07-17

Updated: 2026-11-06

Comments: 21 pages, 9 figures, 13 tables

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 84/100

The gist: Continuous 3-D Latent Diffusion for Medical Image Generation and Reconstruction proposes a continuous 3-D latent diffusion model framework built on a coordinaterendering autoencoder (CoRe-AE) to…

Key concepts

Continuous 3-D Latent Diffusion Model
This is a diffusion model designed to generate high-resolution medical images by learning the underlying data distribution in a compact latent space. It works by iteratively refining random noise into meaningful image representations, enabling both new image creation and recovery from measurements.
Coordinatorendering Autoencoder (CoRe-AE)
This is a specialized autoencoder used to compress 3D medical images into a lower-dimensional latent grid. The encoder reduces spatial resolution first, followed by standard convolutional blocks. This allows the model to operate efficiently in the latent space for generation and reconstruction.
Measurement Guided Reconstruction
This process uses incomplete data (measurements) to recover the full 3D image volume. The framework reformulates this problem in a latent space where it can be solved by modeling how measurements relate to the latent representation, enabling guided recovery.

Terminology

Summary

Continuous 3-D Latent Diffusion for Medical Image Generation and Reconstruction proposes a continuous 3-D latent diffusion model framework built on a coordinaterendering autoencoder (CoRe-AE) to handle high-resolution medical image generation and measurement-guided reconstruction efficiently. p. 0

Where the paper says it

1 Continuous 3-D Latent Diffusion for Medical Image Generation and Reconstruction Youness Mellak, Antoine De Paepe, Dimitris Visvikis, Alexandre Bousse Abstract—High-resolution three-dimensional (3-D) medical latent diffusion remains limited by the cost of decoding full volumes, which existing latent autoencoders (AEs) handle through overlapping sub-volume decoding. We propose a continuous 3- D latent diffusion model framework for computed tomography (CT) and magnetic resonance imaging (MRI) generation and measurement-guided reconstruction, built on a coordinaterendering autoencoder (CoRe-AE).

This design avoids overlapping sub-volumes and remains differentiable with respect to the latent, allowing the decoder to be used inside measurement guided reconstruction. p. 0
Where the paper says it

Its decoder applies its convolutional stage once on the compact latent grid and renders voxel intensities from spatial coordinates through a lightweight implicit head, evaluated in memory-bounded chunks. This design avoids overlapping sub-volumes and remains differentiable with respect to the latent, so that the decoder can be used inside measurementguided reconstruction.

Framework Overview

The framework addresses two tasks under constrained memory: generation of new realistic volumes from a learned data distribution p(x) and recovery of x from incomplete measurements y through the forward model A(x). The problem is reformulated in the latent space Z by solving find z ∼ p(z) s. p. 1

Where the paper says it

(3) The inverse problem (2) can then be reformulated in the latent space by replacing the image prior p(x) with a prior p(z) over latent representations. Specifically, we solve find z ∼ p(z) s.

t. y ≈ A ◦ Dψ(z). p. 1
Where the paper says it

To make both tasks tractable at clinical resolution on a single GPU, x is represented by a compact latent variable z ∈ Z ≜ R Cz×d×h×w, with latent channels Cz and latent spatial dimensions (d, h, w) = (D/r, H/r, W/r) obtained from a spatial downsampling factor r; the values of Cz and r are specified in Section II-B1. The latent is produced and decoded by a jointly trained encoder–decoder pair (Eϕ, Dψ) with trainable parameters ϕ and ψ.

The framework follows a two-stage design where the AE pair (Eϕ, Dψ) is trained first and then frozen. p. 2
Where the paper says it

The framework follows a two-stage design. The AE pair (Eϕ, Dψ) is trained first and then frozen (Section II-B); the diffusion prior is trained on the resulting latents (Section II-C) and reused, without modification, as a prior that regularizes reconstruction from y via decoded data consistency A ◦ Dψ (Section II-D).

The diffusion prior p(z) is then trained on the resulting latents and reused for both unconditional generation and reconstruction. p. 2
Where the paper says it

The framework follows a two-stage design. The AE pair (Eϕ, Dψ) is trained first and then frozen (Section II-B); the diffusion prior is trained on the resulting latents (Section II-C) and reused, without modification, as a prior that regularizes reconstruction from y via decoded data consistency A ◦ Dψ (Section II-D).

Latent Autoencoder Design

The encoder Eϕ has two stages: Eϕ = Eenc ϕ2 ◦ Edown ϕ1. The first stage, Edown ϕ1, reduces the spatial resolution by a factor of r = 4 along each axis using a small stack of lightweight, depthwise-separable convolutional blocks interleaved with two strided convolutions. p. 2

Where the paper says it

, Eϕ = E enc ϕ2 ◦ E down ϕ1 , ϕ = (ϕ1, ϕ2) . (5) The first stage, E down ϕ1 , reduces the spatial resolution by a factor of r = 4 along each axis using a small stack of lightweight, depthwise-separable convolutional blocks interleaved with two strided convolutions: h = E down ϕ1 (x) ∈ R Ch×d×h×w , (6) where Ch is the hidden channel width.

The second stage, Eenc ϕ2, applies standard convolutional residual blocks at full channel width on the already-reduced grid h. p. 2
Where the paper says it

This lets E down ϕ1 stay cheap in activation memory even though it operates on the largest, full-resolution feature maps. The second stage, E enc ϕ2 , applies standard convolutional residual blocks at full channel width on the already-reduced grid h, which contains 64 times fewer voxels than x.

The decoder Dψ is implicit and factors into a feature-decoding stage and a coordinate-decoding stage. p. 2
Where the paper says it

In this view, decoding is no longer the construction of a dense D × H × W tensor, which is what makes conventional decoders expensive, but the evaluation of the continuous image at any queried c, and it decomposes into two operations: describing the local image content at each cell of G˜, and combining the corresponding predictions at the query point. We factor the decoder into a feature-decoding stage and a coordinate-decoding stage, Dψ = Dhead ψ2 ◦ Dfeat ψ1 , ψ = (ψ1, ψ2).

The LIIF head uses the LIIF local ensemble to predict intensities at queried spatial coordinates using normalized trilinear weights. p. 2
Where the paper says it

a) LIIF head: This head uses the LIIF local ensemble. The vector-valued decoder Dhead ψ2 is defined using a shared real-valued MLP, D mlp ψ2 .

The CLIF head replaces the MLP with a small convolutional network applied on the output grid to predict voxel intensities. p. 2
Where the paper says it

b) CLIF head: The LIIF head predicts each voxel independently, so it cannot use the local context of neighboring voxels. The CLIF head, introduced for 2-D images in [24], replaces the MLP by a small convolutional network applied on the output grid.

Training Objectives

The AE is trained with a regularized reconstruction objective L = 1/N Σ xˆ(ci) − x(ci) + λKLDKL(qϕ(· x)∥ N (0Z, IZ). The Latent Diffusion Model is trained as an unconditional denoising diffusion probabilistic model (DDPM) directly on latents z = µϕ(x) from the training set. p. 3

Where the paper says it

(12) where xˆ(ci) is given by (10) and qϕ(· | x) by (7). Latent Diffusion Model With Eϕ and Dψ fixed, a 3-D U-Net noise predictor ϵθ parametrized by θ is trained as an unconditional denoising diffusion probabilistic model (DDPM) directly on latents z = µϕ(x) from the training set, LDDPM = Ez,ϵ,t h ∥ϵ − ϵθ(zt, t)∥ 2 2 i , (13) where ϵ ∼ N (0Z , IZ ), zt is a degraded version of z at uniformly sampled time step t = 1, .

Where the gains come from

The primary computational gain comes from avoiding repeated dense sub-volume processing; instead, the decoder runs its convolutions only once on the latent grid and reuses these features for all output voxels. This factorization establishes that the gain comes from avoiding repeated decoder passes, not from a smaller model. p. 9

Where the paper says it

The gain therefore comes from avoiding repeated decoder passes, not from a smaller model (Supplementary Fig. This factorization also establishes the main limitation of the method.

The LIIF head favours voxel-wise fidelity while the CLIF head favours perceptual quality. p. 4
Where the paper says it

694 3D MedDiffusion 8× 25. 202 clear trade-off: the LIIF head favours voxel-wise fidelity, and the CLIF head favours perceptual quality.

The method achieves the highest PSNR and SSIM among evaluated AEs. p. 0
Where the paper says it

At 5123 , the decoder was 4–18 times faster than the MAISI and 3D MedDiffusion AEs and required the lowest peak graphics processing unit (GPU) memory, including when its gradient was computed. CoRe-AE also achieved the highest peak signal-tonoise ratio (PSNR) and structural similarity index (SSIM) among the evaluated AEs on both modalities.

Reconstruction Performance

The framework is used for reconstruction via the hard-data-consistency procedure of ReSample. The latent prior reconstructs good-quality volumes from severely undersampled measurements (Tables IV and V). p. 7

Where the paper says it

0 CoRe-AE (CLIF) 38. 0 The latent prior reconstructs good-quality volumes from severely undersampled measurements (Tables IV and V).

However, the reconstructions remain 2. p. 7
Where the paper says it

968), so the fine details that are recovered are at the right place. However, our reconstructions keep only 90–92% of the high-frequency energy of the ground truth (PHF), and their texture contrast error is higher (8.

4 to 4.9 dB below the pixel-domain DDS solver. p. 0
Where the paper says it

The resulting frozen 3- D latent prior generated full volumes without patch seams and, without retraining, reconstructed sparse-view CT and accelerated MRI through hard data consistency, within 2. 9 dB PSNR of the pixel-domain decomposed diffusion sampler (DDS).

The reconstruction cost is dominated by the latent-space stage because each iteration requires a decoder pass with gradient. p. 8
Where the paper says it

The latent-space stage, however, needs one decoder pass with gradient per iteration, and with the full schedule it makes the reconstruction slower than DDS (about 1500 s). The full cost analysis is given in Supplementary Section S5.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Continuous 3-D Latent Diffusion for Medical Image Generation and Reconstruction can generate full volumes without patch seams by using a single volumetric latent prior that supports both generation and reconstruction on one GPU.

  2. The system can perform measurement guided reconstruction by being differentiable with respect to the latent, allowing it to be used inside measurement guided reconstruction procedures.

  3. The model can reconstruct sparse-view CT and accelerated MRI through hard data consistency, achieving 2.4–4.9 dB PSNR of the pixel-domain decomposed diffusion sampler DDS.

  4. The system can handle high-resolution medical volumes on a single GPU by using a memory efficient encoder that applies dense, wide-channel layers only after spatial downsampling, ensuring full 5123 volumes fit on one GPU.

  5. The system can render voxel intensities from spatial coordinates through a lightweight implicit head, evaluated in memory bounded chunks to avoid overlapping sub-volumes and remain differentiable with respect to the latent.

Abstract

High-resolution three-dimensional (3-D) medical latent diffusion remains limited by the cost of decoding full volumes, which existing latent autoencoders (AEs) handle through overlapping sub-volume decoding. We propose a continuous 3-D latent diffusion model framework for computed tomography (CT) and magnetic resonance imaging (MRI) generation and measurement-guided reconstruction, built on a coordinate-rendering autoencoder (CoRe-AE). Its decoder applies its convolutional stage once on the compact latent grid and renders voxel intensities from spatial coordinates through a lightweight implicit head, evaluated in memory-bounded chunks. This design avoids overlapping sub-volumes and remains differentiable with respect to the latent, so that the decoder can be used inside measurement-guided reconstruction. The framework was evaluated on CT volumes of 512 voxels and MRI volumes of 256 voxels. At 512, the decoder was 4--18 times faster than the MAISI and 3D MedDiffusion AEs and required the lowest peak graphics processing unit (GPU) memory, including when its gradient was computed. CoRe-AE also achieved the highest peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) among the evaluated AEs on both modalities. The resulting frozen 3-D latent prior generated full volumes without patch seams and, without retraining, reconstructed sparse-view CT and accelerated MRI through hard data consistency, within 2.4--4.9 dB PSNR of the pixel-domain decomposed diffusion sampler (DDS). A single volumetric latent prior thus supports both generation and reconstruction of high-resolution medical volumes on one GPU. Our code and pretrained weights are available at https://github.com/Mellak/continuous-3d-latent-diffusion.

Sources

Related papers