MedGS: Gaussian Splatting for Multi-Modal 3D Medical Imaging

summary

The gist

MedGS presents a novel framework based on Gaussian Splatting designed for high-fidelity 3D anatomical reconstruction by simultaneously performing frame interpolation and segmentation using a unified…

In short

MedGS uses a Gaussian Splatting framework to reconstruct high-fidelity 3D medical anatomy from sequential images. It simultaneously performs frame interpolation and segmentation by coupling visual signals to regularize a unified geometric representation of Folded-Gaussians, achieving superior mesh reconstruction accuracy compared to other methods.

Key concepts

Folded-Gaussian Distribution
This is the core 3D geometric representation used by MedGS. Each Gaussian primitive in this distribution encodes both its spatial location and temporal deformation. Crucially, it carries two sets of color coefficients: one for image intensity and another for the binary segmentation mask, linking geometry directly to visual features.
Multi-task Coupling
MedGS trains a single model to perform multiple related tasks at once: image reconstruction, frame interpolation, and segmentation. The visual data from the image branch is used to regularize the underlying geometric structure. This coupling ensures that the learned 3D shape is consistent with the visual appearance of both frames.
In-Between Frame Regularization (IBFR)
This mechanism enhances frame interpolation by generating a pseudo-ground truth frame between two input slices. It forces the Folded-Gaussians to smoothly approximate the transition between slices rather than just interpolating pixel values. This prevents artifacts and ensures that moving between sequential medical scans results in high-quality, anatomically plausible intermediate frames.

Terminology used across episodes

This episode discusses

The paper

MedGS: Gaussian Splatting for Multi-Modal 3D Medical Imaging · Read on arXiv

Jagiellonian University of Poland · Warsaw University of Technology of Poland Institute for IDEAS Research Institute of Poland Sano Centre for Computational Medicine and Military Institute of Medicine

Multi-modal three-dimensional (3D) medical imaging data, derived from ultrasound, magnetic resonance imaging (MRI), and computed tomography (CT), provide a widely adopted approach for non-invasive anatomical visualization. However, accurate modeling depends on surface reconstruction and frame-to-frame interpolation, where traditional methods often struggle with image noise and incomplete information between sparse frames. To address these challenges, we present, a novel framework based on Gaussian Splatting (GS) designed for high-fidelity 3D anatomical reconstruction. Uniquely, employs a multi-task architecture that simultaneously performs frame interpolation and segmentation using a unified geometric representation. By coupling these tasks, the model leverages dense signals from image synthesis to regularize the geometry, enabling high-quality surface extraction even from a limited number of input frames. Specifically, medical data are modeled as Folded-Gaussians with dual color attributes, supported by an In-Between Frame Regularization (IBFR) mechanism. Experimental results demonstrate that achieves higher metric scores than implicit neural representations and improves interpolation quality. Repository: https://github.com/gmum/MedGS

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "MedGS: Gaussian Splatting for Multi-Modal 3D Medical Imaging".

Jane: Here's the gist of the paper. MedGS presents a novel framework based on Gaussian Splatting designed for high-fidelity 3D anatomical reconstruction by simultaneously performing frame interpolation and segmentation using a unified geometric representation This multi-task coupling leverages the visual signal from the image branch to regularize the underlying Folded-Gaussian distribution By coupling these tasks, the model leverages dense signals from image synthesis to regularize geometry, enabling high-quality surface extraction even from sparse frame sequences Core Framework and Representation MedGS represents volumetric data as sequences of 2D cross-sections in 3D space The core of MedGS is a collection of 3D Folded-Gaussians that share spatial and temporal attributes but possess dual appearance characteristics Formally, the model is defined as a set of primitives GMedGS = nFN (mi, Σi, ai, fi), ρi, c img i, c seg ioN i=1 Here the geometry is parameterized by the Folded-Gaussian distribution FN (defined in Eq.

Tom: 1), comprising the mean mi, covariance Σi, and temporal deformation functions ai, fi Crucially each Gaussian i carries two distinct sets of color coefficients: c img i: Reconstructs the grayscale intensity of the medical scan and c seg i: Reconstructs the binary occupancy mask of the anatomical structure This shared parameterization enforces a strong structural prior where geometric primitives defining the organ’s boundary in segmentation are the same primitives generating visual texture in interpolation Training Signal Construction MedGS employs a multi-task learning strategy that optimizes both shared geometry and distinct color sets simultaneously The total objective function combines losses from both tasks and temporal constraints: Ltotal = Limg + λsegLseg + λσLσ The image loss Limg combines L1 reconstruction, Structural Similarity (SSIM), and the In-Between Frame Regularization (IBFR) term The segmentation loss Lseg minimizes the difference between rendered masks and ground truth masks Finally the model regularizes the time-spread parameter σt of the Folded-Gaussians via Lσ The In-Between Frame Regularization (IBFR) generates a pseudo-ground-truth interpolated frame Itα = α · It + (1 − α) · It+1, where α ∼ U(0.2, 0.8), introducing a regularization loss L∗interp Performance and Results Experimental results demonstrate that MedGS achieves higher metric scores than implicit neural representations and improves interpolation quality For mesh reconstruction on the Prostate Ultrasound dataset Table 1 shows MedGS attains the lowest Chamfer and 95% Hausdorff distances compared to FUNSR and Poisson reconstructions In interpolation evaluations Table 3 confirms that MedGS consistently outperforms linear interpolation and optical flow across PSNR and SSIM metrics Qualitative results in Figure 3 confirm that MedGS maintains sharp anatomical boundaries whereas baselines exhibit significant blurring or distortion Model Variants and Limitations To address data heterogeneity two specialized sub-models are introduced I.

Jane: MedGSi (Interpolation Only) is optimized solely for the view synthesis and denoising of medical slices discarding the mask branch c seg II.

Title and authors: Jane: Okay, so we know it’s about using Gaussian Splatting for medical imaging. What exactly is this unified geometry they are talking about? Is it just some fancy way of storing the shape?

Tom: Think of it like this: they are defining these primitives, which are basically three dimensional Folded-Gaussians, and each one carries two different sets of color coefficients. One set reconstructs the grayscale intensity from the medical scan, and another set reconstructs the binary mask for where the actual anatomy is.

Lu: That’s a crucial detail. By having both those colors tied to the same underlying geometric primitive, they create this strong structural prior that links how things look visually with how they are actually shaped in three dimensional space.

Jane: And that means if the visual signal from the image branch changes, it has to respect those geometric rules, which should help smooth out any noise in the reconstruction process.

Meng: From a deployment standpoint, having this unified representation means you’re not training two separate models for geometry and segmentation; it’s one system doing both tasks simultaneously.

Lalam: It really boils down to making sure the visual texture information is directly informing the shape estimation, which should lead to much more coherent results than doing them separately.

The paper's summary: Tom: So, they’ve got this multi-task learning setup. Their total objective function combines losses from the image task, the segmentation task, and a temporal constraint on how the Gaussians spread over time.

Jane: That temporal constraint is interesting because it’s what drives the frame interpolation part of their work. They use something called In-Between Frame Regularization, or IBFR, which creates a pseudo-ground truth interpolated frame by combining two existing slices.

Lu: The IBFR term is key because it forces those Folded-Gaussians to smoothly approximate the transition between slices instead of just picking an average or collapsing into discrete layers. It regularizes the time spread parameter, sigma t.

Meng: When you’re shipping this model, you’d want to know how stable that temporal regularization is. If it’s too aggressive, it might cause artifacts in the interpolated frames at scale.

Lalam: The paper suggests that this coupling is what enables high-fidelity three dee mesh extraction even when the input slices are sparse <ref:2509.16806#pg1,enables high-fidelity 3D mesh extraction even>. It's about using dense signals from image synthesis to regularize the geometry, which is a powerful idea for low-data scenarios.

The paper's improvements: Jane: Okay, so they show some results on prostate ultrasound data. What’s the main difference between their output and what we see from other methods like FUNSR or Poisson reconstruction?

Tom: They show that MedGS achieves lower Chamfer and ninety five percent Hausdorff distances for mesh reconstruction compared to those baselines. That means their resulting three dee meshes are geometrically more accurate <ref:2509.16806#pg1>.

Lu: Specifically, they claim superior topology preservation and edge continuity in the qualitative results on prostate ultrasound, heart, lung, and kidney data. FUNSR and Poisson produce less smooth meshes and fail when the sampling is sparse.

Meng: Better geometric accuracy is good for surgical planning or anything where you need precise boundaries. But how does this translate to real-world performance when we have noisy medical scans?

Lalam: The improvement isn't just in accuracy, it’s in the quality of the surface geometry itself, which is what matters most for things like monitoring disease progression over time.

Conclusion: Tom: So to wrap up, this paper on "MedGS: Gaussian Splatting for Multi-Modal three dee Medical Imaging" shows that coupling frame interpolation and segmentation through a unified geometric representation works very well <ref:2509.16806#pg1>.

Jane: The main implication is that we can get much higher fidelity three dee anatomical reconstructions from sparse medical image sequences than we could before, because the visual data directly guides the shape estimation <ref:2509.16806#pg1>.

Lu: I’d add that the model variants, like MedGSi which does interpolation only, show robustness when specific segmentation masks aren't available for training. It demonstrates that this representation learning approach is flexible enough to handle different data modalities and constraints.

Meng: From an engineering side, if we can achieve better reconstruction quality with less reliance on perfect segmentation labels during the training process, that could simplify the pipeline for clinical deployment significantly.

Lalam: Ultimately, MedGS proves that this specific coupling of tasks isn't just a neat trick; it’s a robust way to build three dee models that respect both the appearance and the underlying structure of medical data <ref:2509.16806#pg1>.

More episodes

← Home