A Lightweight Fiducial-Based Pipeline for 3D Hyperspectral Mapping of ex-vivo Lumpectomy Specimens
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "A Lightweight Fiducial-Based Pipeline for 3D Hyperspectral Mapping of ex-vivo Lumpectomy Specimens".
Jane: Here's the gist of the paper. Hyperspectral Imaging (HSI) is presented as a promising modality for intraoperative assessment of resection margins in BreastConserving Surgery (BCS), but its clinical translation requires aligning inherently 2D spectral information onto the 3D shape of the excised tissue so that suspicious regions can be precisely localized for targeted follow-up This paper presents a fully automated, calibration-free pipeline that produces a 3D hyperspectral point cloud of an ex-vivo lumpectomy specimen from a set of consumer-camera RGB images and a single top-down HSI acquisition The pipeline achieves a median 3D registration error below 1 mm and a 2D reprojection error below 0.02 mm, with a total per-specimen processing time under 4 minutes on accelerated hardware Pipeline Overview The proposed pipeline is organized into four distinct stages to transform input data into the final spectral point cloud The end-to-end pipeline P can be summarized as C = P(∪ ︁ ︁ N RGB images, HSI image) The input data consists of a set of N RGB images and a single "top-down view" HSI image The output is a 3D point cloud C representing the specimen, where each point has 3D coordinates and the full spectral signature recovered by the HSI scanner Data Acquisition and Geometry Reconstruction The data acquisition involves placing specimens surrounded by four ArUco markers to define a global reference frame and metric scale A set of at least 20 RGB photographs is captured using a consumer-grade smartphone camera from different viewpoints in an "inward-sphere" configuration The single hyperspectral cube is acquired with a Goldeye G-130 VSWIR TEC1 imager covering the 400–1700 nm wavelength range with B = 290 spectral bands The 3D geometry is reconstructed from RGB images using an off-the-shelf StructurefromMotion (SfM) backbone followed by a custom ArUco-driven joint bundle adjustment This reconstruction utilizes the MASt3R-SfM model to produce an initial point cloud C' Registration Mechanism The HSI cube is registered to the 3D mesh without recovering the HSI camera pose The registration relies on ArUco fiducial markers visible in both modalities to define 16 corner correspondences that drive a planar homography This process involves fitting a planar homography using RANSAC on the paired 2D–2D correspondences between the HSI image and the orthographically rendered mesh The final 3D coordinate for each spectral signature is recovered by bilinear interpolation on a fine render map Performance and Validation Evaluation was conducted on two ex-vivo lumpectomy specimens, SB019 and SB020, using a single NVIDIA A100 GPU partition The 2D reprojection error stays well below the registration-grid resolution for both specimens The 3D errors are below 1 mm on average for both specimens Processing time is dominated by the 3D reconstruction stage, completing under 25 s for both specimens The pipeline's geometric stage scales with the number of input views This work presents the first lightweight, calibration-free pipeline combining single-view HSI and 3D photogrammetric reconstruction for ex-vivo lumpectomy specimens The median 3D registration error stays below 1 mm on both specimens and the total per-specimen processing time remains under 4 minutes These results support the feasibility of integrating HSI-guided spatial localization into intraoperative margin assessment workflows for breast-conserving surgery Future work will focus on validating the pipeline on a larger specimen cohort against such an independent ground truth References
1: U.
Tom: Veronesi, N.
Jane: Cascinelli, L.
Tom: Mariani, M.
Paper summary: Tom: So, wrapping up this paper, the main thing is that they’ve delivered a method that takes consumer camera RGB images and one HSI image and spits out a three dee point cloud with spectral signatures in under four minutes processing time for the whole specimen <ref:2606.14534#pg1,consumer camera RGB images and>.
Jane: The authors of "A Lightweight Fiducial-Based Pipeline for three dee Hyperspectral Mapping of ex-vivo Lumpectomy Specimens" are essentially saying they’ve made it possible to get that alignment without needing extensive, manual calibration steps before you even start <ref:2606.14534#pg1,A Lightweight Fiducial-Based Pipeline for 3D Hyperspectral Mapping of ex-vivo>.
Lu: What this means for the broader research area is that they’ve shown a feasible path toward integrating HSI-guided spatial localization into intraoperative margin assessment workflows.
Meng: From an engineering standpoint, if this pipeline works reliably outside of those two validated specimens, it suggests we might be able to deploy these tools on more standard hardware.
Lalam: This work is significant because it tackles the problem of aligning inherently two-dimensional spectral data onto the three-dimensional shape of the tissue for precise localization.
Jane: It moves us closer to a point where HSI data can directly inform where a surgeon needs to look during surgery, which is really what this paper aims to achieve.
Tom: The focus on ArUco markers as a shared reference across modalities is the key mechanical innovation here that makes it lightweight and calibration-free.
Jane: So, for someone listening who isn't reading the math, think of it as building a map of the tissue shape using everyday photos and one special spectral picture, where you use little plastic markers to make sure both maps line up perfectly.
Conclusion: Tom: So what’s the big picture here? We’re talking about a method that takes regular camera photos and one special spectral image to get a full three dee map of a removed tumor specimen.
Jane: Right, so the core idea is getting that spectral information onto the actual shape of the tissue so doctors can pinpoint suspicious areas for follow-up.
Lu: The authors built this pipeline from scratch without needing any complex, pre-calibration steps beforehand. They used ArUco markers to link the camera views and the HSI image together.
Meng: So if I’m thinking about deploying this in a real clinic, how lightweight is it? Because right now, running high-end reconstruction software can take a lot of time and resources.
Lalam: The paper claims this pipeline runs in under four minutes per specimen on accelerated hardware. That’s pretty fast for processing raw spectral data into a usable three dee point cloud.
Tom: Under four minutes is solid, but what about accuracy? Is the registration error low enough to actually matter for surgical planning?
Jane: The authors report a median three dee registration error of less than one millimeter and a two-dimensional reprojection error under zero point zero two millimeters. Those are pretty tight numbers for aligning different types of data.
Lalam: And that means they’re doing this calibration-free, just by finding those matching markers between the RGB photos and the HSI image.
Meng: Calibration-free is huge for adoption, but what about scaling up? Does this pipeline keep working reliably if we have a different type of marker or more complex tissue shapes?
Lu: The methodology relies on fitting a planar homography using RANSAC on those ArUco correspondences. It’s robust for the setup they described, but it is specifically designed for these ex-vivo specimens.
Jane: So, to sum up what they did: they created a single pipeline combining multiple images and one HSI cube to create a three dee spectral point cloud of a lumpectomy specimen without any initial calibration.
Tom: It’s lightweight, fast, and it works with existing consumer camera data. That’s the mechanics we need to focus on next.
Jane: This work suggests that integrating HSI-guided spatial localization into breast-conserving surgery workflows is becoming much more feasible in a practical sense.
Department of Electronics, Information and Bioengineering, Politecnico di Milano Department of Physics, Politecnico di Milano
cs.CV
Submitted: 2026-06-12
Updated: 2026-10-30
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 86/100
The gist: Hyperspectral Imaging (HSI) is presented as a promising modality for intraoperative assessment of resection margins in BreastConserving Surgery (BCS)…
Key concepts
- Hyperspectral Imaging (HSI)
- HSI captures not just color (RGB) but also hundreds of narrow spectral bands across a wide wavelength range. This provides rich chemical and material information about the tissue, which is crucial for identifying cancerous regions in breast specimens.
- 3D Registration
- This is the process of accurately aligning two different sets of data—in this case, a 3D shape built from photos and a 2D spectral image. The method uses visible markers (ArUco) to find matching points between the two views, allowing the spectral information to be correctly placed onto the physical structure.
- Point Cloud
- A point cloud is a collection of discrete 3D coordinates representing individual points in space. In this context, each point not only has an (x, y, z) location in the specimen but also carries its full spectral signature from the HSI scanner.
Terminology
Summary
Hyperspectral Imaging (HSI) is presented as a promising modality for intraoperative assessment of resection margins in BreastConserving Surgery (BCS), but its clinical translation requires aligning inherently 2D spectral information onto the 3D shape of the excised tissue so that suspicious regions can be precisely localized for targeted follow-up This paper presents a fully automated, calibration-free pipeline that produces a 3D hyperspectral point cloud of an ex-vivo lumpectomy specimen from a set of consumer-camera RGB images and a single top-down HSI acquisition The pipeline achieves a median 3D registration error below 1 mm and a 2D reprojection error below 0. p. 0 A Lightweight Fiducial-Based Pipeline for 3D
Hyperspectral Mapping of ex-vivo Lumpectomy
Specimens
Anna Bicchi1,*,†
, Alberto Rota1,†
, Leonardo Passoni1,†
, Nicola Ancellotti1
, Andrea Peroni2
,
Lorenzo Vinco2
, Dario Polli2
and Elena De Momi1
1Authors are with the Department of Electronics, Information and Bioengineering (DEIB), Politecnico di Milano, Milan, Italy
2Authors are with the Department of Physics, Politecnico di Milano, Milan, Italy
Abstract
Hyperspectral Imaging (HSI) is a promising modality for intraoperative assessment of resection margins in BreastConserving Surgery (BCS), but its clinical translation requires aligning the inherently 2D spectral information
onto the 3D shape of the excised tissue so that suspicious regions can be precisely localized for targeted follow-up. We present a fully automated, calibration-free pipeline that produces a 3D hyperspectral point cloud of an ex-vivo
lumpectomy specimen from a set of consumer-camera RGB images and a single top-down HSI acquisition.Where the paper says it
Pipeline Overview
The proposed pipeline is organized into four distinct stages to transform input data into the final spectral point cloud The end-to-end pipeline P can be summarized as C = P(∪ N RGB images, HSI image) The input data consists of a set of N RGB images and a single top-down view
HSI image The output is a 3D point cloud C representing the specimen, where each point has 3D coordinates and the full spectral signature recovered by the HSI scanner
Data Acquisition and Geometry Reconstruction
The data acquisition involves placing specimens surrounded by four ArUco markers to define a global reference frame and metric scale A set of at least 20 RGB photographs is captured using a consumer-grade smartphone camera from different viewpoints in an inward-sphere
configuration The single hyperspectral cube is acquired with a Goldeye G-130 VSWIR TEC1 imager covering the 400–1700 nm wavelength range with B = 290 spectral bands The 3D geometry is reconstructed from RGB images using an off-the-shelf StructurefromMotion (SfM) backbone followed by a custom ArUco-driven joint bundle adjustment This reconstruction utilizes the MASt3R-SfM model to produce an initial point cloud C'
Registration Mechanism
The HSI cube is registered to the 3D mesh without recovering the HSI camera pose The registration relies on ArUco fiducial markers visible in both modalities to define 16 corner correspondences that drive a planar homography This process involves fitting a planar homography using RANSAC on the paired 2D–2D correspondences between the HSI image and the orthographically rendered mesh The final 3D coordinate for each spectral signature is recovered by bilinear interpolation on a fine render map
Performance and Validation
Evaluation was conducted on two ex-vivo lumpectomy specimens, SB019 and SB020, using a single NVIDIA A100 GPU partition The 2D reprojection error stays well below the registration-grid resolution for both specimens The 3D errors are below 1 mm on average for both specimens Processing time is dominated by the 3D reconstruction stage, completing under 25 s for both specimens The pipeline's geometric stage scales with the number of input views
Conclusion
This work presents the first lightweight, calibration-free pipeline combining single-view HSI and 3D photogrammetric reconstruction for ex-vivo lumpectomy specimens The median 3D registration error stays below 1 mm on both specimens and the total per-specimen processing time remains under 4 minutes These results support the feasibility of integrating HSI-guided spatial localization into intraoperative margin assessment workflows for breast-conserving surgery Future work will focus on validating the pipeline on a larger specimen cohort against such an independent ground truth
References
[1] U. Veronesi, N. p. 5 References
[1] U. Veronesi, N. Veronesi, N. Cascinelli, L. Cascinelli, L. Mariani, M. Mariani, M. Saccozzi, A. Saccozzi, A. Aguilar, E.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[2] N. Houssami, P. p. 5 1056/NEJMoa020989. Houssami, P. Houssami, P. Macaskill, M.Where the paper says it
Where the paper says it
[3] A Hoffman, I Ashkenazi
[4] C Krafft, J Popp
[5] E Kho, L L de Boer K K Van de Vijver F van Duijnhoven M J T Vrancken Peeters H J Sterenborg T J Ruers
[6] L J S Jong D Veluponnar F Geldof J Sanders M Da Silva Guimaraes M J T Vrancken Peeters F van Duijnhoven et al.
[7] G Lu L Halig D Wang Z G Chen B Fei Hyperspectral imaging for cancer surgical margin delineation: Registration of hyperspectral and histological images in Medical Imaging 2014: Image-Guided Procedures, Robotic Interventions, and Modeling volume 9036 SPIE 2014 pp. p. 5 1038/s41598-025-94526-9. Fei, Hyperspectral imaging for cancer surgical margin
delineation: Registration of hyperspectral and histological images, in: Medical Imaging 2014:
Image-Guided Procedures, Robotic Interventions, and Modeling, volume 9036, SPIE, 2014, pp.Where the paper says it
[8] M Wahabzada M Besser M Khosravani M T Kuska K Kersting A K Mahlein E Stürmer Monitoring wound healing in a 3D wound model by hyperspectral imaging and efficient clustering PLOS ONE 12 (2017) 1–14
[9] V Lindholm A M Raita-Hakola L Annala M Salmivuori L Jeskanen H Saari S Koskenmies S Pitkänen I Pölönen K Isoherranen A Ranki Differentiating malignant from benign pigmented or non-pigmented skin tumours—a pilot study on 3D hyperspectral imaging of complex skin surfaces and convolutional neural networks Journal of Clinical Medicine 11 (2022) 1914
[10] M Villa J Sancho G Rosa M Chavarrias E Juarez C Sanz HyperMRI: Hyperspectral and magnetic resonance fusion methodology for neurosurgery applications International Journal of Computer Assisted Radiology and Surgery 19 (2024) 1367–1374
[11] V Leroy Y Cabon J Revaud Grounding image matching in 3D with MASt3R in Computer Vision ECCV 2024: 18th European Conference Milan Italy September 29–October 4, 2024 Proceedings Part LXXII Springer-Verlag Berlin Heidelberg, 2024 pp.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Automated 3D Spectral Point Cloud Generation: The improved system can take a set of consumer-camera RGB images and a single top-down hyperspectral acquisition to produce a 3D hyperspectral point cloud of an ex-vivo lumpectomy specimen, effectively aligning the inherently 2D spectral information onto the 3D shape of the excised tissue.
-
Calibration Free Registration: The system achieves registration without recovering the HSI camera pose or requiring intrinsic calibration by using ArUco fiducial markers to define
16 corner correspondences that drive a planar homography,
removing the need for dedicated geometric hardware or cross-modal pose estimation. -
Robust 3D Geometry Estimation: The system employs a deep learning Structure-from-Motion backbone, such as MASt3R-SfM, stabilized by a custom bundle adjustment that
enforces consistency on the corners of four ArUco markers,
ensuring the 3D geometry is reconstructed with a median 3D registration error below 1 mm. -
Direct Spectral Localization: The system maps every spectral signature to a point on the reconstructed surface by using bilinear interpolation from a fine-render homography, allowing for
HSI-guided spatial localization into intraoperative margin assessment workflows.
Abstract
Hyperspectral Imaging (HSI) is a promising modality for intraoperative assessment of resection margins in Breast-Conserving Surgery (BCS), but its clinical translation requires aligning the inherently 2D spectral information onto the 3D shape of the excised tissue so that suspicious regions can be precisely localized for targeted follow-up. We present a fully automated, calibration-free pipeline that produces a 3D hyperspectral point cloud of an ex-vivo lumpectomy specimen from a set of consumer-camera RGB images and a single top-down HSI acquisition. The 3D geometry is reconstructed with a deep-learning Structure-from-Motion backbone, stabilized in a metric reference frame by a custom bundle adjustment that enforces consistency on the corners of four ArUco markers placed around the specimen. The HSI cube is then registered to the reconstruction without recovering the HSI camera pose: the markers, visible in both modalities, define 16 corner correspondences that drive a planar homography, and 3D coordinates are recovered by lookup on an orthographically rendered depth map. Evaluated on two ex-vivo lumpectomy specimens, the pipeline achieves a median 3D registration error below 1 mm and a 2D reprojection error below 0.02 mm, with a total per-specimen processing time under 4 minutes on accelerated hardware. These results support the feasibility of integrating HSI-guided spatial localization into intraoperative margin assessment workflows for breast-conserving surgery.
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs