EXACT: an explainable anomaly-aware vision foundation model for analysis of 3D chest CT

arXiv:2604.24146 · cs.CV · Submitted 2026-04-27 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "EXACT: an explainable anomaly-aware vision foundation model for analysis of 3D chest CT".

Jane: Here's the gist of the paper. EXACT is an explainable anomaly-aware foundation model developed for three-dimensional chest Computed Tomography (CT) designed to address the limitations of scan-level prediction by enabling spatially resolved representation learning, abnormality localization, and interpretable visual evidence.

Tom: The core innovation lies in its ability to learn from paired clinical scans and radiology reports without requiring manual voxel-level annotations.

Jane: The model is pre-trained on 25,692 CT-reports pairs utilizing anatomy-aware weak supervision.

Tom: This process involves first generating coarse organ segmentation masks using the Segment Anything by Text (SAT) model from CT volumes, and then extracting multi-disease pseudo-labels from paired free-text radiology reports using a fine-tuned RadBERT model, mapping these labels to anatomical regions.

Paper summary: Tom: So we're wrapping up on "EXACT: an explainable anomaly-aware vision foundation model for analysis of three dee chest CT" <ref:2604.24146#pg1,EXACT: an explainable anomaly-aware vision foundation model for analysis of 3D>. It’s about moving past just guessing if something is wrong and actually showing you exactly where it is in the scan.

Jane: Exactly. The authors are trying to make sure that when an AI flags a spot on a chest scan, it doesn't just point vaguely at the general area, but gives you a precise location that makes sense anatomically.

Lu: Their core idea is using this unified Y Mamba network to learn from paired clinical scans and those radiology reports without needing those tedious manual voxel level annotations.

Lalam: They pre-trained it on twenty-five thousand six hundred ninety-two CT reports pairs using anatomy aware weak supervision, which means they used image segmentation models and fine tuned language models to create those initial anatomical clues <ref:2604.24146#pg2,on 25,692 CT reports pairs using anatomy aware weak supervision>.

Jane: It means the next big question for us is whether this method can be generalized beyond the specific patterns they learned from those twenty-five thousand six hundred ninety-two reports, or if it’s really just a powerful tool for well documented cases right now <ref:2604.24146#pg2>.

Conclusion: Tom: So we're wrapping up on EXACT, an explainable anomaly-aware vision foundation model for analysis of three dee chest CT. It’s about moving past just guessing if something is wrong and actually showing you exactly where it is in the scan.

Jane: Exactly. The authors are trying to make sure that when an AI flags a spot on a chest scan, it doesn't just point vaguely at the general area, but gives you a precise location that makes sense anatomically.

Lu: Their core idea is using this unified Y Mamba network to learn from paired clinical scans and those radiology reports without needing those tedious manual voxel level annotations.

Meng: So they’re not just looking at pixels; they’re trying to build a structure that understands the actual shape of the chest before it tries to find the disease.

Lalam: They pre-trained it on over twenty-five thousand six hundred ninety-two CT report pairs using anatomy aware weak supervision, which means they used image segmentation models and fine tuned language models to create those initial anatomical clues.

Tom: And their results show that this approach actually beats existing three dimensional medical foundation models in zero shot diagnosis when you use the anatomical maps to guide the search.

Jane: That’s significant because it suggests we can use these large chest scans not just for broad screening, but for pinpointing specific anomalies with a level of detail that matches what a radiologist would look for.

Lu: The way they handled the anatomy aware pre training, using that dynamic loss function that prioritizes learning the lung and pleura first, really sets a strong foundation.

Meng: I’m interested in how this translates to actually deploying something. They mention limitations around needing that fine tuned language model for pseudo labels, which limits how easily you can just drop it into any new scenario without redoing all that mapping work.

Lalam: That's true, the reliance on that specific RadBERT model for extracting the initial disease labels means it's not perfectly plug and play for every single rare condition out there right now.

Tom: So while EXACT is incredibly strong at localization and generating those understandable reports, we still need to figure out how to make that initial setup step less manual if we want this kind of localized precision everywhere.

Jane: It means the next big question is whether this method can be generalized beyond the specific patterns they learned from those twenty-five thousand six hundred ninety-two reports, or if it’s really just a powerful tool for well documented cases right now.

Xuguang Bai, Mingxuan Liu, Tongxi Song, Yifei Chen, Hongjia Yang, Kasidit Anmahapong, Zihan Li

School of Biomedical Engineering, Tsinghua University Department of Radiology, Mianyang Central Hospital

cs.CV

Submitted: 2026-04-27

Updated: 2026-11-04

Code: https://github.com/juntaoJianggavin/RWKV-UNet

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 98/100

The gist: EXACT is an explainable anomaly-aware foundation model developed for three-dimensional chest Computed Tomography (CT) designed to address the limitations of scan-level prediction by enabling…

Key concepts

Anatomy-aware Weak Supervision
This method uses a two-step process: first, an AI model segments organs from CT scans; second, a language model extracts disease labels from radiology reports. This combines automated image segmentation with text interpretation to create initial training data without requiring precise manual annotations for every single voxel.
Unified Y-Mamba Network
This is the core architecture integrating three components: an image encoder (using 3D convolutions and Mamba layers for deep feature extraction), an organ segmentation decoder, and an anomaly detection decoder. It allows the model to process spatial data while simultaneously learning how to identify pathological regions.
Anomaly-Aware Maps (AAmaps)
These are visual outputs generated by the model that highlight potential abnormalities in the CT scan. EXACT creates two versions: a low-resolution map for global context and a high-resolution map for fine detail, which are then combined to provide a comprehensive view of where issues might exist.
EXACT-CHAT
This module uses the frozen image encoder paired with an LLM (like LLaMA) to generate clinical reports. It refines these reports by cross-referencing them against the model's predictions, ensuring accuracy and reducing errors or 'hallucinations' in the final written analysis.

Terminology

Summary

EXACT is an explainable anomaly-aware foundation model developed for three-dimensional chest Computed Tomography (CT) designed to address the limitations of scan-level prediction by enabling spatially resolved representation learning, abnormality localization, and interpretable visual evidence. p. 1 The core innovation lies in its ability to learn from paired clinical scans and radiology reports without requiring manual voxel-level annotations. p. 1

The model is pre-trained on 25,692 CT-reports pairs utilizing anatomy-aware weak supervision. p. 1 This process involves first generating coarse organ segmentation masks using the Segment Anything by Text (SAT) model from CT volumes, and then extracting multi-disease pseudo-labels from paired free-text radiology reports using a fine-tuned RadBERT model, mapping these labels to anatomical regions. p. 4

The architecture employs a unified Y-Mamba network, which integrates an image encoder with an organ segmentation decoder and an anomaly detection decoder. p. 4 The image encoder uses 3D convolutional layers (Conv3D) with LeakyReLU activation and Instance Normalization (InstanceNorm3D), enhanced by Gated Spatial Convolution (GSC) and Mamba layers at deeper levels for hierarchical feature extraction. p. 4 The organ segmentation decoder utilizes an encoder-decoder structure with skip connections, while the anomaly detection decoder generates 18-channel Anomaly-Aware Maps (AAmaps) at two resolutions: a low-resolution branch (D/2 times W/2 times H/2, top- k with k=3) capturing global salient regions, and a high-resolution branch (D times W times H, k=24) preserving fine spatial detail. p. 4 The final AAmmap is obtained by element-wise summation of these branches, combining global context with fine detail. p. 8

A crucial aspect of EXACT is its anatomy-aware pre-training strategy. p. 6 It employs an anatomy-constrained Masked Instance Learning (MIL) framework, treating each organ region as an instance bag to confine disease evidence exclusively to anatomically plausible tissue. p. 1 This approach is coupled with a dynamic loss function (L STSMU = lambda V times L WXY + L MNO), where the weighting coefficient lambda V decays exponentially, enforcing a curriculum that prioritizes robust anatomical prior learning before shifting capacity toward fine-grained pathological pattern learning. p. 3

EXACT achieves several key capabilities:

  1. Anatomical Prior Learning: It learns robust anatomical priors across six thoracic structures, achieving high Dice Similarity Coefficients (DSC) for the lung and pleura (up to 0.966 plus or minus 0.088).

  2. Multi-Disease Diagnosis: EXACT demonstrates notable improvement in multi-disease diagnosis, outperforming existing three-dimensional medical foundation models in both zero-shot settings (based on anatomy-guided top- k aggregation of AAmaps and organ masks) and fine-tuning settings where a dedicated encoder/classifier is trained on EXACT's generated AAmaps.

  3. Voxel-Level Anomaly Localization: EXACT is presented as the first 3D chest CT foundation model to achieve intrinsic voxel-level anomaly localization entirely through automated text-derived weak supervision, without manual voxel-level masks. During zero-shot inference, it generates 18 disease-specific AAmaps and applies an explicit anatomical constraint (element-wise multiplication with organ masks) to eliminate background noise before binarization. p. 8

  4. Supervised Localization Improvement: For supervised anomaly localization, a hybrid loss function combining Tversky loss and Focal loss yields EXACT-Seg, showing statistically significant improvements over baselines for localizing small and morphologically diverse lesions.

  5. Explainable Report Generation (EXACT-CHAT): EXACT-CHAT integrates the frozen EXACT image encoder with a multimodal projector module and LLaMA-3.1-8BInstruct for report generation. p. 1 It further refines reports using an optional GPT-4. p. 11 step that cross-references generated content against upstream disease predictions, correcting missed findings and suppressing hallucinations, leading to the best clinical efficacy across all datasets tested.

In retrospective multinational and multi-center evaluations on datasets like CT-RATE, RAD-ChestCT, and MianYang, EXACT consistently outperformed State-of-the-Art (SOTA) 3D medical foundation models. p. 5 For instance, in zero-shot anomaly localization on the COVID-19 dataset, EXACT achieved a DSC of 0. p. 8435, rivaling fully supervised models (0. p. 3476). Furthermore, in external validation on the RAD-ChestCT and MianYang datasets, EXACT-CHAT (Refined) significantly outperformed baselines such as Hulu-Med in terms of RadBERT-F1 scores (e. p. 5g., 0.441 on RAD-ChestCT versus 0. p. 5279).

Limitations include reliance on a fine-tuned RadBERT model for pseudo-label extraction and the manual pre-definition required for mapping target abnormalities to disease/organ pairs, which constrains scalability to rare pathologies. Prospective clinical validation is also necessary to confirm translational utility beyond retrospective settings. p. 1

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. An explainable anomaly-aware foundation model for three-dimensional chest CT that learns spatially resolved representations from paired clinical scans and radiology reports can provide intrinsic spatial interpretability without manual voxel-level annotations. This allows the system to generate organ-specific anomaly-aware maps that assign each voxel a disease-specific anomaly score confined to its corresponding anatomy, jointly encoding lesion extent and organ-level context.

  2. The improved system can perform zero-shot anomaly localization by simultaneously generating 18 disease specific AAmaps and anatomical organ masks from 3D CT volumes, which are then element wise multiplied and aggregated via top k pooling to obtain the final prediction score. This capability allows for zero shot anomaly localization across diverse datasets like ReXGroundingCT, COVID-19, and MosMed without any manual voxel level annotations.

  3. The system can achieve clinically accurate and visually grounded radiology report generation by integrating a frozen EXACT image encoder with a large language model to generate reports, which are then post hoc calibrated by an optional GPT-4 refinement step against upstream disease predictions. This ensures that EXACT-CHAT (Refined) demonstrates the highest consistency with the reference, successfully identifying complex pathologies without omissions or hallucinations.

  4. The system can perform multi-disease diagnosis under zero shot and fine tuning settings, achieving superior performance on external datasets compared to existing models like CT-CLIP and fVLM. For instance, in the zero shot setting on the MianYang dataset, EXACT (Zero-shot) achieved an AUROC of 0.758, surpassing fVLM (Zero-shot) (AUROC, 0.716).

  5. The improved system can provide fine tuned segmentation for anomaly localization by utilizing a hybrid loss function combining Tversky loss and Focal loss to address class imbalance and prioritize recall for small lesions. This results in EXACT-Seg achieving a DSC of 0.215 on the ReX-Val internal validation dataset, outperforming SOTA segmentation models like SegMamba46.

Sources

Related papers