OraRefer: Segmentation-Guided Multi-Photograph Fusion for Oral Cancer Screening with Guaranteed Patient-Level Sensitivity

arXiv:2611.08360 · cs.CV, eess.IV · Submitted 2026-11-05 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "OraRefer: Segmentation-Guided Multi-Photograph Fusion for Oral Cancer Screening with Guaranteed Patient-Level Sensitivity".

Jane: Here's the gist of the paper. Oral cancer and the oral potentially malignant disorders (OPMD) that can precede it are visible on the oral mucosa, which makes smartphone photographs a promising tool for screening <ref:2611.08360#pg2>.

Tom: OraRefer proposes a patient-level screening model that reads each photograph with a segmentation-guided foundation model that conditions its class decision on the predicted lesion, combines all photographs of a patient with risk factors, and refers through a threshold with a finite-sample sensitivity guarantee <ref:2611.08360#pg2>.

Jane: On 3000 photographs of 714 patients, patient-level fusion reaches an AUROC of 0.919 and outperforms attention-based multiple-instance learning <ref:2611.08360#pg3>.

Tom: Model Architecture and Training Signal The image model is a segmentation-guided foundation model that reads each photograph in two passes to return class probabilities, a lesion mask, and an uncertainty estimate <ref:2611.08360#pg4>.

Title and authors: Tom: So we're looking at the paper called OraRefer: Segmentation-Guided Multi-Photograph Fusion for Oral Cancer Screening with Guaranteed Patient-Level Sensitivity, and the title itself tells you a lot about what they did. It's a patient-level screening model that uses segmentation to guide its decision on oral cancer and potentially malignant disorders.

Jane: That sounds really complex, Tom. When you say patient-level screening, what does that mean for someone listening right now? Is it just looking at one picture, or are they looking at a whole history of photos?

Tom: It means the model takes all the photographs a single patient has—maybe many of them—and combines that information with their known risk factors like age or smoking to make a single referral decision for that person.

Lu: Exactly. The paper focuses on making sure that this decision is robust, which is why they include "guaranteed sensitivity" in the title. It's about setting a hard limit on how often you miss someone who actually has the disease.

Meng: I wonder how they managed to fuse all those different photos and risk factors into one score without just getting lost in the noise? That sounds like a huge engineering challenge at scale.

Lalam: From what I'm seeing, OraRefer is designed so that it reads each photo individually first, then fuses everything together later. It seems their main goal is to create a reliable referral decision based on all that evidence combined >

The paper's summary: Jane: So the authors summarize the approach by saying they have this segmentation-guided foundation model that reads each photo in two passes. They get class probabilities, a lesion mask, and an uncertainty estimate from each individual image.

Tom: Right, and then they use that information to guide the final classification decision on whether there's a lesion or not. The key is they crop the mouth first with a pass one so you don't need any manual annotation for inference when you actually use it.

Lu: And then they have this decoder part that reassembles tokens from the encoder blocks into a dense map, and within that map, they use region-affinity attention to aggregate information based on learned weights. This helps them segment the background, mouth area, and the lesion itself >

Meng: So it’s like they are teaching a model not just what a lesion looks like in general, but exactly where to look on any given photo? That seems much more robust than just running one model over everything.

Lalam: It sounds like they're training the classifier to pay attention specifically to the predicted lesions and their uncertainty, which helps when you're dealing with uncertain medical images >

Tom: And then they take all those per-photo outputs—the class probabilities, lesion areas, and uncertainty—and feed them into a fusion model that combines them with the patient's risk factors using an l2 regularized logistic regression to get one final patient score >

The paper's improvements: Jane: So what are the actual improvements they point out? They highlight that their method is better than just taking the outputs from a single photo and some simple attention-based learning methods.

Tom: They show that their patient-level fusion strategy, combining all photos with risk factors into one score, reaches an AUROC of zero point nine one nine on three thousand photographs of seven hundred fourteen patients, which they say outperforms attention-based multiple instance learning >

Lu: That number is pretty solid for a patient-level approach. They also emphasize that their guaranteed sensitivity thresholding method lets them set a specific operating point without having to retrain the model on new data, which is a big practical win >

Meng: But what about those guarantees? How do they ensure that this threshold works reliably when you're deploying it in a real clinic where the patient population might be different than the test set?

Lalam: They use something called an order-statistic bound or Mondrian conformal triage to set that threshold, and they calibrate it on held-out patients without retraining, which is what gives them that finite-sample sensitivity guarantee >

Conclusion: Tom: So to wrap up the OraRefer paper, they show a way to turn raw smartphone photos into a referral decision with a guaranteed sensitivity. It's about localization and fusion for oral cancer screening.

Jane: The implication for me is that this means clinics could use these tools to make sure no one who has disease gets missed, which is crucial in screening applications.

Lu: It’s interesting because they made the model local to the image by making it predict the mouth mask first so inference needs no annotation, and then they used a sophisticated fusion strategy with risk factors >

Meng: From a deployment standpoint, I'm still thinking about how much computation this two-pass segmentation requires for every single photo. It’s fast enough for testing, but at scale, running that whole pipeline repeatedly might be costly >

Lalam: The paper is available online and the code is public, which makes it accessible for other researchers to test these ideas on their own data and see if they get similar results >

Tom: Yeah, OraRefer turns a patient’s raw smartphone photographs into a referral with guaranteed sensitivity. That's the main point here.

Rupam Mukherjee, Mehakdeep Singh, Shirin Dasgupta, Subhamoy Mandal

Department of Medical Science and Technology · Dr. B. C. Roy Multi-Speciality Medical Research Centre Indian Institute of Technology Kharagpur

cs.CV, eess.IV

Submitted: 2026-11-05

Updated: 2026-11-05

Code: https://github.com/ZenAesthete/OraRefer

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: Oral cancer and the oral potentially malignant disorders (OPMD) that can precede it are visible on the oral mucosa…

Key concepts

Segmentation-Guided Foundation Model
This is an image model that performs two tasks simultaneously: it segments the background, mouth area, and the lesion itself. This segmentation output shapes the features used by the classifier to make a diagnosis. A second pass crops out the mouth area, meaning inference doesn't require manual annotation.
Patient-Level Fusion Strategy
This component combines all photographs of a single patient with their known risk factors (like age or smoking) into one comprehensive score. This score is then used to predict referral likelihood. It employs an L2-regularized logistic regression to map these combined features to a final patient risk score.
Guaranteed Sensitivity Thresholding
Instead of relying on retraining, the system sets a referral threshold based on held-out patients using methods like order-statistic bounds or Mondrian conformal triage. This guarantees that the system maintains a minimum required sensitivity ($\geq 1 - \delta$) for new patients, ensuring reliable detection.
Mondrian Conformal Triage
This is an advanced method used to set the referral threshold. It ensures that every class of lesion has a high probability (p-value) above a certain level ($\alpha$). This guarantees that the system covers all relevant classes with at least $1 - \alpha$ expected coverage.

Terminology

Summary

Oral cancer and the oral potentially malignant disorders (OPMD) that can precede it are visible on the oral mucosa, which makes smartphone photographs a promising tool for screening <ref:2611. p. 008360#pg2>. OraRefer proposes a patient-level screening model that reads each photograph with a segmentation-guided foundation model that conditions its class decision on the predicted lesion, combines all photographs of a patient with risk factors, and refers through a threshold with a finite-sample sensitivity guarantee <ref:2611. p. 008360#pg2>. On 3000 photographs of 714 patients, patient-level fusion reaches an AUROC of 0. p. 0919 and outperforms attention-based multiple-instance learning <ref:2611. p. 008360#pg3>.

Model Architecture and Training Signal

The image model is a segmentation-guided foundation model that reads each photograph in two passes to return class probabilities, a lesion mask, and an uncertainty estimate <ref:2611.08360#pg4>. The encoder is DINOv2 ViT-B/14 which is fine-tuned end to end with layer-wise learning-rate decay <ref:2611. p. 208360#pg4>. The DPT decoder reassembles four encoder blocks into a dense map that segments background, mouth, and lesion <ref:2611. p. 208360#pg4>. Within the decoder, region-affinity attention aggregates each reassembled token over its 3 × 3 neighborhood with learned affinity weights <ref:2611. p. 208360#pg4>. The model is trained with a loss function L = Lseg + λclsLcls + λconLcon + λraaLraa + Laux <ref:2611. p. 208360#pg4>.

Patient-Level Fusion Strategy

The fusion model turns the outputs for all photographs of a patient, together with the patient’s risk factors, into one score <ref:2611.08360#pg4>. For each photograph, features include the four-class posterior, lesion area, and uncertainty estimates <ref:2611. p. 208360#pg5>. These are summarized by the maximum, mean, and top-two mean of qi j, the fraction with qi j > 0. p. 25, and the noisy-OR <ref:2611. p. 008360#pg5>. These summaries are added to the risk factors ri (age, sex, smoking, betel quid, and alcohol use) to map them to a patient score si using an l2-regularized logistic regression with inner cross-validated penalty and Platt calibration <ref:2611. p. 208360#pg5>.

Guaranteed Sensitivity Thresholding

The referral decision is made through a threshold calibrated on held-out patients without retraining, using either an order-statistic bound or Mondrian conformal triage <ref:2611.08360#pg5>. The guaranteed-sensitivity threshold t = s(k∗) ensures that Pr[sensitivity ≥ tau] ≥ 1 − delta for new patients, where k∗ is determined by the target sensitivity and desired miss probability <ref:2611. p. 308360#pg5>. Mondrian conformal triage adds a third option where a prediction set keeps every class with p-value above α, ensuring coverage per class is at least 1 − α in expectation <ref:2611. p. 308360#pg5>.

Performance and Comparison

OraRefer outperforms frozen foundation-model probes by 0.069–0.084 in macro AUROC <ref:2611. p. 508360#pg6>. Patient-level fusion with risk factors reaches an AUROC of 0. p. 5919, the highest in Table 2, above each untrained aggregate <ref:2611. p. 508360#pg5>. The guaranteed-sensitivity threshold detects 0. p. 0950 [0.934, 0.966] of referable patients against a target of 0. p. 590, with a specificity of 0. p. 0610 <ref:2611. p. 008360#pg8>. The missed cancers are classification errors, not localization errors, as all three were localized and segmented well <ref:2611. p. 508360#pg8>.

Conclusion and Clinical Utility

OraRefer turns a patient’s raw smartphone photographs into a referral with guaranteed sensitivity <ref:2611.08360#pg9>. Since the thresholds are calibrated without retraining, a new clinic can set its own guaranteed operating point from a representative local sample of patients with OPMD or cancer <ref:2611. p. 708360#pg9>. This work motivates prospective evaluation in community screening <ref:2611. p. 708360#pg9>. The code of OraRefer is available at https://github. p. 7com/ZenAesthete/OraRefer. p. 7

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Segmentation-Guided Image Model This model trains one network jointly to segment background, mouth, and lesion while classifying its decision on predicted lesions; this shapes features for the classifier and crops the mouth in a second pass so inference needs no annotation.

  2. Self-Localization at Test Time This system performs a first pass on the full frame to predict the mouth mask, which is then used as a crop for the second pass, ensuring that inference needs no annotation at test time.

  3. Patient-Level Fusion and Referral This component fuses all photographs of a patient with risk factors using an L2-regularized logistic regression to generate a patient score, which is then referred through a distribution-free sensitivity threshold and Mondrian conformal triage, both calibrated on held-out patients without retraining.

Sources

Related papers