CORE-Seg: Reasoning-Driven Segmentation for Complex Lesions via Reinforcement Learning

arXiv:2603.05911 · cs.CV, cs.AI · Submitted 2026-03-06 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "CORE-Seg: Reasoning-Driven Segmentation for Complex Lesions via Reinforcement Learning".

Jane: The gist: CORE-Seg introduces an end-to-end reasoning framework that integrates multimodal large language models with reinforcement learning to achieve state-of-the-art performance in complex lesion segmentation.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: We're wrapping up our look at "CORE-Seg: Reasoning-Driven Segmentation for Complex Lesions via Reinforcement Learning." The authors are essentially arguing that reasoning driven segmentation is the way to go for complex lesions because standard visual pattern matching simply isn't enough.

Jane: They introduce ComLesion-14K, which is this first large scale Chain-of-Thought benchmark, and then they build CORE-Seg to handle that complexity using an end-to-end framework integrating reasoning with segmentation <ref:2603.05911#pg1,an end-to-end framework integrating reasoning with segmentation>.

Lu: The core idea hinges on the Semantic Guided Prompt Adapter, which uses the MLLM's internal state as a semantic anchor to guide the visual segmentation model, rather than just relying on simple input prompts.

Meng: It’s about moving from models that guess based on general knowledge to models that follow a structured line of thought when they encounter something visually ambiguous or heterogeneous.

Tom: So, what does this mean for the bigger picture? It shows that combining the linguistic power of MLLMs with visual segmentation tools can lead to much more robust solutions for difficult medical imaging tasks.

Jane: It suggests we are moving toward AI systems where the process is as important as the final output, not just the final result itself.

Lu: They also pointed out some limitations in their work. Specifically, CORE-Seg currently mainly targets 2D images and doesn't support three dee volumetric medical data which would let it exploit inter-slice context <ref:2603.05911#pg2>.

Meng: And they mentioned that inference can still be slow because of the explicit reasoning steps, which adds computational cost for real-time use in a clinical setting.

Tom: So while the performance numbers are impressive—like that mean Dice of thirty-seven point zero six percent—the authors are being realistic about where the method stops working right now.

Jane: The implication is that future work needs to focus on extending this framework to volumetric segmentation and finding ways to make those reasoning steps run faster for clinical application.

Conclusion: Tom: So we've seen how CORE-Seg takes that idea of using reasoning to tackle complex medical images and it's pretty impressive what they achieved here on ComLesion-14K <ref:2603.05911#pg1>

Jane: Yeah, it seems like they're really pushing past just looking at a picture and starting to think about what the picture means in a bigger context

Lu: They built this whole system using an end-to-end approach where the MLLM's thinking actually guides the segmentation model directly through this semantic prompt adapter

Meng: That part about that adapter, it sounds like they’re trying to bridge the gap between words and pixels without just throwing a random text prompt at it

Lalam: From my perspective, this shows how language can be structured to provide a really specific visual cue that the segmentation model can latch onto instead of getting confused by noise

Tom: So what does this mean for the people who actually have to look at these scans every day? Are we talking about faster diagnoses or something else

Jane: Well, it suggests that if we could get models to reason like this, they could potentially help reduce the guesswork involved in labeling those tricky lesions where things are really hard to see

Lu: It moves the focus from just pattern matching pixels to understanding the actual pathology behind them which is a much deeper level of analysis

Meng: But I gotta ask, how do we know this reasoning actually holds up when it goes into a real clinical setting? The authors mention some limitations in their paper that we should keep an eye on

Lalam: They did point out that right now it's really focused on two-dimensional images and they aren't fully tackling three-dimensional medical data yet which is a big limitation for comprehensive care

Tom: Exactly, so while the numbers are great for 2D cases, we have to remember that this is just the first step and we need to see how this reasoning framework scales up <ref:2603.05911#pg2>

Jane: Definitely. It sets a new bar for what's possible in reasoning-driven segmentation but it also shows us exactly where the next big challenges lie

Lu: It's fascinating because it really formalizes the idea that complex tasks need a structured thought process instead of just brute force data processing

Meng: So we're looking at an architecture that balances the linguistic understanding with the pixel precision in a way I hadn't seen before for this kind of problem

Lalam: It really highlights how layering different types of information—the text, the visual features, and then the reinforcement learning feedback—can create a very powerful tool

Tom: So we've seen the benchmark, we've seen the architecture and we’ve looked at what it means for practice but where do we go from here with this CORE-Seg approach?

School of Computer Science and Engineering, Southeast University · School of Computer Science and Engineering, Nanjing University of Science and Technology · Zhongda Hospital

cs.CV, cs.AI

Submitted: 2026-03-06

Updated: 2026-10-09

Comments: Accepted by Computational Visual Media

Project page: https://xyx1024.github.io/CORE-Seg.github.io

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 88/100

The gist: The gist: CORE-Seg introduces an end-to-end reasoning framework that integrates multimodal large language models with reinforcement learning to achieve state-of-the-art performance in complex lesion

Key concepts

ComLesion-14K Benchmark: This is a large dataset of 14,000 complex medical images across 31 diseases. It was specifically created to test models on scenarios where standard visual pattern matching fails, forcing the model to use reasoning instead of simple recognition.
CORE-Seg Architecture: This is an end-to-end system that links language understanding with image segmentation. It uses an MLLM (like Qwen2.5-VL) to reason about the lesion and then a 'Semantic-Guided Prompt Adapter' to translate that reasoning into visual cues for the segmentation model.
Dual-Granularity Mask Reward: This is a special reward function used during reinforcement learning training. It helps the model learn not just correct segmentation, but also to generate accurate reasoning chains and precise boundary predictions simultaneously, improving overall performance.

Terminology

Core — 3× in this paper · explained in 2 episodes
Benchmark — 1× in this paper · explained in 2 episodes
Chain-of-Thought (CoT) — 1× in this paper · explained in 5 episodes
GRPO — 1× in this paper · explained in 6 episodes
Interpretability — 1× in this paper · explained in 15 episodes
Reinforcement Learning — 1× in this paper · explained in 4 episodes
Sparsity — 1× in this paper · explained in 4 episodes

Summary

The gist: CORE-Seg introduces an end-to-end reasoning framework that integrates multimodal large language models with reinforcement learning to achieve state-of-the-art performance in complex lesion segmentation.

Introduction and Problem Statement

Medical image segmentation is shifting from conventional visual pattern matching to cognitive reasoning analysis, as existing models often perform poorly when segmenting complex or ambiguous lesions that require expert judgment The gist: Medical image segmentation is undergoing a paradigm shift from conventional visual pattern matching to cognitive reasoning analysis.

The paper addresses the gap in current research by focusing on Complex Lesion Segmentation, which presents a triad of challenges: (1) Noisy and Distorted Imaging, (2) Lesion Diversity, and (3) Pathological Heterogeneity These characteristics result in uncertain visual evidence that impedes the stable perception of conventional pattern-matching models. Existing MLLM-based solutions generally follow two technical paths: supervised finetuned (SFT), end-toend dense prediction architecture, which lacks explainability, and reinforcement learning-driven, cascaded reasoning framework, which suffers from severe error propagation.

ComLesion-14K Benchmark

To systematically solve these challenges, the authors introduce ComLesion-14K, the first largescale Chain-of-Thought (CoT) driven benchmark explicitly designed for such scenarios. This benchmark comprises 14k cases across 31 diseases, selected from 300k images to focus on scenarios where conventional models often fail. The construction pipeline includes difficulty-aware filtering, VQA and CoT generation for structured descriptions and reasoning, and automated quality assurance to ensure annotation consistency.

CORE-Seg Architecture

CORE-Seg is an end-toend complex-lesion centric reasoning segmentation architecture that integrates the coherence of end-toend models with the explicit reasoning of reinforcement learning. The framework operates through a unified pipeline comprising three components: the Multimodal Reasoning Module, the Semantic-Guided Prompt Adapter, and the Segmentation Module.

Multimodal Reasoning Module: This module employs a MLLM, specifically Qwen2.5-VL-3B, to generate a structured output sequence Oˆ formatted as RˆAˆ. The hidden state of the special token serves as a condensed semantic anchor.
Semantic-Guided Prompt Adapter: This module extracts the final-layer hidden state of the special token as a condensed semantic anchor P, which is then projected into a visually-aligned manifold via stacked Residual MLP blocks to produce the refined feature P˜. Subsequently, an attention-based module interacts with P˜ to extract and reconstruct the segmentation prompt Pˆ.
Segmentation Model: The model uses the pre-trained Segment Anything Model (SAM) as the backbone, where it takes (Zimg, Pˆ) as inputs to yield the binary segmentation mask Mˆ.

Two-Stage Training Framework

The training consists of two distinct stages designed for progressive refinement.

Stage 1: CoT-Based Semantic Alignment Stage, where Supervised fine-tuning establishes the mapping between clinical reasoning and lesion localization using the token. This stage utilizes a composite loss function LS1 = λ1Ltxt(O, R, A ˆ) + λ2LDiceCE(M, M ˆ), balancing text generation loss with pixel-level segmentation precision.
Stage 2: RL-Based Reasoning Exploration and Segmentation Refinement, where the model is further optimized using Group Relative Policy Optimization (GRPO). This stage employs a composite reward function R tailored for complex medical scenarios, which includes a Format Reward (rfmt), a Bipartite Matching Reward (rbbox), and a Dual-Granularity Mask Reward (rmask). The final objective combines the RL policy gradient with the supervised segmentation loss: LS2 = λ3LGRP O(πθ; R, DKL) + λ4LDiceCE(M, M ˆ), jointly optimizing reasoning chains and boundary predictions.

Key Results and Contributions

CORE-Seg achieves state-of-the-art performance with a mean Dice of 37.06% (14.89% higher than the second-best baseline), while reducing the failure rate to 18.42%. The framework establishes a new state-of-the-art for complex lesion segmentation with 37.06% mDice and 27.79% mIoU, outperforming the second-best LISA-3B by +14.89%. CORE-Seg demonstrates superior generalization capabilities, achieving state-of-the-art performance particularly in MRI, CT, X-Ray and Ultrasound. Crucially, CORE-Seg achieves SOTA performance with exceptional efficiency, operating with minimal trainable parameters and computational overhead compared to massive models like Qwen2.5-VL-72B.

The key contributions of this work are fourfold: (1) We define a new task paradigm—Complex Lesion Segmentation, which necessitates reasoning-driven understanding of lesion regions characterized by high visual ambiguity and heterogeneity. (2) A first complex lesion segmentation benchmark, ComLesion-14K, is constructed that systematically mines clinical scenarios where existing models usually fail to localize lesions. (3) We propose CORE-Seg, an end-toend reasoning framework anchored by a Semantic-Guided Prompt Adapter that bridges MLLM reasoning with visual segmentation to eliminate cascading errors. (4) CORE-Seg establishes a new state-of-the-art in complex lesion segmentation.

Conclusion and Limitations

CORE-Seg proves an end-toend RL-driven paradigm effectively balances precise localization and generalization, suggesting its potential for clinical use. Despite the encouraging results, several limitations remain. The current framework mainly targets 2D images and does not support 3D volumetric medical data, limiting its ability to exploit inter-slice context. In addition, inference can be slow due to explicit reasoning, which increases computational cost. Future work will extend the framework to volumetric segmentation and improve efficiency while maintaining accuracy and clinical reliability.

The gist: CORE-Seg introduces an end-toend reasoning framework that integrates multimodal large language models with reinforcement learning to achieve state-of-the-art performance in complex lesion segmentation.

How it works

The framework operates through a unified pipeline comprising three components: the Multimodal Reasoning Module, the Semantic-Guided Prompt Adapter, and the Segmentation Module. The reasoning module first interprets the medical context and identifies potential lesions; the adapter then functions as a bridge to translate these semantic insights into visual cues; and finally, the segmentation module executes precise pixel-level delineation based on these cues.

The Multimodal Reasoning Module employs a MLLM, specifically Qwen2.5-VL-3B, to generate a structured output sequence Oˆ formatted as RˆAˆ. The hidden state of the special token serves as a condensed semantic anchor.

The Semantic-Guided Prompt Adapter extracts the final-layer hidden state of the special token as a condensed semantic anchor P, which is then projected into a visually-aligned manifold via stacked Residual MLP blocks to produce the refined feature P˜. Subsequently, an attention-based module interacts with P˜ to extract and reconstruct the segmentation prompt Pˆ.

The Segmentation Model uses the pre-trained Segment Anything Model (SAM) as the backbone, where it takes (Zimg, Pˆ) as inputs to yield the binary segmentation mask Mˆ.

Training Strategy

The training consists of two distinct stages designed for progressive refinement.

Reward Mechanism Details

The composite reward function R is tailored for complex medical scenarios. It includes a Format Reward (rfmt) which strictly enforces the structural integrity of the reasoning process, requiring the special token to be present for a positive reward. It also includes a Bipartite Matching Reward (rbbox)

Improvements for AI systems

Improved AI system can perform state-of-the-art complex lesion segmentation by reducing failure rate to 18.42% and achieving a mean Dice of 37.06%, specifically by integrating a Semantic-Guided Prompt Adapter to project the hidden states of special tokens from the MLLM’s textual space into SAM’s visual feature space, removing box-based error propagation.
Improved AI system can generate interpretable diagnostic assessments by utilizing a progressive training pipeline evolving from SFT to GRPO, equipped with an adaptive dual-granularity reward mechanism which ensures that the reward is set to zero for invalid formats and uses a reward formulation like "rmask = (GIoU(BMˆ, BM), if Dice < 0.05) 1 + λ · Dice, otherwise to amplify the reward for high-quality segmentations."
Improved AI system can achieve superior generalization across diverse clinical scenarios by employing a two-stage training strategy where the RL stage further enhances generalization and segmentation performance, as demonstrated by achieving +10.66% on TN-SCUI, +9.4% on ISPY, and +14.04% on CVC-ClinicDB when compared to the SFT baseline.
Improved AI system can handle multi-lesion reasoning robustly by using a Bipartite Matching Reward (rbbox) which minimizes global IoU cost via the Hungarian algorithm to balance localization precision with detection completeness.
Improved AI system can maintain efficiency and scalability by operating with minimal trainable parameters and achieving SOTA performance despite being 24× smaller than the massive Qwen2.5-VL-72B, proving that explicit reasoning alignment outweighs sheer parameter scaling.