Robust Reasoning Segmentation for Open-World MLLMs without Catastrophic Forgetting

arXiv:2510.16785 · cs.CV · Submitted 2025-10-19 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Robust Reasoning Segmentation for Open-World MLLMs without Catastrophic Forgetting".

Jane: Here's the gist of the paper. Integrating diverse visual capabilities into a unified model is a significant trend in Multimodal Large Language Models (MLLMs) To equip MLLMs with pixel-level segmentation abilities, prevailing methods require finetuning the model to produce specific outputs compatible with a mask decoder LENS introduces a novel plug-and-play solution that attaches a lightweight, trainable head to a completely frozen MLLM By refining the spatial cues embedded in attention maps, LENS extracts keypoints and describes them into point-wise features directly compatible with the mask decoder Extensive experiments validate our approach, showing that LENS achieves segmentation performance competitive with or superior to retraining-based methods while fully preserving the MLLM’s generalization capabilities.

Tom: Pruning redundant reasoning tokens The paper argues against prior fine-tuning methods because the dual-objective training introduces an inherent tension between model’s capabilities, where generative tasks thrive on abstract, sparse semantics and segmentation requires direct, spatial features Conventional paradigms couple these objectives in a way that leads to incoherent training This dual-objective training increases optimization complexity and sensitivity to configurations.

Jane: LENS resolves this by decoupling the roles, allowing a frozen MLLM to focus on reasoning while a lightweight head is trained exclusively for segmentation How the training signal is built The training process for LENS employs a composite loss function consisting of two components: an attention loss Lattn and a segmentation loss Lseg The Attention Loss uses binary cross-entropy (BCE) to enforce alignment between the attention map A1c and the ground-truth binary mask M The Segmentation Loss follows a practice similar to LISA, adopting a combination of Dice loss and BCE applied to the final predicted mask Mˆ The overall objective combines these losses as L = Lseg(M, M ˆ) + Lattn(A 1c, M).

Tom: Where the gains come from The core innovation of LENS is using keypoints from the MLLM’s internal attention maps as a bridge to unify the stages of segmentation The second stage extracts points from high-value regions of A1c, and these are described using a descriptor model that leverages cross-attention between global semantic features and local keypoint features This process results in keypoint set P and description set D, which are then used to generate the final mask Mˆ via the SAM decoder State-of-the-art performance and efficiency LENS achieves state-of-the-art segmentation with lower training cost while preserving general abilities LENS is highly efficient because the MLLM is used only for inference, reducing training costs significantly Furthermore, LENS avoids the multi-objective trade-off by being trained exclusively on segmentation data and remaining decoupled from generative learning This plug-and-play design offers a practical and scalable solution for unified vision models.

Title and authors: Tom: Alright team, let's talk about this paper: "Robust Reasoning Segmentation for Open-World MLLMs without Catastrophic Forgetting". The authors are looking at how to give these multimodal models pixel-level segmentation without messing up their main reasoning abilities.

Jane: Exactly. It sounds like they're tackling that big problem where you need both detailed spatial information and high-level semantic understanding in one model, and they're trying to solve it by creating a setup where the two goals don't fight each other during training.

Lu: The core idea is using keypoints from the MLLM’s internal attention maps as a bridge to unify the stages of segmentation, which allows for keypoint extraction and description guided by semantic features from the MLLM.

Meng: But what does that actually mean for us? If we can keep the main model frozen and only train a small part, that sounds like it could drastically cut down on training costs when we deploy these systems.

Lalam: LENS is this plug-and-play architecture that delivers state-of-the-art segmentation performance while fully preserving the MLLM’s broad capabilities and cutting training costs by a large margin.

Tom: So, to summarize what they did, they introduced LENS as a way to attach a lightweight head to a frozen MLLM instead of retraining the whole thing for segmentation.

Jane: Right. And they achieved this by refining the spatial cues embedded in those attention maps to extract keypoints and then using those keypoints with descriptions to guide the final mask decoder.

Lu: They built this by decoupling the roles, so a frozen MLLM is dedicated to reasoning while a lightweight head is trained exclusively for segmentation, which avoids that incoherent training problem seen in earlier methods like LISA.

Meng: I’m interested in how they handle the training signal then; if you have two objectives, how do you keep them from fighting each other during optimization?

Jane: They use a composite loss function with two parts: an attention loss that aligns the attention map with the ground truth mask, and a segmentation loss that follows a practice similar to LISA using both dice loss and binary cross-entropy on the final predicted mask.

Tom: So, we have this dual training signal setup to make sure both spatial alignment and accurate prediction happen simultaneously. It’s a specific way of forcing coherence.

Title and authors: Lu: The second part is where they use keypoints from high-value regions of the attention map and describe them using a descriptor model that leverages cross-attention between global semantic features and those local keypoint features, resulting in a set of keypoints and descriptions.

Meng: That sounds like a lot of work on the representation side. What's the real gain here compared to just training one big model for everything?

Jane: The main gain is that this process creates keypoint–description pairs that act as direct prompts for the SAM decoder, effectively bridging those internal MLLM representations with what the segmentation model actually needs.

Tom: So, it’s not just feeding raw pixels into a separate network; it's using the model's own thinking—its attention maps—to find where to look and what to describe. That’s a clever way to leverage existing knowledge.

Lu: Yes, they call this architecture LENS because it leverages keypoints for MLLMs’ Segmentation, and they claim this design keeps the backbone entirely frozen while delivering substantial efficiency gains because the MLLM is used purely in inference mode.

Meng: Efficiency is good, but what about deployment? If we need to switch between dialogue and vision tasks quickly, does this plug-and-play nature actually help us in a real agent system?

Lalam: Yes, they designed it so the MLLM can be invoked through question answering mechanisms rather than relying on special tokens, meaning the model decides when to activate the segmentation head on demand.

Tom: That modularity is key for agents. So, they're not forcing a segmentation task onto every single prompt; it’s optional and controlled by the system logic.

Jane: And they show that this approach preserves general abilities, which is crucial because prior methods often saw severe degradation in those core skills when trying to add vision capabilities.

Lu: They also found that removing either the global description or the keypoint description hurts performance, which shows both components of that bridge are actually essential for getting those best results.

Meng: That’s a fair caveat, though. It means we can't just strip things out without seeing a drop in performance, so the balance between the two descriptions matters a lot.

Tom: So, to wrap up this paper "Robust Reasoning Segmentation for Open-World MLLMs without Catastrophic Forgetting", LENS is this method that lets you get pixel segmentation on an MLLM without having to retrain the whole thing or lose its general reasoning skills?

Jane: Exactly. It’s a way to get that fine-grained visual detail while keeping the model smart for other things.

Title and authors: Lu: The paper argues against prior methods because they introduce this tension between generative tasks and segmentation, which makes the training very unstable, but LENS solves that by decoupling them.

Meng: From a deployment standpoint, it’s good because you only train a small head, so the cost to run it is low compared to retraining the whole massive model.

Lalam: I'm confident in this direction; LENS establishes state-of-the-art results on reasoning and referring segmentation while keeping accuracy on par with other large models.

Tom: Those numbers are impressive, but what about the mechanism? How exactly does it bridge that gap between high-level language and low-level pixels?

Lu: The core idea is using keypoints from the MLLM's internal attention maps as a bridge to unify the stages of segmentation. It extracts these points and describes them using cross-attention between global semantic features and those local keypoints.

Jane: That’s interesting, so it’s not just feeding pixels into a separate network, but using the model's own thinking—its attention maps—to find where to look.

Tom: Right, so the training loss has two parts: one part for aligning the model's attention map with the ground truth mask, and another part for actually making the final prediction.

Jane: That composite loss function is what makes it coherent; it forces both parts of the system to work together during training.

Meng: I worry about scaling that; if you have to run inference on two separate stages, does it add too much latency compared to just using a standard segmentation head?

Tom: The authors show it's highly efficient because the MLLM is only used for inference, reducing the overall training cost significantly.

Lu: They also showed that removing either the global description or the keypoint description hurts performance, so both components are actually essential for getting those best results.

Jane: It sounds like they’ve really nailed a way to keep the MLLM's broad capabilities intact while adding this powerful new visual tool.

Tom: So that’s "Robust Reasoning Segmentation for Open-World MLLMs without Catastrophic Forgetting". That’s a lot of work done here.

Jane: It is. And it shows we can integrate complex vision tasks into these large language models in a much smoother way than before.

Meng: I'm curious to see how this modular approach plays out when we start putting these kinds of tools into real agent systems for autonomous tasks.

The paper's summary: Tom: So we're diving into "Robust Reasoning Segmentation for Open-World MLLMs without Catastrophic Forgetting." This paper is about how to give multimodal models pixel-level segmentation without messing up their main reasoning skills.

Jane: Right, it’s tackling that big problem where you need both detailed spatial information and high-level semantic understanding in one model structure, and they're trying to solve it by creating this decoupled setup where the two goals don't fight each other during training.

Lu: The core innovation is using keypoints from the MLLM’s internal attention maps as a bridge to unify the stages of segmentation, which allows for keypoint extraction and description guided by semantic features from the MLLM.

Tom: So they aren't just tacking on a separate vision module; they are using the model's own internal thinking—its attention maps—to find where to look and what to describe. That’s a clever way to leverage existing knowledge, right?

Jane: Exactly. And this process creates keypoint-description pairs that act as direct prompts for the final mask decoder, essentially bridging those internal representations with what the segmentation model actually needs.

Lu: They built this by decoupling roles so a frozen MLLM focuses on reasoning while a lightweight head is trained exclusively for segmentation, which avoids that incoherent training problem seen in earlier methods like LISA.

Tom: That sounds like it solves the optimization complexity issue, because they aren't trying to force two totally different kinds of learning—generative and spatial—to happen at the same time in a messy way.

Jane: They do use a composite loss function with two parts: an attention loss that aligns the attention map with the ground truth mask, and a segmentation loss that follows LISA using both dice loss and binary cross-entropy on the final predicted mask.

Lu: That dual training signal is what makes it coherent; it forces both spatial alignment and accurate prediction to happen together during training.

Tom: The benefit here is efficiency; they keep the heavy MLLM frozen, so you're not retraining the whole thing, and they claim this still gets segmentation performance competitive with or better than methods that require full retraining.

Jane: But there's a catch because the authors found that removing either the global description or the keypoint description hurts performance, which means both parts of that bridge are actually essential for getting those best results.

Lu: That’s a fair caveat; it shows that we can't just strip things out without seeing a drop in accuracy, so the balance between those two descriptions matters a lot for their setup.

Tom: So to summarize this paper, LENS is this plug-and-play architecture that brings segmentation into MLLMs without compromising their general reasoning skills by successfully bypassing the objective conflict that hampers prior fine-tuning based approaches.

Jane: It means we can build multimodal AI systems that have both deep reasoning and fine-grained visual perception without sacrificing the general purpose intelligence of the base model, which is a really important step forward for real world applications.

The paper's summary: Lu: They establish this plug-and-play architecture that delivers state-of-the-art segmentation performance while fully preserving the MLLM’s broad capabilities and cutting training costs by a large margin, which is what makes it practical.

Tom: So we have a method where the MLLM keeps its general reasoning skills, and we get pixel segmentation performance competitive with or better than retraining methods. That's what this paper is all about.

Jane: It’s a way to get that fine-grained visual detail while keeping the model smart for other things, but it’s important to remember that LENS is designed to be invoked by the MLLM through question answering mechanisms rather than relying on special tokens.

Lu: That modularity is key for agents; they designed it so the MLLM decides when to activate the segmentation head on demand through tool frameworks or thinking-based control.

Tom: That means we aren't forcing a segmentation task onto every single prompt; it’s optional and controlled by the system logic, which sounds much better for deployment than having a fixed, mandatory module.

Jane: And they showed that this approach preserves general abilities, which is crucial because prior methods often saw severe degradation in those core skills when trying to add vision capabilities.

Lu: They also found that removing either the global description or the keypoint description hurts performance, so both components are actually essential for getting those best results.

Tom: Those numbers on reasoning and referring segmentation are impressive, but what about the mechanism? How exactly does it bridge that gap between high-level language and low-level pixels?

Lu: The core idea is using keypoints from the MLLM's internal attention maps as a bridge to unify the stages of segmentation. It extracts these points and describes them using cross-attention between global semantic features and those local keypoints.

Jane: That’s interesting, so it’s not just feeding pixels into a separate network, but using the model's own "thinking" process—its attention maps—to find where to look.

Tom: Right, so the training loss has two parts: one part for aligning the model's attention map with the ground truth mask, and another part for actually making the final prediction.

Jane: That composite loss function is what makes it coherent; it forces both parts of the system to work together during training.

Meng: I worry about scaling that; if you have to run inference on two separate stages, does it add too much latency compared to just using a standard segmentation head?

Tom: The authors show it's highly efficient because the MLLM is only used for inference, reducing the overall training cost significantly.

Lu: They also showed that removing either the global description or the keypoint description hurts performance, so both components are actually essential for getting those best results.

Jane: It sounds like they’ve really nailed a way to keep the MLLM's broad capabilities intact while adding this powerful new visual tool, and it shows we can integrate complex vision tasks into these large language models in a much smoother way than before.

The paper's improvements: Tom: So we're talking about how they fixed their setup for "Robust Reasoning Segmentation for Open-World MLLMs without Catastrophic Forgetting." It sounds like they’re suggesting a few tweaks to make this system even better than what's already out there.

Jane: Right, the paper suggests adding two specific technical improvements to boost the quality of those keypoints and descriptions they're using as a bridge between the language and vision parts.

Lu: One improvement is introducing sub-pixel refinement; they propose applying a regularized Newton update to shift grid-aligned coordinates toward underlying peak locations.

Tom: So instead of just getting an approximate keypoint in pixel space, you get this refined sub-pixel keypoint that’s much more accurate, which should help with precise segmentation masks.

Jane: That makes sense because better coordinates mean the final mask prediction will be sharper, and they say this helps improve coordinate accuracy.

Lu: They also mention a way to manage the training signal itself by using a dropout mechanism during training where with a probability of zero point five, we randomly use only the global description or use both during training.

Tom: That sounds like a safety net; if one component isn't working well at any given moment, you still get some learning signal from the other part of the system.

Jane: It’s a way to generally ensure better performance, and it addresses that tension we talked about earlier by making the training process more robust to instability.

Lu: These changes show they are really focusing on making that keypoint-to-description link stronger so the overall segmentation task doesn't suffer much when it has to operate in a complex open world.

Tom: So, these tweaks suggest that getting better spatial detail and a more stable training process are key to making this approach reliable for real-world use.

Jane: They’re essentially taking their existing architecture and adding these small, targeted refinements to make the segmentation part of it really perform at its best.

Lu: This modular design with those specific fixes is what makes LENS so practical; it’s not just an idea on paper, it’s a functional system that can be plugged in and run.

Tom: And since they're focused on these improvements, I wonder if they plan to test these new components against some real-world data like the reasoning segmentation benchmarks they mentioned earlier?

Conclusion: Tom: So we're wrapping up "Robust Reasoning Segmentation for Open-World MLLMs without Catastrophic Forgetting." Essentially, this paper shows how to give multimodal models pixel segmentation without forcing them to forget how to reason.

Jane: It is a big deal because it proves you can integrate detailed visual perception into those large language models while keeping their general intelligence intact, which is really what we want for useful AI.

Lu: The paper argues against prior methods because they introduce that tension between generative tasks and segmentation, which makes the training very unstable, but LENS solves that by decoupling them.

Meng: From a deployment standpoint, it’s good because you only train a small head, so the cost to run it is low compared to retraining the whole massive model.

Lalam: I'm confident in this direction; LENS establishes state-of-the-art results on reasoning and referring segmentation while keeping accuracy on par with other large models.

Tom: Those numbers are impressive, but what about the mechanism? How exactly does it bridge that gap between high-level language and low-level pixels?

Lu: The core idea is using keypoints from the MLLM's internal attention maps as a bridge to unify the stages of segmentation. It extracts these points and describes them using cross-attention between global semantic features and those local keypoints.

Jane: That’s interesting, so it’s not just feeding pixels into a separate network, but using the model's own "thinking" process—its attention maps—to find where to look.

Tom: Right, so the training loss has two parts: one part for aligning the model's attention map with the ground truth mask, and another part for actually making the final prediction.

Jane: That composite loss function is what makes it coherent; it forces both parts of the system to work together during training.

Meng: I worry about scaling that; if you have to run inference on two separate stages, does it add too much latency compared to just using a standard segmentation head?

Tom: The authors show it's highly efficient because the MLLM is only used for inference, reducing the overall training cost significantly.

Lu: They also showed that removing either the global description or the keypoint description hurts performance, so both components are actually essential for getting those best results.

Jane: It sounds like they’ve really nailed a way to keep the MLLM's broad capabilities intact while adding this powerful new visual tool.

Tom: So that’s "Robust Reasoning Segmentation for Open-World MLLMs without Catastrophic Forgetting." That’s a lot of work done here.

Jane: It is. And it shows we can integrate complex vision tasks into these large language models in a much smoother way than before.

Meng: I'm curious to see how this modular approach plays out when we start putting these kinds of tools into real agent systems for autonomous tasks.

The Hong Kong University of Science and Technology

cs.CV

Submitted: 2025-10-19

Updated: 2026-11-11

Comments: Published in International Journal of Computer Vision (IJCV), 2026

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 94/100

The gist: Integrating diverse visual capabilities into a unified model is a significant trend in Multimodal Large Language Models (MLLMs) To equip MLLMs with pixel-level segmentation abilities…

Key concepts

Catastrophic Forgetting
This occurs when a neural network, trained on a new task, rapidly loses its previously learned knowledge from earlier tasks. In MLLMs, this means the model might forget how to reason or understand language after being specifically trained on a new visual segmentation style.
Decoupled Adaptation
Instead of fine-tuning the entire MLLM for every new task, this approach separates the general reasoning components from the task-specific modules. This allows the core model to retain its broad capabilities while only updating small, dedicated parts for a new skill like segmentation.
Open-World Reasoning
This refers to an MLLM's ability to handle novel visual concepts or tasks it was not explicitly trained on. The goal is to enable the model to apply its existing reasoning skills flexibly across a wide range of unseen visual and language challenges without needing complete retraining.

Terminology

Summary

Integrating diverse visual capabilities into a unified model is a significant trend in Multimodal Large Language Models (MLLMs) To equip MLLMs with pixel-level segmentation abilities, prevailing methods require finetuning the model to produce specific outputs compatible with a mask decoder LENS introduces a novel plug-and-play solution that attaches a lightweight, trainable head to a completely frozen MLLM By refining the spatial cues embedded in attention maps, LENS extracts keypoints and describes them into point-wise features directly compatible with the mask decoder Extensive experiments validate our approach, showing that LENS achieves segmentation performance competitive with or superior to retraining-based methods while fully preserving the MLLM’s generalization capabilities. p. 0

Pruning redundant reasoning tokens

The paper argues against prior fine-tuning methods because the dual-objective training introduces an inherent tension between model’s capabilities, where generative tasks thrive on abstract, sparse semantics and segmentation requires direct, spatial features Conventional paradigms couple these objectives in a way that leads to incoherent training This dual-objective training increases optimization complexity and sensitivity to configurations. LENS resolves this by decoupling the roles, allowing a frozen MLLM to focus on reasoning while a lightweight head is trained exclusively for segmentation

How the training signal is built

The training process for LENS employs a composite loss function consisting of two components: an attention loss Lattn and a segmentation loss Lseg The Attention Loss uses binary cross-entropy (BCE) to enforce alignment between the attention map A1c and the ground-truth binary mask M The Segmentation Loss follows a practice similar to LISA, adopting a combination of Dice loss and BCE applied to the final predicted mask Mˆ The overall objective combines these losses as L = Lseg(M, M ˆ) + Lattn(A 1c, M).

Where the gains come from

The core innovation of LENS is using keypoints from the MLLM’s internal attention maps as a bridge to unify the stages of segmentation The second stage extracts points from high-value regions of A1c, and these are described using a descriptor model that leverages cross-attention between global semantic features and local keypoint features This process results in keypoint set P and description set D, which are then used to generate the final mask Mˆ via the SAM decoder

State-of-the-art performance and efficiency

LENS achieves state-of-the-art segmentation with lower training cost while preserving general abilities LENS is highly efficient because the MLLM is used only for inference, reducing training costs significantly Furthermore, LENS avoids the multi-objective trade-off by being trained exclusively on segmentation data and remaining decoupled from generative learning This plug-and-play design offers a practical and scalable solution for unified vision models.

Usage in agent systems

LENS is designed to be invoked by the MLLM through question answering mechanisms rather than relying on special tokens This allows the MLLM to autonomously decide whether to activate the segmentation head when needed, such as via chain-of-thought reasoning The architecture can be integrated into agent frameworks using tool frameworks or thinking-based control.

Evaluation results

LENS establishes state-of-the-art results on multiple segmentation benchmarks, including reasoning segmentation and referring segmentation On ReasonSeg, LENS reaches 65.3 cIoU on validation and 57. p. 83 on test, outperforming LISA under the same backbone On RefCOCO(+/g), LENS achieves 76. p. 85 cIoU on validation and 70. p. 83 on test, exceeding LISA-7B Crucially, MMBench accuracy remains on par with LLaVA-1. p. 75-7B, unlike prior methods which suffer severe degradation. p. 7

Ablation study findings

The ablation study revealed that removing either the global description or the keypoint description leads to a performance drop, indicating that both components contribute significantly to the overall performance The steepest drop occurs when shifting the head’s input from the 14th to the 30th layer, which reduces cIoU from 57.3 to 40.0, underscoring that features from deeper layers lose spatial detail

Conclusion

LENS establishes a plug-and-play architecture that brings segmentation into MLLMs without compromising their general-purpose abilities The design successfully bypasses the objective conflict that hampers prior fine-tuning based approaches LENS achieves state-of-the-art segmentation performance while fully preserving the MLLM’s broad capabilities and cutting training costs by a large margin.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Decouple segmentation from generative training by introducing LENS, which attaches a lightweight, trainable head to a completely frozen MLLM. This allows for a decoupled, coherent objectives setup, preserving the MLLM’s general purpose abilities while adding pixel-level segmentation.

  2. Leverage internal attention mechanisms as a bridge between high-level reasoning and dense prediction by using keypoints from the MLLM’s internal attention maps as a bridge that intrinsically unifies the stages. This allows for keypoint extraction and description to be guided by semantic features from the MLLM.

  3. Implement sub-pixel refinement to improve coordinate accuracy by applying a regularized Newton update to shift grid-aligned coordinates toward underlying peak locations, resulting in refined sub-pixel keypoints in pixel space.

  4. Create a modular, plug-and-play capability where the MLLM decides when to activate the segmentation head through tool frameworks or thinking-based control, enabling seamless switching between general dialogue and vision-centric segmentation tasks.

  5. Ensure robust performance by using a dropout mechanism during training where with a probability p = 0.5, we randomly use only the global description or use both during training, which is found to generally ensures better performance.

Sources

Related papers