MedPixel: A Unified Pixel-Language Model for Medical Reasoning and Segmentation

MedPixel: A Unified Pixel-Language Model for Medical Reasoning and Segmentation

Abstract

Reliable medical image understanding requires models to connect clinical language and visual reasoning with pixel-level grounding. Yet medical vision-language models often lack precise localization, whereas medical segmenters typically rely on explicit target categories or precise spatial prompts. This divide is reinforced by a supervision mismatch: segmentation datasets provide precise masks but little language supervision, whereas medical vision-language data rarely pair language with dense spatial annotations. To address this gap, we present MedPixel, a unified medical pixel-language model built around a shared language–mask interface. To provide scalable supervision, we introduce MedPLG-440K, comprising approximately 440K pixel-language task samples constructed through a clinically motivated synthesis process without external LLM annotation. MedPixel is trained with joint multi-task supervised fine-tuning followed by Pixel-Level Preference Optimization, which uses ground-truth masks as offline verifiers to derive response preferences from mask quality. MedPixel supports a broad spectrum of tasks spanning explicit grounding, implicit reasoning, spatial interaction, grounded explanation, and medical VQA. Across this task spectrum, MedPixel achieves strong performance in both pixel-level prediction and response generation, together with effective zero-shot transfer to external grounding benchmarks and robustness to imperfect spatial prompts.

Introduction

Medical image analysis is evolving from isolated recognition and segmentation tasks toward general systems that can support broader clinical decision making. Clinicians synthesize imaging findings with anatomical knowledge, clinical context, and diagnostic intent to identify relevant abnormalities, interpret their significance, and determine where they occur. This evolution expands the role of medical models from processing fixed image inputs to understanding images, language, and spatial prompts, and from producing fixed outputs to supporting reasoning, interaction, localization, and explanation. The paper summarizes these practical requirements into five representative medical task formats: explicit grounding, implicit reasoning, spatial interaction, grounded explanation, and medical visual question answering.

Current medical models emphasize different aspects of this functionality. Mainstream medical vision-language models focus primarily on image-level understanding and text generation, including medical question answering, report generation, and clinical reasoning. Medical segmentation models provide accurate pixel-level predictions across diverse imaging modalities, but usually rely on predefined target categories or explicit spatial prompts. Recent pixel-language models, including LISA, PixelLM, SAM4MLLM, and UniPixel, connect multimodal language understanding with dense prediction through shared language–mask representations, enabling natural-language grounding, reasoning segmentation, and visual-prompt interaction. Related medical models extend pixel-level grounding to biomedical images and increasingly support complex or implicit clinical queries that require target inference before localization. Despite this progress, unified support for diverse medical pixel-language interactions remains limited.

As the capability space broadens, supervision becomes a new bottleneck. Different forms of language–pixel interaction often require different types of supervision. This challenge is particularly pronounced in medicine, where existing data are highly asymmetric. Segmentation datasets provide abundant pixel-accurate masks but little language supervision, whereas medical vision-language datasets contain rich questions and answers but rarely pair them with dense spatial annotations. Although recent efforts have expanded biomedical datasets with spatial grounding and reasoning supervision, constructing dedicated supervision for each interaction form remains costly. The central challenge is therefore not simply to add more capabilities, but to obtain diverse medical pixel-language supervision in a scalable manner.

This motivates the authors to reconsider the role of existing segmentation masks. A mask is conventionally treated as the target of a segmentation objective, yet the corresponding annotated region also provides a spatial anchor for connecting image content with language. Together with image, category, and modality information, it can support descriptions of where a finding appears, how it is visually characterized, and what target is being referred to. Inspired by clinical image analysis, these cues can be organized from visual observation and characterization toward target interpretation and spatial grounding. Existing medical masks therefore constitute an underused source of structured pixel-language supervision.

Based on this insight, the authors construct MedPLG-440K (Medical Pixel-Language Grounding 440K), comprising approximately 440K pixel-language task samples across four grounded interaction formats. These samples are synthesized from existing medical segmentation annotations through a clinically motivated process without external LLM annotation. They separately incorporate medical VQA data to preserve general image-level understanding.

Built on this supervision, the authors introduce MedPixel, a unified medical pixel-language model based on Qwen2.5-VL and SAM2. A shared language–mask interface connects language generation with dense mask decoding through a special token, enabling diverse interaction forms to share a common backbone and segmentation pathway.

The shared language–mask interface also creates an alignment problem. The hidden representation of each generated token directly conditions mask decoding, yet standard supervised fine-tuning optimizes reference responses and target masks without comparing the spatial consequences of alternative generations. Responses that are similarly plausible in language may therefore produce different representations and masks of substantially different quality, creating a mismatch between response likelihood and pixel-level correctness. The authors introduce Pixel-Level Preference Optimization (PLPO) to address this mismatch. Ground-truth masks act as task-native offline verifiers that rank candidate responses by the quality of their induced masks. These rankings are converted into response preferences, aligning language generation with pixel-level outcomes without requiring an external reward model.

Another practical consideration is the quality of user-provided spatial prompts. Promptable segmentation methods are typically trained and evaluated with precise point or box prompts, whereas real users may provide only approximate spatial guidance. As prompt quality decreases, segmentation performance can degrade substantially. By combining semantic descriptions with spatial cues, MedPixel can use language to clarify the intended target when the spatial prompt is incomplete or inaccurate.

The contributions are summarized as follows:

• The authors introduce MedPLG-440K, comprising approximately 440K pixel-language samples across four grounded interaction formats, synthesized without external LLM annotation.

• They present MedPixel, a unified model for four pixel-level interactions and medical VQA, together with PLPO for aligning language generation with mask quality.

• MedPixel achieves leading localization and response-generation performance, with effective zero-shot transfer and robustness to imperfect box prompts.

Method

Task Formulation. Given a medical image I, a task instruction q, and, for interactive segmentation, an optional spatial reference r, MedPixel generates a textual response y and, when required, a segmentation mask M̂. The first four tasks produce mask-grounded responses, whereas medical VQA produces text-only answers.

Language–Mask Interface. MedPixel couples Qwen2.5-VL with a SAM2 mask decoding branch. Qwen2.5-VL performs multimodal understanding and language generation, while the mask branch produces dense segmentation outputs. All interaction formats are processed through the same multimodal backbone. Segmentation-oriented responses contain a special token, while interactive inputs additionally contain a spatial reference token.

For segmentation-oriented tasks, MedPixel generates a special token at the position where mask prediction is required. Let h denote the hidden state of this token. A learnable projection Φseg maps it into the SAM2 prompt embedding space: zseg = Φseg(h). Given the SAM2 image embedding FI, the mask decoder predicts M̂ = DSAM2(FI, zseg), where DSAM2 denotes the mask decoding pathway. This shared language–mask interface allows different query and response formats to use a common dense prediction mechanism.

Spatial Reference Injection. Interactive segmentation incorporates point or box guidance through a special token. Given a spatial reference r, a trainable prompt encoder converts it into a sparse spatial representation, which is then projected to the language-model hidden dimension: zref = Φref(Eref(r)), where Eref denotes the spatial prompt encoder and Φref the projection into the Qwen2.5-VL embedding space. Before the sequence is processed by the language model, zref replaces the original token embedding at the position. Spatial guidance therefore enters the same multimodal sequence as image and text inputs, allowing the subsequent representation to integrate semantic and spatial information before mask decoding.

MedPLG-440K Construction

MedPLG-440K is constructed from existing medical segmentation annotations. Each source sample provides a medical image I, a ground-truth mask M, a target category c, and metadata m. Based on these annotations, the authors first perform visual attribute extraction to obtain a visual descriptor d, and then synthesize four segmentation-oriented interaction formats: (I, M, c, m) → d → (xt, yt, M) t=1 T, where A denotes visual attribute extraction and T denotes task-specific synthesis.

Visual Attribute Extraction. For each annotated region, the authors derive a structured visual descriptor from mask geometry and image statistics. The descriptor captures location, size, shape, contrast, component structure, boundary, and uniformity. Location and size are obtained from the mask centroid and relative area; shape and component structure are characterized by region geometry and connected components; boundary regularity is estimated from contour measurements. Contrast and uniformity are computed from differences between the foreground and background and from variation within the region. Appearance attributes are expressed using terminology appropriate to the imaging modality. For example, appearance characteristics may be described in terms of density for CT, signal intensity for MRI, echogenicity for ultrasound, or opacity for X-ray. This process yields a compact visual descriptor aligned with the annotated region.

Task-Specific Synthesis. The source annotations and visual descriptor are combined according to the target interaction format. For T1 referring segmentation, the target category and metadata are used to construct explicit referring prompts. For T2 reasoning segmentation, category information is paired with curated templates to form implicit queries that require target inference before localization. For T3 interactive segmentation, points or bounding boxes are sampled from the ground-truth mask, while the visual descriptor is converted into a concise region description. For T4 explanatory segmentation, the visual descriptor is combined with concise clinical context to construct an explanatory response grounded in the target region. Each synthesized sample is paired with the original ground-truth mask to form a pixel-language training pair.

Through this pipeline, the authors construct MedPLG-440K, comprising approximately 440K pixel-language task samples. During Stage 1, MedPLG-440K is further combined with medical VQA and auxiliary clinical reasoning data to support joint training across five tasks.

Two-Stage Training

Stage 1: Joint Multi-Task SFT. Stage 1 jointly trains MedPixel on a unified mixture of all tasks. The objective combines autoregressive language modeling with mask supervision: LSFT = Lce + λseg Lseg, where Lce denotes the language modeling loss and Lseg denotes the pixel-level segmentation loss. For samples with segmentation supervision, the mask objective is Lseg = λfocal Lfocal + λdice Ldice + λiou Liou + λobj Lobj. For samples without mask supervision, the mask term is omitted. This joint objective allows the five task formats to share the same language–mask interface while learning both language generation and pixel-level prediction.

Stage 2: Pixel-Level Preference Optimization. Building on the Stage 1 checkpoint, the authors construct response preference pairs from reasoning segmentation samples according to the quality of the masks induced by alternative generations. For each input xi, they sample N candidate responses from πSFT and decode the mask associated with each generated representation. The quality of the n-th candidate is measured against the corresponding ground-truth mask: si(n) = Dice(M̂i(n), Migt), where M̂i(n) denotes the mask induced by the n-th candidate response. Ground-truth masks are used only for offline scoring and are never exposed during candidate generation. This process converts pixel-level outcomes into response preference supervision without requiring an additional reward model or online rollouts.

The authors discard empty responses and candidates that do not produce a valid output from which a mask can be decoded. Among the remaining candidates, the highest-scoring response is selected as the chosen response yi+, while a lower-scoring valid response is selected as the rejected response yi−. To avoid weak or ambiguous preference supervision, they retain a pair only when s+i ≥ τmin and s+i − s−i ≥ τgap, where τmin ensures sufficient quality of the chosen response and τgap enforces a meaningful quality difference between the chosen and rejected responses. Inputs that do not satisfy these criteria are discarded.

Given the preference dataset P, the authors optimize MedPixel using Direct Preference Optimization. Let πθ denote the trainable policy and πref = πSFT the frozen reference policy. The preference objective is LPLPO = −EP[log σ(β∆θ)] + λnll Lnll(y+ x), where β controls the preference strength and the NLL term anchors the model to the chosen response.

During Stage 2, the segmentation projector Φseg and the SAM2 mask decoder remain frozen. Preference optimization therefore reshapes the response distribution and the resulting representations while preserving the dense decoding pathway learned in Stage 1. In this way, MedPixel converts differences in pixel-level mask quality into response-level preferences, encouraging generations whose representations induce more accurate segmentation outcomes.

Experiments

Training data. Stage 1 trains MedPixel on a 1.84M-sample mixture of medical VQA (64%), segmentation (32%), and clinical reasoning (4%) data. MedPLG-440K provides the four segmentation-oriented tasks (T1–T4), which are incorporated through task-specific sampling and replication. The VQA component is aggregated from public medical QA datasets, while the reasoning component combines HuatuoGPT-o1, Medical-R1-Distill, Citrus-S3, and image-grounded reasoning synthesized from PubMedVision. The resulting checkpoint is used to mine offline preference pairs for Stage 2.

Evaluation Sets. T1–T4 are derived from the held-out official BiomedParse test split, which is excluded from training. T1 evaluates explicit referring segmentation on 24,391 representative slices selected to reduce redundancy in volumetric studies. T2 and T4 contain 2,500 samples each for implicit reasoning and grounded explanation, respectively. T3 contains 2,000 interactive samples, with point and box prompts derived from ground-truth masks. T5 contains 10,267 image-based and text-based multiple-choice questions from publ…

Based on this paper, here are the specific improvements I can implement in AI systems, along with the resulting capabilities:

1. Unified Language–Mask Interface for Multi-Task Medical AI

• Improvement: Integrate a shared token-based interface (e.g., and tokens) that couples a vision-language backbone with a mask-decoding branch (like SAM2). This allows a single model to handle text generation, spatial reasoning, and dense segmentation without task-specific architectures.

• Capability: The AI can simultaneously answer clinical questions, reason about findings, generate explanations, and produce pixel-accurate masks in one forward pass—eliminating the need for separate models for VQA, segmentation, and reasoning.

2. Scalable Pixel-Language Supervision from Existing Masks

• Improvement: Automatically synthesize 440K+ pixel-language training samples from existing segmentation masks by extracting visual descriptors (location, size, shape, contrast, boundary, uniformity) and generating task-specific prompts (referring, reasoning, interactive, explanatory) using rule-based templates—no external LLM annotation required.

• Capability: The AI can be trained on diverse grounded interaction formats without costly manual annotation, enabling rapid adaptation to new medical domains where only masks are available.

3. Pixel-Level Preference Optimization (PLPO) for Outcome-Aligned Generation

• Improvement: After supervised fine-tuning, sample multiple candidate responses, decode their induced masks, score them against ground-truth masks using Dice, and construct preference pairs (chosen vs. rejected) for Direct Preference Optimization—while freezing the segmentation branch.

• Capability: The AI learns to generate responses whose associated masks are more accurate, directly aligning language output with pixel-level correctness. This reduces the mismatch between plausible text and wrong localization in medical reasoning.

4. Robustness to Imperfect Spatial Prompts via Semantic-Spatial Fusion

• Improvement: Inject spatial references (points/boxes) as token embeddings into the language model, allowing the AI to combine semantic descriptions with spatial cues. Train with both precise and perturbed prompts to handle real-world user input variability.

• Capability: The AI maintains high segmentation accuracy even when users provide loose, tight, or partially incorrect bounding boxes—unlike specialist segmenters that degrade sharply—making it practical for clinical use where prompts are often approximate.

5. Zero-Shot Transfer Across Modalities and Benchmarks

• Improvement: Train on a diverse mixture of tasks (VQA, segmentation, reasoning) and evaluate directly on unseen benchmarks (e.g., MeCoVQA-G+, U-MRG-14K) without fine-tuning, using the unified interface and multi-task supervision.

• Capability: The AI generalizes to new medical modalities (CT, MRI, ultrasound, X-ray) and unseen reasoning-grounding tasks, producing masks and explanations without benchmark-specific adaptation—critical for deployment in varied clinical settings.

6. Joint Multi-Task Training with Balanced Supervision

• Improvement: Combine 1.84M samples (64% VQA, 32% segmentation, 4% reasoning) with task-specific sampling and replication, optimizing a combined loss (language modeling + focal/dice/IoU/object losses) for all five tasks simultaneously.

• Capability: The AI learns shared representations across tasks, improving performance on each—especially on reasoning and explanatory segmentation where gains exceed 29–40 Dice points over baselines—while preserving VQA accuracy.

7. Scale-Aware Capability Enhancement

• Improvement: Train at 3B and 7B scales with LoRA on attention projections, showing that scaling improves semantic reasoning and response generation without sacrificing dense grounding.

• Capability: The AI can be deployed at different compute budgets, with the 7B version offering superior clinical reasoning and explanation quality while maintaining segmentation accuracy—useful for resource-constrained vs. high-end environments.

What the Improved AI System Can Do:

• Answer medical VQA questions with high accuracy (e.g., >75% on image-based MCQs).

• Segment anatomical structures from a natural-language query (explicit or implicit) with Dice >85% on standard benchmarks.

• Explain why a region is abnormal, with both text and mask output, grounded in visual evidence.

• Accept a rough point or box from a clinician and still produce a correct mask, even if the prompt is imprecise.

• Transfer zero-shot to new modalities and unseen reasoning tasks without retraining.

• Learn from existing mask datasets alone, without needing paired language annotations, making it scalable to large unlabeled medical image repositories.

• Operate as a single unified model for clinical decision support, reducing the need for multiple specialized tools.

Reliable medical image understanding requires models to connect clinical language and visual reasoning with pixel-level grounding. Yet medical vision-language models often lack precise localization, whereas medical segmenters typically rely on explicit target categories or precise spatial prompts. This divide is reinforced by a supervision mismatch: segmentation datasets provide precise masks but little language supervision, whereas medical vision-language data rarely pair language with dense spatial annotations. To address this gap, we present MedPixel, a unified medical pixel-language model built around a shared language--mask interface. To provide scalable supervision, we introduce MedPLG-440K, comprising approximately 440K pixel-language task samples constructed through a clinically motivated synthesis process without external LLM annotation. MedPixel is trained with joint multi-task supervised fine-tuning followed by Pixel-Level Preference Optimization, which uses ground-truth masks as offline verifiers to derive response preferences from mask quality. MedPixel supports a broad spectrum of tasks spanning explicit grounding, implicit reasoning, spatial interaction, grounded explanation, and medical VQA. Across this task spectrum, MedPixel achieves strong performance in both pixel-level prediction and response generation, together with effective zero-shot transfer to external grounding benchmarks and robustness to imperfect spatial prompts. Code and model checkpoints will be released at https://github.com/yhy-whu/Medpixel.