VITAL: Visual-Semantic Dual Supervision for Enhanced and Interpretable Latent Reasoning in Medical MLLMs

arXiv:2605.28422 · cs.CV, cs.AI · Submitted 2026-05-27 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "VITAL: Visual-Semantic Dual Supervision for Enhanced and Interpretable Latent Reasoning in Medical MLLMs".

Jane: The gist: VITAL proposes a latent-space reasoning framework for medical MLLMs that introduces visual-semantic dual supervision to prevent modality collapse while maintaining strict train–inference parity and zero inference overhead.

Tom: First, who's behind it and why it matters.

Title and authors: Jane: So we know they are focusing on latent-space reasoning for medical MLLMs, and this paper, "VITAL: Visual-Semantic Dual Supervision for Enhanced and Interpretable Latent Reasoning in Medical MLLMs," is about solving the issues of modality collapse and lack of transparency that come with those internal states >

Tom: Exactly. The paper sets out to introduce a latent-space reasoning framework for medical MLLMs that includes visual-semantic dual supervision, meaning they’re adding two specific training signals to guide the model >

Lu: The core idea is an auxiliary text decoder that reconstructs the reasoning chains from each latent state, and a visual projector that regresses region of interest features from a frozen, independent medical vision encoder >

Meng: It seems like the big shift they are proposing is moving away from those opaque latent states to something more structured that can be both read and seen >

Lalam: They also mention this framework allows for both modules to be discarded at inference with zero overhead, but you can re-attach them later if you want dual interpretability, which is a huge plus for clinical use >

The paper's summary: Tom: Now let's talk about what the VITAL framework actually does according to the summary. It maintains strict train-inference parity through a single recurrent loop, which is super important because it means you don't get that mismatch you see in some other methods >

Jane: So they keep the training path and inference path identical by using this recurrent latent loop, which generates K continuous states where the language model only does a single token forward pass at each step >

Lu: This means instead of discrete tokens, the model is performing an autoregressive causal chain over these continuous vectors >

Meng: And for decoding the final answer, they predict that first answer token directly from the very last latent state zK >

Lalam: The summary highlights that this approach prevents modality collapse by supervising every latent state zk from two directions: semantic and visual >

The paper's improvements: Tom: So what are the actual improvements they claim in terms of performance? They test it on seven benchmarks and they say VITAL consistently outperforms the backbone, all latent reasoning baselines, and medical MLLMs trained on much larger data >

Jane: It sounds like a solid jump for their performance metrics, but they also mention that VITAL achieves state-of-the-art results competitive with trillion-parameter proprietary models while being way ahead on visual grounding >

Lu: They claim this dual supervision leads to better visual grounding by a large margin, which is significant because it addresses the issue of the model not fully understanding where things are in the image >

Meng: And they mention a curriculum learning strategy, Phase one with K equal to one, Phase two with K less than or equal to two, and Phase three with all K > <ref:2605.28422#pg1>

Lalam: That curriculum learning helps them avoid overfitting to the training data distribution shift because it consistently improves over the backbone on every single benchmark >

Conclusion: Tom: So, to wrap up this discussion on VITAL: they’ve given us a latent-space reasoning framework for medical MLLMs that uses visual-semantic dual supervision to keep the model aligned and interpretable >

Jane: It seems like the main idea is that by supervising the latent states both textually and visually, they get a system where you can reconstruct the reasoning chain and see what the model is looking at >

Lu: And because they kept train-inference parity through that single recurrent loop, it avoids those train-test mismatches we've seen before >

Meng: From an engineering standpoint, being able to re-attach those modules post hoc for dual interpretability is a practical feature that really helps build trust in the model’s output >

Lalam: It gives you both a textual window showing what the model reasons and a visual window showing where the model looks, which is a remarkable level of transparency for latent reasoning >

Tom: That's it for this paper on VITAL: it’s about taking continuous latent states and making them robust and understandable in medical contexts >

Zhejiang University · Shanghai AI Laboratory

cs.CV, cs.AI

Submitted: 2026-05-27

Updated: 2026-10-09

Comments: EMNLP 2026 Oral

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: The gist: VITAL proposes a latent-space reasoning framework for medical MLLMs that introduces visual-semantic dual supervision to prevent modality collapse while maintaining strict train–inference

Key concepts

Recurrent Latent Reasoning: Instead of discrete tokens, VITAL uses a recurrent loop that generates a sequence of continuous latent states. The language model performs one step at each state, creating an autoregressive chain over these vectors. Crucially, training and inference follow the exact same path to maintain consistency.
Visual-Semantic Dual Supervision: This technique stabilizes reasoning by supervising every latent state from two directions: semantically (via a text decoder reconstructing chains) and visually (via a projector mapping states to ROI features from a frozen vision encoder). This dual supervision prevents the model from collapsing into only one modality.
Train-Inference Parity: VITAL ensures that the process used during training is identical to the one used during inference. This is achieved by having both training and testing traverse the same recurrent path, eliminating common mismatch issues between training data and real-world application.

Terminology

Latent Reasoning — 4× in this paper · explained in 3 episodes
Interpretability — 2× in this paper · explained in 15 episodes
Benchmark — 1× in this paper · explained in 2 episodes

Summary

The gist: VITAL proposes a latent-space reasoning framework for medical MLLMs that introduces visual-semantic dual supervision to prevent modality collapse while maintaining strict train–inference parity and zero inference overhead.

VITAL Framework Overview

VITAL is a latent-space reasoning framework for medical MLLMs with visual-semantic dual supervision, where an auxiliary text decoder reconstructs reasoning chains from latent states, while a visual projector regresses ROI features from a frozen, independent medical vision encoder. Both modules are discarded at inference with zero overhead, yet can be re-attached post-hoc for dual interpretability (Page 1). The framework maintains strict train–inference parity through a single recurrent loop (Page 3).

Recurrent Latent Reasoning

The recurrent latent loop produces K continuous states where the language model performs a single-token forward pass at each step, creating an autoregressive causal chain over continuous vectors rather than discrete tokens (Page 3). A critical design is that both training and inference traverse the identical recurrent path, eliminating train–test mismatch (Page 3). The answer decoding is auto-regressive, where the first answer token is predicted directly from the final latent state zK (Page 3).

Visual-Semantic Dual Supervision

VITAL prevents modality collapse by supervising every latent state zk from two directions: semantic and visual (Page 4). Semantic supervision involves an auxiliary text decoder that reconstructs reasoning chains, while visual supervision involves a visual projector that regresses ROI features from a frozen, independent medical vision encoder (Page 3). The visual projector uses a norm-stabilized residual MLP to map the LLM hidden state to the vision encoder’s feature space, ensuring stable L1 regression (Page 4).

Data and Training

VITAL requires each training sample to be a five-tuple: (image, question, answer, K-step reasoning chain, ROI visual feature) (Page 11). The dataset is constructed from two public medical segmentation datasets, MSD and BiomedParse, spanning 9 imaging modalities (Page 14). Teacher distillation is employed using a state-of-the-art proprietary MLLM as the teacher model to generate high-quality reasoning chains (Page 13).

Performance and Interpretability

Experiments on 7 benchmarks show that VITAL consistently and substantially outperforms the backbone, all latent reasoning baselines, and medical MLLMs trained on far larger data (Page 4). VITAL achieves state-of-the-art results competitive with trillion-parameter proprietary models while exceeding them on visual grounding by a large margin (Page 4). Dual interpretability allows for re-attaching the text decoder and visual projector post-hoc, yielding both a textual window and a visual window into the model’s implicit thought process (Page 5).

Limitations

Limitations include data scale and modality coverage, as the 61K five-tuple dataset spans only 9 imaging modalities (Page 9). The quality ceiling of VITAL’s latent states is bounded by the teacher’s medical reasoning ability due to distillation (Page 9). Furthermore, VITAL's open-ended Token-F1 trails medical MLLMs specifically optimized for verbose freetext generation (Page 7).

Future Work

Promising directions include adaptive reasoning depth to allow the model to allocate fewer steps to simple questions and more to complex clinical reasoning (Page 9). Broader data coverage could be achieved by incorporating radiology reports as an additional supervision source (Page 9). Reinforcement learning, such as outcome-based or process-based RL rewards, could push latent reasoning beyond the teacher distillation ceiling (Page 9).

Ethical Considerations

VITAL is not intended for clinical deployment and requires rigorous clinical validation and regulatory approval (Page 9). Users should not treat the decoded reasoning chains as the sole basis for clinical judgment (Page 9). The interpretability outputs are approximate visualizations of latent states rather than faithful representations of the model’s true decision process (Page 9).

References

Anthropic. 2026. Introducing claude opus 4.6

Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald M Summers and 1 others. 2022. The medical segmentation decathlon

Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge and 1 others. 2025. Qwen3-vl technical report

Huatuogpt-vision

Ankan Deria, Komal Kumar, Adinath Madhavrao Dukre, Eran Segal, Salman Khan, Imran Razzak and 1 others. 2026. Medmo: Grounding and understanding multimodal large language model for medical images

Lin Fan, Yafei Ou, Zhipeng Deng, Pengyu Dai, Hou Chongxian, Jiale Yan, Yaqian Li, Kaiwen Long, Xun Gong, Masayuki Ikebe and 1 others. 2026. Step-cot: Stepwise visual chain-of-thought for medical visual question answering

Xiaotang Gai, Chenyi Zhou, Jiaxiang Liu, Yang Feng, Jian Wu and Zuozhu Liu. 2025. MedThink: A rationale-guided framework for explaining medical visual question answering

Juntao Jiang, Jiangning Zhang, Yali Bi, Jinsheng Bai, Weixuan Liu, Weiwei Jin, Zhucun Xue, Yong Liu, Xiaobin Hu and Shuicheng Yan. 2026a. M3cotbench: Benchmark chain-of-thought of mllms in medical image understanding

Yankai Jiang, Qiaoru Li, Binlu Xu, Haoran Sun, Chao Ding, Junting Dong, Yuxiang Cai, Xuhong Zhang and Jianwei Yin. 2026b. Ibisagent: Reinforcing pixel-level visual reasoning in mllms for universal biomedical object referring and segmentation

Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu and Jiaya Jia. 2024. Lisa: Reasoning segmentation via large language model

Jason J Lau, Soumya Gayen, Asma Ben Abacha and Dina Demner-Fushman. 2018. A dataset of clinically generated visual questions and answers about radiology images

Khai Le-Duc, Duy MH Nguyen, Phuong TH Trinh, Tien-Phat Nguyen, Nghiem T Diep, An Ngo, Tung Vu, Trinh Vuong, Anh-Tien Nguyen, Mau Nguyen and 1 others. 2025. S-chain: Structured visual chain-of-thought for medicine

Bangzheng Li, Ximeng Sun, Jiang Liu, Ze Wang, Jialian Wu, Xiaodong Yu, Hao Chen, Emad Barsoum, Muhao Chen and Zicheng Liu. 2025. Latent visual reasoning

Binxu Li, Tiankai Yan, Yuanting Pan, Jie Luo, Ruiyang Ji, Jiayuan Ding, Zhe Xu, Shilong Liu, Haoyu Dong and 1 others. 2024. Mmedagent: Learning to use medical tools with multi-modal agent

Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann Hoifung Poon and Jianfeng Gao. 2023.

Improvements for AI systems

Continuous latent reasoning for medical VQA without inference overhead: VITAL maintains strict train–inference parity through a single recurrent loop, allowing it to achieve substantially higher accuracy across all benchmarks while being ∼97× faster at inference compared to explicit Chain-of-Thought models.
Enhanced visual grounding via independent supervision: VITAL prevents modality collapse by using visual-semantic dual supervision, where a visual projector regresses ROI features from a frozen, independent medical vision encoder, ensuring latent states are anchored to fine-grained visual evidence.
Dual interpretability for clinical trust: The framework provides both textual and visual explanations through re-attachable modules, yielding a textual window (what the model reasons) and a visual window (where the model looks), which is described as a remarkable level of transparency for latent reasoning.
Robustness against training distribution shift: By utilizing a curriculum learning strategy (Phase 1 (K=1), Phase 2 (K≤2), Phase 3 (all K)), VITAL avoids overfitting to the training data, demonstrating that it consistently improves over the backbone on every single benchmark.
Adaptive reasoning depth: The system can be optimized for question complexity by implementing a learned early-exit mechanism, allowing the model to allocate steps adaptively, moving beyond the fixed maximum of K=4.