Slot2Text: Object-Centric Visual Tokenization for Efficient and Spatially Traceable Surgical MLLMs
cs.CV, cs.CL, cs.LG
Submitted: 2026-08-02
Updated: 2026-09-10
Comments: 17 pages, 8 Figures
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Multimodal large language models (MLLM) for surgical scene understanding typically inject hundreds of dense visual tokens into a language model, leading to costly inference and limited spatial
Terminology
Abstract
Multimodal large language models (MLLM) for surgical scene understanding typically inject hundreds of dense visual tokens into a language model, leading to costly inference and limited spatial traceability for generated answers. We present Slot2Text, a dual-mode surgical MLLM that replaces dense representations of visual input with a compact set of regions encoded as slot latents. Instead of relying on contrastive alignment of the visual encoder with language, Slot2Text groups self-supervised vision features into a few regions--slots that are consumed by the language model as area-labeled visual tokens. Slot2Text-Fast uses the slot prefix to answer surgical questions. Slot2Text-Reason also identifies and locates areas relevant for reasoning, linking language outputs to corresponding slot tokens, masks or regions. Experiments on multiple visual question answering and visual grounding benchmarks show that Slot2Text-Fast is competitive with state-of-the-art baseline at a much lower cost, reducing the average total token consumption by a 91.8% and the visual prefix from 1,295 to 47 tokens (a 96.4% reduction). Slot2Text-Reason trades additional tokens and latency for explicit area identities, locations, and traceable spatial evidence. These results establish compact slot latents as an efficient default visual interface for surgical MLLMs, with grounded reasoning invoked when greater spatial traceability is required.
Sources
- 2017 Robotic Instrument Segmentation Challenge
- Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic
- Slot-MLLM: Object-Centric Visual Tokenization for Multimodal LLM
- Zero-Shot Object-Centric Representation Learning
- LoRA: Low-Rank Adaptation of Large Language Models
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- Slot-BERT: Self-supervised Object Discovery in Surgical Video
- Kosmos-2: Grounding Multimodal Large Language Models to the World
- DINOv3
- Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgery
- DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models
- BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models