WM-VLM: Probing Internal World Models for Interleaved Visual-Textual Reasoning
cs.CV
Submitted: 2026-09-28
Updated: 2026-09-28
Comments: 21 pages, 11 figures
License: http://creativecommons.org/licenses/by/4.0/
The gist: Humans often solve spatial problems by mentally simulating visual transformations.
Terminology
Abstract
Humans often solve spatial problems by mentally simulating visual transformations. In contrast, conventional vision-language models (VLMs) reason primarily through language. We investigate whether VLMs can solve spatial problems by reasoning with both text and generated visual states. To this end, we introduce WM-VLM, which equips a pretrained VLM with a lightweight world model branch for generating intermediate visual states. Our two-stage training first teaches the model to generate the next visual state and then to use that state for reasoning. We programmatically construct spatial reasoning tasks with verifiable intermediate visual states. These tasks allow us to evaluate how well the model generates visual states and how much it relies on them to answer the question. On 2D and 3D mental rotation tasks, WM-VLM consistently outperforms the supervised fine-tuned backbone, with gains of up to 39.25 percentage points. Ablations suggest that these gains depend on the generated visual states, as removing or corrupting them sharply reduces performance. Together, these results suggest that internal world models offer a promising path toward VLMs that reason in both language and visual space.
Sources
- Qwen2.5-VL Technical Report
- Chart-R1: Chain-of-Thought Supervision and Reinforcement for Advanced Chart Reasoner
- ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generation
- Thinking with Generated Images
- Emerging Properties in Unified Multimodal Pretraining
- Seeing Through Their Eyes: Evaluating Visual Perspective Taking in Vision Language Models
- Bridging Interleaved Multi-Modal Reasoning as a Unified Decision Process
- DreamPlan: Efficient Reinforcement Fine-Tuning of Vision-Language Planners via Video World Models
- LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Model
- Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models
- BigCharts-R1: Enhanced Chart Reasoning with Visual Reinforcement Finetuning
- Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models
- What's Holding Back Latent Visual Reasoning?
- Visual Generation Unlocks Human-Like Reasoning through Multimodal World Models
- VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs
- When and How Much to Imagine: Adaptive Test-Time Scaling with World Models for Visual Spatial Reasoning
- Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs
- Thinking with Imagination: Agentic Visual Spatial Reasoning with World Simulators
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models