Beyond Static Visual Tokens: Structured Sequential Visual Chain-of-Thought Reasoning
cs.CV, cs.AI
Submitted: 2026-03-21
Updated: 2026-08-29
Comments: Accepted to Findings of EMNLP 2026
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Current multimodal LLMs encode images as static visual prefixes and rely on text-based reasoning, lacking goal-driven and adaptive visual access.
Terminology
Abstract
Current multimodal LLMs encode images as static visual prefixes and rely on text-based reasoning, lacking goal-driven and adaptive visual access. Inspired by human visual perception-where attention is selectively and sequentially shifted from the most informative regions to secondary cues-we propose Structural Sequential Visual CoT SSV-CoT. First, a question-relevant saliency map identifies and organizes key visual regions, explicitly modeling the spatial distribution of visual importance. Second, reasoning is performed following this discriminative order, inducing a curriculum-like semantic progression from primary to secondary cues. This method is trained end-to-end, using text cot and answer supervision, without relying on region-level annotations or specialized external tools. Experiments on diverse visual reasoning benchmarks show gains, validating structured and sequential visual cognition.
Sources
- VIPER: Visual Perception and Explainable Reasoning for Sequential Decision-Making
- Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic
- MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoning
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
- CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers
- Taming Transformers for High-Resolution Image Synthesis
- VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction
- Interleaved-Modal Chain-of-Thought
- MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding
- Weakly Supervised Grounding for VQA in Vision-Language Transformers
- Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Models
- AIM-CoT: Active Information-driven Multimodal Chain-of-Thought for Vision-Language Reasoning
- REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering
- Visual Instruction Tuning
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- POINTS1.5: Building a Vision-Language Model towards Real World Applications
- Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
- Ovis: Structural Embedding Alignment for Multimodal Large Language Model
- Compositional Chain-of-Thought Prompting for Large Multimodal Models
- Learning Transferable Visual Models From Natural Language Supervision
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models