Multimodal Thinking with Renderable Programs
cs.CV, cs.CL
Submitted: 2026-09-24
Updated: 2026-09-24
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Current vision-language models (VLMs) excel at visual content understanding and text-based reasoning, yet their structure limits the advancement of incorporating images into the reasoning chain.
Terminology
Abstract
Current vision-language models (VLMs) excel at visual content understanding and text-based reasoning, yet their structure limits the advancement of incorporating images into the reasoning chain. Though Omnimodal models have made efforts in unifying text and image generation, they focus on visual tasks in the open-domain, lacking tractability due to rasterized or latent representations of images. We introduce SVGLM, a framework that uses scalable vector graphics (SVG) primitives to connect text and image in reasoning tasks. We exploit the duality of SVG as both image description and text instructions, yielding a more compact, interpretable solution to equip general VLMs with the capability of generating images within the reasoning process. We provide a large curated dataset of SVG-based image editing dataset, as well as the paradigm to tune open-source VLMs. Experiments on a mathematical reasoning benchmark demonstrate that SVGLM achieves strong SVG generation power as well as think-with-image intelligence. Our results highlight SVG as a suitable medium for building more robust digital domain agents, bridging the gap between text-based thinking and pixel-based images.
Sources
- Exploring Visual Prompts for Adapting Large-Scale Models
- Qwen3-VL Technical Report
- Qwen2.5-VL Technical Report
- Emerging Properties in Unified Multimodal Pretraining
- CodePlot-CoT: Mathematical Visual Reasoning by Thinking with Code-Driven Images
- Visually Prompted Benchmarks Are Surprisingly Fragile
- Inner Monologue: Embodied Reasoning through Planning with Language Models
- Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Models
- Evaluating Object Hallucination in Large Vision-Language Models
- VRPTEST: Evaluating Visual Referring Prompting in Large Multimodal Models
- More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models
- V-Thinker: Interactive Thinking with Images
- RECODE: Reasoning Through Code Generation for Visual Question Answering
- MathCanvas: Intrinsic Visual Chain-of-Thought for Multimodal Mathematical Reasoning
- OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learning
- ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models
- Gemini: A Family of Highly Capable Multimodal Models
- Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- Monet: Reasoning in Latent Visual Space Beyond Images and Language
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models