VisualSphinx: Large-Scale Synthetic Vision Logic Puzzles for RL
cs.CV, cs.AI, cs.LG
Submitted: 2025-05-29
Updated: 2026-10-30
Comments: DMLR; Project page at https://visualsphinx.github.io/
Project page: https://visualsphinx.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: Vision language models (VLMs) are expected to perform effective multimodal reasoning and make logically coherent decisions…
Terminology
Abstract
Vision language models (VLMs) are expected to perform effective multimodal reasoning and make logically coherent decisions, which is critical to tasks such as diagram understanding and spatial problem solving. However, current VLM reasoning lacks large-scale and well-structured training datasets. To bridge this gap, we propose VisualSphinx, a first-of-its-kind large-scale synthetic visual logical reasoning training dataset. To tackle the challenge of image synthesis with grounded answers, we propose a rule-to-image synthesis pipeline that extracts and expands puzzle rules from seed questions and generates executable rendering programs for puzzle assembly. Using standard GRPO with binary correctness rewards as a validation recipe, experiments show that VLMs trained on VisualSphinx benefit from the logical coherence and readability of our dataset and exhibit improved performance on logical reasoning tasks. The enhanced reasoning capabilities developed from VisualSphinx also benefit other reasoning tasks such as algebraic reasoning, arithmetic reasoning and geometry reasoning.
Sources
- ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models
- Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyond
- Evaluating Large Language Models Trained on Code
- Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models
- Training Verifiers to Solve Math Word Problems
- VILA$^2$: VILA Augmented VILA
- OpenAI o1 System Card
- DeepSeek-V3 Technical Report
- LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
- MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct
- MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learning
- VGRP-Bench: Visual Grid Reasoning Puzzle Benchmark for Large Vision-Language Models
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledge
- WizardLM: Empowering large pre-trained language models to follow complex instructions
- VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models
- Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing
- R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization
- Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Model
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs