RoSeViT: Role-Separated Vision Transformers for ARC Visual Reasoning
cs.AI, cs.CV, cs.LG
Submitted: 2026-01-20
Updated: 2026-10-28
Comments: NeurIPS 2026 poster (camera-ready). Code, checkpoints and predictions: https://github.com/lz7fd/RoSeViT
Code: https://github.com/lz7fd/Reasoning
License: http://creativecommons.org/licenses/by/4.0/
The gist: ARC tasks can be formulated as visual transformation problems, where a model must infer a high-level transformation rule from a small set of demonstrations and apply it consistently across a grid.
Terminology
Abstract
ARC tasks can be formulated as visual transformation problems, where a model must infer a high-level transformation rule from a small set of demonstrations and apply it consistently across a grid. While dense Vision Transformers (ViTs) perform well in this setting, they entangle local visual state and global transformation logic within a shared token representation, which can hinder consistent reasoning. In this paper, we introduce RoSeViT, a role-separated Vision Transformer that explicitly decouples high-level instruction inference from visual workspace computation. RoSeViT uses controller tokens to aggregate transformation-level information and workspace tokens to represent the visual grid, with a structured attention pass in which nonlocal workspace interactions are routed through the controller. This design is motivated by our formal analysis showing that enforcing a shared instruction representation can reduce the effective complexity of visual transformation modeling. RoSeViT outperforms dense ViT baselines under the standard VARC training and evaluation protocol for ARC. In a controlled backbone replacement setting, it improves ARC-1 accuracy from 54.5 to 56.6, and with recurrent refinement reaches 58.8. When integrated into the full system, RoSeViT-Ensemble improves performance from 60.4 to 62.6 on ARC-1. These results demonstrate that explicitly separating instruction and workspace representations leads to more effective visual reasoning on ARC tasks. GitHub repository: https://github.com/lz7fd/RoSeViT.
Sources
- gpt-oss-120b & gpt-oss-20b Model Card
- The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
- On the Measure of Intelligence
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents