Geometric Similarity in VLM Low-Level Vision Representations
cs.CV, cs.AI
Submitted: 2026-10-01
Updated: 2026-10-01
Comments: First version: 10 pages
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Vision-language models (VLMs) have emerged as powerful candidates for universal vision backbones, with representative architectures including autoregressive (AR) models and diffusion transformers
Terminology
Abstract
Vision-language models (VLMs) have emerged as powerful candidates for universal vision backbones, with representative architectures including autoregressive (AR) models and diffusion transformers (DiTs). Yet, adapting them efficiently for all-in-one low-level image restoration remains a challenge. Crucially, the field lacks an understanding of how VLMs organize hidden-layer representations and whether these structurally distinct paradigms share a common geometric organization for pixel-level perception. Such shared organization is a prerequisite for building highly transferable, unified restoration VLMs and adapters. In this paper, we systematically investigate representational similarity across 24 low-level tasks spanning 5 categories. We propose GeoSim, a unified four-level framework that analyzes task-conditioned representations from global similarity, local geometry, sparse feature decomposition, and topological verification perspectives. Our formulation applies to the analysis of hidden states in AR models and feature maps in DiTs across same- and cross-task/model settings. Our results reveal the organizing principles of low-level visual representations while exposing their limits in cross-task and cross-model agreement. Ultimately, GeoSim provides an interpretability lens for probing latent transferability in low-level vision and diagnosing model limitations in task- or model-specific scenarios.
Sources
- Qwen3-VL Technical Report
- Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
- ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generation
- Emu3.5: Native Multimodal Models are World Learners
- VIDIT: Virtual Image Dataset for Illumination Transfer
- When Models Manipulate Manifolds: The Geometry of a Counting Task
- Investigate the Low-level Visual Perception in Vision-Language based Image Quality Assessment
- Reasoning emerges from constrained inference manifolds in large language models
- UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
- Effects of Degradations on Deep Neural Network Architectures
- Steering Vision-Language Models with Joint Sparse Autoencoders
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- Emu3: Next-Token Prediction is All You Need
- Qwen-Image Technical Report
- T2T-VICL: Cross-Task Visual In-Context Learning via Implicit Text-Driven VLMs
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models