GViT: Representing Images as Gaussians for Visual Recognition
cs.CV, cs.LG
Submitted: 2025-06-30
Updated: 2026-10-27
Comments: - added new explainability experiments - more clarification about our method - more comparisons with other methods
License: http://creativecommons.org/licenses/by/4.0/
The gist: We introduce GViT, a supervised classification framework that uses patch-conditioned 2D Gaussian primitives as input tokens for ViT backbones.
Terminology
Abstract
We introduce GViT, a supervised classification framework that uses patch-conditioned 2D Gaussian primitives as input tokens for ViT backbones. A ViT-S-sized encoder predicts each primitive's position, scale, orientation, color, and opacity in one forward pass. The same parameters are constrained to reconstruct the image through a differentiable renderer, as well as serve as classifier inputs. During training, classifier-gradient guidance shapes primitive geometry and appearance under an image reconstruction constraint. On ImageNet-1K, GViT-B achieves 76.9% top-1 accuracy, compared with 79.8% for ViT-B/16 trained with the same recipe. Gaussian tokens also provide explicit spatial support for explanation maps. Grad-CAM over Gaussians improves on patch-based Grad-CAM in segmentation and token-deletion probes, while other probe comparisons are comparable to a ViT. A covariance-based foreground diagnostic on the PASCAL VOC segmentation task further measures the geometry learned through supervised guidance. These results establish the feasibility of recognition from renderable Gaussian parameters and characterize how task supervision shapes this inspectable representation.
Sources
- BEiT: BERT Pre-Training of Image Transformers
- Class-Discriminative Attention Maps for Vision Transformers
- ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
- Explaining and Harnessing Adversarial Examples
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- PyTorch Distributed: Experiences on Accelerating Data Parallel Training
- Decoupled Weight Decay Regularization
- Enhance the Visual Representation via Discrete Adversarial Training
- Gaussian Masked Autoencoders
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Beyond Language Models: Byte Models are Digital World Simulators
- Vector-quantized Image Modeling with Improved VQGAN
- A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs