Shape-based Figure-Ground Organization in Vision Transformers
summary
The gist
Shape-based figure-ground organization in Vision Transformers is studied to determine if these models encode human-like shape cues from natural scene statistics The results show that ViTs robustly…
In short
This study investigates if Vision Transformers (ViTs) learn human-like shape cues like surroundedness and convexity from natural images. Researchers used linear probes trained on frozen ViT features to see if these intermediate layers can classify shapes. Results show ViTs robustly encode these cues, and probes trained on natural scenes generalize effectively to unseen artificial shapes across different models.
Key concepts
- Surroundedness
- This refers to a shape being completely enclosed by the background, meaning there is no part of the foreground touching the image edges. The study tests if ViT features capture this fundamental spatial relationship as a generic shape cue.
- Convexity
- Convexity describes whether a shape's boundary curves inward toward its center. In this research, it was tested by separating foreground and background using a random parabolic arc to see if the model recognizes this curvature as a distinct feature.
- Linear Probes
- These are simple classifiers trained on top of frozen ViT features. They act as tools to test what information the intermediate layers have learned. By training these probes, researchers can measure how well specific shape cues like surroundedness or convexity are represented in the network's internal data.
- Zero-Shot Generalization
- This means testing if a model trained on one type of natural image can correctly identify shapes it has never seen before. The study found that probes trained on natural images successfully recognized artificial stimuli, demonstrating the models learned general shape principles.
Terminology used across episodes
This episode discusses
- Shape-based Figure-Ground Organization in Vision Transformers · Paper Radio
- Adam: A Method for Stochastic Optimization
The paper
Shape-based Figure-Ground Organization in Vision Transformers · Read on arXiv
University of Toronto · Vector Institute · McGill University · MILA Institute of Artificial Intelligence Research at the University of California, Irvine
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Shape-based Figure-Ground Organization in Vision Transformers".
Jane: Here's the gist of the paper. Shape-based figure-ground organization in Vision Transformers is studied to determine if these models encode human-like shape cues from natural scene statistics The results show that ViTs robustly encode surroundedness and convexity, and that probes trained on natural images generalize zero-shot to artificial stimuli across several models Evaluation Methodology The study evaluates whether the intermediate features of pretrained ViTs encode surroundedness, convexity and symmetry as generic shape cues for figure-ground assignment To this end, linear probes are trained to classify each patch as figure or ground based on frozen features from pre-trained ViTs Patches that overlap both foreground and background in the original mask receive ambiguous labels and are excluded from training and evaluation The probe definition involves fitting a logistic regression model that predicts the probability of a patch belonging to the foreground Evaluation is performed using Information Gain Explained (IGE), which measures how much the probe improves over the spatial prior baseline Stimuli Construction The synthetic stimuli are constructed so that semantic content, texture, and region size are uninformative, ensuring that probe performance reflects genuine sensitivity to shape The three conditions tested are: 1.
Tom: Surroundedness: The foreground is a random shape from the Infinite DSprites dataset positioned with at least a one-pixel margin to all image boundaries so that it is fully surrounded by the background 2.
Jane: Convexity: Foreground and background are separated by a random parabolic arc, v = ku2+c, in a randomly oriented coordinate frame, with k ∼ Uniform(1.0, 3.0) and c chosen to ensure equal region size 3.
Title and authors: Tom: So, looking at the title, "Shape-based Figure-Ground Organization in Vision Transformers," it sounds very specific to what they are trying to measure Jane It’s taking something that humans do intuitively—figuring out what's a shape against its background—and seeing if the AI has mapped those specific cues onto its internal representations Lu The authors are Matthias Tangemann, Benjamin Lo, Zygmunt Pizlo, Kaleem Siddiqi, and Dirk B. Walther Tom They’re bringing together people from different places, which often means a broader look at the problem. Jane What this title really boils down to is they want to see if these deep neural networks are just grabbing textures or semantic things, or if they are actually learning the same fundamental geometric rules we use for shape recognition Lu It’s about moving past just saying "this model is good" and figuring out *why* it's good at separating things in a human-like way Tom
The paper's summary: Jane: So, the core finding they present is that these ViTs are robust enough to encode surroundedness and convexity very well, even when they are trained on natural scenes Tom They found that if you train linear probes on features from these models, those probes can successfully classify patches as figure or ground based on those shape cues Lu What’s interesting here is that this works across several different ViTs, including both supervised and self-supervised training objectives Tom This suggests the ability to find these cues isn't tied to just one specific way a model is trained. Jane And they also showed that probes trained on natural images can generalize zero-shot to artificial stimuli, which means if you train it on real photos, it can figure out shape cues in completely synthetic pictures it hasn't seen before Lu That generalization capability is pretty significant for how we think about these models.
The paper's improvements: Tom: Now they suggest a few ways to make this better. They focus on using linear probes with position-dependent biases initialized with the spatial prior, which means you start the probe knowing where things *should* be based on basic spatial layout Lu This way, any improvement the probe shows over that starting point is clearly due to learned shape information rather than just guessing based on location Jane That’s a clean way to attribute success to the learned representations. Tom And they have a few other points. They argue that you should look at the intermediate layers of these ViTs for finding these figure-ground cues because the strongest representations seem to emerge there, not in the final output layer Lu This helps narrow down where we should even be looking inside the network when we try to find these shape signals Jane
Conclusion: Tom: So, wrapping this up, the main implication is that ViTs are learning shape cues like surroundedness and convexity from natural data, which gives us a way to test if they are mimicking human visual processing Lu It shows they can do this across different training setups and even generalize to new shapes they haven't seen before Jane This gives us a better idea of what kind of visual knowledge these models actually build up during training. Tom Lu, from your side, does this mean we should start designing our next layers specifically to look for these geometric properties instead of just relying on the final prediction? Lu I think it means we have a better target. If we know exactly what's being learned—surroundedness and convexity—we can design training that pushes the model to explicitly learn those things, not just whatever correlations happen in the raw data Jane Meng, from a production standpoint, does this mean we can trust these models more when they are deployed on novel scenes? Meng It means we have a better diagnostic tool. If we know *where* in the network it’s learning these fundamental cues, it helps us debug why a model might fail on weird images at scale Tom Lalam, what's your take on this finding regarding the "Shape-based Figure-Ground Organization in Vision Transformers"? Lalam I’m quite sure this finding is solid because the mechanism being tested—surroundedness and convexity—is fundamental to how we structure perception, and seeing it emerge robustly in a ViT architecture is a pretty strong indicator of deep structural understanding Lu
More episodes
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI