A squared: Smaller Self-Supervised ViTs Localize Better than Larger Ones
summary
The gist
Robust visual classification often depends on localizing the main foreground objects in an image while ignoring contextual distractors Smaller self-supervised ViTs localize foreground objects better…
In short
A2 is a simple method that improves visual classification by decoupling where to look from what to extract features. It leverages the finding that smaller self-supervised Vision Transformers (ViTs) localize foreground objects better than larger ones. A2 achieves this by cropping around attention peaks of a small model and embedding those crops with a large model, creating robust representations.
Key concepts
- Inverse Scaling Finding
- This is the observation that smaller self-supervised ViTs are better at localizing main objects than larger ones. This trend runs contrary to the usual belief that bigger models are always superior. This finding is central to A2's design, suggesting a specific scaling relationship between model size and localization quality.
- A2 Method
- A2 is a simple technique that separates the attention mechanism from the feature extraction. It works by using a small attention model to identify relevant image regions (crops) and then feeding those specific crops into a larger embedding model for final classification. This process helps filter out distracting background information.
- Decoupling Attention and Embedding
- This is the core mechanism of A2, where the two main components are separated. The small attention model determines 'where to look' by highlighting important areas in an image using its attention map. The large embedding model then focuses only on extracting features from those selected, relevant regions, effectively ignoring irrelevant context.
Terminology used across episodes
This episode discusses
- A squared: Smaller Self-Supervised ViTs Localize Better than Larger Ones · Paper Radio
- Zero-Shot Robustification of Zero-Shot Models
- SAM 3: Segment Anything with Concepts
- Emerging Properties in Self-Supervised Vision Transformers
- Reproducible scaling laws for contrastive language-image learning
- Invariant Causal Mechanisms through Distribution Matching
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
- Shortcut Learning in Deep Neural Networks
- Supervised Contrastive Learning
- Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations
- MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflicts
- DINOv2: Learning Robust Visual Features without Supervision
- Learning Transferable Visual Models From Natural Language Supervision
- SAM 2: Segment Anything in Images and Videos
- Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
- When Do We Not Need Larger Vision Models?
- DINOv3
- Deep CORAL: Correlation Alignment for Deep Domain Adaptation
- Attention Is All You Need
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
The paper
A squared: Smaller Self-Supervised ViTs Localize Better than Larger Ones · Read on arXiv
Columbia University · Stanford University
Robust visual classification often depends on localizing the main foreground objects in an image while ignoring contextual distractors. Surprisingly, we find that the last layer attention maps of smaller self-supervised ViTs localize foreground objects better than those of larger ViTs. However, we still need large ViTs, because they extract richer representations from each patch. To get the best of both worlds, good localization and rich representations, we propose A squared, a simple method that leverages this inverse scaling finding by decoupling where to look (a small attention model) from what to extract (a large embedding model): we crop around the attention peaks of a small model and embed the crops with a larger model. A squared uses entirely pretrained features, requires no group labels, and does not require per-dataset attention or backbone training. Across 5 benchmarks, A squared is competitive with backbone-matched loss-level methods like DFR, and outperforms end-to-end attention training under stronger distribution shifts.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "A squared: Smaller Self-Supervised ViTs Localize Better than Larger Ones".
Jane: Here's the gist of the paper. Robust visual classification often depends on localizing the main foreground objects in an image while ignoring contextual distractors Smaller self-supervised ViTs localize foreground objects better than those of larger ViTs To get the best of both worlds, we propose A2, a simple method that leverages this inverse scaling finding by decoupling where to look (a small attention model) from what to extract (a large embedding model Pruning redundant reasoning tokens The paper introduces A2, a simple method that decouples where to look (the attention model) from what to extract (the embedding model by cropping around the attention peaks of a small model and embedding the crops with a larger model This approach uses entirely pretrained features, requires no group labels, and does not require per-dataset attention or backbone training A2 is designed to remove spatially spurious correlations by automatically selecting crops based on a pre-trained model’s attention map How the training signal is built A2 operates in two stages: 1) selection of crops and 2) classification based on crops The main idea of the approach is that the crops are computed using the attention maps from a second (possibly different) transformer Specifically, we crop the image around the highest-attention regions in an attention map This converts the transformer’s soft attention to hard attention, allowing only the most attended-to and relevant regions to enter the downstream representation Where the gains come from The performance of A2 is tied to localization quality, as we observe that downstream A2 performance mimics the trends seen in Section 2 We find that localization ability of the underlying self-supervised model is indicative of downstream task performance The best configuration pairs a small attention model with a large embedder (ViT-S × ViT-G) outperforms using the large model for both (ViT-G × ViT-G) on every dataset Pruning redundant reasoning tokens The paper finds that smaller pretrained self-supervised ViTs localize foreground objects substantially better than larger ones This inverse scaling trend runs contrary to the common belief that larger parameter-count models are always better The trend holds across five self-supervised families, with ViT-S/16 putting 74.6% of its mass inside the ground truth object box on ImageNet Val versus 67.6% for ViT-G Where the gains come from A2 improves robustness and complements loss-level methods, showing that A2 + DFR adds another +13.9 points WGA over DFR alone on Spawrious O2O Hard Furthermore, A2 outperforms end-to-end attention training (iFAM) under stronger distribution shifts on MetaShift Animals Pruning redundant reasoning tokens The study demonstrates that for OpenCLIP, the trend does not hold, with ViT-B/16 being both the smallest and the worst at localization with a particularly low hit rate of 0.166 compared to ViT-L/14 (0.815) The paper also shows that A2 can be used with existing methods for spurious correlations that operate at the loss level Where the gains come from The MLP adapter network is introduced to address limitations where pretrained attention misaligns, improving worst-group accuracy on CelebA by 35.01% compared to a baseline logistic regression fit on ViT-S embeddings The final results show that A2 outperforms iFAM under stronger distribution shifts on MetaShift Animals, where train and test contexts are disjoint The paper concludes that pretrained attention generalizes better than fine-tuned attention, which tends to overfit train-task distribution biases Pruning redundant reasoning tokens The final conclusion is that Self-supervised ViTs show inverse scaling of localization, and this trend directly translates into A2, with performance decreasing as the size of the model used to select crops increases By preventing distractors from entering the model through hard crops, A2 outperforms loss-level baselines and beats end-to-end attention training (iFAM) under stronger distribution shifts while being simple and interpretable Where the gains come from A2's cross-model performance shows that ViT-S attention is best for 3/4 embedding columns on MetaShift Cat vs.
Tom: Dog, and ViT-S attention is best for 3/4 embedding columns on MetaShift Animals The paper also shows that A2's performance on Waterbirds improves over standard CLIP classification by 5.8 points in accuracy and 3.5 points in WGA The study suggests that increased model capacity is not enough on its own to prevent spurious correlations from entering the representations The authors conclude that A2 allows for improving the robustness of existing models to spurious correlations, without either, which can be a helpful bridge while waiting to collect new data or update the model Pruning redundant reasoning tokens The paper details the implementation of TTR and A2 using the OpenCLIP ViT-L-14 backbone for fair comparison across different tasks The results show that DINOv2 attention scaling shows a mild downward WGA trend with size, while OpenCLIP attention shows large positive offsets at ViT-L/14 and larger A2's zero-shot variant outperforms TTR on 4/5 datasets The paper also notes that the center crop performs well for Waterbirds because the dataset is heavily biased toward placing the main subject at the center of the image Where the gains come from The analysis of per-layer attention mass reveals a premature peak in localization ability in larger ViTs, coinciding with their shift to register tokens A2 cropping of spurious spatial correlations and the shape bias of ViTs are complementary, preventing the spurious feature from entering, and then the backbone’s shape bias helps make the representation robust to color or texture level shifts The authors conclude that A2 is simple and interpretable Pruning redundant reasoning tokens The paper shows that A2's performance on MetaShift Animals, where train and test contexts are disjoint, beats iFAM by up to 2.3 points in accuracy and 2 points in worst-class accuracy The authors also show that A2’s drop is roughly half as steep compared to iFAM when the test distribution strays further from train This suggests that frozen pretrained DINOv2 attention generalizes across context shifts in a way that iFAM’s per-dataset training does not The paper also notes that A2's performance on MetaShift Animals, where train and test contexts are disjoint, beats iFAM by up to 2.
Paper summary: Tom: So, we're talking about this paper called "A squared: Smaller Self-Supervised ViTs Localize Better than Larger Ones." It looks like the main idea is that usually, for good visual classification, you need to find the main foreground objects in an image while ignoring all the distracting background stuff.
Jane: Right, and this paper claims that when you look at how these models localize those objects, smaller self-supervised Vision Transformers actually do a better job than larger ones do.
Lu: What's interesting here is that this isn't just about size; it points to an inverse scaling trend where smaller ViTs localize foreground objects better than larger ones, which goes against the common idea that bigger models are always superior <ref:2606.03148#pg1>.
Tom: So the core proposal they have is something called A squared, which is a simple method designed to get the best of both worlds: good localization and rich representations. How does this method actually work in simple terms?
Lu: A2 decouples where you look from what you extract. Instead of using just one model, they use a small attention model to decide where to look, then they crop those areas and feed them into a much larger embedding model for the actual classification <ref:2606.03148#pg1>.
Jane: So if I'm trying to understand this for someone who doesn't read papers, it sounds like they are using a tiny scout to pick interesting spots in a picture, and then they use a big brain to deeply analyze those chosen spots. Does that make sense?
Tom: Exactly. The paper explains that A2 uses entirely pre-trained features so you don't need any group labels or training specific attention maps for every dataset <ref:2606.03148#pg1>. That’s a big deal for deployment because it’s pretty plug-and-play, right?
Meng: From an engineering standpoint, that sounds promising if it means less training overhead. But what about the trade off? If you use a small model to select crops, aren't you losing some of the fine detail that a larger model might capture on its own <ref:2606.03148#pg2>?
Lu: That’s exactly the tension they address. They show that for ImageNet Val, the ViT-S model puts seventy-four point six percent of its attention mass inside the ground truth object box, compared to sixty-seven point six percent for the ViT-G <ref:2606.03148#pg3>.
Paper summary: Jane: Seventy-four point six percent is a solid number. So, what does this actually mean for someone just using an AI model for visual tasks? It means the way the model focuses its attention is more accurate when it's smaller, provided you pair it correctly with a larger embedding model <ref:2606.03148#pg1>.
Tom: And they show this pattern holds across five different self-supervised families, like DINOv1, DINOv2, DINOv3, MAE and iBOT <ref:2606.03148#pg2>. That's pretty strong evidence that this isn't just a fluke on one model type.
Meng: But what about the real world deployment? If we use ViT-S for localization and ViT-G for embedding, does that mean we can keep the computational load low while maintaining accuracy? Or is the process of cropping and re-embedding adding significant latency that we need to worry about when shipping this to production <ref:2606.03148#pg1>?
Lu: They are testing this on different benchmarks like Waterbirds and ImageNet Val, and they find that A2 is competitive with other loss level methods, like DFR <ref:2606.03148#pg1>. It also shows that A2 outperforms end-to-end attention training under stronger distribution shifts <ref:2606.03148#pg1>.
Jane: So, if a user is working on a system where the data changes or drifts over time, A2 seems to be more robust than methods that just train the attention mechanism from scratch for every new situation <ref:2606.03148#pg3>. That addresses one of those big headaches in real-world AI.
Tom: And they actually show that A2 plus another technique called DFR adds an extra thirteen point nine WGA over DFR alone on the Spawrious O2O Hard benchmark <ref:2606.03148#pg1>. That suggests A2 isn't just a minor tweak; it brings tangible performance gains when you combine it with other ideas.
Meng: I’m curious about the limitations they mention. They did test this on OpenCLIP, and the trend doesn't hold there at all, showing that ViT-B/sixteen can actually be the worst at localization with a very low hit rate of zero point one six six compared to ViT-L/fourteen <ref:2606.03148#pg2>. That variability makes deployment tricky.
Lu: Yes, and they also show that for OpenCLIP, the trend isn't monotonic across all models <ref:2606.03148#pg2>. But the authors introduce a small MLP adapter network to fix issues where the pre-trained attention doesn't align well, improving worst group accuracy on CelebA by thirty-five point zero one percent compared to a baseline logistic regression fit on ViT-S embeddings <ref:2606.03148#pg3>.
Paper summary: Jane: That sounds like they are acknowledging that the pre-trained attention isn't perfect, so they added a little layer to smooth things out for models like ViT-S when things get really hard <ref:2606.03148#pg3>. So, what’s the big picture here regarding how we build these visual systems going forward?
Tom: The main conclusion is that the inverse scaling of localization in self-supervised ViTs directly translates into A squared's performance, meaning performance drops as you use a bigger model to select the crops <ref:2606.03148#pg1>. By cropping out those distractors with hard crops, A2 beats loss level baselines and end-to-end attention training under tougher distribution shifts <ref:2606.03148#pg1>.
Lu: They also found that A2's cross model performance shows that the small ViT attention is best for three out of four embedding columns on MetaShift Cat versus Dog, and same for Animals <ref:2606.03148#pg3>. That suggests a very specific pairing strategy can give you better results than just using the biggest model available.
Jane: So, to wrap up, this paper shows that we don't always need the biggest model to get the best localization; sometimes a smaller one is better when you use it smartly with a bigger one for the final classification step <ref:2606.03148#pg1>. What does this mean for people who are building these systems?
Tom: It means they can build more robust visual systems without needing to constantly fine-tune the attention mechanism from scratch on every single new task <ref:2606.03148#pg3>. This is a simple and interpretable approach, which is what we want when we're shipping models to users.
Meng: I guess for me, it means we can deploy models that are simpler in structure but still get better performance on challenging data shifts than the end-to-end attention methods <ref:2606.03148#pg1>. We need to see how much of that complexity we can cut down while keeping this level of accuracy.
Lu: The authors conclude that pretrained attention generalizes better than fine-tuned attention, which tends to overfit the training task distribution biases <ref:2606.03148#pg3>. This supports the idea that leveraging existing pre-trained knowledge is a really solid path forward for this kind of problem.
Jane: So, in short, A squared gives us a simple way to improve robustness by leveraging how self-supervised ViTs naturally focus their attention during pretraining <ref:2606.03148#pg1>. That’s the main thing we’ve heard about this work today.
Conclusion: Tom: So we're wrapping up on A squared, which is about how smaller self-supervised Vision Transformers actually locate foreground objects better than their bigger counterparts.
Jane: It’s interesting that they found this inverse scaling trend holds across different types of models, not just one specific architecture.
Lu: Yeah, it suggests the way these models focus their attention during pretraining is fundamentally tied to their size in a way we didn't expect.
Meng: From what I see, the big takeaway for me is that this method lets us build something robust without needing massive amounts of training data just to fix those localization errors.
Tom: It sounds like they’re showing that you don't always need the biggest model available to get the best results in visual classification.
Jane: Exactly, and A squared gives us a simple way to improve robustness by leveraging how self-supervised Vision Transformers naturally focus their attention during pretraining.
Lu: The authors point out that this direct relationship between model size and localization quality is what makes A squared so powerful for our applications.
Meng: It’s about being able to deploy simpler structures that still perform better on challenging data shifts without the massive overhead of end-to-end attention training.
Tom: So, in simple terms, the authors are saying that tuning your model size for localization is actually counterintuitive; smaller models with the right setup win here.
Jane: And what this means for us is that we might find more efficient and reliable ways to build visual systems that can handle messy real-world data better.
Lu: We've seen some limitations, like it doesn't work perfectly on every single model, but the core mechanism of separating where to look from what to extract seems solid.
Meng: It’s still a bit of an experiment for me, though seeing this trend hold across different self-supervised families is a strong signal that this idea has real potential.
Tom: Before we move on, I want to talk about how they specifically paired the small attention model with the large embedding model to get those gains.
More episodes
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI