What DINO saw: ALiBi positional encoding reduces positional bias in Vision Transformers

summary

The gist

Vision transformers (ViTs) often display positional biases and artifacts independent of semantic content…

In short

Vision Transformers often have positional biases that hinder zero-shot adaptation, especially in material science imaging. This work found these biases across various models and reduced them by finetuning with 2D ALiBi positional encoding. The resulting unbiased features retain good semantics and improve segmentation accuracy, even in low-label regimes.

Key concepts

Positional Bias
This is an artifact in Vision Transformers where the model's output features are strongly influenced by the position of pixels in the input image rather than just what those pixels represent. It causes features to look like positional ramps regardless of the actual content, making it hard for models to generalize.
Linear Probing
This is a technique used by researchers to analyze how different parts of a pre-trained model's internal features respond to specific tasks. The authors used linear probing across various objectives and encodings to systematically identify which feature channels were purely positional ramps.
ALiBi Positional Encoding
ALiBi is a type of relative positional encoding that calculates the distance between every pair of tokens in an image. It is used here to replace learned positional encodings, aiming to remove inherent biases by making the model's understanding position-aware but content-agnostic.

Terminology used across episodes

This episode discusses

The paper

What DINO saw: ALiBi positional encoding reduces positional bias in Vision Transformers · Read on arXiv

Center for Materials Research, Justus Liebig University of Giessen · Dyson School of Design Engineering, Imperial College London · Finden ltd, Rutherford Appleton Laboratory

Vision transformers (ViTs) - especially feature foundation models like DINOv2 - learn rich representations useful for many downstream tasks. However, architectural choices (such as positional encoding) can lead to these models displaying positional biases and artefacts independent of semantic content. This makes zero-shot adaption difficult in fields like material science, where images are often cross-sections of homogeneous microstructure (i.e. having no preferred direction). In this work, we investigate the positional bias in ViTs via linear probing, finding it present across a range of objectives and positional encodings, and subsequently reduce it by finetuning models to use ALiBi relative positional encoding. We demonstrate that these models retain desirable general semantics and their unbiased features can be used successfully in trainable segmentation of complex microscopy images.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "What DINO saw".

Jane: Here's the gist of the paper. Vision transformers (ViTs) often display positional biases and artifacts independent of semantic content, which makes zero-shot adaptation difficult in fields like material science where images are often homogeneous cross-sections <ref:2603.16840#pg3>.

Tom: This work investigates the positional bias in ViTs via linear probing, finding it present across various objectives and encodings, and subsequently reduces it by finetuning models to use ALiBi relative positional encoding <ref:2603.16840#pg2>.

Jane: The authors demonstrate that these models retain desirable general semantics and their unbiased features can be used successfully in trainable segmentation of complex microscopy images <ref:2603.16840#pg2>.

Tom: Characterization of Positional Bias The authors first characterize the positional bias across various literature ViTs using linear probing, finding some channels of the output features which are almost purely positional ramp functions regardless of input image <ref:2603.16840#pg3>.

Title and authors: Tom: Let’s look at the title and who wrote this. It’s titled "What DINO saw: ALiBi positional encoding reduces positional bias in Vision Transformers," and it was written by Pawlowsky, Vamvakeros, Weiss, Bielefeld, Cooper, and Docherty.

Jane: That title is pretty straightforward. It tells you exactly what they investigated: that using a specific type of positional encoding called ALiBi can fix the positional bias that affects Vision Transformers.

Lu: The core idea they are pushing is that these feature foundation models, which are trained on massive datasets, often have these inherent biases from the positional encodings they use, and this paper tests whether changing that encoding helps when you try to adapt them to new tasks.

Meng: So the goal isn't just to fix one model; it’s about understanding why these features behave this way across the board so we can build more reliable systems for things like material science analysis.

Lalam: It points out that Vision transformers are everywhere now, with great performance and scalability, but the success relies on frozen features that can be used in pipelines without further training >

The paper's summary: Tom: The summary of this paper explains that they investigated positional bias by using linear probing across different literature vision transformers and found specific channels whose output features were almost purely positional ramp functions regardless of the input image.

Jane: That means when you probe these models, some parts of the feature space are just showing position information, not actual picture content >

Lu: They found that these biases exist in both MAE and DINO models, across different learned or RoPE positional encodings, but they were absent in supervised vision transformers. This suggests the issue is inherent to the self-supervised training process itself rather than just the final objective function.

Meng: That’s a big deal if it means we can’t trust any frozen feature backbone without knowing where those biased channels are, because I have to worry about what breaks when we try to deploy that model.

Lalam: They also found distinct peaks in the per-channel R squared scores, which indicated that some specific channels were highly correlated with those different one dimensional ramps >

The paper's improvements: Tom: Now for the fix they propose. They reduce this positional bias by finetuning the models to use 2D aware ALiBi positional encodings, complete with cylindrical boundary conditions and normalization to help interpolate between positions <ref:2603.16840#pg1>.

Jane: So they aren’t just throwing out the old encoding; they are training a new one that handles the spatial structure of an image better, which is a big step because it’s not just about replacing text in a formula.

Lu: Surprisingly, once they swap out the learned positional encoding for this unbiased ALiBi positional encoding, it turns out you only need to use those original biased embeddings from models like DINOv2 or DINOv3 as your training target to recover the semantics >

Meng: That’s a clever way to save computation. So instead of retraining a whole massive model, you can just fine-tune the new encoding on top of what’s already learned, which makes sense for deployment cost and time.

Lalam: They also zeroed out those four most positional channels they identified earlier, which reduced the reappearance of weak long range biases in those features >

Conclusion: Tom: To wrap this up, the main point is that biased vision transformer embeddings can actually be used as a training target to produce a feature model that ends up being homogeneous and without positional biases.

Jane: So these models retain the general semantics of the original DINOv2 or DINOv3 features, but they get rid of those problematic positional artifacts, which is key for applying them successfully in trainable segmentation of complex microscopy images.

Lu: The authors found that this replacement doesn't damage semantics; it retains feature similarities across changes in style and pose, and the ALiBi-Dv2 model is generally more robust to transformations than DINOv2 or NoPE >

Meng: If you can get better segmentations with less label data because the features are more consistent, that’s good for production. The paper shows ALiBi-Dv2 produces features of complicated microstructural concepts like particle size and gaps, which is what we need for materials analysis >

Lalam: They also showed this positional bias is a more general property of self supervision rather than something isolated to the DINO objective or the learned positional encodings, meaning altering the encoding seems to damage things enough that they can’t re-learn those biases during finetuning >

More episodes

← Home