What DINO saw: ALiBi positional encoding reduces positional bias in Vision Transformers
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "What DINO saw".
Jane: Here's the gist of the paper. Vision transformers (ViTs) often display positional biases and artifacts independent of semantic content, which makes zero-shot adaptation difficult in fields like material science where images are often homogeneous cross-sections <ref:2603.16840#pg3>.
Tom: This work investigates the positional bias in ViTs via linear probing, finding it present across various objectives and encodings, and subsequently reduces it by finetuning models to use ALiBi relative positional encoding <ref:2603.16840#pg2>.
Jane: The authors demonstrate that these models retain desirable general semantics and their unbiased features can be used successfully in trainable segmentation of complex microscopy images <ref:2603.16840#pg2>.
Tom: Characterization of Positional Bias The authors first characterize the positional bias across various literature ViTs using linear probing, finding some channels of the output features which are almost purely positional ramp functions regardless of input image <ref:2603.16840#pg3>.
Title and authors: Tom: Let’s look at the title and who wrote this. It’s titled "What DINO saw: ALiBi positional encoding reduces positional bias in Vision Transformers," and it was written by Pawlowsky, Vamvakeros, Weiss, Bielefeld, Cooper, and Docherty.
Jane: That title is pretty straightforward. It tells you exactly what they investigated: that using a specific type of positional encoding called ALiBi can fix the positional bias that affects Vision Transformers.
Lu: The core idea they are pushing is that these feature foundation models, which are trained on massive datasets, often have these inherent biases from the positional encodings they use, and this paper tests whether changing that encoding helps when you try to adapt them to new tasks.
Meng: So the goal isn't just to fix one model; it’s about understanding why these features behave this way across the board so we can build more reliable systems for things like material science analysis.
Lalam: It points out that Vision transformers are everywhere now, with great performance and scalability, but the success relies on frozen features that can be used in pipelines without further training >
The paper's summary: Tom: The summary of this paper explains that they investigated positional bias by using linear probing across different literature vision transformers and found specific channels whose output features were almost purely positional ramp functions regardless of the input image.
Jane: That means when you probe these models, some parts of the feature space are just showing position information, not actual picture content >
Lu: They found that these biases exist in both MAE and DINO models, across different learned or RoPE positional encodings, but they were absent in supervised vision transformers. This suggests the issue is inherent to the self-supervised training process itself rather than just the final objective function.
Meng: That’s a big deal if it means we can’t trust any frozen feature backbone without knowing where those biased channels are, because I have to worry about what breaks when we try to deploy that model.
Lalam: They also found distinct peaks in the per-channel R squared scores, which indicated that some specific channels were highly correlated with those different one dimensional ramps >
The paper's improvements: Tom: Now for the fix they propose. They reduce this positional bias by finetuning the models to use 2D aware ALiBi positional encodings, complete with cylindrical boundary conditions and normalization to help interpolate between positions <ref:2603.16840#pg1>.
Jane: So they aren’t just throwing out the old encoding; they are training a new one that handles the spatial structure of an image better, which is a big step because it’s not just about replacing text in a formula.
Lu: Surprisingly, once they swap out the learned positional encoding for this unbiased ALiBi positional encoding, it turns out you only need to use those original biased embeddings from models like DINOv2 or DINOv3 as your training target to recover the semantics >
Meng: That’s a clever way to save computation. So instead of retraining a whole massive model, you can just fine-tune the new encoding on top of what’s already learned, which makes sense for deployment cost and time.
Lalam: They also zeroed out those four most positional channels they identified earlier, which reduced the reappearance of weak long range biases in those features >
Conclusion: Tom: To wrap this up, the main point is that biased vision transformer embeddings can actually be used as a training target to produce a feature model that ends up being homogeneous and without positional biases.
Jane: So these models retain the general semantics of the original DINOv2 or DINOv3 features, but they get rid of those problematic positional artifacts, which is key for applying them successfully in trainable segmentation of complex microscopy images.
Lu: The authors found that this replacement doesn't damage semantics; it retains feature similarities across changes in style and pose, and the ALiBi-Dv2 model is generally more robust to transformations than DINOv2 or NoPE >
Meng: If you can get better segmentations with less label data because the features are more consistent, that’s good for production. The paper shows ALiBi-Dv2 produces features of complicated microstructural concepts like particle size and gaps, which is what we need for materials analysis >
Lalam: They also showed this positional bias is a more general property of self supervision rather than something isolated to the DINO objective or the learned positional encodings, meaning altering the encoding seems to damage things enough that they can’t re-learn those biases during finetuning >
Center for Materials Research, Justus Liebig University of Giessen · Dyson School of Design Engineering, Imperial College London · Finden ltd, Rutherford Appleton Laboratory
cs.CV, cond-mat.mtrl-sci
Submitted: 2026-03-17
Updated: 2026-11-12
Code: https://github.com/facebookresearch/dinov3
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 97/100
The gist: Vision transformers (ViTs) often display positional biases and artifacts independent of semantic content…
Key concepts
- Positional Bias
- This is an artifact in Vision Transformers where the model's output features are strongly influenced by the position of pixels in the input image rather than just what those pixels represent. It causes features to look like positional ramps regardless of the actual content, making it hard for models to generalize.
- Linear Probing
- This is a technique used by researchers to analyze how different parts of a pre-trained model's internal features respond to specific tasks. The authors used linear probing across various objectives and encodings to systematically identify which feature channels were purely positional ramps.
- ALiBi Positional Encoding
- ALiBi is a type of relative positional encoding that calculates the distance between every pair of tokens in an image. It is used here to replace learned positional encodings, aiming to remove inherent biases by making the model's understanding position-aware but content-agnostic.
Terminology
Summary
Vision transformers (ViTs) often display positional biases and artifacts independent of semantic content, which makes zero-shot adaptation difficult in fields like material science where images are often homogeneous cross-sections <ref:2603. p. 0 However, architectural choices (such
as positional encoding) can lead to these models displaying positional biases and artefacts
independent of semantic content. This makes zero-shot adaption difficult in fields like material
science, where images are often cross-sections of homogeneous microstructure (i. having no
preferred direction). In this work, we investigate the positional bias in ViTs via linear probing,
finding it present across a range of objectives and positional encodings, and subsequently
reduce it by finetuning models to use ALiBi relative positional encoding. In this work, we investigate the positional bias in ViTs via linear probing,
finding it present across a range of objectives and positional encodings, and subsequently
reduce it by finetuning models to use ALiBi relative positional encoding. We demonstrate
that these models retain desirable general semantics and their unbiased features can be used
successfully in trainable segmentation of complex microscopy images.Where the paper says it
Where the paper says it
Where the paper says it
Characterization of Positional Bias
The authors first characterize the positional bias across various literature ViTs using linear probing, finding some channels of the output features which are almost purely positional ramp functions regardless of input image <ref:2603.16840#pg3>. They found that these biases exist in both MAE and DINO models, and across different positional encodings (learned or RoPE), although not in supervised ViTs <ref:2603. p. 1 First, we characterise the positional bias across various literature ViTs using linear
probes, finding some channels of the output features
which are almost purely positional ramp functions regardless of input image. These exist in both MAE and
DINO models, and across different positional encodings (learned or RoPE), though not in supervised ViTs. sunsets, the sky, field-of-view) skewing the
results. Of note are the distinct peaks in the per-channel R
2
scores, indicating that some channels were highly correlated with the different 1D ramps. We found biases across all models in the DINO series,
including DINOv3, which used a relative positional
encoding (RoPE). We also observed biases in a ViT
trained with masked autoencoding [4].Where the paper says it
Where the paper says it
Where the paper says it
The Training Signal is Built
To reduce the positional bias, the authors finetune ViT checkpoints to use 2D-aware ALiBi positional encodings, with cylindrical boundary conditions and normalization to enable positional encoding interpolation <ref:2603.16840#pg2>. Surprisingly, once the learned PE is replaced with the unbiased ALiBi PE it is sufficient to use the (biased) embeddings of the original ViT (DINOv2-S or DINOv3-S+) as a training target to recover semantics <ref:2603. p. 1 Next, we finetune ViT checkpoints (DINOv2 & DINOv3) to use 2D-aware ALiBi positional encodings,
with cylindrical boundary conditions and normalisation to enable positional encoding interpolation [31]. Surprisingly, once the learned PE is replaced with the
unbiased ALiBi PE it is sufficient to use the (biased)
embeddings of the original ViT as a training target to
recover semantics. We experimented with producing ‘cleaned’ target embeddings by
directly averaging over the features of transformations
of the input images, but these were costly to produce
and did not improve performance. During training, we
zeroed the four most positional channels as identified
in Sec. During training, we
zeroed the four most positional channels as identified
in Sec. 1 - this reduced the re-emergence of weak
long-range biases.Where the paper says it
Where the paper says it
Where the paper says it
Where the Gains Come From
The gains in feature homogeneity are attributed to the ALiBi encoding specifically, as compared to other methods like channel blanking or fixed sinusoidal PE <ref:2603.16840#pg10>. The ALiBi-Dv2 model produces features which are semantically rich but display less positional bias and artifacts <ref:2603. p. 8 WHAT DINO SAW PREPRINT
DINOv2 DVT ALiBi-Dv2 DINOv2 DVT ALiBi-Dv2
DINOv2 DVT ALiBi-Dv2 DINOv2 DVT ALiBi-Dv2
Figure 5: Feature PCA comparisons for DINOv2-S, DVT and our ALiBi-Dv2. ALiBi-Dv2 produces features which are
semantically rich but that display less positional bias and artefacts. Cooper* 2, and Ronan Docherty2, 4
1Center for Materials Research, Justus Liebig University, 35392 Giessen
2Dyson School of Design Engineering, Imperial College London, London
3Finden ltd, Building R71, Rutherford Appleton Laboratory, Harwell Science and Innovation Campus
4Department of Materials, Imperial College London, London
Train classifier (XGB) to map features → labels
Image
(+labels) Features Pred Features Pred
ALiBi-Dv2
(frozen)
DINOv2
(no learned PE)
L
Relative distance matrix
Attention
layers
DINOv2
• Learned PE
• Positional features
→Biased segmentation
ALiBi-Dv2
• Finetune with ALiBi PE
• Homogenous features
→Unbiased segmentation
(a) (b) (c)
Figure 1: Summary of our contribution. (a) DINOv2’s learned positional encoding (PE) leads to positionally-biased features,
which causes poor segmentations when used in zero-shot segmentation of out-of-distribution images.Where the paper says it
Where the paper says it
Benchmarking Performance
The ALiBi-Dv2 model performs similarly to or better than DINOv2 and NoPE in linear probe semantic segmentation across VOC, ADE20K, and ADE20K datasets <ref:2603.16840#pg12>. In trainable segmentation of materials science micrographs, the ALiBi models’ predictions are far more homogeneous than those from DINOv2 and DVT features <ref:2603. p. 1 However, materials science
images, typically captured with a Scanning or Transmission Electron Microscope (SEM/TEM), are fundamentally different to natural images: they are grayscale,
are sometimes hundreds of megapixels large, and are
often homogeneous cross-sections. This homogeneity
is especially challenging given the clear positional biases seen in ViTs like DINOv2, which has been show to
have negative impacts on trainable segmentation [26]. Cooper* 2, and Ronan Docherty2, 4
1Center for Materials Research, Justus Liebig University, 35392 Giessen
2Dyson School of Design Engineering, Imperial College London, London
3Finden ltd, Building R71, Rutherford Appleton Laboratory, Harwell Science and Innovation Campus
4Department of Materials, Imperial College London, London
Train classifier (XGB) to map features → labels
Image
(+labels) Features Pred Features Pred
ALiBi-Dv2
(frozen)
DINOv2
(no learned PE)
L
Relative distance matrix
Attention
layers
DINOv2
• Learned PE
• Positional features
→Biased segmentation
ALiBi-Dv2
• Finetune with ALiBi PE
• Homogenous features
→Unbiased segmentation
(a) (b) (c)
Figure 1: Summary of our contribution. (a) DINOv2’s learned positional encoding (PE) leads to positionally-biased features,
which causes poor segmentations when used in zero-shot segmentation of out-of-distribution images. 2), and therefore elected to
use ALiBi encodings. We used Euclidean distance as
the distance metric with wrap boundary conditions in
order to avoid asymmetries in the distance matrix.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
In summary, biased ViT embeddings could be used as a training target to produce a homogeneous feature model <ref:2603.16840#pg2>. Once trained, these models produce rich features that retain the semantics and useful properties of the original ViT (DINOv2 or DINOv3) without the problematic positional biases <ref:2603. p. 0 ABSTRACT
Vision transformers (ViTs) - especially feature foundation models like DINOv2 - learn rich
representations useful for many downstream tasks. However, architectural choices (such
as positional encoding) can lead to these models displaying positional biases and artefacts
independent of semantic content. In this work, we investigate the positional bias in ViTs via linear probing,
finding it present across a range of objectives and positional encodings, and subsequently
reduce it by finetuning models to use ALiBi relative positional encoding. We demonstrate
that these models retain desirable general semantics and their unbiased features can be used
successfully in trainable segmentation of complex microscopy images. 5 Feature artefacts & positional bias in ViTs
Despite their impressive performance on benchmarks,
the DINO features (and ViT features more broadly)
exhibit certain biases and artefacts. The authors of ref. S12 for feature PCAs with
scaling. It is reasonable to ask the following question: does
the improved homogeneity after finetuning stem from
the ALiBi encoding specifically, or is it that altering
the positional encoding is sufficiently damaging to the
model’s representations that it cannot re-learn the biases
during finetuning, regardless of choice of encoding? For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth
echoing the results of Ref. As expected,
the absolute positional encodings (learned, sinusoidal)
can trivially learn the mapping. Notably, both variants
of RoPE can learn to reproduce the grid - this suggests
that, despite being a relative positional encoding, it
is capable of modelling long-range positional biases,
potentially in low-frequency rotations. As expected,
the absolute positional encodings (learned, sinusoidal)
can trivially learn the mapping. Notably, both variants
of RoPE can learn to reproduce the grid - this suggests
that, despite being a relative positional encoding, it
is capable of modelling long-range positional biases,
potentially in low-frequency rotations. As expected,
the absolute positional encodings (learned, sinusoidal)
can trivially learn the mapping. Notably, both variants
of RoPE can learn to reproduce the grid - this suggests
that, despite being a relative positional encoding, it
is capable of modelling long-range positional biases,
potentially in low-frequency rotations. As expected,
the absolute positional encodings (learned, sinusoidal)
can trivially learn the mapping. Notably, both variants
of RoPE can learn to reproduce the grid - this suggests
that, despite being a relative positional encoding, it
is capable of modelling long-range positional biases,
potentially in low-frequency rotations. 5 Feature artefacts & positional bias in ViTs
Despite their impressive performance on benchmarks,
the DINO features (and ViT features more broadly)
exhibit certain biases and artefacts. The authors of ref. WHAT DINO SAW: ALIBI POSITIONAL ENCODING REDUCES
POSITIONAL BIAS IN VISION TRANSFORMERS
PREPRINT - AUGUST 12, 2026
Moritz Pawlowsky1
, Antonis Vamvakeros2, 3, Alexander Weiss1
, Anja Bielefeld1
,
Samuel J. Cooper* 2, and Ronan Docherty2, 4
1Center for Materials Research, Justus Liebig University, 35392 Giessen
2Dyson School of Design Engineering, Imperial College London, London
3Finden ltd, Building R71, Rutherford Appleton Laboratory, Harwell Science and Innovation Campus
4Department of Materials, Imperial College London, London
Train classifier (XGB) to map features → labels
Image
(+labels) Features Pred Features Pred
ALiBi-Dv2
(frozen)
DINOv2
(no learned PE)
L
Relative distance matrix
Attention
layers
DINOv2
• Learned PE
• Positional features
→Biased segmentation
ALiBi-Dv2
• Finetune with ALiBi PE
• Homogenous features
→Unbiased segmentation
(a) (b) (c)
Figure 1: Summary of our contribution. For the synthetic ‘square-circle’ example the resulting features
are sharper and more separated, whereas for DINOv2
and DVT, albeit to a lesser extent, the square and circle
features appear closer, either due to the influence of the
positional encoding or weak artefacts. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2
are similar; this was more often seen in in-distribution
images, where the positional bias was expressed less
strongly (in relative terms) to the semantic content. The
‘bimodal cathode’ is a scanning electron microscope
image of a lithium-ion battery cathode with a mix of
different particle sizes - these have better packing and
therefore promise higher energy densities. Analysing
them is therefore of great interest, and the ALiBi models produce homogenous features of complicated microstructural concepts such as particle size, cracking
and gaps (pores) in the structure. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. DINOv2 DVT ALiBi-Dv2 DINOv3
Homogeneity
ALiBi-Dv3
Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2
and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM
image of a Li-ion battery cathode [53]. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. As expected, the finetuned sinusoidal checkpoint is
capable of re-learning the biases present in the original features. We present feature PCAs comparisons
between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig. 2), and therefore elected to
use ALiBi encodings. We used Euclidean distance as
the distance metric with wrap boundary conditions in
order to avoid asymmetries in the distance matrix.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Positional bias reduction in feature foundation models allows for successful zero-shot adaptation in material science imaging by finetuning models with 2D ALiBi positional encoding, which
retains desirable general semantics and their unbiased features can be used successfully in trainable segmentation of complex microscopy images.
-
AI systems can perform more accurate trainable segmentation of complex microscopy images, such as those involving bimodal cathodes or silicongraphite anodes, by utilizing ALiBi-Dv2 features that
produce less positionally biased (and therefore higher quality) segmentations
compared to DINOv2 or DVT. -
Foundation models can be effectively applied in low- or no-training pipelines like trainable segmentation by using homogeneous features, as the paper demonstrates that
the ALiBi models produce homogenous features of complicated microstructural concepts such as particle size, cracking and gaps (pores) in the structure.
-
Models can better handle input image transformations, such as rolling or flipping a material sample during analysis, because ALiBi-Dv2 is
effectively equivariant
due to the wrap conditions in the ALiBi distance matrix. -
Trainable segmentation performance improves significantly in sparse, low-label regimes; specifically,
ALiBi-Dv2 improves the mIoU compared to DINOv2 (best: 0.75 vs 0.69), especially at the start (0.57 vs 0.29) - we attribute this to the lack of positional bias.
Abstract
Vision transformers (ViTs) - especially feature foundation models like DINOv2 - learn rich representations useful for many downstream tasks. However, architectural choices (such as positional encoding) can lead to these models displaying positional biases and artefacts independent of semantic content. This makes zero-shot adaption difficult in fields like material science, where images are often cross-sections of homogeneous microstructure (i.e. having no preferred direction). In this work, we investigate the positional bias in ViTs via linear probing, finding it present across a range of objectives and positional encodings, and subsequently reduce it by finetuning models to use ALiBi relative positional encoding. We demonstrate that these models retain desirable general semantics and their unbiased features can be used successfully in trainable segmentation of complex microscopy images.
Sources
- CoCAViT: Compact Vision Transformer with Robust Global Coordination
- Scaling Vision Transformers
- Masked Autoencoders Are Scalable Vision Learners
- iBOT: Image BERT Pre-Training with Online Tokenizer
- Emerging Properties in Self-Supervised Vision Transformers
- DINOv2: Learning Robust Visual Features without Supervision
- DINOv3
- Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
- Understanding Self-Supervised Features for Learning Unsupervised Instance Segmentation
- Deep ViT Features as Dense Visual Descriptors
- Segment Anything
- EVA-CLIP: Improved Training Techniques for CLIP at Scale
- Vision Transformers Need Registers
- Denoising Vision Transformers
- Into the Rabbit Hull: From Task-Relevant Concepts in DINO to Minkowski Geometry
- Bi-Orthogonal Factor Decomposition for Vision Transformers
- Localizing Objects with Self-Supervised Transformers and no Labels
- MOST: Multiple Object localization with Self-supervised Transformers for object discovery
- Unsupervised Object Localization in the Era of Self-Supervised ViTs: A Survey
- Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localization
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs