What DINO saw: ALiBi positional encoding reduces positional bias in Vision Transformers

arXiv:2603.16840 · cs.CV, cond-mat.mtrl-sci · Submitted 2026-03-17 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "What DINO saw".

Jane: Here's the gist of the paper. Vision transformers (ViTs) often display positional biases and artifacts independent of semantic content, which makes zero-shot adaptation difficult in fields like material science where images are often homogeneous cross-sections <ref:2603.16840#pg3>.

Tom: This work investigates the positional bias in ViTs via linear probing, finding it present across various objectives and encodings, and subsequently reduces it by finetuning models to use ALiBi relative positional encoding <ref:2603.16840#pg2>.

Jane: The authors demonstrate that these models retain desirable general semantics and their unbiased features can be used successfully in trainable segmentation of complex microscopy images <ref:2603.16840#pg2>.

Tom: Characterization of Positional Bias The authors first characterize the positional bias across various literature ViTs using linear probing, finding some channels of the output features which are almost purely positional ramp functions regardless of input image <ref:2603.16840#pg3>.

Title and authors: Tom: Let’s look at the title and who wrote this. It’s titled "What DINO saw: ALiBi positional encoding reduces positional bias in Vision Transformers," and it was written by Pawlowsky, Vamvakeros, Weiss, Bielefeld, Cooper, and Docherty.

Jane: That title is pretty straightforward. It tells you exactly what they investigated: that using a specific type of positional encoding called ALiBi can fix the positional bias that affects Vision Transformers.

Lu: The core idea they are pushing is that these feature foundation models, which are trained on massive datasets, often have these inherent biases from the positional encodings they use, and this paper tests whether changing that encoding helps when you try to adapt them to new tasks.

Meng: So the goal isn't just to fix one model; it’s about understanding why these features behave this way across the board so we can build more reliable systems for things like material science analysis.

Lalam: It points out that Vision transformers are everywhere now, with great performance and scalability, but the success relies on frozen features that can be used in pipelines without further training >

The paper's summary: Tom: The summary of this paper explains that they investigated positional bias by using linear probing across different literature vision transformers and found specific channels whose output features were almost purely positional ramp functions regardless of the input image.

Jane: That means when you probe these models, some parts of the feature space are just showing position information, not actual picture content >

Lu: They found that these biases exist in both MAE and DINO models, across different learned or RoPE positional encodings, but they were absent in supervised vision transformers. This suggests the issue is inherent to the self-supervised training process itself rather than just the final objective function.

Meng: That’s a big deal if it means we can’t trust any frozen feature backbone without knowing where those biased channels are, because I have to worry about what breaks when we try to deploy that model.

Lalam: They also found distinct peaks in the per-channel R squared scores, which indicated that some specific channels were highly correlated with those different one dimensional ramps >

The paper's improvements: Tom: Now for the fix they propose. They reduce this positional bias by finetuning the models to use 2D aware ALiBi positional encodings, complete with cylindrical boundary conditions and normalization to help interpolate between positions <ref:2603.16840#pg1>.

Jane: So they aren’t just throwing out the old encoding; they are training a new one that handles the spatial structure of an image better, which is a big step because it’s not just about replacing text in a formula.

Lu: Surprisingly, once they swap out the learned positional encoding for this unbiased ALiBi positional encoding, it turns out you only need to use those original biased embeddings from models like DINOv2 or DINOv3 as your training target to recover the semantics >

Meng: That’s a clever way to save computation. So instead of retraining a whole massive model, you can just fine-tune the new encoding on top of what’s already learned, which makes sense for deployment cost and time.

Lalam: They also zeroed out those four most positional channels they identified earlier, which reduced the reappearance of weak long range biases in those features >

Conclusion: Tom: To wrap this up, the main point is that biased vision transformer embeddings can actually be used as a training target to produce a feature model that ends up being homogeneous and without positional biases.

Jane: So these models retain the general semantics of the original DINOv2 or DINOv3 features, but they get rid of those problematic positional artifacts, which is key for applying them successfully in trainable segmentation of complex microscopy images.

Lu: The authors found that this replacement doesn't damage semantics; it retains feature similarities across changes in style and pose, and the ALiBi-Dv2 model is generally more robust to transformations than DINOv2 or NoPE >

Meng: If you can get better segmentations with less label data because the features are more consistent, that’s good for production. The paper shows ALiBi-Dv2 produces features of complicated microstructural concepts like particle size and gaps, which is what we need for materials analysis >

Lalam: They also showed this positional bias is a more general property of self supervision rather than something isolated to the DINO objective or the learned positional encodings, meaning altering the encoding seems to damage things enough that they can’t re-learn those biases during finetuning >

Center for Materials Research, Justus Liebig University of Giessen · Dyson School of Design Engineering, Imperial College London · Finden ltd, Rutherford Appleton Laboratory

cs.CV, cond-mat.mtrl-sci

Submitted: 2026-03-17

Updated: 2026-11-12

Code: https://github.com/facebookresearch/dinov3

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 97/100

The gist: Vision transformers (ViTs) often display positional biases and artifacts independent of semantic content…

Key concepts

Positional Bias
This is an artifact in Vision Transformers where the model's output features are strongly influenced by the position of pixels in the input image rather than just what those pixels represent. It causes features to look like positional ramps regardless of the actual content, making it hard for models to generalize.
Linear Probing
This is a technique used by researchers to analyze how different parts of a pre-trained model's internal features respond to specific tasks. The authors used linear probing across various objectives and encodings to systematically identify which feature channels were purely positional ramps.
ALiBi Positional Encoding
ALiBi is a type of relative positional encoding that calculates the distance between every pair of tokens in an image. It is used here to replace learned positional encodings, aiming to remove inherent biases by making the model's understanding position-aware but content-agnostic.

Terminology

Summary

Vision transformers (ViTs) often display positional biases and artifacts independent of semantic content, which makes zero-shot adaptation difficult in fields like material science where images are often homogeneous cross-sections <ref:2603. p. 0

Where the paper says it

However, architectural choices (such as positional encoding) can lead to these models displaying positional biases and artefacts independent of semantic content. This makes zero-shot adaption difficult in fields like material science, where images are often cross-sections of homogeneous microstructure (i.

16840#pg3>. This work investigates the positional bias in ViTs via linear probing, finding it present across various objectives and encodings, and subsequently reduces it by finetuning models to use ALiBi relative positional encoding <ref:2603. p. 0
Where the paper says it

having no preferred direction). In this work, we investigate the positional bias in ViTs via linear probing, finding it present across a range of objectives and positional encodings, and subsequently reduce it by finetuning models to use ALiBi relative positional encoding.

16840#pg2>. The authors demonstrate that these models retain desirable general semantics and their unbiased features can be used successfully in trainable segmentation of complex microscopy images <ref:2603. p. 0
Where the paper says it

In this work, we investigate the positional bias in ViTs via linear probing, finding it present across a range of objectives and positional encodings, and subsequently reduce it by finetuning models to use ALiBi relative positional encoding. We demonstrate that these models retain desirable general semantics and their unbiased features can be used successfully in trainable segmentation of complex microscopy images.

16840#pg2>.

Characterization of Positional Bias

The authors first characterize the positional bias across various literature ViTs using linear probing, finding some channels of the output features which are almost purely positional ramp functions regardless of input image <ref:2603.16840#pg3>. They found that these biases exist in both MAE and DINO models, and across different positional encodings (learned or RoPE), although not in supervised ViTs <ref:2603. p. 1

Where the paper says it

First, we characterise the positional bias across various literature ViTs using linear probes, finding some channels of the output features which are almost purely positional ramp functions regardless of input image. These exist in both MAE and DINO models, and across different positional encodings (learned or RoPE), though not in supervised ViTs.

16840#pg3>. The linear probe analysis revealed distinct peaks in the per-channel R squared scores, indicating that some channels were highly correlated with the different 1D ramps <ref:2603. p. 5
Where the paper says it

sunsets, the sky, field-of-view) skewing the results. Of note are the distinct peaks in the per-channel R 2 scores, indicating that some channels were highly correlated with the different 1D ramps.

16840#pg5>. Furthermore, they found that biases across all models in the DINO series, including DINOv3 which used a relative positional encoding (RoPE), were observed <ref:2603. p. 5
Where the paper says it

We found biases across all models in the DINO series, including DINOv3, which used a relative positional encoding (RoPE). We also observed biases in a ViT trained with masked autoencoding [4].

16840#pg8>.

The Training Signal is Built

To reduce the positional bias, the authors finetune ViT checkpoints to use 2D-aware ALiBi positional encodings, with cylindrical boundary conditions and normalization to enable positional encoding interpolation <ref:2603.16840#pg2>. Surprisingly, once the learned PE is replaced with the unbiased ALiBi PE it is sufficient to use the (biased) embeddings of the original ViT (DINOv2-S or DINOv3-S+) as a training target to recover semantics <ref:2603. p. 1

Where the paper says it

Next, we finetune ViT checkpoints (DINOv2 & DINOv3) to use 2D-aware ALiBi positional encodings, with cylindrical boundary conditions and normalisation to enable positional encoding interpolation [31]. Surprisingly, once the learned PE is replaced with the unbiased ALiBi PE it is sufficient to use the (biased) embeddings of the original ViT as a training target to recover semantics.

16840#pg2>. They also zeroed the four most positional channels as identified in Sec. p. 4
Where the paper says it

We experimented with producing ‘cleaned’ target embeddings by directly averaging over the features of transformations of the input images, but these were costly to produce and did not improve performance. During training, we zeroed the four most positional channels as identified in Sec.

4.1 - this reduced the re-emergence of weak long-range biases <ref:2603. p. 4
Where the paper says it

During training, we zeroed the four most positional channels as identified in Sec. 1 - this reduced the re-emergence of weak long-range biases.

16840#pg8>.

Where the Gains Come From

The gains in feature homogeneity are attributed to the ALiBi encoding specifically, as compared to other methods like channel blanking or fixed sinusoidal PE <ref:2603.16840#pg10>. The ALiBi-Dv2 model produces features which are semantically rich but display less positional bias and artifacts <ref:2603. p. 8

Where the paper says it

WHAT DINO SAW PREPRINT DINOv2 DVT ALiBi-Dv2 DINOv2 DVT ALiBi-Dv2 DINOv2 DVT ALiBi-Dv2 DINOv2 DVT ALiBi-Dv2 Figure 5: Feature PCA comparisons for DINOv2-S, DVT and our ALiBi-Dv2. ALiBi-Dv2 produces features which are semantically rich but that display less positional bias and artefacts.

16840#pg10>. This improved homogeneity leads to better resulting segmentations in trainable segmentation of complex microscopy images, where DINOv2 and DVT features were clearly positionally biased <ref:2603. p. 0
Where the paper says it

Cooper* 2, and Ronan Docherty2, 4 1Center for Materials Research, Justus Liebig University, 35392 Giessen 2Dyson School of Design Engineering, Imperial College London, London 3Finden ltd, Building R71, Rutherford Appleton Laboratory, Harwell Science and Innovation Campus 4Department of Materials, Imperial College London, London Train classifier (XGB) to map features → labels Image (+labels) Features Pred Features Pred ALiBi-Dv2 (frozen) DINOv2 (no learned PE) L Relative distance matrix Attention layers DINOv2 • Learned PE • Positional features →Biased segmentation ALiBi-Dv2 • Finetune with ALiBi PE • Homogenous features →Unbiased segmentation (a) (b) (c) Figure 1: Summary of our contribution. (a) DINOv2’s learned positional encoding (PE) leads to positionally-biased features, which causes poor segmentations when used in zero-shot segmentation of out-of-distribution images.

16840#pg14>.

Benchmarking Performance

The ALiBi-Dv2 model performs similarly to or better than DINOv2 and NoPE in linear probe semantic segmentation across VOC, ADE20K, and ADE20K datasets <ref:2603.16840#pg12>. In trainable segmentation of materials science micrographs, the ALiBi models’ predictions are far more homogeneous than those from DINOv2 and DVT features <ref:2603. p. 1

Where the paper says it

However, materials science images, typically captured with a Scanning or Transmission Electron Microscope (SEM/TEM), are fundamentally different to natural images: they are grayscale, are sometimes hundreds of megapixels large, and are often homogeneous cross-sections. This homogeneity is especially challenging given the clear positional biases seen in ViTs like DINOv2, which has been show to have negative impacts on trainable segmentation [26].

16840#pg31>. This improved homogeneity leads to higher mIoU in sparse, low-label regimes where labels are randomly distributed across the image. p. 0
Where the paper says it

Cooper* 2, and Ronan Docherty2, 4 1Center for Materials Research, Justus Liebig University, 35392 Giessen 2Dyson School of Design Engineering, Imperial College London, London 3Finden ltd, Building R71, Rutherford Appleton Laboratory, Harwell Science and Innovation Campus 4Department of Materials, Imperial College London, London Train classifier (XGB) to map features → labels Image (+labels) Features Pred Features Pred ALiBi-Dv2 (frozen) DINOv2 (no learned PE) L Relative distance matrix Attention layers DINOv2 • Learned PE • Positional features →Biased segmentation ALiBi-Dv2 • Finetune with ALiBi PE • Homogenous features →Unbiased segmentation (a) (b) (c) Figure 1: Summary of our contribution. (a) DINOv2’s learned positional encoding (PE) leads to positionally-biased features, which causes poor segmentations when used in zero-shot segmentation of out-of-distribution images.

The ALiBi-Dv2 model is more robust to adversarial transformations, especially roll - thanks to wrap conditions in the ALiBi distance matrix it is effectively equivariant <ref:2603. p. 4
Where the paper says it

2), and therefore elected to use ALiBi encodings. We used Euclidean distance as the distance metric with wrap boundary conditions in order to avoid asymmetries in the distance matrix.

16840#pg8>.

Conclusion

In summary, biased ViT embeddings could be used as a training target to produce a homogeneous feature model <ref:2603.16840#pg2>. Once trained, these models produce rich features that retain the semantics and useful properties of the original ViT (DINOv2 or DINOv3) without the problematic positional biases <ref:2603. p. 0

Where the paper says it

ABSTRACT Vision transformers (ViTs) - especially feature foundation models like DINOv2 - learn rich representations useful for many downstream tasks. However, architectural choices (such as positional encoding) can lead to these models displaying positional biases and artefacts independent of semantic content.

16840#pg6>. This meant they could be applied successfully to trainable segmentation of complex microscopy images that previous models struggled with <ref:2603. p. 0
Where the paper says it

In this work, we investigate the positional bias in ViTs via linear probing, finding it present across a range of objectives and positional encodings, and subsequently reduce it by finetuning models to use ALiBi relative positional encoding. We demonstrate that these models retain desirable general semantics and their unbiased features can be used successfully in trainable segmentation of complex microscopy images.

16840#pg14>. The authors found evidence that this positional bias was a more general property of self-supervision (rather than being isolated to the DINO objective or learned positional encodings) <ref:2603. p. 2
Where the paper says it

5 Feature artefacts & positional bias in ViTs Despite their impressive performance on benchmarks, the DINO features (and ViT features more broadly) exhibit certain biases and artefacts. The authors of ref.

16840#pg13>. This suggests that altering the positional encoding is sufficiently damaging to the model’s representations that it cannot re-learn the biases during finetuning, regardless of choice of encoding <ref:2603. p. 8
Where the paper says it

S12 for feature PCAs with scaling. It is reasonable to ask the following question: does the improved homogeneity after finetuning stem from the ALiBi encoding specifically, or is it that altering the positional encoding is sufficiently damaging to the model’s representations that it cannot re-learn the biases during finetuning, regardless of choice of encoding?

16840#pg10>. The ALiBi-Dv2 model avoids these problems <ref:2603. p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

p. 7
Where the paper says it

For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content. The ‘token geometry’ (3D plot of patch features according using PCA coordinates) remains locally smooth echoing the results of Ref.

16840#pg10>. This suggests that the replacement of the learned PE has not damaged semantics <ref:2603. p. 7
Where the paper says it

As expected, the absolute positional encodings (learned, sinusoidal) can trivially learn the mapping. Notably, both variants of RoPE can learn to reproduce the grid - this suggests that, despite being a relative positional encoding, it is capable of modelling long-range positional biases, potentially in low-frequency rotations.

p. 7
Where the paper says it

As expected, the absolute positional encodings (learned, sinusoidal) can trivially learn the mapping. Notably, both variants of RoPE can learn to reproduce the grid - this suggests that, despite being a relative positional encoding, it is capable of modelling long-range positional biases, potentially in low-frequency rotations.

p. 7
Where the paper says it

As expected, the absolute positional encodings (learned, sinusoidal) can trivially learn the mapping. Notably, both variants of RoPE can learn to reproduce the grid - this suggests that, despite being a relative positional encoding, it is capable of modelling long-range positional biases, potentially in low-frequency rotations.

p. 7
Where the paper says it

As expected, the absolute positional encodings (learned, sinusoidal) can trivially learn the mapping. Notably, both variants of RoPE can learn to reproduce the grid - this suggests that, despite being a relative positional encoding, it is capable of modelling long-range positional biases, potentially in low-frequency rotations.

16840#pg14>. The authors found that the addition of ViT features improves the mIoU drastically, as ALiBi-Dv2 improves the mIoU compared to DINOv2 (best: 0. p. 2
Where the paper says it

5 Feature artefacts & positional bias in ViTs Despite their impressive performance on benchmarks, the DINO features (and ViT features more broadly) exhibit certain biases and artefacts. The authors of ref.

75 vs 0.69) - we attribute this to the lack of positional bias in its features. p. 0
Where the paper says it

WHAT DINO SAW: ALIBI POSITIONAL ENCODING REDUCES POSITIONAL BIAS IN VISION TRANSFORMERS PREPRINT - AUGUST 12, 2026 Moritz Pawlowsky1 , Antonis Vamvakeros2, 3, Alexander Weiss1 , Anja Bielefeld1 , Samuel J. Cooper* 2, and Ronan Docherty2, 4 1Center for Materials Research, Justus Liebig University, 35392 Giessen 2Dyson School of Design Engineering, Imperial College London, London 3Finden ltd, Building R71, Rutherford Appleton Laboratory, Harwell Science and Innovation Campus 4Department of Materials, Imperial College London, London Train classifier (XGB) to map features → labels Image (+labels) Features Pred Features Pred ALiBi-Dv2 (frozen) DINOv2 (no learned PE) L Relative distance matrix Attention layers DINOv2 • Learned PE • Positional features →Biased segmentation ALiBi-Dv2 • Finetune with ALiBi PE • Homogenous features →Unbiased segmentation (a) (b) (c) Figure 1: Summary of our contribution.

The ALiBi-Dv2 model offers improved performance over DINOv2 with less labels due to the lack of positional bias in its features. p. 7
Where the paper says it

For the synthetic ‘square-circle’ example the resulting features are sharper and more separated, whereas for DINOv2 and DVT, albeit to a lesser extent, the square and circle features appear closer, either due to the influence of the positional encoding or weak artefacts. For some images, such as the dog, the PCAs of DVT and ALiBi-Dv2 are similar; this was more often seen in in-distribution images, where the positional bias was expressed less strongly (in relative terms) to the semantic content.

The ALiBi-Dv2 model produces homogenous features of complicated microstructural concepts such as particle size, cracking and gaps (pores) in the structure <ref:2603. p. 8
Where the paper says it

The ‘bimodal cathode’ is a scanning electron microscope image of a lithium-ion battery cathode with a mix of different particle sizes - these have better packing and therefore promise higher energy densities. Analysing them is therefore of great interest, and the ALiBi models produce homogenous features of complicated microstructural concepts such as particle size, cracking and gaps (pores) in the structure.

16840#pg10>. This suggests that the replacement of the learned PE has not damaged semantics <ref:2603.16840#pg14>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

p. 9
Where the paper says it

DINOv2 DVT ALiBi-Dv2 DINOv3 Homogeneity ALiBi-Dv3 Figure 7: Cosine similarity scores for DINOv2 and our ALiBi models for a given query patch (red inset square). ALiBi-Dv2 and -Dv3 have more uniform similarities for increasingly homogenous (and out-of-distribution) images, such as the SEM image of a Li-ion battery cathode [53].

<ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603. p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

p. 8
Where the paper says it

As expected, the finetuned sinusoidal checkpoint is capable of re-learning the biases present in the original features. We present feature PCAs comparisons between DINOv2, a sinuosidal DINOv2, a NoPE DINOv2 and our ALiBi-Dv2 in Fig.

16840#pg8>. The ALiBi-Dv2 is more robust to adversarial transformations, especially roll - thanks to wrap conditions in the ALiBi distance matrix it is effectively equivariant <ref:2603. p. 4
Where the paper says it

2), and therefore elected to use ALiBi encodings. We used Euclidean distance as the distance metric with wrap boundary conditions in order to avoid asymmetries in the distance matrix.

16840#pg8>. This suggests that the replacement of the learned PE has not damaged semantics <ref:2603.16840#pg14>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. This suggests that the replacement of the learned PE has not damaged semantics <ref:2603.16840#pg14>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc. <ref:2603.16840#pg14>. The ALiBi-Dv2 is generally more robust to transformations than DINOv2 or NoPE <ref:2603.16840#pg8>. The ALiBi-Dv2 model avoids these problems <ref:2603.16840#pg10>. The ALiBi-Dv2 model retains DINOv2’s feature similarities across changes in style, pose, etc.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Positional bias reduction in feature foundation models allows for successful zero-shot adaptation in material science imaging by finetuning models with 2D ALiBi positional encoding, which retains desirable general semantics and their unbiased features can be used successfully in trainable segmentation of complex microscopy images.

  2. AI systems can perform more accurate trainable segmentation of complex microscopy images, such as those involving bimodal cathodes or silicongraphite anodes, by utilizing ALiBi-Dv2 features that produce less positionally biased (and therefore higher quality) segmentations compared to DINOv2 or DVT.

  3. Foundation models can be effectively applied in low- or no-training pipelines like trainable segmentation by using homogeneous features, as the paper demonstrates that the ALiBi models produce homogenous features of complicated microstructural concepts such as particle size, cracking and gaps (pores) in the structure.

  4. Models can better handle input image transformations, such as rolling or flipping a material sample during analysis, because ALiBi-Dv2 is effectively equivariant due to the wrap conditions in the ALiBi distance matrix.

  5. Trainable segmentation performance improves significantly in sparse, low-label regimes; specifically, ALiBi-Dv2 improves the mIoU compared to DINOv2 (best: 0.75 vs 0.69), especially at the start (0.57 vs 0.29) - we attribute this to the lack of positional bias.

Abstract

Vision transformers (ViTs) - especially feature foundation models like DINOv2 - learn rich representations useful for many downstream tasks. However, architectural choices (such as positional encoding) can lead to these models displaying positional biases and artefacts independent of semantic content. This makes zero-shot adaption difficult in fields like material science, where images are often cross-sections of homogeneous microstructure (i.e. having no preferred direction). In this work, we investigate the positional bias in ViTs via linear probing, finding it present across a range of objectives and positional encodings, and subsequently reduce it by finetuning models to use ALiBi relative positional encoding. We demonstrate that these models retain desirable general semantics and their unbiased features can be used successfully in trainable segmentation of complex microscopy images.

Sources

Related papers