Understanding Transformer-based Vision Models through Inversion

The gist: This study introduces a novel, modular feature inversion method that enables significantly more efficient application of feature inversion to large-scale transformer-based vision models, specifically Detection Transformer and Vision Transformer, uncovering key insights into how these models encode contextual shape and image details through systematic analysis of reconstructed images.

Introduction and Motivation

Understanding the internal mechanisms of transformer-based vision models remains a fundamental challenge in machine learning and computer vision The opaqueness prevents an interpretation and a clear understanding of how predictions are made within these networks Feature inversion, which attempts to reconstruct images from intermediate representations using trained inverse neural networks, is an early technique to interpret the processing capabilities of DNNs for vision While feature inversion was successfully applied to AlexNet, it has not seen widespread adoption in the context of modern DNNs for vision because individual inverse networks for each layer are computationally demanding In this work, the authors revisit feature inversion and apply it to two widely used TVMs (see Figure 1 for an illustration): Detection Transformer (DETR) and Vision Transformer (ViT) They introduce a novel feature inversion approach that builds on the classical idea of training layer-wise inverse networks but adopts a modular strategy that inverts only specific components of the model Their analysis reveals key properties of DETR and ViT, as well as fundamental architectural differences

Modular Feature Inversion Method

The authors introduce a novel highly efficient feature inversion method based on modular, independently trained inverse components, which they empirically test and validate Instead of training inverse components to map from Xj to X0 directly, they train local inverse components, learning N−1j:j−1 for all j ∈ P The formal definition of the reconstruction is given by xˆ0:j:= (N−1j:i ◦ · · · ◦ N−1j:j−1)(xj) This modular approach offers several advantages, including biasing the inverse solution space of inverse components by independently training them, which maintains greater symmetry between the processing stages of the forward and backward path Furthermore, computational efficiency is greatly improved because fewer, smaller components are required compared to training separate inverse components attempting to inverse the entire forward path for each stage The parameter count for the modular method totals p parameters, constant in p and independent of n

Application and Qualitative Analysis

The modular feature inversion was applied to DETR and ViT, analyzing four processing stages for DETR (P:= (1, 2, 3, 4)) and two stages for ViT (P:= (1, 2)) Early-stage image reconstructions from both architectures maintain the overall scene layout and coarse object structure However, ViT preserved fine-grained details more accurately than DETR reconstructions show early signs of blurring even at the backbone stage In deeper stages, DETR reconstructions progressively lose structural detail and introduce color shifts, suggesting a systematic abstraction process The modular approach enables reconstructions that retain stage-specific transformations without collapsing into a globally averaged output

Comparative Analysis of DETR and ViT

The analysis reveals key properties of DETR and ViT, as well as fundamental architectural differences Among the findings are contrasts in the preservation of contextual shape and image detail, inter-layer representational correlations, and robustness to color perturbations Notably, DETR gradually transforms object shapes and colors into more prototypical representations at higher layers, while ViT retains fine-grained visual detail throughout all layers For DETR, objects undergo significant alterations in later stages, including changes in size and shape In contrast, ViT reconstructions show little structural change across stages, suggesting it retains low-level visual and semantic information without applying the same degree of abstraction observed in DETR

Color and Spatial Correlation Analysis

Systematic analysis of color processing showed that for DETR, color perturbations are preserved in backbone representations but gradually fade or disappear almost entirely in encoder representations For ViT, color perturbations remain visible in image reconstructions from all processing stages Quantifying response to color perturbations showed that for DETR, the average pairwise MSE decreases progressively from x0 to xˆenc:0, indicating increasing similarity For ViT, the preservation of color perturbations throughout the architecture is reflected in an almost constant pairwise MSE across processing stages Furthermore, DETR diffuses spatial information more globally, whereas ViT encodes spatial information in a localized manner

Conclusion and Future Directions

The work concludes that both DETR and ViT refine their representations gradually across layers, a pattern consistent with prior ViT studies (Raghu et al., 2021) and now extended to DETR The findings suggest that both models exhibit gradually evolving representations across layers, yet differ in their treatment of color and structural image information Future work could examine how the training dataset influences feature inversion results by retraining both architectures on the same dataset The authors propose that modular feature inversion is both more efficient and more naturally aligned with the architecture of TVMs than the classical approach This approach shows promise for error detection by comparing reconstructed images to their inputs, where discrepancies may serve as indicators of detection failures The method is also expected to be valuable for analyzing modern CNN-based models such as ConvNeXt V2 or YOLOv8

Acknowledgments

This work was supported by grants from the German Research Foundation (DFG), project number 397530566 – FOR 2812, P5

References

Samira Abnar and Willem Zuidema. Quantifying Attention Flow in Transformers, 2020. arXiv:2005.00928 Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation. PLOS ONE, 10(7):e0130140 Adrien Bibal, Rémi Cardon, David Alfter, Rodrigo Wilkens, Xiaoou Wang, Thomas François, and Patrick Watrin. Is Attention Explanation? An Introduction to the Debate. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3889–3900 Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-End Object Detection with Transformers, 2020. arXiv:2005.12872 [cs] Hila Chefer, Shir Gur, and Lior Wolf. Transformer Interpretability Beyond Attention Visualization. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 782–791 Alexey Dosovitskiy and Thomas Brox. Inverting Visual Representations with Convolutional Networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4829–4837 Dumitru Erhan, Y. Bengio, Aaron Courville, and Pascal Vincent. Visualizing Higher-Layer Features of a Deep Network. Technical Report, Univeristé de Montréal, 2009 Fenglei Fan, Jinjun Xiong, Mengzhou Li, and Ge Wang. On Interpretability of Artificial Neural Networks: A Survey, 2021. arXiv:2001.02522 Paolo Fantozzi and Maurizio Naldi. The Explainability of Transformers: Current Status and Directions.

1. The modular feature inversion framework can be used as a diagnostic tool for detection failures by comparing reconstructed images to their inputs, discrepancies may serve as indicators of detection failures. This allows researchers to pinpoint exactly which stages in the processing pipeline are responsible for misinterpretations or omissions.

2. AI systems can be trained to exhibit more robust color processing, as DETR reconstructions progressively shift object colors toward prototypical representations, whereas ViT preserves original color information throughout. This insight can guide model training to either preserve specific visual fidelity or encourage abstraction based on the desired application.

3. The system can be designed with a trade-off mechanism where reconstruction quality is balanced against task performance, as evidenced by the finding that as λ decreases, reconstruction error increases but both detection and classification performances improve. This enables the selection of an optimal model configuration that balances interpretability depth with operational accuracy.

4. The system can be specialized for object detection error analysis by observing how objects that are ultimately not detected (e.g., the bicycle in the second row or the potted plants in the third row) are gradually suppressed across the processing stages. This provides a visual trail of where critical information is lost during object query formation or matching.

5. The system can be adapted to distinguish between different architectural philosophies by analyzing spatial correlations, as ViT encodes spatial information in a localized manner while DETR diffuses spatial information more globally. This informs the design choice between architectures based on whether fine-grained detail preservation or global context distribution is required for the specific task.

6. The system can be used to diagnose model weaknesses by analyzing how false positives behave, as Reconstructions from later stages reveal a shift toward features associated with incorrect classes, suggesting that misinterpretations are consolidated into prototypical representations. This guides corrective refinements to prevent the model from learning spurious correlations during detection.

Understanding the internal mechanisms of deep neural networks remains a central challenge in machine learning. In computer vision, one promising yet only preliminarily explored approach is feature inversion via inverse networks, which reconstructs images from intermediate representations using trained inverse networks. In this study, we revisit feature inversion via inverse networks, introducing a novel, modular variant that enables a computationally more efficient application of the technique while, in some cases, producing semantically more coherent image reconstructions. We apply our method to large-scale transformer-based vision models, specifically Detection Transformer, Vision Transformer, Swin Transformer, and Data-Efficient Image Transformer, analyzing the resulting reconstructions across network depth. Our main analysis compares Detection Transformer and Vision Transformer, which exhibit the most informative differences among the evaluated architectures. At the same time, results for Swin Transformer and Data-Efficient Image Transformer support the broader applicability of our framework. Our findings reveal gradual representational changes across transformer layers as a shared characteristic of Detection Transformer and Vision Transformer, as well as systematic differences in how they preserve contextual shape and fine-grained image details, and in their robustness to color perturbations. These findings deepen understanding of transformer-based vision models and demonstrate the utility of modular feature inversion as an interpretability tool.