Causal Attribution via Activation Patching
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Causal Attribution via Activation Patching".
Jane: Here's the gist of the paper. Attribution methods for Vision Transformers (ViTs) aim to identify image regions that influence model predictions, but producing faithful and well-localized attributions remains challenging The key challenge is that class-relevant evidence is formed through interactions between patch tokens across layers; methods that operate only on input changes, attention weights, or backward relevance signals may therefore provide indirect proxies for patch importance rather than directly testing the predictive effect of contextualized patch representations We propose Causal Attribution via Activation Patching (CAAP), which estimates the contribution of individual image patches to the ViT’s prediction by directly intervening on internal activations rather than using learned masks or synthetic perturbation patterns The resulting attribution map reflects the causal contribution of patch-associated internal representations on the model’s prediction.
Tom: Pruning redundant reasoning tokens Existing attribution methods face several limitations, with gradient-based, relevance-propagation, and attention-based methods relying on local approximations The key challenge is that class-relevant evidence is formed through interactions between patch tokens across layers; methods that operate only on input changes, attention weights, or backward relevance signals may therefore provide indirect proxies for patch importance rather than directly testing the predictive effect of contextualized patch representations.
Jane: Gradient-based and relevance-propagation methods provide efficient class-specific signals, but they measure local sensitivity or propagate relevance through the computation graph rather than directly testing whether a contextualized patch representation is sufficient to support the prediction.
Tom: Attention-based methods expose token-mixing patterns, but attention weights or attention flow are indirect proxies for feature contribution, especially after information from different tokens has been repeatedly mixed across layers.
Title and authors: Tom: So we're talking about this paper, "Causal Attribution via Activation Patching," and what it does to how we figure out which parts of an image matter for a Vision Transformer.
Jane: It seems like the main problem they're tackling is that existing methods often give us indirect hints instead of actually testing if a specific part of the image truly drives the model's prediction.
Lu: Yeah, they point out that class-relevant evidence isn't just about what a single patch looks like in isolation; it’s about how those patches interact across many layers through self attention, which makes methods relying only on input changes or attention weights kind of an approximation.
Meng: So, if I understand this right, traditional methods are giving us some local sensitivity or looking at where the tokens are paying attention, but they aren't really measuring the direct causal effect of that specific patch representation.
Lalam: Exactly. The core idea here is to move the intervention away from the input pixels or even just the attention weights and right into the internal activations of a specific image patch itself.
Tom: And their proposed solution, Causal Attribution via Activation Patching, essentially estimates that causal contribution by directly intervening on those internal activations instead of using learned masks or synthetic patterns.
Jane: That sounds like they are taking those source-image activations for a chosen patch and injecting them into a neutral target context over several layers to see what happens to the final prediction score.
Lu: They construct this patched context, which we can call c star of pi, by replacing the original activations with those from the source image at that specific patch position across all layers L in the model.
Meng: So, they are essentially creating a modified version of the target context where only one patch's internal representation is swapped in, and then running a full forward pass to see how much that single patch influences the final class score.
Tom: The attribution score for that specific patch pi with respect to a target class y is defined as the prediction score you get from that patched context when predicting y, which they call A of pi equals f of c star of pi at y.
Jane: And they argue that because these patched activations come directly from the original context, it avoids introducing inputs that are completely out-of-distribution in a way input perturbation methods can.
Lu: That's an important distinction because it means the patched activations already encode those interactions between patches that we were worried about ignoring when using simpler methods.
Tom: The paper shows that this causal intervention serves as a principled measure of patch influence because it captures the semantic evidence after the initial representation formation, which is a big step forward.
Title and authors: Jane: But they also have to be careful about where they stop patching; if you patch too many layers, you risk losing spatial specificity because of late-layer global mixing.
Lu: They actually looked at attention patterns and found that in the middle layers, where intra-object grouping is happening, the information enrichment from other patches increases while after the middle layers it decreases.
Meng: So they suggest a reasonable choice for that layer range L is to include the early and intermediate layers up to around two thirds of L because that's when you still have strong object-level information but it's not getting too mixed with background stuff.
Tom: And on the practical side, they tackle efficiency by suggesting two things: parallel computation of the classification tokens, or CLS tokens, and precomputation of blank context attention.
Jane: That precomputation lets them reuse activations from a blank image xzero across samples and then compute all patch scores in parallel during attribution time.
Lu: They use a dedicated classification token, CLSi, with the same positional encoding as the original CLS token to manage this intervention process efficiently.
Meng: From an engineering standpoint, that parallelization is huge because it means for every single image we analyze, we only need one forward pass for the source image and one pass using those precomputed contexts.
Tom: And the cost per sample drops from something like N cubed operations in a naive implementation to approximately N operations when you use that blank context precomputation.
Jane: So, the main gain they claim is that this design lets them isolate the causal contribution of class-relevant patch representations without relying on indirect evidence like attention weights or approximate gradient signals.
Lu: They also mentioned they considered a padded variant where the selection operator S of pi includes not just the patch pi but its immediate spatial neighbors to better capture feature spread across those tokens.
Meng: That makes sense for locality, but how much extra computation does adding those neighbors cost in practice?
Tom: Well, they found that faithfulness is largely insensitive to the choice of the target blank patches or even the exact selection operator, as long as you use a padded version instead of just a single patch.
Jane: That's reassuring because it means their results aren't overly dependent on those specific implementation details we might change later.
Lu: They also tested different layer depths and found that the performance, especially the insertion score, stayed stable across a broad range of intermediate layers, not just one exact cutoff point.
Meng: That stability is good for deployment because it means we don't have to be super picky about tuning that layer depth parameter when we move to production models.
Title and authors: Tom: The results on CNN architectures also show that even with these transformer approaches, CAAP still achieves the best overall results in most cases compared to other intervention-based methods.
Jane: So, if you look at the metrics they report, like Deletion AUC and Insertion AUC, they consistently get lower Deletion AUC and higher Insertion AUC across multiple ViT backbones on ImageNet-1K <ref:2603.13652#pg1>.
Lu: And they found that this difference in insertion versus deletion scores is notably larger compared to prior methods, which suggests the regions ranked by their attribution maps actually reflect the model’s true decision-making process.
Meng: That's what I care about—if the map is accurate, it means we can actually trust it for debugging or auditing.
Tom: They also reported improvements in localization metrics like AUPR1, AUPR0, and Pointing Game on most models because the patch-level interventions align better with actual object regions rather than just highlighting a few very discriminative patches.
Jane: It seems they're producing more compact and coherent attribution maps that line up closely with the target object, which is much better than having diffuse attention signals.
Lu: They also found that for convolutional architectures like ResNet-eighteen or VGG-sixteen while their overall performance might be lower than transformer results, CAAP still achieves the best overall results in most cases when comparing intervention methods to gradient or relevance propagation.
Meng: That's interesting because I was worried about distribution shifts on CNNs, but they still beat those other local sensitivity measures.
Tom: The main point is that they shift the intervention site from external variables or early model states into internal representations that are closer to the final decision computation, which is a really helpful perspective.
Jane: So, to wrap this up, this paper on Causal Attribution via Activation Patching proposes a framework based on activation patching where you inject source activations into a target context to see the causal effect on the prediction score for each patch individually.
Lu: It moves us past relying on indirect evidence like attention weights or input masks and directly tests the predictive effect of contextualized patch representations within the model’s layers.
Meng: And they make it practical by suggesting parallel computations using CLS tokens and precomputed blank context to keep the cost manageable for real-world use.
Tom: Ultimately, they show that this method consistently achieves lower Deletion AUC and higher Insertion AUC across multiple ViT backbones on ImageNet-1K, leading to more faithful and localized attributions <ref:2603.13652#pg1>.
Jane: It gives us a new tool for interpretability that moves from looking at inputs or attention patterns to actually probing the internal representations that the model is using for classification.
The paper's summary: Tom: So, to recap, this paper is proposing a way to find out which part of an image matters for an AI model's decision by directly poking at the internal representations instead of using some kind of approximation or mask.
Jane: Right, and they do that by inserting the actual activations from a specific patch into a neutral context over several layers and seeing how that changes the final prediction score.
Lu: The big idea here is moving past looking at things like attention weights or simple input changes, because those are just indirect clues about what's important in the model's deep layers.
Meng: So, instead of guessing where a patch is important based on some learned score, they are testing the actual causal effect of that patch's internal signal on the final outcome.
Tom: Exactly. They call it Causal Attribution via Activation Patching, and they argue this lets them capture the true causal contribution of those representations as they form through self-attention across the layers.
Jane: It’s a big shift because it tries to connect what happens deep inside the neural network—those patches interacting across many layers—to an interpretable map we can actually see on an image.
Lu: They’re careful about how many layers they patch, though. They found that patching too deep into the model causes the spatial information to get washed out by things from other parts of the scene, which is a problem in itself.
Tom: So they settled on patching up to around two thirds of the layers because that's where you still have strong object-level information while it's not getting too mixed with background stuff.
Jane: That sounds like a practical compromise, balancing capturing the object's identity with keeping it spatially specific.
Lu: And they made things practical by suggesting ways to speed this up massively, like using precomputed blank context and parallel classification tokens to make the whole process much faster for big Vision Transformers.
Meng: Because I’m thinking about deployment, if an attribution method takes too long to run on a large model, it just isn't going into production. Speed matters when you’re running these kinds of analyses on real traffic data.
Tom: Yeah, and they found that with that precomputation trick, the cost per sample drops from something like N cubed operations down to roughly N operations for an image with N tokens.
Jane: That’s a huge win for scaling this kind of analysis because it makes it feasible to run these kinds of detailed causal tests on much larger models than we could before.
Lu: The results they're showing, especially on ImageNet-1K, show that this method consistently gets lower Deletion AUC and higher Insertion AUC compared to existing methods <ref:2603.13652#pg1>.
Tom: And what that means is that the regions they rank as important by their attribution maps actually reflect the model’s true decision-making process more accurately.
Jane: That's a big deal for debugging, because if the map is accurate, you can actually trust it to figure out why an AI made a certain choice.
Lu: They also found that these localized regions are generally more compact and coherent, which means the attribution maps aren't just scattered noise; they’re actually pointing to meaningful object areas.
Meng: So, this isn't just a theoretical improvement in accuracy metrics; it’s producing maps that are useful for actual debugging workflows where you need to pinpoint a specific object or region.
Tom: And they even checked convolutional networks like ResNet and VGG, and CAAP still achieved the best overall results in most cases when compared to other intervention methods on those architectures.
Jane: It seems like this approach is robust across different model types, which makes it more of a general tool for interpreting Vision Transformers than something specific to just one architecture.
Lu: The authors also pointed out that they don't need to be super picky about the exact number of layers they patch; the performance stayed stable across a broad range of intermediate layers.
Tom: That stability is good for us because it means when we move to new, bigger models, we don't have to redesign the entire intervention strategy from scratch just because the layer count changes a little.
Jane: It gives researchers a lot more flexibility in choosing where they want to look inside the model's thinking process.
Lu: So, what this paper really brings to the table is a principled causal framework for attribution that moves us away from indirect proxies and toward directly testing how internal representations affect predictions.
The paper's improvements: Tom: So, after showing us their core method, they also suggest some practical improvements to make it actually work in the real world.
Jane: What kind of improvements are we talking about now? Are they just tweaking the math, or are they changing how it’s actually run?
Lu: They focus a lot on making sure the selection operator S of pi doesn't just look at one single patch, but includes its immediate spatial neighbors.
Tom: So they broaden the scope of what gets patched; instead of just one token, they consider that whole little area to capture more feature spread across the image.
Jane: That makes sense because a single patch can only tell you about itself in isolation, so including neighbors helps get a better picture of the local context.
Lu: And they also looked at where they should stop patching layers; instead of picking one fixed number, they found that performance stays stable across a whole range of intermediate layers.
Tom: That’s good for deployment because it means we don't have to be super precise about that layer cutoff when we move to new models later on.
Jane: And then they tackled the speed issue again by proposing precomputing blank context attention, which lets them reuse activations from a blank image across different samples.
Lu: That’s the efficiency boost; it turns what could be a very slow process into something that scales much better, with the cost per sample dropping to roughly N operations.
Tom: So they are basically trying to make this method fast enough for actual use on large models, which is a huge step for any practical application.
Jane: And they also mentioned that when comparing the different types of blank context or selection operators, the faithfulness of their results doesn't really change much, as long as you use that padded version with neighbors.
Lu: They also pointed out that when looking at how information flows through layers, the most important stuff is actually happening in those early and middle layers where things are still grouping objects together.
Tom: So they’re confirming that focusing on those specific layer ranges is the right move because it aligns with how the model actually builds its understanding of objects.
Jane: It seems like this paper isn't just about finding a better score; it’s about creating a more principled and efficient way to probe the internal workings of these massive models.
Lu: They are moving us toward methods that test causal contributions directly, rather than relying on those indirect signals from attention weights or input perturbations.
Conclusion: Tom: So we’re wrapping up this paper on Causal Attribution via Activation Patching by summarizing what it actually means for us in practical terms and saying goodbye to it.
Jane: Right, so at its heart, this is a method that moves past just looking at input changes or attention patterns to directly testing the causal effect of an internal patch representation on the final prediction score.
Lu: The authors have shown that by intervening on these activations across several layers and using those clever parallelization tricks with blank context, we can get very faithful and localized attribution maps.
Meng: For us in production, that means we get a much more reliable signal for debugging what’s actually causing the model to make a mistake, instead of just guessing based on where it looked.
Lalam: I’m confident in saying that this approach provides a principled way to measure patch influence by directly testing the predictive effect of those representations.
Tom: It really does shift us from looking at external evidence or early model variables to probing internal representations that are closer to the final decision computation.
Jane: And they found it works across different Vision Transformer backbones, even showing better results than other intervention methods on ImageNet-1K, especially with deletion and insertion metrics <ref:2603.13652#pg1>.
Lu: The result is maps that are more compact and coherent, which means they aren't just diffuse signals; they align closely with the actual object regions we see.
Meng: That’s what I care about—if the map is accurate, it means we can actually trust it for debugging or auditing when dealing with complex vision systems in sensitive areas.
Tom: So, this paper on Causal Attribution via Activation Patching gives us a new tool for interpretability that lets us understand *why* an AI makes a decision by looking inside the model’s layers.
Jane: It moves us past relying on indirect evidence and shows how to use causal intervention to get cleaner, more semantically meaningful explanations.
Lu: The authors also pointed out the limitations: they said they didn't test every single layer depth or every possible selection operator, but even with those caveats, the method holds up well.
Meng: So we can use this as a solid starting point for our next experiments instead of trying to tweak every single parameter just to get a tiny bit better performance on one metric.
Tom: Exactly. It’s about building an attribution framework that is both accurate and efficient enough for real-world engineering challenges.
Jane: This work opens up a new direction for researchers who want to understand the causal pathway of information flow within these large vision models.
Sharif University of Technology
cs.CV
Submitted: 2026-03-13
Updated: 2026-11-10
Comments: Accepted at NeurIPS 2026
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
Importance score: 92/100
The gist: Attribution methods for Vision Transformers (ViTs) aim to identify image regions that influence model predictions…
Key concepts
- Activation Patching
- CAAP works by taking the actual internal feature activations from a source image's patches and inserting them into a neutral target context over several layers of the ViT. This simulates what happens when a specific patch's information is present, allowing researchers to see how that specific patch representation affects the final prediction score.
- Causal Attribution
- This approach aims to establish a direct causal link between an image patch and the model's output. Unlike indirect methods like attention weights, CAAP tests the predictive effect of the patch's internal features after they have been processed through multiple layers, providing a principled measure of influence.
- Pruning Redundant Reasoning Tokens
- The method considers spatial feature spread by including immediate neighbors when patching a single patch. This ensures that the intervention captures not just the isolated patch but also its surrounding context, leading to more faithful and localized measures of contribution.
Terminology
Summary
Attribution methods for Vision Transformers (ViTs) aim to identify image regions that influence model predictions, but producing faithful and well-localized attributions remains challenging The key challenge is that class-relevant evidence is formed through interactions between patch tokens across layers; methods that operate only on input changes, attention weights, or backward relevance signals may therefore provide indirect proxies for patch importance rather than directly testing the predictive effect of contextualized patch representations We propose Causal Attribution via Activation Patching (CAAP), which estimates the contribution of individual image patches to the ViT’s prediction by directly intervening on internal activations rather than using learned masks or synthetic perturbation patterns The resulting attribution map reflects the causal contribution of patch-associated internal representations on the model’s prediction. p. 0 The key
challenge is that class-relevant evidence is formed through interactions between patch
tokens across layers; methods that operate only on input changes, attention weights,
or backward relevance signals may therefore provide indirect proxies for patch
importance rather than directly testing the predictive effect of contextualized patch
representations. We propose Causal Attribution via Activation Patching (CAAP),
which estimates the contribution of individual image patches to the ViT’s prediction
by directly intervening on internal activations rather than using learned masks or
synthetic perturbation patterns.Where the paper says it
Pruning redundant reasoning tokens
Existing attribution methods face several limitations, with gradient-based, relevance-propagation, and attention-based methods relying on local approximations The key challenge is that class-relevant evidence is formed through interactions between patch tokens across layers; methods that operate only on input changes, attention weights, or backward relevance signals may therefore provide indirect proxies for patch importance rather than directly testing the predictive effect of contextualized patch representations. Gradient-based and relevance-propagation methods provide efficient class-specific signals, but they measure local sensitivity or propagate relevance through the computation graph rather than directly testing whether a contextualized patch representation is sufficient to support the prediction. p. 1 This creates limitations for several attribution families. Gradient-based and relevance-propagation
methods provide efficient class-specific signals, but they measure local sensitivity or propagate relevance
through the computation graph rather than directly testing whether a contextualized patch representation
is sufficient to support the prediction [7–10, 12]. Gradient-based and relevance-propagation
methods provide efficient class-specific signals, but they measure local sensitivity or propagate relevance
through the computation graph rather than directly testing whether a contextualized patch representation
is sufficient to support the prediction [7–10, 12]. Attention-based methods expose token-mixing patterns,
but attention weights or attention flow are indirect proxies for feature contribution, especially after
information from different tokens has been repeatedly mixed across layers [13–15]. Attention-based methods expose token-mixing patterns,
but attention weights or attention flow are indirect proxies for feature contribution, especially after
information from different tokens has been repeatedly mixed across layers [13–15]. Perturbation and masking methods measure causal output changes under interventions, but input-space
perturbations can introduce distribution shift [17–20]. Perturbation and masking methods measure causal output changes under interventions, but input-space
perturbations can introduce distribution shift [17–20]. Input-based deletion does not account for
redundancy across patches: when multiple patches encode similar class evidence, removing each patch
from a group of redundant patches may cause little change in the output, leading to underestimation of
the group’s importance.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
How the training signal is built
For each patch, CAAP inserts the corresponding source-image activations into a neutral target context over an intermediate range of layers and uses the resulting target-class score as the attribution signal The patched context c∗(pi) is constructed by replacing activations as h l(c∗(pi))S(pi) = h l(x)S(pi), hl l(c∗(pi))¬S(pi) = h l(x0)¬S(pi), ∀l ∈ L The attribution score for patch pi with respect to class y is defined as the prediction score obtained from the patched context, A(pi) = f(c∗(pi))y. An important property of activation patching, compared to input-based perturbation methods, is that the patched activations are taken directly from the original context, so they already encode interactions between patches and avoid introducing OOD inputs in the source context. p. 2 Repeating
this procedure for all patches yields a patch-level attribution map over the input image. An important property of activation patching, compared to input-based perturbation methods, is that
the patched activations are taken directly from the original context, so they already encode interactions
between patches and avoid introducing OOD inputs in the source context.Where the paper says it
Where the gains come from
The causal intervention serves as a principled measure of patch influence by capturing semantic evidence after initial representation formation, while avoiding late-layer global mixing that can reduce spatial specificity CAAP consistently outperforms existing methods in various settings and produces more faithful and localized attributions. Empirical results across multiple Vision Transformer backbones show that our method improves over existing approaches on most evaluated settings, achieving state-of-the-art results on deletion, insertion, and foreground and background localization metrics. p. 1 The procedure can be efficiently parallelized, enabling
practical patch-level attribution for large ViT backbones. Empirical results across multiple Vision
Transformer backbones show that our method improves over existing approaches on most evaluated
settings, achieving state-of-the-art results on deletion, insertion, and foreground and background
localization metrics. Localization results are reported in Table 2. CAAP achieves the best overall performance across most backbones.Where the paper says it
Where the paper says it
In the naive implementation, the patch selection operator S(pi) corresponds exactly to the patch pi However, patching a single token often does not transfer sufficient predictive information from the source context To account for spatial feature spread across neighboring tokens, we consider a padded variant in which S(pi) includes the patch pi and its immediate spatial neighbors This choice preserves both contextual information and spatial locality, making the patch-level intervention a more faithful measure of localized contribution.
ViTs build patch representations through repeated self-attention, causing each patch token to encode information aggregated from other patches In a single attention head, the attention update for patch i can be written as z l+1 i = X N / N j=1 α l ijv l j, (4) where v l j are value vectors derived from the token representations and α l ij are attention weights defined by α l ij = exp ⟨q li, klj⟩/√d m=1 exp ⟨q li, klm⟩/√d. We empirically show that in the early layers, where intra-object grouping occurs, the quantity E j∈O[α l ij] − E j /∈O[α l ij] increases, while after the middle layers it decreases. p. 4 The attention update can be
decomposed as
z˜
l+1
i =
X
j∈O
α
l
ijv
l
j +
X
j /∈O
α
l
ijv
l
j
. (6)
We empirically show that in the early layers, where intra-object grouping occurs, the quantity
Ej∈O[α
l
ij ] − Ej /∈O[α
l
ij ] increases, while after the middle layers it decreases. (6)
We empirically show that in the early layers, where intra-object grouping occurs, the quantity
Ej∈O[α
l
ij ] − Ej /∈O[α
l
ij ] increases, while after the middle layers it decreases. This enrichment of
intra-object grouping supports the motivation for layer-wise activation patching rather than input-level
insertion.Where the paper says it
Where the paper says it
We next propose two practical techniques to further accelerate the intervention process: parallel computation of CLS tokens and precomputation of blank-context attention This parallel construction of CLS tokens yields the same attribution map (no approximation) while computing all patch scores in parallel. The total cost per sample is one forward pass for the source image and one forward pass with multiple CLS tokens. p. 4 This parallel construction of CLS tokens yields the same attribution map (no approximation) while
computing all patch scores in parallel. The total cost per sample is one forward pass for the source image
and one forward pass with multiple CLS tokens.Where the paper says it
We next propose two practical techniques to further accelerate the intervention process: parallel computation of CLS tokens and precomputation of blank-context attention For the blank image x0, we perform one forward pass and cache its activations h l(x0), l∈L, reused across samples. We then build a patched computation containing both activation sets. p. 4 For each source image x, we run a standard forward pass and cache {h
l
(x)}l∈L. We
then build a patched computation containing both activation sets. For each patch pi
, we introduce
a dedicated classification token CLSi with the same positional encoding as the original CLS. The
attention mask ensures that CLSi attends to source activations at S(pi) and to target activations at
¬S(pi).Where the paper says it
Where the paper says it
We therefore precompute two quantities for each layer and attention head using the blank context. First, the normalization contribution of blank tokens, Z l 0 = X j /∈S(pi) exp(⟨q c′CLS, kl 0 j⟩/√d), (8) which is reused as a fixed term in the softmax denominator. p. 1 CLS
Selected Token Activations
Source Context
Classification
Head
cat
Softmax
Probability
Repeat for each
patch to form
attribution map
CLS
Target Context
CLS
Patched Activations
Patched Context
Forward Pass
𝒍𝒆
Forward Pass
Figure 1: Overview of CAAP. Given a source image (top) and a blank target image (middle), we extract
internal activations from the source image corresponding to a selected patch and a specified range
of layers. Insertion into a blank or blurred image overlooks interactions among patches
and may fail to reproduce the functional role a patch plays in the full image context. More ViT-specific
perturbation or optimization methods operate over tokens, attention mechanisms, patch embeddings,
or attribution metrics [21–23], but they still do not directly evaluate the class evidence carried by the
original internal patch representation within a controlled target context. To avoid
recomputing these vectors from hidden states during activation patching, we reuse cached key and
value projections associated with the intervened hidden states, rather than recomputing them from
residual streams. Only the query vectors of the additional CLS tokens are computed dynamically in
the patched context.Where the paper says it
Where the paper says it
Where the paper says it
This design allows CAAP to better isolate the causal contribution [25, 26] of class-relevant patch representations to the final prediction. It derives patch scores directly from information already encoded in contextualized token representations, rather than relying on indirect evidence such as attention weights, approximate gradient signals, input-space masks, or metric-optimized surrogate maps. p. 1 Causal intervention offers a practical advantage for CAAP over existing attribution techniques. It derives
patch scores directly from information already encoded in contextualized token representations, rather
than relying on indirect evidence such as attention weights, approximate gradient signals, input-space
masks, or metric-optimized surrogate maps. It derives
patch scores directly from information already encoded in contextualized token representations, rather
than relying on indirect evidence such as attention weights, approximate gradient signals, input-space
masks, or metric-optimized surrogate maps. The procedure can be efficiently parallelized, enabling
practical patch-level attribution for large ViT backbones.Where the paper says it
Where the paper says it
We next propose two practical techniques to further accelerate the intervention process: parallel computation of CLS tokens and precomputation of blank-context attention This approximation substantially reduces computational complexity. A naive implementation requires approximately O(N 3) operations for an image with N tokens, due to N patch interventions combined with N × N attention interactions With precomputation, the contribution of blank tokens becomes a constant term (O(1)), and only the N patch interventions are evaluated dynamically, reducing the overall cost to approximately O(N). p. 4 We next propose two practical techniques to further accelerate the intervention
process: parallel computation of CLS tokens and precomputation of blank-context attention. The patch-wise formulation requires one forward pass per patch, which scales linearly with the number
of tokens in the image.Where the paper says it
In addition to gradient-based attribution, attention-based attribution, and perturbation-based attribution and masking in ViTs, CAAP measures the effect of patch-associated representations on the model output through an explicit activation intervention. It complements masking- and sampling-based perturbation methods by moving the intervention from the input or token level to the representation level It is also distinct from metric-driven approaches that optimize attribution maps to fit evaluation criteria. p. 1 It derives
patch scores directly from information already encoded in contextualized token representations, rather
than relying on indirect evidence such as attention weights, approximate gradient signals, input-space
masks, or metric-optimized surrogate maps. The procedure can be efficiently parallelized, enabling
practical patch-level attribution for large ViT backbones. The key
challenge is that class-relevant evidence is formed through interactions between patch
tokens across layers; methods that operate only on input changes, attention weights,
or backward relevance signals may therefore provide indirect proxies for patch
importance rather than directly testing the predictive effect of contextualized patch
representations. We propose Causal Attribution via Activation Patching (CAAP),
which estimates the contribution of individual image patches to the ViT’s prediction
by directly intervening on internal activations rather than using learned masks or
synthetic perturbation patterns. The key
challenge is that class-relevant evidence is formed through interactions between patch
tokens across layers; methods that operate only on input changes, attention weights,
or backward relevance signals may therefore provide indirect proxies for patch
importance rather than directly testing the predictive effect of contextualized patch
representations. We propose Causal Attribution via Activation Patching (CAAP),
which estimates the contribution of individual image patches to the ViT’s prediction
by directly intervening on internal activations rather than using learned masks or
synthetic perturbation patterns.Where the paper says it
Where the paper says it
Where the paper says it
CAAP consistently achieves the lowest Deletion AUC and the highest or competitive Insertion AUC on all backbones. In particular, CAAP improves the Insertion−Deletion with a notable margin compared to prior methods, indicating that the regions ranked as important by our attribution maps more accurately reflect the model’s true decision-making process Localization metrics such as AUPR1, AUPR0, and Pointing Game also improve, indicating that patch-level interventions better align with object regions rather than highlighting only a few highly discriminative patches. p. 5 In
particular, CAAP improves the Insertion−Deletion with a notable margin compared to prior methods,
indicating that the regions ranked as important by our attribution maps more accurately reflect the
model’s true decision-making process. Localization. As illustrated, baseline methods often produce diffuse
or fragmented attribution maps that extend beyond the target object into the background, or highlight
only a small subset of discriminative patches. In contrast, CAAP tends to generate more compact,
coherent, and semantically meaningful attributions that align closely with the target object.Where the paper says it
Where the paper says it
We ablate three design choices in CAAP on ImageNet: (i) the type of the target’s blank patches, (ii) the type of the selection operator, and (iii) the choice of intervened layers Fig. 4a shows that faithfulness is largely insensitive to this choice, as Insertion, Deletion, and Insertion−Deletion remain close across blank types. p. 7 Gaussian noise to the target and then applies a blur filter. 4a shows that faithfulness is largely
insensitive to this choice, as Insertion, Deletion, and Insertion−Deletion remain close across blank
types. 4b compares
four neighborhood variants. The no-padding variant performs the worst, while all padded alternatives
improve both faithfulness and localization.Where the paper says it
Where the paper says it
To analyze the effect of layer depth, we first examine attention patterns that govern information flow (Section 3.2) Fig. 5a shows that E j∈O[α l ij] − E j /∈O[α l ij] peaks in the middle layers. p. 7 To analyze the effect of layer depth, we first examine attention patterns
that govern information flow (Section 3. 5a shows that Ei
[Ej∈O[α
l
ij ] − Ej /∈O[α
l
ij ]] peaks in the
middle layers. Around the middle layers, this gap reaches its peak. In
later layers, intra-object attention decreases while object-to-background attention increases, reflecting
growing extra-object contextualization and reduced spatial specificity.Where the paper says it
Where the paper says it
The performance (particularly Insertion AUC) remains stable across a broad range of intermediate layers, indicating that the method is not highly sensitive to the exact cutoff choice (results on other backbones are provided in Appendix D.1). This choice balances intra-object grouping and extra-object contextualization and performs near-optimally across models. p. 7 2, we fix le = ⌈
2
3
L⌉ as the default setting for all architectures. This choice balances
intra-object grouping and extra-object contextualization and performs near-optimally across models.Where the paper says it
We further evaluate the intervention strategies on convolutional architectures, including ResNet-18, ResNet-50 [41], DenseNet-161 [42], and VGG-16 [43]. Compared with the transformer results reported earlier, the overall performance of intervention-based methods is lower across both faithfulness and localization metrics One possible explanation is that transformer architectures are generally more robust to distribution shifts than convolutional networks. Within the CNN architectures, CAAP still achieves the best overall results in most cases. p. 5 CAAP achieves the best overall performance across most backbones. In particular, the higher PG and AUPR1 values indicate that the regions
identified as most relevant are consistently located within the annotated foreground areas.Where the paper says it
In contrast to gradient- and relevance-propagation methods, which trace local sensitivity or backward relevance through the computation graph, CAAP directly intervenes on the forward activations of contextualized patch tokens It complements masking- and sampling-based perturbation methods by moving the intervention from the input or token level to the representation level CAAP can be viewed as an interventional attribution approach that assigns relevance by replacing internal activations and measuring the resulting target-class score.
CAAP shifts the intervention site for ViT attribution from external or early model variables to internal representations that are closer to the model’s final decision computation It complements attention-derived explanations by treating attention as part of the computation that forms the intervened states rather than as the explanatory signal itself CAAP derives each score from an explicit intervention within the classifier itself, without learning an external explainer, mask generator, or metric-specific surrogate. The key distinction is that CAAP directly tests the predictive effect of intermediate residual representations associated with image patches.
The overall, CAAP consistently outperforms both input-based baselines As shown in Table 13, CAAP achieves substantially lower Deletion AUC and higher Insertion AUC, leading to much larger Insertion−Deletion scores across all architectures. Localization metrics in Table 14 show the same pattern CAAP yields higher AUPR1, AUPR0, and PG on most models, indicating better alignment with object regions. p. 8 Empirically, CAAP improves faithfulness across multiple backbones, including CLIP, DINOv2, ViT,
and DeiT3, by achieving lower Deletion AUC and higher Insertion AUC. Localization metrics such
as AUPR1, AUPR0, and Pointing Game also improve, indicating that patch-level interventions better
align with object regions rather than highlighting only a few highly discriminative patches.Where the paper says it
The results suggest that manipulating pixels alone does not reconstruct the internal evidence used by the model In contrast, intervening on activations across layers preserves contextualized patch representations formed through self-attention, which leads to more faithful and spatially consistent attribution.
This work contributes to interpretability and attribution methods for Vision Transformers by proposing a causal intervention framework based on activation patching Improved attribution quality may help researchers better understand model behavior, identify reliance on spurious correlations, and support debugging and auditing of vision systems in sensitive applications. The paper introduces a new attribution method but does not release new datasets or generative models. p. 2 reliable and effective approach to spatial attribution in ViTs, offering a new perspective based on
mechanistic interpretability tools. 2 Preliminaries
2. ∗Equal contribution. arXiv:2603.Where the paper says it
Where the paper says it
All experiments use publicly available datasets and pretrained models, including ImageNet-1K, ImageNet-S, Oxford-IIIT-Pet, Food-101, CLIP, DINOv2, DeiT3, and ViT models The paper introduces a new attribution method but does not release new datasets or generative models.
The paper is 2603.13652 The paper introduces a new attribution method but does not release new datasets or generative models. p. 2 reliable and effective approach to spatial attribution in ViTs, offering a new perspective based on
mechanistic interpretability tools. 2 Preliminaries
2. reliable and effective approach to spatial attribution in ViTs, offering a new perspective based on
mechanistic interpretability tools. 2 Preliminaries
2. reliable and effective approach to spatial attribution in ViTs, offering a new perspective based on
mechanistic interpretability tools. 2 Preliminaries
2.Where the paper says it
Where the paper says it
Where the paper says it
The paper is 2603.13652 The paper introduces a new attribution method but does not release new datasets or generative models.
The paper is 2603.13652 The paper introduces a new attribution method but does not release new datasets or generative models.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Causal Attribution via Activation Patching (CAAP) directly intervenes on internal activations rather than using learned masks or synthetic perturbation patterns to estimate patch contribution. This allows it to capture
the causal contribution of patch-associated internal representations on the model’s prediction
by inserting source-image activations into a neutral target context. -
CAAP is designed to avoid late-layer global mixing that reduces spatial specificity by applying activation patching up to intermediate layers, specifically selecting layers up to 2/3 L, where
patch representations contain strong object-level information while preserving spatial specificity.
-
The method can be parallelized by computing all patch attribution scores in parallel within a single forward pass using dedicated classification tokens (CLS) with attention masks, reducing the total cost per sample to approximately O(N).
-
CAAP provides more faithful and localized attributions than existing methods, consistently achieving
lower Deletion AUC and higher Insertion AUC
across multiple ViT backbones on ImageNet-1K. -
The method produces more compact and coherent attribution maps, evidenced by consistently lower entropy and higher Gini scores compared to baselines. This indicates that the attribution is
more compact and coherent,
better aligning with object regions rather than being diffuse.
Abstract
Attribution methods for Vision Transformers (ViTs) aim to identify image regions that influence model predictions, but producing faithful and well-localized attributions remains challenging. Existing attribution methods face several limitations, with gradient-based, relevance-propagation, and attention-based methods relying on local approximations, while perturbation- or optimization-based methods intervene on inputs, tokens, or surrogates rather than internal patch representations. The key challenge is that class-relevant evidence is formed through interactions between patch tokens across layers; methods that operate only on input changes, attention weights, or backward relevance signals may therefore provide indirect proxies for patch importance rather than directly testing the predictive effect of contextualized patch representations. We propose Causal Attribution via Activation Patching (CAAP), which estimates the contribution of individual image patches to the ViT's prediction by directly intervening on internal activations rather than using learned masks or synthetic perturbation patterns. For each patch, CAAP inserts the corresponding source-image activations into a neutral target context over an intermediate range of layers and uses the resulting target-class score as the attribution signal. The resulting attribution map reflects the causal contribution of patch-associated internal representations on the model's prediction. The causal intervention serves as a principled measure of patch influence by capturing semantic evidence after initial representation formation, while avoiding late-layer global mixing that can reduce spatial specificity. Across multiple ViT backbones and standard metrics, CAAP consistently outperforms existing methods in various settings and produces more faithful and localized attributions.
Sources
- How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers
- Learning Transferable Visual Models From Natural Language Supervision
- DINOv2: Learning Robust Visual Features without Supervision
- DeiT III: Revenge of the ViT
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Not Just a Black Box: Learning Important Features Through Propagating Activation Differences
- AtMan: Understanding Transformer Predictions Through Memory Efficient Attention Manipulation
- ViT-CX: Causal Explanation of Vision Transformers
- What's the Point: Semantic Segmentation with Point Supervision
- ImageNet Large Scale Visual Recognition Challenge
- ImageNet-Hard: The Hardest Images Remaining from a Study of the Power of Zoom and Spatial Biases in Image Classification
- SmoothGrad: removing noise by adding noise
- Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks
- Very Deep Convolutional Networks for Large-Scale Image Recognition
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs