Does Target Alignment Mean Target Recovery? An Evidence-Ladder Study of Adversarial Claims on Contrastive Encoders

arXiv:2610.11938 · cs.CV · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Does Target Alignment Mean Target Recovery? An Evidence-Ladder Study of Adversarial Claims on Contrastive Encoders".

Tom: The gist The score—victim-space target alignment (VTS)—predicts recovery of the target by an independent model only within a fixed robust encoder, leaving practical recovery undecided.

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So let's start with "Does Target Alignment Mean Target Recovery? An Evidence-Ladder Study of Adversarial Claims on Contrastive Encoders." The authors are Tao Yang and Jianying Zhou from Singapore University of Technology and Design. They are trying to figure out if just because a model aligns with an attacked text target means that the model can actually recover the original image.

Jane: Right, so what they found is pretty nuanced. It seems the simple idea of victim-space alignment isn't enough on its own to guarantee recovery by an independent judge >

Lu: They build this measurement instrument using real target blends and shuffled negatives to define a reference level based on where a fifty percent blend lands for those judges >

Meng: And they found that no single cell ever actually reaches that reference level even when the attack budget is pushed really high, like epsilon sixty-four over two hundred fifty-five >

Tom: That means practical recovery stays undecided, doesn't it? Even with these robustly trained encoders, the alignment score isn't a perfect predictor of real target evidence transfer >

Jane: The paper introduces this three-level taxonomy—L0 intent, L1 victim-space alignment, and L2 independent recovery—to organize all this stuff >

Lu: L1 is that cosine similarity to the frozen target text embedding in the victim model, while L2 is the shuffle-corrected change in the logit margin on an independent judge >

Tom: And they found that within a robust encoder cell, alignment gain predicts evidence gain with a rho of zero point two four to zero point five one, but not in vanilla CLIP at all >

Jane: That tells us that just because the model is robustly fine-tuned doesn't mean its alignment score reliably predicts how much independent target evidence it will get >

Meng: It sounds like we need to look at the actual evidence transfer rate rather than just the alignment score when we’re evaluating these models >

Tom: Exactly. And they also found that between different victim models in Study B, the ranking of victim-space similarity actually inverted compared to the independent recovery ranking with a rho of negative one point zero >

Jane: That cancellation across all nine cells suggests that while alignment might look good in isolation, it doesn't translate into consistent recovery performance when you compare different setups >

Lu: The authors then propose a reporting protocol for adversarial claims, suggesting we need to scope alignment claims to the specific encoder and report at least one validated independent endpoint with graded controls >

Tom: They also stress that for robust encoders, you have to evaluate them right at their training budget and just beyond it to get the full picture >

Jane: What this means for us is that we shouldn't rely on a single metric like victim-space alignment when deciding if a model is safe against these kinds of attacks >

Lu: This study highlights that robust features are better at transferring independent evidence, but even then the alignment score doesn't guarantee successful recovery across different setups >

The paper's summary: Tom: We just talked about how they built this whole measurement system—this evidence ladder—to see if just aligning a model to a target text actually means that the model can recover that real image.

Jane: Right, and what they found is pretty nuanced. It seems the simple idea of victim-space alignment isn't enough on its own to guarantee recovery by an independent judge >

Lu: Basically, they set up this rigorous test with real target blends and shuffled negatives to define a reference level based on where a fifty percent blend lands for those judges >

Meng: And they found that no single cell ever actually reaches that reference level even when the attack budget is pushed really high, like epsilon sixty-four over two hundred fifty-five >

Tom: That means practical recovery stays undecided, doesn't it? Even with these robustly trained encoders, the alignment score isn't a perfect predictor of real target evidence transfer >

Jane: The paper introduces this three-level taxonomy—L0 intent, L1 victim-space alignment, and L2 independent recovery—to organize all this stuff >

Lu: L1 is that cosine similarity to the frozen target text embedding in the victim model, while L2 is the shuffle-corrected change in the logit margin on an independent judge >

Tom: And they found that within a robust encoder cell, alignment gain predicts evidence gain with a rho of zero point two four to zero point five one, but not in vanilla CLIP at all >

Jane: That tells us that just because the model is robustly fine-tuned doesn't mean its alignment score reliably predicts how much independent target evidence it will get >

Meng: It sounds like we need to look at the actual evidence transfer rate rather than just the alignment score when we’re evaluating these models >

Tom: Exactly. And they also found that between different victim models in Study B, the ranking of victim-space similarity actually inverted compared to the independent recovery ranking with a rho of negative one point zero >

Jane: That cancellation across all nine cells suggests that while alignment might look good in isolation, it doesn't translate into consistent recovery performance when you compare different setups >

Lu: The authors then propose a reporting protocol for adversarial claims, suggesting we need to scope alignment claims to the specific encoder and report at least one validated independent endpoint with graded controls >

Tom: They also stress that for robust encoders, you have to evaluate them right at their training budget and just beyond it to get the full picture >

Jane: What this means for us is that we shouldn't rely on a single metric like victim-space alignment when deciding if a model is safe against these kinds of attacks >

Lu: This study highlights that robust features are better at transferring independent evidence, but even then the alignment score doesn't guarantee successful recovery across different setups >

The paper's improvements: Tom: So we just looked at what the study found about alignment versus recovery, and now they’re talking about how we should actually use this kind of evidence in practice.

Jane: They're proposing a reporting protocol for adversarial claims, which basically means that if you want to claim something about an attacked model, you have to follow these steps.

Lu: They suggest scoping the alignment claims specifically to the encoder you are evaluating and reporting at least one validated independent endpoint with graded real-target positive controls and shuffled-target negatives >

Meng: That sounds like a way to make sure we aren't just looking at a single, potentially misleading score from an attack run.

Tom: And they say for robust encoders, you have to evaluate them both at their training budget and just beyond it to get the full picture of their performance >

Jane: It’s about moving beyond just victim-space target alignment as a standalone metric so we actually get actionable evidence instead >

Lu: They also want us to establish that measurable reference level based on a fifty percent target-image blend to see if an adversarial claim passes the L2 evidence test, rather than relying on subjective scores >

Tom: That means deployment should only happen when the mean independent evidence meets or exceeds that specific reference level >

Jane: What this changes for us is that we can finally start distinguishing between alignment that looks good on paper and actual recovery happening in real conditions >

Lu: This approach helps us select which types of models are fundamentally more reliable when they are being tested under these kinds of adversarial scenarios >

Tom: It sounds like they’re giving engineers a much clearer way to audit model robustness rather than just relying on surface-level metrics >

Jane: And it seems like the next big step is developing this reporting protocol into a community benchmark that covers more attack families and different encoder architectures >

Conclusion: Tom: So to wrap up, this paper, "Does Target Alignment Mean Target Recovery? An Evidence-Ladder Study of Adversarial Claims on Contrastive Encoders," shows that victim-space alignment isn't a guarantee for independent target recovery by an outside judge >

Jane: Basically, they built this evidence ladder and found that without specific controls like graded blends, the alignment score alone doesn't tell you if the model is actually doing its job >

Lu: The main thing is that robust features are better at transferring independent evidence than non-adversarially trained ones, but even then there’s a gap at the reference level >

Meng: From an engineering side, it confirms that we need to be much more careful when we choose which models we trust based only on their alignment scores during adversarial testing >

Lalam: It means that for AI culture, this study pushes us to move past easy metrics and focus on verifiable evidence of actual capability >

Tom: They propose a reporting protocol where you have to scope the claims to the encoder and show at least one validated independent endpoint with graded controls >

Jane: That’s smart because it forces researchers not just to report a number, but to show how that number was measured against real conditions >

Lu: The study is useful for understanding the internal mechanics of these encoders and how their features transfer evidence under pressure across different training regimes >

Meng: It helps us see where the model stops working, which is important when we’re trying to build reliable systems in the field >

Lalam: I think this kind of deep diagnostic work is what will help large language models become more trustworthy as they get integrated into our daily lives >

Tom: So that’s the gist of it, a lot of nuance about alignment versus actual recovery in contrastive encoders.

Jane: Yeah, it really highlights that the way we measure success needs to be tied directly to what happens on independent judges under controlled conditions >

Lu: It opens up new avenues for developing better benchmarking systems that look at both L1 and L2 evidence simultaneously >

Meng: I'm looking forward to seeing how this reporting protocol gets adopted by the wider community for stress-testing these vision models.

Tao Yang, Jianying Zhou

Singapore University of Technology and Design

cs.CV

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: 15 pages

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 87/100

The gist: The gist The score—victim-space target alignment (VTS)—predicts recovery of the target by an independent model only within a fixed robust encoder, leaving practical recovery undecided.

Key concepts

Victim-Space Target Alignment (VTS)
This is a measurement of how similar the attacked image is to the original target text embedding within the victim model itself. It serves as an initial, internal indicator of alignment after an attack has occurred.
Independent Target Evidence (ITE)
This measures whether an independent model can recover the original target. It is calculated by correcting for shuffling and measuring how much a judge's prediction margin changes when comparing the attacked image to the real target.
Evidence-Ladder Taxonomy
This framework organizes claims about adversarial attacks into three levels: L0 (target intent), L1 (victim-space alignment/VTS), and L2 (independent recovery/ITE). This helps researchers systematically assess the quality and type of evidence presented in attack reports.

Terminology

Summary

The gist The score—victim-space target alignment (VTS)—predicts recovery of the target by an independent model only within a fixed robust encoder, leaving practical recovery undecided.

How it works

Adversarial attacks on vision–language models optimize an image toward a text target, then cite the attacked model’s similarity score as evidence of success The practice is widespread because the quantity is cheap to compute, directly optimized, and available to any white-box attack But it is computed by the same encoder being optimized, so it rises under attack regardless of whether any other model can recognize the intended target.

The study builds a measurement instrument that pairs supervised judges outside the attacked geometry with graded real-target blend controls and shuffled-target negatives, anchored to a reference level derived from a 50% target-image blend. Two preregistered studies then compare six contrastive encoders under a matched attack at three perturbation budgets.

Key Findings and Evidence Taxonomy

Three main results emerged from the comparison of encoders. First, robustly fine-tuned encoders transfer more independent target evidence than non-adversarially trained ones, with all eight robust–versus–naive contrasts rejecting at the bootstrap floor Second, no cell’s mean evidence reaches the blend reference level, although the three nearest cells fall within its replication band, leaving practical recovery undecided. Third, alignment gain predicts evidence gain within robust encoders (ρ = 0.24–0.51) but not within vanilla CLIP (consistent with zero under a post-hoc approximation), and no monotone alignment–evidence relation exists across all encoders.

The paper introduces an evidence taxonomy to frame claims, separating them into levels L0 target intent, L1 victim-space alignment (VTS), and L2 independent recovery (ITE). VTS is defined as cosine to the frozen target text embedding in the victim, while ITE is the shuffle-corrected change in the target’s logit margin on an independent judge.

Instrument Validation and Robustness Comparisons

Study A validated the L2 independent-evidence instrument, showing that the primary judge recovers real target evidence monotonically in blend dose (ρ = 1.0). The anchor-dose ITE is 5.71 [95% CI 5.12, 6.32] on held-out data, and the reference level is set at Smin = 5.371. Study B confirmed that independent target evidence rises with budget for every encoder At ϵ=4, all six encoders cluster near zero ITE (0.08–0.39), while at ϵ=64, the gap widens significantly for robust recipes like TeCoA (+4.45 ITE units).

Diagnostic Findings on Alignment and Evidence

Within every robust-encoder cell, per-sample alignment gain predicts evidence gain (ρ = 0.24–0.51), whereas within vanilla CLIP, the observed ρ is consistent with zero under a post-hoc approximation. Between victims in Study B, the mean-VTS ranking inverts the mean-ITE ranking (ρ = −1.0; n=3, descriptive). Pooling the nine cells shows a cancellation of correlations where VTS and ITE levels cancel each other out.

Reporting Protocol and Limitations

The authors propose a reporting protocol for adversarial semantic claims, recommending that one should scope alignment claims to the evaluated encoder, report at least one validated independent endpoint with graded controls, state the optimization recipe and its distance to stronger attacks, and evaluate robust encoders at and just beyond their training budget. Limitations include the lack of matched training interventions to isolate causal pathways and that no cell’s mean reaches the reference level even at ϵ=64/255. The results are scoped to machine-level evidence (L1 and L2) under supervised ImageNet-1k judges.

The protocol adds graded real-target blend controls that establish the judge’s sensitivity to actual target evidence, shuffled-target negatives that correct for non-specific logit-mass movement, and a pre-specified reference level that anchors the verdict to a measurable quantity rather than a threshold chosen post hoc. The observed pattern is consistent with robust features and off-manifold accounts but no mechanism quantity was measured confirmatorily. The paper concludes that the standard evaluation in this space reports attack success rate or victimside alignment, but the instrument adds three things: graded real-target blend controls that establish the judge’s sensitivity to actual target evidence, shuffled-target negatives that correct for non-specific logit-mass movement, and a pre-specified reference level that anchors the verdict to a measurable quantity rather than a threshold chosen post hoc. The mechanism is consistent with robust features and off-manifold accounts but no mechanism quantity was measured confirmatorily. The limitations include six encoders, one dataset (ImageNet-100), three budgets, and the fact that no cell’s mean reaches the reference level even at ϵ=64/255. The future work suggests extending the instrument to additional judge paradigms and developing a human-grounded study to determine if the blend-derived reference level corresponds to any perceptual threshold. The reporting protocol could be developed into a community benchmark covering additional attack families, encoder architectures, and perturbation norms. The checkpoint provenance question depends on resolving the training budget attribution and the regime-mismatch interpretation. The uncertainty is approximately 15% between draws, which is comparable to the gap between these cells and the reference level. The final conclusion is that facts together limit VTS to a within-encoder diagnostic for robust checkpoints.

--- Page 5 ---

Table 5: Study B (confirmatory): independent target evidence ITESa per cell, pooled over mapping seeds 101/202/303 (n=300/cell; R50 judge; held-out ImageNet-100 rows). Rule: L2-positive at the anchor ⇐⇒ mean ≥ Smin = 5.371 and class-cluster bootstrap 95% CI lower bound > 0. ITR = anchor-in-top-5 rate VTS = victim-space target similarity after attack (L1 diagnostic)

Table 6: Two-level diagnosticity of victim-space alignment (confirmatory). Within-cell: Spearman ρ between per-sample ∆VTS and ITESa (all permutation p ≤ 0.0005 except vanilla, ns) Between-victim: victimmean VTS vs victim-mean ITE Pooled cell-level (9 cells): ρ = +0.083 (cancellation)

Table 7: Study C per-cell results by budget (confirmatory; n=300/cell; R50 judge; cf. Holm-4 and H C2/H C3 in the main text)

Table 8: Recipe-frontier check (exploratory split; identical cohort, anchors, and judge to the calibration pilot)

Table 9: Study A: validation of the L2 independent-evidence instrument (held-out split, n=250). The pre-specified reference level is anchored at the α∗=0.

Improvements for AI systems

  1. Continuous calibration for adversarial claims: Implement a reporting protocol that requires "(i) scope alignment claims to the evaluated encoder; (ii) report at least one validated independent endpoint, with graded real-target positive controls, shuffled-target negatives, and a judge competence check; (iii) state the optimization recipe and its distance to standard stronger attacks; (iv) for robust encoders, evaluate at and just beyond the training budget. This moves beyond victim-space target alignment (VTS)" as a standalone metric to provide actionable evidence.

  2. Robustness-aware model selection: Prioritize robustly trained encoders (FARE, TeCoA, PMG, TRADES) over vanilla models when assessing adversarial claims. This improves system reliability by ensuring that the reported evidence is derived from models capable of transferring independent target evidence more effectively than non-adversarially trained ones.

  3. Evidence thresholding for deployment: Establish a measurable reference level based on a 50% target-image blend to determine if an adversarial claim passes L2 evidence. A system should only be deemed safe or successful when its mean independent evidence (ITE) meets or exceeds this level, rather than relying on subjective alignment scores.

  4. Diagnostic monitoring for encoder performance: Within robust encoders, monitor the correlation between per-sample alignment gain and evidence gain (ρ = 0.24–0.51) to detect when increased optimization efforts are actually yielding independent target evidence transfer, distinguishing it from within-optimization replication of the L1⊥L2 dissociation.

  5. Cross-encoder validation refinement: When using cross-encoder VTS for decision-making, account for the observed pattern that the between-victim means (van/FARE/TeCoA): VTS 0.465/0.323/0.262 vs ITE 0.52/2.05/2.90; ρ = −1.0 (n=3, descriptive). This prevents misinterpreting the pooled cell-level ρ = +0.08 as a general positive correlation when comparing different encoder regimes or victim models.

Sources

Related papers