Breaking Adversarial Transferability in Fine-Tuned Speech Recognition

arXiv:2610.09109 · cs.LG · Submitted 2026-10-06 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Breaking Adversarial Transferability in Fine-Tuned Speech Recognition".

Jane: As an excellent, fastidious, and diligent researcher,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: To wrap up "Breaking Adversarial Transferability in Fine-Tuned Speech Recognition," the authors show that adversarial transferability is a real threat when deploying ASR models fine-tuned on public bases >

Jane: The authors propose TransferBreaker, which uses Base-AFT, Latent Jacobian Regularization, and HybridGrad-AFT to defend against this by combining different techniques into one unified framework >

Lu: It establishes that white-box access to a base model can lead to performance degradation from sixteen point four up to ninety-two point six Word Error Rate on the target model > <ref:2610.09109#pg1>

Meng: For those of us shipping models, it means we have a new layer of defense we need to consider when choosing our deployment strategy >

Lalam: This work confirms that adversarial transferability is a paramount security consideration in the deployment pipeline for publicly available ASR models >

Conclusion: Tom: So we’ve been deep into TransferBreaker, and now we need to talk about what this whole paper is actually trying to prove with that title, "Breaking Adversarial Transferability in Fine-Tuned Speech Recognition."

Jane: Basically, it's saying that when you take a speech recognition model that someone has already fine-tuned on a public base model—like Whisper—you can make little changes to the base and those changes jump over to your target model, messing up its performance badly.

Lu: That’s the core vulnerability they are exposing. The authors show how this happens, quantifying it with a big gap in Word Error Rate, moving from sixteen point four up to ninety-two point six under attack conditions.

Meng: It sounds like a problem for anyone shipping models where you don't control the entire training pipeline; if a base model is public, that vulnerability is real.

Lalam: I’m seeing this as a major security consideration because it means an attacker doesn't need to train on your specific target model directly to cause damage; they can use knowledge from the public base and transfer it.

Tom: Exactly. The authors are proposing TransferBreaker, which uses three specific training tweaks—Base-AFT, Latent Jacobian Regularization, and HybridGrad-AFT—to stop that transfer from happening.

Jane: They aren't just patching things up; they’re showing how to build a unified defense framework that handles the static attacks and the adaptive ones at the same time. It’s about making sure those latent directions in the model stay stable even when you introduce noise.

Lu: The methodology is pretty solid because they have a theoretical justification for each part, like connecting Latent Jacobian sensitivity back to how much noise actually matters in the representation space.

Meng: From an engineering standpoint, I’m looking at how much computational overhead those three components add to the fine-tuning job compared to just doing standard adversarial training. That’s where my concern lies for real deployment scenarios.

Lalam: The authors did look at that cost trade-off in the appendix, comparing it against Standard Adversarial Fine-Tuning, showing a measurable difference in training time and resources used.

Tom: It’s a significant result because they show that this isn't just an academic curiosity; it’s a concrete failure mode when deploying these models publicly. This paper really forces everyone in the ASR space to rethink their deployment security protocols.

Jane: Right, and if we can stop these transfers, we start building more trustworthy systems for things like in-vehicle voice interfaces where those errors could be serious.

Lu: The paper lays out a path forward for model developers who are relying on open or semi-open models to make sure their final product is actually robust against these kinds of attacks.

Meng: So the implication is that if you’re deploying something public, you have to bake in this kind of multi-layered defense into your fine-tuning process before you even consider putting it into production. That’s a pretty heavy lift for production pipelines.

Lalam: It confirms that adversarial transferability isn't just a theoretical risk; it's a practical security flaw we have to address right now across the entire ASR deployment pipeline.

Mojtaba Nafez, Aref Mousavi, Mohammad Ebrahim Mahdavi, Mobina Poulaei, Kiarash Kiani Feriz, Mohammad Hossein Rohban

Idiap Research Institute of École Polytechnique Fédérale de Lausanne of Sharif University of Technology and University of Isfahan

cs.LG

Submitted: 2026-10-06

Updated: 2026-10-06

Comments: 39 pages, 5 figures

Code: https://github.com/rohban-lab/TransferBreaker

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 89/100

The gist: As an excellent, fastidious, and diligent researcher, I have meticulously analyzed both provided texts regarding the paper "Breaking Adversarial Transferability in Fine-Tuned Speech Recognition." The

Key concepts

Adversarial Transferability: This is the vulnerability where an attack crafted for one speech recognition model can easily be used against a different, fine-tuned model. The paper shows this transfer causes severe performance degradation, meaning security assumptions about limited access are often false.
Base Adversarial Fine-Tuning (Base-AFT): This technique restricts adversarial training so that perturbations are only trained on those effective for the original base model. This prevents the over-constraining effects seen in standard methods when fine-tuning complex ASR tasks.
Latent Jacobian Regularization (LJ-Regularization): This mechanism enforces latent-space invariance by actively suppressing directions in the model's internal representation that are highly sensitive to adversarial noise. It neutralizes these specific, vulnerable directions in the model's learned space.

Terminology

Adversarial Transferability — 1× in this paper · explained in 2 episodes
Transferability — 1× in this paper · explained in 5 episodes

Summary

As an excellent, fastidious, and diligent researcher, I have meticulously analyzed both provided texts regarding the paper Breaking Adversarial Transferability in Fine-Tuned Speech Recognition. The following synthesis combines the technical details from Section A with the ethical/methodological justifications from Section B to provide a comprehensive and detailed summary suitable for high-stakes review.

Comprehensive Research Summary: TransferBreaker Framework for Suppressing Adversarial Transferability in ASR Models

This research introduces TransferBreaker, a novel, unified fine-tuning framework specifically designed to suppress the critical vulnerability of adversarial transferability when deploying publicly available pre-trained Automatic Speech Recognition (ASR) models in black-box settings. The core finding is that the assumption of protection afforded by limited access is fragile: adversarial perturbations crafted on a base public model can effectively transfer to downstream fine-tuned target models, severely degrading performance and posing significant risks for safety-critical applications.

Technical Contributions and Methodology

TransferBreaker integrates three distinct components into a single training procedure to achieve robust defense against adversarial transfer:

Base Adversarial Fine-Tuning (Base-AFT): This component is designed to restrict the adversarial training process, ensuring that perturbations are only trained on those effective for the base model, thereby avoiding the over-constraining behavior often seen in Standard Adversarial Fine-Tuning (Standard-AFT) within complex ASR tasks.
Latent Jacobian Regularization (LJ-Regularization): This mechanism enforces latent-space invariance by actively suppressing directions in the latent space that are highly sensitive to adversarial perturbations, effectively neutralizing adversarially sensitive directions.
HybridGrad-AFT: To counter adaptive attacks—where adversaries tailor their attacks specifically to the target model—this component improves robustness by interpolating transferable perturbation signals derived from both the base and target gradients.

The overall training objective is defined as:

L Total = L ft(x), y + lambda 1 L ft(x + delta), y + lambda 2 1 over d h t(x + delta) - h b(x) 2

where delta is a hybrid perturbation constructed as delta = I delta hybrid + (1 - I) delta base, and I about Bernoulli(p).

Performance and Evaluation:

The framework was rigorously evaluated across three languages (Polish, Portuguese, and Arabic) using four state-of-the-art ASR backbones: Whisper-largev3-turbo, wav2vec2-large [8], MMS-1B [9], and Whisper-medium. The results demonstrate a dramatic reduction in adversarial performance degradation. Specifically, the framework reduced the average adversarial Word Error Rate (WER) from a severe baseline of 92.6 down to 27.8 on base-model attacks. Furthermore, it significantly improved Character Error Rate (CER), dropping from 4.3 to 9.0 under adversarial conditions, representing a substantial 64.8-point improvement over fine-tuned variants under attack (where the transferred attack increases WER from 16.4 to 92.6).

Model Agnosticism and Robustness:

A key strength of TransferBreaker is its modelagnostic nature; it is architecture-independent, allowing it to maintain effectiveness even when perturbations are generated directly on the target models, where it consistently remains within plus or minus0.6 WER across all four tested settings.

Critical Findings and Contributions

The research establishes several critical findings:

Exposing Vulnerability (Contribution i): The authors expose a critical vulnerability where white-box access to a base model enables near–white-box attacks on target models, which is quantified by an increase in average WER from 16.4 to 92.6.
Refined Training Strategy (Contribution ii): Base-AFT is shown to be superior to Standard-AFT for complex ASR tasks because it restricts adversarial training only to base-effective perturbations, preventing undesirable over-constraining effects.
Latent Space Control (Contribution iii): LJ-Regularization provides a principled method for enforcing latent-space invariance by suppressing directions sensitive to adversarial noise, supported by theoretical characterization of directional latent-Jacobian sensitivity (Equation 16).
Adaptive Attack Defense (Contribution iv): HybridGrad-AFT is developed to specifically enhance robustness against adaptive attacks by strategically interpolating transferable perturbations from both the base and target gradients.
Theoretical Foundation (Contribution v): The framework is supported by a thorough theoretical justification across all components, including analyses of Base-AFT vs. Standard-AFT, LJ-Regularization's connection to latent invariance, and the motivation for HybridGrad-AFT against adaptive attacks.

Ethical and Societal Context

The research operates with a strong commitment to responsible AI development:

Ethical Adherence: The work fully adheres to the NeurIPS Code of Ethics by utilizing publicly available datasets and proactively aiming to expose vulnerabilities, thereby improving the security and trustworthiness of real-world ASR systems.
Societal Impact: The authors explicitly discuss the societal impact in Section 1, highlighting severe risks associated with adversarial vulnerabilities in safety-critical applications such as in-vehicle voice interfaces. TransferBreaker is positioned as a defensive framework that mitigates these threats, and the authors do not release high-risk models or datasets intended for misuse.
Data Usage: The research utilizes only publicly available datasets and open-source models, respecting all existing licenses (e.g., CC0, CC-BY-4.0, MIT). No new assets are introduced; no crowdsourcing or human subjects were involved in the core methodology, rendering IRB approvals inapplicable.
Transparency: The authors provide extensive documentation for reproducibility, including a public GitHub repository and detailed implementation instructions (Section K), alongside an ablation study comparing training computational costs against Standard-AFT (Appendix L).

Conclusion:

TransferBreaker successfully addresses the urgent security concern of adversarial transferability in ASR. By combining Base-AFT, Latent Jacobian Regularization, and HybridGrad-AFT into a unified framework, the authors provide a robust defense that significantly enhances model reliability against both static and adaptive adversarial attacks across diverse languages and architectures. The findings establish adversarial transferability as a paramount security consideration in the deployment pipeline for publicly available ASR models.

Improvements for AI systems

Based on TransferBreaker, an improved ASR system can be deployed in black-box settings with significantly degraded performance compared to current methods; specifically, adversarial WER from 92.6 to 27.8 across three languages and four models is achieved by integrating Base Adversarial Fine-Tuning, which restricts adversarial training to base-effective perturbations.
The system can be made robust against adaptive attackers who are fully aware of the defense strategy by implementing HybridGrad-AFT, which improves robustness against adaptive attacks by interpolating transferable perturbations from base and target gradients.
A system incorporating Latent Jacobian Regularization can achieve stronger robustness by enforcing latent-space invariance to adversarial perturbations through the objective: LLJ−Regularization = (1/d) ht(x + δ) − hb(x)2.
The improved system can generate highly intelligible adversarial audio, as its perturbations are characterized by a low coefficient of variation in frame energy, resulting in STOI = 0.97 [38], indicating near-perfect speech intelligibility and preserved phonetic content despite the noise.

Abstract

Many organizations fine-tune publicly available pretrained Automatic Speech Recognition (ASR) models and deploy them in black-box settings, assuming limited access provides protection. We show this assumption is fragile: adversarial perturbations crafted on the public base model transfer effectively to fine-tuned target models, severely degrading performance and posing concerns for safety-critical applications. We propose TransferBreaker, a unified fine-tuning framework that suppresses adversarial transfer by integrating Base Adversarial Fine-Tuning, which restricts adversarial training to base-effective perturbations; Latent Jacobian Regularization, which enforces latent-space invariance by suppressing adversarially sensitive directions; and HybridGrad-AFT, which improves robustness against adaptive attacks by interpolating transferable perturbations from base and target gradients. We theoretically justify all components and evaluate TransferBreaker across three languages and four large ASR models, reducing adversarial WER from 92.6 to 27.8. Our code is publicly available at https://github.com/rohban-lab/TransferBreaker.