Beyond Speech: Dual-Domain SSL Fusion for Unified All-Type Audio Deepfake Detection

arXiv:2608.29021 · eess.AS · Submitted 2026-08-29 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: We're looking at the paper: "Beyond Speech: Dual-Domain SSL Fusion for Unified All-Type Audio Deepfake Detection".

Dev: Here's the gist of the paper. Unified all-type audio deepfake detection aims to determine whether an input clip is real or fake when its audio type may be speech, environmental sound, singing voice, or music The core detector uses EAT-large and wav2vec 2.0 XLS-R-300M as complementary SSL feature sources to map heterogeneous audio into a shared binary authenticity space Dual-Domain Feature Fusion The method employs a dual-domain SSL fusion system that utilizes EAT-large and wav2vec 2.0 XLS-R-300M as complementary front-ends for one unified detector Both streams are optimized toward the same shared binary authenticity space rather than separate type-specific decision paths.

Rosa: Learnable layer-wise weighted fusion aggregates hidden states from multiple transformer layers, allowing the model to use low-level spectral artifacts, intermediate acoustic or event inconsistencies, and higherlevel semantic cues.

Dev: The layerfused token sequences are then concatenated along the temporal dimension to form a unified feature pool, which avoids forcing frame-level alignment between the two SSL streams.

Rosa: Unified Feature Representation The dual-domain SSL feature fusion involves concatenating the EAT and XLS-R token sequences along the temporal or token dimension to form a unified feature pool.

Title and authors: Rosa: So, to summarize "Beyond Speech: Dual-Domain SSL Fusion for Unified All-Type Audio Deepfake Detection," this paper takes all sorts of audio—speech, music, even just background noise—and figures out if it's real or fake using one single detector instead of a collection of different ones for each type <ref:2608.29021#pg1>.

Dev: Right. The central concept is mapping all that messy audio data into this one shared space where real and fake are just two points apart, no matter what kind of sound it is <ref:2608.29021#pg3>.

Taro: They start by using two different large models, EAT-large and wav2vec 300M, as the starting points for extracting the features <ref:2608.29021#pg3>.

Rosa: And they don't just randomly throw those features together. They employ this dual-domain SSL fusion system to combine them in a way that lets them learn from each other without forcing the audio streams to match up perfectly in time <ref:2608.29021#pg3>.

Dev: I read it, and it uses this layer-wise weighted fusion, which means you're aggregating features from different layers of the transformer using weights that are learned during training <ref:2608.29021#pg3>.

Taro: That means the model gets to choose which parts of the audio signal—the low-level noise stuff, or the higher-level meaning—are most important for spotting a fake <ref:2608.29021#pg3>.

Rosa: Exactly. They concatenate those features along the time axis first, creating one big pool before they do any summarizing <ref:2608.29021#pg3>.

Dev: And then they run that pool through a small projection based on SwiGLU before doing the attention aggregation, which helps clean up the signals and focus on what matters <ref:2608.29021#pg3>.

Taro: It sounds like a very robust way to handle inconsistent input types because you aren't relying on strict frame-level alignment between two different training setups <ref:2608.29021#pg3>.

Rosa: And the final step is using multi-head attentive statistics pooling to summarize everything, which gives them a weighted average and a weighted standard deviation of the features <ref:2608.29021#pg3>.

Dev: The results are pretty solid too, achieving ninety-five point five eight percent Macro F1 on their test set and ranking second in the competition <ref:2608.29021#pg3>.

Taro: That’s a big number, but I still look at the audio types themselves. The t-SNE plots show that while they are all separated by label, you can see that speech, sound, singing voice, and music are all clustered together in different areas <ref:2608.29021#pg3>.

Rosa: It confirms the idea of a unified space where those different audio domains are jointly embedded <ref:2608.29021#pg3>. And they show that this fusion actually gives them better scores across all four types compared to their previous method, S2 <ref:2608.29021#pg3>.

Dev: They say the improvement for speech is one point three one, and for music it's zero point eight seven points relative to that baseline <ref:2608.29021#pg3>.

Taro: So what this really means is that a system built on this shared detector can handle diverse audio inputs much better than having separate systems for each one <ref:2608.29021#pg3>.

Rosa: It shows you can use the type information to guide the detection, but you don't have to build entirely separate detection routes for every single thing you want to check <ref:2608.29021#pg1>.

Dev: And they do add this conservative speech refinement on top, which only triggers if auxiliary cues agree that they are hearing recognizable speech <ref:2608.29021#pg3>.

Taro: That’s a good safety mechanism, I guess. It keeps the system from making weird calls on random noise when it's not supposed to be listening for speech <ref:2608.29021#pg3>.

The paper's summary: Rosa: So, looking at how they suggest improving this work, they are talking about taking that core fusion idea and adding a speech-specific refinement strategy to the very end of the process <ref:2608.29021#pg3>.

Dev: What does that mean for us in terms of real-time performance? Does that extra step add too much latency when we need a fast loop <ref:2608.29021#pg3>?

Taro: It means they're conditionally adding a speech specific detector only if two other cues already agree that the audio is recognizable speech <ref:2608.29021#pg3>.

Rosa: Exactly, so it’s not always on, which keeps things fast for environmental sounds or music <ref:2608.29021#pg3>.

Dev: But when it does trigger, they're using joint auxiliary cues and frozen audio reconstruction models to help confirm the speech <ref:2608.29021#pg3>. That's a bit more computation, right?

Taro: It is more computation, but they framed it as a conservative approach that only activates when there's high confidence that recognizable speech is present <ref:2608.29021#pg3>.

Rosa: The goal is to use this for enhancing performance on speech detection specifically, without making the whole system overly complicated for every single input <ref:2608.29021#pg3>.

Dev: I see the trade-off. We gain a bit more accuracy on speech by adding that extra step, but we have to manage that added processing load in our real-time control loop <ref:2608.29021#pg3>.

Taro: It really shows that they aren't just pushing for the highest possible number on every single audio type <ref:2608.29021#pg3>. They’re being smart about where to spend their resources <ref:2608.29021#pg3>.

Rosa: So it’s about getting a strong baseline for everything, and then applying a targeted boost only when the input looks like speech <ref:2608.29021#pg3>. That feels like a sensible way to manage the complexity of all those audio types <ref:2608.29021#pg1>.

Dev: It’s a pragmatic addition, I guess. It keeps the unified core detector running on everything, but gives us a specialized tool ready for speech when it needs to be deployed <ref:2608.29021#pg3>.

Taro: The implication is that you can have one general-purpose audio detector that’s highly accurate across the board, and then layer on specific checks for things like clear human voices if the context demands it <ref:2608.29021#pg3>.

Rosa: That means less maintenance down the line because you don't have to update four separate detectors for speech, music, sound, and environmental noise <ref:2608.29021#pg1>.

Dev: So it moves the goal closer to a single model architecture that handles all these different audio domains in one go <ref:2608.29021#pg3>.

Taro: But the question remains about those edge cases, what happens when the world throws us something totally unexpected that doesn't fit the speech or music categories at all <ref:2608.29021#pg3>.

The paper's improvements: Rosa: So, wrapping up this look at "Beyond Speech: Dual-Domain SSL Fusion for Unified All-Type Audio Deepfake Detection," the main message is that you can merge different audio detectors into one shared space for real versus fake detection across speech, music, and noise <ref:2608.29021#pg3>.

Dev: Basically, they proved you can use this unified approach without needing a separate detector for every single type of sound <ref:2608.29021#pg1>.

Taro: The main thing is that they proved you can use this unified approach without needing a separate detector for every single type of sound <ref:2608.29021#pg3>.

Rosa: And the results showed that this method actually improves performance on speech, singing voice, environmental sound, and music compared to previous methods like S2 <ref:2608.29021#pg3>.

Dev: They did it by using a dual domain SSL fusion system that merges features without having to perfectly align the time frames between the audio inputs <ref:2608.29021#pg3>.

Taro: That part about avoiding frame level alignment is what makes it interesting for real-world use because you don't have to worry about perfect synchronization across different recording conditions <ref:2608.29021#pg1>.

Rosa: Right, and they even added that speech specific refinement as a cautious layer on top just to boost the numbers when it matters most <ref:2608.29021#pg3>.

Dev: So the overall implication is that we can build more flexible AI systems for audio without having to maintain a huge library of specialized detectors <ref:2608.29021#pg3>.

Taro: It means the system can be more adaptable to different types of inputs, which is crucial when you're testing autonomy in unpredictable environments <ref:2608.29021#pg1>.

Rosa: Yeah, and while they hit ninety-five point five eight percent Macro F1 on their test set, it’s a strong result for a unified system <ref:2608.29021#pg3>.

Dev: It's definitely not the highest number in the world, but it shows that this fusion architecture works well across different audio domains <ref:2608.29021#pg3>.

Taro: I’m still curious about those extreme edge cases where the world throws us something totally outside what they trained on <ref:2608.29021#pg3>.

Rosa: That’s definitely the next frontier, figuring out how gracefully this unified system handles truly novel audio that doesn't fit any of the known categories <ref:2608.29021#pg1>.

Dev: We’ll have to see if those improvements hold up when we move this from controlled lab trials to chaotic, real-world deployment loops <ref:2608.29021#pg3>.

Conclusion: Rosa: So we’re wrapping up our look at "Beyond Speech: Dual-Domain SSL Fusion for Unified All-Type Audio Deepfake Detection." This paper takes all those different audio inputs—speech, music, noise—and puts them into one shared space to figure out if they're real or fake.

Dev: Basically, the authors proved you can use this unified approach without needing a separate detector for every single type of sound <ref:2608.29021#pg3>. It’s about merging those different detectors into one shared space for real or fake detection across speech, music, and noise.

Taro: The main thing is that they proved you can use this unified approach without needing a separate detector for every single type of sound <ref:2608.29021#pg1>. That’s pretty significant for autonomy research because it means the system can be more adaptable to different types of inputs, which is crucial when you're testing something in unpredictable environments.

Rosa: Yeah, and they showed that this method actually improves performance on speech detection by one point three one, environmental sound by one point zero nine, singing voice by one point three three, and music detection by zero point eight seven points compared to the previous method <ref:2608.29021#pg3>.

Dev: It did it by using a dual domain SSL fusion system that merges features without having to perfectly align the time frames between the audio inputs <ref:2608.29021#pg3>. That part about avoiding frame level alignment is what makes it interesting for real-world use because you don't have to worry about perfect synchronization across different recording conditions.

Taro: And they even added that speech specific refinement as a cautious layer on top just to boost the numbers when it matters most <ref:2608.29021#pg3>. That makes sense for real-world deployment because you don't want the system over-analyzing non-speech just to be safe.

Rosa: Right, and while they hit ninety-five point five eight percent Macro F1 on their test set, it’s a strong result for a unified system <ref:2608.29021#pg3>. It shows that a shared detector can handle heterogeneity well without building typedependent inference routes.

Dev: It's definitely not the highest number in the world, but it shows that this fusion architecture works well across different audio domains <ref:2608.29021#pg3>. We’ll have to see if those improvements hold up when we move this from controlled lab trials to chaotic, real-world deployment loops.

Taro: I’m still curious about those extreme edge cases where the world throws us something totally outside what they trained on <ref:2608.29021#pg3>. That’s definitely the next frontier, figuring out how gracefully this unified system handles truly novel audio that doesn't fit any of the known categories.

Rosa: That sounds like a good direction for future work, exploring those kinds of edge cases where things get really unpredictable <ref:2608.29021#pg3>.

Dev: We’ll have to see if those improvements hold up when we move this from controlled lab trials to chaotic, real-world deployment loops <ref:2608.29021#pg3>. That's what we'll be focusing on next.

Anhui University · State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology, School of Computer Science and Technology, Anhui University · Anhui Laboratory for Safe Artificial Intelligence in the Yangtze River Delta · Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering · University of Science and Technology of China

eess.AS

Submitted: 2026-08-29

Updated: 2026-11-11

Comments: 8 pages, 5 figures; accepted to the AT-ADD Grand Challenge at ACM Multimedia 2026 (MM '26)

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: Unified all-type audio deepfake detection aims to determine whether an input clip is real or fake when its audio type may be speech, environmental sound, singing voice…

Key concepts

Dual-Domain SSL Fusion
This technique uses two separate, powerful self-supervised models (EAT and wav2vec 2.0) as front-ends for one detector. Instead of having separate paths for each audio type, they are trained to map all audio inputs into the same shared mathematical space representing authenticity.
Layer-wise Weighted Fusion
The model combines information from different layers within the SSL streams using learnable weights. This allows it to selectively use low-level details like spectral artifacts and higher-level semantic cues, creating a rich feature pool without needing to perfectly align the time steps of the two input streams.
Unified Feature Representation
The features from both models are concatenated along the temporal dimension to form one large feature pool. A lightweight projection then processes this combined sequence before attention pooling summarizes the most important discriminative cues, capturing evidence from both sources effectively.

Terminology

Summary

Unified all-type audio deepfake detection aims to determine whether an input clip is real or fake when its audio type may be speech, environmental sound, singing voice, or music The core detector uses EAT-large and wav2vec 2. p. 0

Where the paper says it

cn Xin Fang University of Science and Technology of China Hefei, Anhui, China xinfang@iflytek. com Abstract Unified all-type audio deepfake detection aims to determine whether an input clip is real or fake when its audio type may be speech, environmental sound, singing voice, or music.

0 XLS-R-300M as complementary SSL feature sources to map heterogeneous audio into a shared binary authenticity space

Dual-Domain Feature Fusion

The method employs a dual-domain SSL fusion system that utilizes EAT-large and wav2vec 2.0 XLS-R-300M as complementary front-ends for one unified detector Both streams are optimized toward the same shared binary authenticity space rather than separate type-specific decision paths. p. 1

Where the paper says it

EAT-large takes time-frequency features and provides broad acoustic and event-level representations, whereas XLS-R300M takes the raw waveform and provides waveform-level, vocal, and speech-sensitive representations. Both streams are optimized toward the same shared binary authenticity space rather than separate type-specific decision paths.

Learnable layer-wise weighted fusion aggregates hidden states from multiple transformer layers, allowing the model to use low-level spectral artifacts, intermediate acoustic or event inconsistencies, and higherlevel semantic cues. p. 1
Where the paper says it

Instead of treating them as separate detectors, both streams are optimized toward the same shared binary authenticity space. Learnable layer-wise weighted fusion aggregates hidden states from multiple transformer layers, allowing the model to use low-level spectral artifacts, intermediate acoustic or event inconsistencies, and higherlevel semantic cues [15, 16].

The layerfused token sequences are then concatenated along the temporal dimension to form a unified feature pool, which avoids forcing frame-level alignment between the two SSL streams. p. 1
Where the paper says it

Learnable layer-wise weighted fusion aggregates hidden states from multiple transformer layers, allowing the model to use low-level spectral artifacts, intermediate acoustic or event inconsistencies, and higherlevel semantic cues [15, 16]. The layer-fused token sequences are then concatenated along the temporal dimension to form a unified feature pool, which avoids forcing frame-level alignment between SSL streams.

Unified Feature Representation

The dual-domain SSL feature fusion involves concatenating the EAT and XLS-R token sequences along the temporal or token dimension to form a unified feature pool. This concatenation avoids assuming frame-level alignment between SSL streams with different inputs and pretraining objectives. p. 2

Where the paper says it

The shared SwiGLU layer is the first learned transformation of the combined pool. Token-level concatenation avoids assuming frame-level alignment between SSL streams with different inputs and pretraining objectives.

A lightweight projection based on SwiGLU processes the concatenated sequence before attention aggregation, which enhances useful responses and suppresses less informative activations. p. 2
Where the paper says it

A lightweight projection based on SwiGLU [17] processes the concatenated sequence before attention aggregation. This gated nonlinear layer enhances useful responses and suppresses less informative activations.

Multi-head attentive statistics pooling is used to summarize the unified feature pool, capturing dominant discriminative cues via a weighted mean and reflecting variation through a weighted standard deviation. p. 3
Where the paper says it

This statistical pooling step is useful because fake artifacts may be sparse, local, or distributed unevenly across time. The weighted mean captures the dominant discriminative cues, while the weighted standard deviation reflects the variation and instability of the selected tokens.

Final Decision Mechanism

The pooled representation is finally passed to an MLP classifier to produce binary real/fake logits. The final utterance-level representation is obtained by concatenating the mean and standard deviation statistics from all attention heads. p. 3

Where the paper says it

In implementation, this is computed equivalently as the attention-weighted second moment minus the squared mean. The final utterance-level representation is obtained by concatenating the mean and standard deviation statistics from all attention heads: 𝑟 = [𝜇1; .

A fixed threshold of 0. p. 4
Where the paper says it

41 final logit is zfinal = ( (z2 + z5 + z10)/3, 𝑞 = 1, (z2 + z5)/2, 𝑞 = 0. (8) A fixed threshold of 0.

5 is applied to the real-class probability obtained from zfinal, while S2 and S5 remain active for every sample, and S10 is conditionally added as a conservative speech-specific refiner. p. 4
Where the paper says it

5 is applied to the real-class probability obtained from zfinal. Thus, S2 and S5 remain active for every sample, while S10 is conditionally added as a conservative speech-specific refiner.

Performance and Analysis

The submitted system achieves 95.58% Macro-F1 on the AT-ADD Track 2 evaluation set and ranks second in the challenge. p. 0

Where the paper says it

3836631 this unified core detector, the submitted system achieves 95. 58% Macro-F1 on the AT-ADD Track 2 evaluation set and ranks second in the challenge.

The t-SNE visualization shows a clear separation between authenticity labels, while coloring by audio type reveals heterogeneous but jointly embedded audio domains. p. 4
Where the paper says it

The t-SNE visualization in Figure 2 is computed on dev1 and is used only to inspect the learned binary-classification embedding. Coloring the same embedding by real/fake label shows a clear separation between authenticity labels, while coloring it by audio type reveals heterogeneous but jointly embedded audio domains.

The results support dual-domain fusion with conservative refinement across heterogeneous audio. p. 5
Where the paper says it

87 points on speech, sound, singing, and music. The results support dual-domain fusion with conservative refinement across heterogeneous audio.

The final fusion improves speech, sound, singing, and music by 1. p. 4
Where the paper says it

The final row uses the stricter S5–S9 intersection gate defined above and averages S2, S5, and S10 only for the selected speech-refinement subset. Relative to S2, the final fusion improves speech, sound, singing, and music by 1.

31, 1.09, 1.33, and 0.87 points relative to S2. p. 4
Where the paper says it

Relative to S2, the final fusion improves speech, sound, singing, and music by 1. 87 points, respectively, yielding a 1.

Conclusion

The paper presents a unified AT-ADD Track 2 detector that combines EAT-large and XLS-R300M without frame alignment. The final submission conditionally adds a speech refiner using joint auxiliarytype and frozen-ASR cues, while S2 and S5 remain active for every sample. p. 5

Where the paper says it

Layer-wise weighting and token-level fusion combine EAT-large and XLS-R300M without frame alignment; shared SwiGLU, attentive statistics pooling, and a binary head retain one all-type path. The final submission conditionally adds a speech refiner using joint auxiliarytype and frozen-ASR cues, while S2 and S5 remain active for every sample.

The results demonstrate that a shared detector can exploit type-sensitive cues without introducing typedependent inference routes. p. 5
Where the paper says it

The embedding and layer-weight analyses further show complementary SSL cues within the shared authenticity space. Together, these results demonstrate that a shared detector can exploit type-sensitive cues without introducing typedependent inference routes.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Dual-domain SSL Fusion for Unified All-Type Audio Deepfake Detection: The system can determine whether an input clip is real or fake when its audio type may be speech, environmental sound, singing voice, or music by mapping heterogeneous audio into a shared binary authenticity space.

  2. Layer-wise weighted fusion and token-level fusion: This strategy builds a unified feature pool for multi-head attentive statistics pooling without requiring frame-level alignment between the two SSL streams, while retaining each front-end’s native token order and complementary temporal evidence during aggregation.

  3. Conservative speech refinement strategy: A conservative speech refinement strategy is added on top of the unified core detector, conditionally adding a speech-specific refiner only when two auxiliary cues agree that recognizable speech is present to enhance performance.

Abstract

Unified all-type audio deepfake detection aims to determine whether an input clip is real or fake when its audio type may be speech, environmental sound, singing voice, or music. Existing speech-centric or type-dependent solutions are insufficient for this setting because the test-time audio type is unknown, while the required output is still a single binary decision. To address these issues, this paper proposes a dual-domain SSL fusion method that maps heterogeneous audio into a shared binary authenticity space. EAT-large and wav2vec 2.0 XLS-R-300M are used as complementary SSL feature sources, providing broad acoustic and event-level representations as well as waveform-level, vocal, and speech-sensitive representations. Layer-wise weighted fusion integrates multi-level artifacts from different transformer depths, while token-level fusion forms a unified feature pool without enforcing frame-level alignment between the two SSL streams. The fused tokens are summarized by multi-head attentive statistics pooling and classified with a binary MLP head. With conservative speech refinement applied on top of this unified core detector, the submitted system achieves 95.58% Macro-F1 on the AT-ADD Track 2 evaluation set and ranks second in the challenge.

Sources

Related papers