MADBench: A Benchmark for Modality-Aware Audio Deepfake Detection

MADBench: A Benchmark for Modality-Aware Audio Deepfake Detection

Abstract

Recent advances in speech synthesis and audio generation have made high-fidelity acoustic forgery low-cost and difficult to attribute, enabling a realistic attack scenario in which speech and background audio are independently manipulated over otherwise authentic video. Yet existing research either focuses on visual manipulation, addresses speech detection in isolation, or conflates speech and non-speech audio as a single undifferentiated audio stream, overlooking the distinct forensic challenges posed by background audio. This conflation is consequential: the two acoustic components arise from fundamentally different generative mechanisms, exhibit distinct artifact profiles, and pose different challenges to detection systems. We introduce MADBench, the first benchmark that treats speech and environmental audio as distinct acoustic components, enabling component-aware evaluation of audio deepfake detection across independently manipulated forgery sources. We benchmark representative state-of-the-art detectors and multimodal large language models under a unified protocol. Our experiments reveal that environmental audio manipulation is more detectable than synthetic speech across general-purpose encoders, while existing pretrained detectors fail on both acoustic components, and manipulated environmental audio asymmetrically degrades speech deepfake detection, findings entirely invisible under the single-label paradigm of prior benchmarks. MADBench establishes a rigorous foundation for future research into robust, component-aware audio deepfake detection.

Introduction

The rapid advancement of artificial intelligence has altered the threat landscape of synthetic media. Deepfakes, AI-manipulated content designed to deceive, have evolved from face-swap techniques into sophisticated, multimodal fabrications capable of fooling both human perception and automated detection systems alike. While significant research effort has been devoted to detecting visual forgeries, a quieter and arguably more insidious manipulation vector has received far less scrutiny: the replacement or synthesis of audio in otherwise authentic video.

Consider a realistic and increasingly common attack scenario. An adversary obtains a genuine video recording, a politician’s speech, a corporate announcement, a personal conversation, and leaves the visual content entirely intact. Instead, they replace or synthesize the speech to alter what is being said, and manipulate the background audio to make the fabrication acoustically coherent and convincing. The resulting deepfake is visually authentic by construction, yet entirely deceptive in meaning. This setting is operationally attractive precisely because visual forgery detection has matured considerably, making audio-domain manipulation a lower-cost, lower-risk avenue for bad actors.

This attack surface exposes a critical and underappreciated distinction that the research community has largely failed to formalize: speech and environmental audio are distinct acoustic components. Speech carries linguistic and paralinguistic content, what is said and how it is said, and is generated by fundamentally different synthesis pipelines than non-speech audio. Background soundscapes, ambient acoustics, environmental noise, and foley-style effects constitute a separate generative and perceptual domain, one that plays an equally important role in establishing the perceived authenticity of a video. A fabricated crowd reaction, a synthetically generated room ambiance, or an artificially composed acoustic scene can all serve to reinforce a deceptive narrative, yet none of these are captured by speech-centric detection models.

Despite the growing importance of audio-only deepfakes, no existing benchmark is designed for authentic-video scenarios with independently manipulated speech and non-speech audio. Existing datasets either focus on visual manipulations, treat speech as representative of the entire acoustic stream, or collapse all audio manipulations into a single label. As a result, the research community currently lacks the tools to answer even basic questions: How detectable are synthesized speech tracks when the video is genuine? How do non-speech audio artifacts differ from speech artifacts under detection? Does the presence of manipulated background audio confound or assist speech deepfake detectors? Without benchmarks that explicitly disentangle these acoustic components, it is impossible to systematically evaluate detector robustness or understand the limitations of current approaches.

To address this gap, we introduce the first benchmark designed specifically for audio-component deepfake detection in authentic-video settings, where speech and non-speech audio are manipulated independently while the visual content remains unchanged. Unlike existing datasets, our benchmark explicitly annotates these two acoustic sources as separate forgery components, enabling fine-grained evaluation of component-specific detection performance as well as their interactions. This formulation provides a more realistic and challenging testbed for developing next-generation audio-visual deepfake detectors. Extensive experiments reveal a striking finding: although audio-visual encoders consistently detect environmental audio manipulation more reliably than speech, existing pretrained detectors fail on both acoustic components, and manipulated environmental audio degrades speech deepfake detection without the converse holding, a cross-component effect that remains invisible when speech and non-speech audio are collapsed into a single label, leading to an incomplete assessment of detector robustness.

The contributions are summarized as follows:

• A new benchmark. We introduce MADBench, the first benchmark for audio-component deepfake detection in authentic-video settings, establishing a realistic evaluation task where speech and non-speech audio are manipulated independently.

• A modality-aware dataset. We present a dataset with independently synthesized and annotated speech and non-speech audio, enabling fine-grained evaluation of acoustic deepfakes beyond speech-centric settings.

• A comprehensive testbed. We benchmark representative state-of-the-art deepfake detection models under a unified evaluation protocol, providing strong baselines for future research.

Related Work

Audio-Visual Deepfake Benchmarks: Early benchmarks such as FaceForensics++ and DeeperForensics established face-forgery detection as the dominant paradigm, but their evaluation targets remain visual or binary real/fake. Recent audio-visual benchmarks extend the task beyond visual-only forgery. FakeAVCeleb combines fake faces with synthesized or cloned lip-synced audio, while LAV-DF and AV-Deepfake1M introduce content-driven audio-visual manipulation and temporal localization. These benchmarks are important for multimodal fake detection, but they generally treat audio manipulation as speech, identity, or content manipulation rather than separating speech and environmental audio as independent components.

Speech and Environmental Audio Deepfakes: Speech-only spoofing benchmarks have supported research on detecting synthetic and converted speech, replay attacks, and partially spoofed utterances. However, these benchmarks focus on speech and do not evaluate non-speech background audio. Recent advances in text-to-audio generation have made background-sound manipulation increasingly realistic, creating a forensic challenge that speech-centric benchmarks are not designed to address. This has given rise to a nascent line of work on environmental audio deepfake detection: EnvSDD and Compspoof focus on audio-only environmental sound deepfake detection, while VCapAV studies audio-visual environmental manipulation. However, EnvSDD and Compspoof provide no visual grounding, while VCapAV does not independently control speech and environmental audio as separate forgery components. Neither benchmark supports the systematic evaluation of cross-component interactions. MADBench addresses these gaps by jointly controlling speech manipulation, environmental audio manipulation, and scene consistency within a single component-level audio-visual benchmark.

Deepfake Detection Models: Deepfake detectors use different cues depending on modality. Speech anti-spoofing models such as AASIST focus on synthetic or converted speech artefacts, while audio-visual methods such as AVForensics compare cross-modal signals through lip-sync, speech-mouth alignment, or multimodal fusion. However, none of these methods can attribute a manipulation to a specific acoustic component, nor can they be directly evaluated on environmental audio deepfakes that MADBench is designed to expose. Beyond task-specific detectors, pretrained audio and multimodal encoders such as CLAP and ImageBind provide general-purpose representations that serve as non-task-specific baselines for probing whether component-level deepfake signals are recoverable without forensic fine-tuning. Recent omni models such as MiniCPM-o also represent a qualitatively different detection paradigm: rather than relying on learned forensic features, they apply large-scale multimodal pretraining to reason about audio-visual content zero-shot. Evaluating these models on MADBench directly tests whether general multimodal intelligence transfers to component-level forensic detection, a question that existing benchmarks cannot address due to their lack of component-level annotation.

Dataset Construction

Overall Design Principles: MADBench follows a modality-component hierarchy: each video comprises audio and visual modalities, while the audio modality is further decomposed into speech and environmental audio components. The visual stream is held fixed across all variants of a given source clip, while manipulation is applied exclusively to the speech component, the environmental audio component, or both. This design choice ensures that any difference in model behavior across variants is attributable to audio manipulation alone, rather than being confounded by visual artifacts.

To further control evaluation difficulty and probe distinct model capabilities, we introduce a scene-consistency axis orthogonal to the manipulation type. Manipulated samples are categorized as either scene-matched, where the substituted environmental audio remains semantically plausible given the visible scene, or scene-mismatched, where it is intentionally inconsistent with the visual context. This distinction is critical because it separates two fundamentally different detection strategies: recognizing low-level synthesis artifacts or component-level manipulation cues, versus reasoning about high-level semantic coherence between the audio and visual streams. Together, these two axes, manipulation component and scene consistency form the structural foundation of the benchmark.

Source Preprocessing and Splitting: We build MADBench from AVSpeech, which comprises real-world YouTube videos with visible speakers and naturally paired speech. For each source clip, the original video stream is retained, while MossFormer2 estimates a speech stem ŝ from the mixed waveform x. We define the non-speech residual as ê = x − ŝ, following the mixture-consistency principle that the separated components should reconstruct the original mixture. After quality control, ŝ and ê serve as the source-derived real speech and environmental-audio components.

We validate the decomposition using independent speech-content and leakage checks. All speech stems are successfully transcribed with 95.9% mean ASR coverage, while independent VAD and ASR checks detect no speech in any environmental residual. All residuals also satisfy a minimum environmental-energy criterion. These checks ensure that the retained clips contain intelligible speech and perceptible environmental audio without detectable speech leakage.

Before fake-component generation, source clips are assessed for benchmark eligibility. We retain clips with valid media, usable English transcripts, sufficient spoken content, perceptible environmental audio, and no substantial music or media playback. The retained clips are approximately 4–12 seconds long, making them suitable for controlled audio replacement and final audio-visual assembly. To reduce speaker and source leakage, all clips from the same detected speaker cluster or related source identifiers are assigned to the same training, validation, or test split.

After source-level quality control, 1,892 clips remain eligible for subsequent four-way benchmark construction. These clips are partitioned approximately 7:1:2 into training, validation, and test sets. The split is fixed before fake-component generation, and all derived samples inherit the split assignment of their source clip.

Fake Speech Generation: Fake speech is generated to match the duration of the source real speech and placed into the original speech regions, with non-speech regions left silent. We use three speech synthesis techniques to ensure broad coverage of diverse speaker manipulation strategies.

Speech synthesis. We first use same-identity Text-to-Speech (TTS) models, which synthesises the source transcript using the source speaker as reference, preserving apparent speaker identity. F5-TTS and IndexTTS2 are used for this branch. Cross-identity TTS synthesises the same transcript with a different target speaker. Voice conversion transforms the source speech toward a target speaker’s timbre while preserving source content and timing. SeedVC and Vevo-Timbre are used for this branch. Together, the three branches yield six generation settings, reducing dependence on any single manipulation mechanism.

Speaker pairing. Cross-identity generation follows a fixed source-target pairing table, constrained to same-gender pairs from different source videos and different detected speaker clusters. Speaker embeddings are used to exclude near-duplicate voices, and target-speaker usage is balanced to prevent a small number of identities from dominating the pool. Each source clip is paired with four target speakers, providing redundancy for QC-based selection. ECAPA similarity between source and target speakers is low (mean = 0.0268, max = 0.3040), confirming acoustic distinctiveness.

Quality control. Generated speech components are assessed against automatic QC criteria covering generation duration, timeline alignment and ASR-based transcript preservation. Furthermore, same-identity TTS is checked for source-speaker preservation; cross-identity TTS and voice conversion are checked for target-speaker similarity and source-speaker leakage.

Fake Environmental Audio Generation: Environmental-audio generation produces full-duration background components to replace the real environmental track. To construct scene-matched and scene-mismatched variants systematically, we require a shared definition of what sounds belong to each type of visible scene. We therefore build a visual-scene taxonomy to guide both audio generation and mismatch selection. The pipeline comprises three stages: taxonomy construction, audio generation, and QC.

Taxonomy Construction. To ensure reproducible scene matching, we first define a taxonomy of visual scenes, where each label specifies (i) expected environmental sounds, (ii) excluded sound categories, and (iii) mismatch scene types. The taxonomy is constructed exclusively from the training and validation set. Qwen2.5-VL-32B summarizes the visual scene, expected environmental sounds, and potential audio confounders (e.g., speech or music), providing standardized descriptions for taxonomy construction. The resulting taxonomy is refined through an LLM-assisted review followed by manual verification to remove redundant labels, identify missing scene categories, and validate mismatch relationships. The final taxonomy contains 10 coarse and 63 fine-grained scene categories spanning indoor, outdoor, transportation, industrial, educational, and natural environments. Since the environmental component represents background ambience only, all labels explicitly exclude speech, singing, music, and media playback.

After the taxonomy is finalized, all source clips ar…

Based on the paper, here are the specific improvements you can make to AI systems and what the improved systems can do:

1. Component-Aware Audio Deepfake Detection

• Improvement: Train detectors to explicitly separate and analyze speech and environmental audio as independent forensic signals, rather than treating audio as a single stream. Use separate detection heads or loss terms for each component.

• What the improved system can do: Detect whether speech, background audio, or both have been manipulated in a video, even when only one component is fake. It can attribute a deepfake to a specific acoustic source, enabling more precise forensic reporting and response.

2. Cross-Component Interference Modeling

• Improvement: Design models that are robust to the presence of fake environmental audio when detecting fake speech. This can be achieved by training with paired augmentation (e.g., randomly mixing fake environmental audio with real speech) or by using disentangled representations that isolate speech cues from background context.

• What the improved system can do: Maintain high speech deepfake detection accuracy even when background audio is also manipulated, avoiding the degradation observed in current models. This is critical for real-world attacks where both components are often altered together.

3. Scene-Consistency Reasoning for Audio-Visual Verification

• Improvement: Integrate cross-modal scene-consistency modules that compare environmental audio semantics with visual scene content, using fine-grained scene labels (e.g., indoor office vs. outdoor street) rather than coarse categories. Train on paired real and synthetic audio with both matched and mismatched scenes.

• What the improved system can do: Detect deepfakes where the background audio is semantically inconsistent with the visible scene (e.g., a beach soundscape in an office video), even when the audio itself is high-quality and artifact-free. It can also flag scene-mismatched manipulations that audio-only detectors miss.

4. Zero-Shot Forensic Attribution with Multimodal LLMs

• Improvement: Fine-tune omni models (e.g., Qwen2.5-Omni, MiniCPM-o) on component-level forensic tasks using instruction tuning with explicit prompts like Is the speech fake? Is the background audio fake? Are they consistent with the scene? Include synthetic and real examples with component-level labels.

• What the improved system can do: Perform zero-shot or few-shot deepfake detection and component attribution on unseen video types without task-specific training, making it deployable in dynamic, real-world scenarios where new manipulation methods emerge.

5. Transferable Forensic Representations via Broad Audio-Visual Pretraining

• Improvement: Pretrain encoders on diverse audio-visual data that includes both speech and environmental sounds (e.g., using contrastive learning on scene-audio pairs), rather than face-centric or speech-only datasets. Then fine-tune lightweight heads for specific forensic tasks.

• What the improved system can do: Transfer effectively to new deepfake benchmarks and manipulation types, outperforming task-specific detectors that fail on out-of-distribution data. It can generalize across different generators, scenes, and acoustic conditions.

6. Audio-Only Forensic Backbone with Optional Visual Grounding

• Improvement: Build a two-stream system: a strong audio-only forensic backbone (e.g., using broad audio encoders like CLAP or DF-Arena) for primary manipulation detection, and a separate visual-scene module for consistency checking. Fuse them only when scene grounding is needed.

• What the improved system can do: Achieve high detection accuracy using audio alone, while optionally leveraging video to catch scene-inconsistent manipulations. This reduces computational cost and improves robustness in scenarios where video is missing or corrupted.

7. Fine-Grained Scene Taxonomy Integration

• Improvement: Train models with a hierarchical scene taxonomy (coarse and fine-grained labels) that specifies expected environmental sounds, excluded sounds (e.g., speech, music), and valid mismatch pairs. Use this taxonomy to generate training data and as a conditioning signal for detection.

• What the improved system can do: Distinguish between subtle scene mismatches (e.g., a quiet library vs. a busy café) and avoid false positives from plausible but slightly different background audio. It can also provide explainable outputs by indicating which scene label is violated.

8. Robustness to Generator Diversity

• Improvement: Train detectors on a balanced mix of synthetic speech (TTS, voice conversion) and environmental audio (text-to-audio, video-to-audio, audio-to-audio) from multiple generators, as done in MADBench, to prevent overfitting to a single synthesis artifact.

• What the improved system can do: Detect deepfakes from unseen generators or new synthesis methods, maintaining high performance across a wide range of attack vectors without retraining.

9. Confidence Calibration for Forensic Decisions

• Improvement: Implement calibrated confidence scores for component-level predictions, using validation-set thresholds per task (binary, four-way, speech, environmental). Output uncertainty estimates for each acoustic component.

• What the improved system can do: Provide reliable confidence levels for forensic analysts, enabling them to prioritize high-confidence alarms and avoid acting on uncertain detections. This is essential for legal or security applications where false positives are costly.

10. Scene-Consistency-Aware Training for Synthetic Audio

• Improvement: Train models specifically on pairs of synthetic environmental audio with both scene-matched and scene-mismatched videos, as in the D-S condition, to learn to separate synthesis artifacts from scene-correspondence cues.

• What the improved system can do: Detect deepfakes where both audio components are synthetic but scene-consistent, a setting where current models fail. It can also identify the specific reason for a deepfake (artifact-based vs. scene-mismatch-based), aiding in forensic analysis.

Summary of Capabilities of the Improved AI System:

• Detects and attributes deepfakes to specific audio components (speech vs. background) with high accuracy.

• Remains robust when both components are manipulated, avoiding cross-component interference.

• Uses visual scene information to catch semantic inconsistencies, not just acoustic artifacts.

• Transfers to new manipulation methods and generators without retraining.

• Works in zero-shot or few-shot settings for rapid deployment.

• Provides explainable, component-level outputs with calibrated confidence, suitable for real-world forensic use.

Recent advances in speech synthesis and audio generation have made high-fidelity acoustic forgery low-cost and difficult to attribute, enabling a realistic attack scenario in which speech and background audio are independently manipulated over otherwise authentic video. Yet existing research either focuses on visual manipulation, addresses speech detection in isolation, or conflates speech and non-speech audio as a single undifferentiated audio stream, overlooking the distinct forensic challenges posed by background audio. This conflation is consequential: the two acoustic components arise from fundamentally different generative mechanisms, exhibit distinct artifact profiles, and pose different challenges to detection systems. We introduce MADBench, the first benchmark that treats speech and environmental audio as distinct acoustic components, enabling component-aware evaluation of audio deepfake detection across independently manipulated forgery sources. We benchmark representative state-of-the-art detectors and multimodal large language models under a unified protocol. Our experiments reveal that environmental audio manipulation is more detectable than synthetic speech across general-purpose encoders, while existing pretrained detectors fail on both acoustic components, and manipulated environmental audio asymmetrically degrades speech deepfake detection, findings entirely invisible under the single-label paradigm of prior benchmarks. MADBench establishes a rigorous foundation for future research into robust, component-aware audio deepfake detection.