End-to-end Domain-Adversarial Voice Activity Detection
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.
Rosa: We're looking at the paper: "End-to-end Domain-Adversarial Voice Activity Detection".
Dev: Here's the gist of the paper. Voice activity detection is addressed by designing an end-to-end neural network combining trainable filters and recurrent layers to tackle voice activity detection directly from the waveform <ref:1910.10655#pg3>.
Rosa: The proposed domain-adversarial approach shows that it brings a relative improvement of more than 10% in out-domain scenarios <ref:1910.10655#pg5>.
Dev: End-to-end Model Design The authors propose an end-to-end model that takes the raw waveform as input and passes it through feature extraction, voice activity detection, and domain classification branches <ref:1910.10655#pg3>.
Rosa: The architecture involves SincNet convolutional layers for feature extraction, a stack of two bi-directional LSTMs followed by three feed-forward layers for the voice activity detection branch, and a unidirectional LSTM followed by max-pooling and one feed-forward layer for the domain classification branch <ref:1910.10655#pg3>.
Paper summary: Rosa: So this paper is about this thing called End-to-end Domain-Adversarial Voice Activity Detection. The main idea they're pushing here is designing a neural network that takes the raw audio waveform directly and figures out voice activity detection from that, instead of using traditional methods like handcrafted features.
Dev: Yeah, it claims this end-to-end model does better than existing variants when tested on the DIHARD dataset. Specifically, it outperforms a version where they just swap out those trainable filters for standard cepstral coefficients <ref:1910.10655#pg3>.
Taro: But they don't stop there; their second big contribution is making this model robust to domain mismatch. They add a domain classification branch to the network so it learns features that aren't tied to a specific recording environment.
Rosa: Exactly, and they use a gradient reversal layer between the voice activity detection part and the domain classification part. This forces the network to extract features that are independent of which specific domain you're testing it on <ref:1910.10655#pg4>.
Dev: It seems like they structure it with three main branches: feature extraction, voice activity detection, and domain classification. The upper branch is trained to minimize a cross-entropy loss using standard back-propagation <ref:1910.10655#pg3>.
Taro: And the domain classification branch minimizes a mean squared error loss where each domain label is one-hot encoded <ref:1910.10655#pg4>. The whole setup is designed to push the system toward finding those domain-independent features.
Rosa: So, what does this mean for us in practical terms? It suggests that by training the model with this adversarial setup, we can get better performance when we move to domains we haven't seen during training <ref:1910.10655#pg5>.
Dev: The numbers they show are pretty compelling; the domain-adversarial approach brings a relative improvement of more than ten percent in out-domain scenarios <ref:1910.10655#pg5>. That's a significant jump over the standard models that ignore domain information.
Conclusion: Rosa: So, wrapping this up, the paper "End-to-end Domain-Adversarial Voice Activity Detection" is about combining trainable filters with recurrent layers to handle voice activity directly from raw waveforms <ref:1910.10655#pg1>.
Dev: The core of the work is that adding that domain classification branch and using the gradient reversal layer helps extract features that generalize across different recording domains <ref:1910.10655#pg4>.
Taro: When you look at the results, it seems like muting or ignoring the specific domain information actually works for extracting robust features that generalize well to new, unseen domains <ref:1910.10655#pg3>.
Rosa: It really suggests that this method could reduce the need for having tons of labeled data for every single environment we want our system to work in <ref:1910.10655#pg3>.
Dev: And because it improves performance compared to standard voice activity detection models that don't use domain information, this strategy might help improve downstream tasks like speaker diarization or speech recognition <ref:1910.10655#pg3>.
Taro: It’s a fully reproducible open-source pipeline they provide, which is great for testing and adapting it to other datasets <ref:1910.10655#pg3>.
Rosa: Overall, this paper shows that the end-to-end approach combined with domain adversarial learning is a viable path toward more robust voice activity detection systems <ref:1910.10655#pg3>.
Marvin Lavechin, Marie-Philippe Gill, Ruben Bousbib, Herve Bredin, Leibny Paola Garcia-Perera
Cognitive Machine Learning team, Ecole Normale Supérie/INRIA, PSL, Paris, France · Ecole de Technologie Supérie, Université du QuĜbec, Montreal, Canada · LIMSI, CNRS, Univ. Paris-Sud, Universite Paris-Saclay
eess.AS
Submitted: 2019-10-23
Updated: 2020-05-26
Comments: submitted to Interspeech 2020
Journal ref: Proc. Interspeech 2020, 3685-3689
DOI: 10.21437/Interspeech.2020-2285
Code: https://github.com/hbredin/DomainAdversarialVoiceActivityDetection
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 91/100
The gist: Voice activity detection is addressed by designing an end-to-end neural network combining trainable filters and recurrent layers to tackle voice activity detection directly from the waveform…
Key concepts
- End-to-end Model Design
- The proposed architecture takes raw audio waveforms as input and processes them through three main branches: feature extraction using SincNet convolutional layers, voice activity detection via two bi-directional LSTMs, and domain classification. The upper branch is trained to minimize cross-entropy loss.
- Domain Adversarial Training Mechanism
- This technique involves training a separate domain classification branch to minimize a mean squared error loss against the main network. A gradient reversal layer is used to update the main network weights, compelling it to extract features that are invariant or independent of the specific domain they are tested on.
- SincNet Convolutional Layers
- These layers are employed in the feature extraction branch of the model. SincNet is a type of convolutional layer specifically designed for audio processing, helping the network learn meaningful representations directly from the raw waveform data before passing them to the detection and classification parts.
- Domain-Independent Features
- The goal of this adversarial training is to make sure the features extracted by the model are not specific to a particular recording environment or domain. By minimizing domain loss, the network learns representations that generalize better when applied to new, previously unseen domains.
Terminology
Summary
Voice activity detection is addressed by designing an end-to-end neural network combining trainable filters and recurrent layers to tackle voice activity detection directly from the waveform <ref:1910. p. 0 First, we design a neural network combining trainable filters and recurrent layers to tackle
voice activity detection directly from the waveform. Experiments on the challenging DIHARD dataset show that the proposed end-to-end model reaches state-of-the-art performance
and outperforms a variant where trainable filters are replaced
by standard cepstral coefficients. In the in-domain
scenario where the training and test sets cover the exact same
domains, we show that the domain-adversarial approach does
not degrade performance of the proposed end-to-end model. In
the out-domain scenario where the test domain is different from
training domains, it brings a relative improvement of more than
10%.Where the paper says it
Where the paper says it
End-to-end Model Design
The authors propose an end-to-end model that takes the raw waveform as input and passes it through feature extraction, voice activity detection, and domain classification branches <ref:1910.10655#pg3>. The architecture involves SincNet convolutional layers for feature extraction, a stack of two bi-directional LSTMs followed by three feed-forward layers for the voice activity detection branch, and a unidirectional LSTM followed by max-pooling and one feed-forward layer for the domain classification branch <ref:1910. p. 1 The voice activity detection branch is made of a
stack of two bi-directional LSTMs, followed by three feed-forward layers. The domain classification branch is made of one unidirectional LSTM, followed by max-pooling along the time axis and one feed-forward layer outputting the probability distribution over
the domains. ture extraction and sequence labeling steps. The upper branch is
trained to minimize the cross-entropy loss, using standard gradient back-propagation to update the weights of the feature extraction network θf and the voice activity detection network θv
(using abusive notation to highlight function composition):
Lv(θf , θv) = −
1
N
XN
i=1
XT
t=1
yit
· log θv (θf (xi))t
(1)
At test time, audio files are processed using overlapping
sliding sub-sequences of the same length as the one used in
training.Where the paper says it
Where the paper says it
Domain Adversarial Training Mechanism
The domain classification branch is trained in conjunction with the rest of the network to minimize a mean squared error loss, MSE, where each domain label is one-hot encoded <ref:1910.10655#pg4>. Instead of simply summing the two losses, a gradient reversal layer is added in front of the domain classification branch to update the weights as follows: θv ← θv − ∂Lv/∂θv, θd ← θd − ∂Ld/∂θd, and θf ← θf − λ∂Ld/∂θf <ref:1910. p. 1 Instead of simply summing the two losses Lv and Ld as in a standard multi-task learning scenario, a gradient reversal layer [2]
is added in front of the domain classification branch. The backpropagation update rules become:
θv ← θv −
∂Lv
∂θv
θd ← θd −
∂Ld
∂θd
θf ← θf −
∂Lv
∂θv
−λ
∂Ld
∂θd
where is the learning rate and λ is a scaling factor controlling
the importance of the domain classification loss Ld with respect
to the main voice activity detection loss Lv. Such as ours, their
architecture relies on a first branch responsible for solving the
main task, and a secondary branch responsible for classifying
the domain. The latter goes through a gradient reversal layer,
forcing the network to extract domain-independent features.Where the paper says it
Where the paper says it
Performance Gains on Unseen Data
The domain-adversarial multi-task learning strategy greatly improves performances compared to the standard VAD model which does not use the domain information when applied on unseen domains. Specifically, lines A and D show that the performances are always better when models are tested on target domains that have been seen during training <ref:1910. p. 2 Domain-adversarial voice activity detection
Returning to the speech activity detection task, Table 2 shows
performances of our domain-adversarial architecture on target
domains that have been seen, or not, during the training phase. Unsurprisingly, lines A and D show that the performances
are always better when models are tested on target domains that
have been seen during training. 13
Average relative improvement (in %)
Figure 3: Relative improvement (in %) in terms of detection
error rate for different values of λ compared to the single-task
voice activity detection in the leave-one-out setup. In Figure 3, we show relative improvement, aggregated
across domains, in terms of detection error rate for different values of λ, the weight associated to the domain classification loss.Where the paper says it
Where the paper says it
Reproducibility and Open Source
The authors provide a fully reproducible open-source pipeline that can be easily adapted to other datasets. All the code has been implemented using pyannote. p. 3 Best results are obtained
for λ = 1 and λ = 10 that decrease, on average, the detection
error rate by 11. Reproducible research
All the code has been implemented (and integrated into) using
pyannote. audio [19], a python toolkit to build neural networks for the speaker diarization task. A Github page1
provides
instructions to reproduce results, along with additional results
such as a per-domain analysis as well as ready-to-use pretrained
models. Furthermore, when applied on unseen domains, the
domain-adversarial multi-task learning strategy greatly improves the performances compared to the standard VAD model
which does not use the domain information. Therefore, on the
voice activity detection task, it seems that the strategy of muting the domain information appears as viable for extracting robust features that generalize well to new unseen domains.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
The paper explores the learning of filters using the SincNet model in conjunction with domain-adversarial neural networks for explicitly extracting domain-independent features. The end-to-end voice activity detection leads to a significant improvement compared to models based on handcrafted features <ref:1910. p. 3 Conclusions
This paper explores the learning of filters using the SincNet model [11], in conjunction with the use of domainadversarial neural networks [12] for explicitly extracting
domain-independent features. We show that end-to-end voice activity detection leads to
a significant improvement compared to models based on handcrafted features. We show that end-to-end voice activity detection leads to
a significant improvement compared to models based on handcrafted features. Furthermore, when applied on unseen domains, the
domain-adversarial multi-task learning strategy greatly improves the performances compared to the standard VAD model
which does not use the domain information. Furthermore, when applied on unseen domains, the
domain-adversarial multi-task learning strategy greatly improves the performances compared to the standard VAD model
which does not use the domain information. Therefore, on the
voice activity detection task, it seems that the strategy of muting the domain information appears as viable for extracting robust features that generalize well to new unseen domains. Therefore, on the
voice activity detection task, it seems that the strategy of muting the domain information appears as viable for extracting robust features that generalize well to new unseen domains. This
method potentially reduces the need for labelled data and can
improve performances on downstream tasks such as the speaker
diarization or the speech recognition task that require robust
voice activity detection systems. This
method potentially reduces the need for labelled data and can
improve performances on downstream tasks such as the speaker
diarization or the speech recognition task that require robust
voice activity detection systems. Finally, we provide a fully reproducible open-source
pipeline that can be easily adapted to other datasets as well as
ready-to-use pretrained models.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 2 ---
Lv(θf, θv) = −
1
N
XN
i=1
XT
t=1
yit· log θv (θf (xi))t <ref:1910.10655#pg3>
--- Page 3 ---
Ld(θf, θd) = 1
N
XN
i=1
kdi − θd (θf (xi))k 2/2 <ref:1910.10655#pg4>
--- Page 4 ---
Table 2: Comparison of regular and domain-adversarial voice activity detection. Results are calculated on the test set for λ = 1. p. 3 Table 2: Comparison of regular and domain-adversarial voice activity detection. Results are calculated on the test set for λ = 1. Table 2: Comparison of regular and domain-adversarial voice activity detection. Results are calculated on the test set for λ = 1. Table 2: Comparison of regular and domain-adversarial voice activity detection. Results are calculated on the test set for λ = 1. Table 2: Comparison of regular and domain-adversarial voice activity detection. Results are calculated on the test set for λ = 1. Results are calculated on the test set for λ = 1. Line
A corresponds to 11 models trained, tuned and tested separately on each of the 11 domains of the dataset. Results are calculated on the test set for λ = 1. Line
A corresponds to 11 models trained, tuned and tested separately on each of the 11 domains of the dataset. Results are calculated on the test set for λ = 1. Line
A corresponds to 11 models trained, tuned and tested separately on each of the 11 domains of the dataset. Results are calculated on the test set for λ = 1. Line
A corresponds to 11 models trained, tuned and tested separately on each of the 11 domains of the dataset.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 5 ---
Figure 3: Relative improvement (in %) in terms of detection error rate for different values of λ compared to the single-task voice activity detection in the leave-one-out setup <ref:1910.10655#pg5>
--- Page 7 ---
Table 2: Comparison of regular and domain-adversarial voice activity detection. Results are calculated on the test set for λ = 1. Line A corresponds to 11 models trained, tuned and tested separately on each of the 11 domains of the dataset <ref:1910.10655#pg5>
--- Page 7 ---
Figure 3: Relative improvement (in %) in terms of detection error rate for different values of λ compared to the single-task voice activity detection in the leave-one-out setup <ref:1910.10655#pg5>
--- Page 8 ---
Table 2: Comparison of regular and domain-adversarial voice activity detection. Results are calculated on the test set for λ = 1. Line A corresponds to 11 models trained, tuned and tested separately on each of the 11 domains of the dataset <ref:1910.10655#pg5>
--- Page 8 ---
Figure 3: Relative improvement (in %) in terms of detection error rate for different values of λ compared to the single-task voice activity detection in the leave-one-out setup <ref:1910.10655#pg5>
--- Page 8 ---
Table 2: Comparison of regular and domain-adversarial voice activity detection. Results are calculated on the test set for λ = 1. Line A corresponds to 11 models trained, tuned and tested separately on each of the 11 domains of the dataset <ref:1910.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
End-to-end waveform processing for Voice Activity Detection (VAD) can be implemented by combining trainable filters and recurrent layers to tackle VAD directly from the raw audio stream, as shown in the proposed architecture. This end-to-end model reaches state-of-the-art performance on challenging datasets like DIHARD.
-
Domain classification branches can be added to the network and trained in an adversarial manner to make the VAD model robust to domain mismatch. This strategy allows models to extract
domain independent features
by forcing them against a domain classification loss, which can lead to a relative improvement of more than 10% in out-domain scenarios. -
The system can be fully reproducible and open-source using a pipeline implemented with pyannote.audio, allowing it to be easily adapted to other datasets for downstream tasks like speaker diarization or speech recognition.
Abstract
Voice activity detection is the task of detecting speech regions in a given audio stream or recording. First, we design a neural network combining trainable filters and recurrent layers to tackle voice activity detection directly from the waveform. Experiments on the challenging DIHARD dataset show that the proposed end-to-end model reaches state-of-the-art performance and outperforms a variant where trainable filters are replaced by standard cepstral coefficients. Our second contribution aims at making the proposed voice activity detection model robust to domain mismatch. To that end, a domain classification branch is added to the network and trained in an adversarial manner. The same DIHARD dataset, drawn from 11 different domains is used for evaluation under two scenarios. In the in-domain scenario where the training and test sets cover the exact same domains, we show that the domain-adversarial approach does not degrade performance of the proposed end-to-end model. In the out-domain scenario where the test domain is different from training domains, it brings a relative improvement of more than 10%. Finally, our last contribution is the provision of a fully reproducible open-source pipeline than can be easily adapted to other datasets.
Sources
Related papers
- WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling
- From perception to production: how acoustic invariance facilitates articulatory learning in a self-supervised vocal imitation model
- Read What You Hear: Reference-Free Hypotheses Evaluation with Acoustic Discrepancy
- pyFDN: A Python Toolbox for Feedback Delay Network Design, Analysis, and Optimization
- GigaSpeechBench: A Real-World Multilingual Speech-to-Text Benchmark
- DM-ASR: Diarization-aware Multi-speaker ASR with Unified Offline and Online Decoding with Large Language Models