End-to-end Domain-Adversarial Voice Activity Detection

arXiv:1910.10655 · eess.AS · Submitted 2019-10-23 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: We're looking at the paper: "End-to-end Domain-Adversarial Voice Activity Detection".

Dev: Here's the gist of the paper. Voice activity detection is addressed by designing an end-to-end neural network combining trainable filters and recurrent layers to tackle voice activity detection directly from the waveform <ref:1910.10655#pg3>.

Rosa: The proposed domain-adversarial approach shows that it brings a relative improvement of more than 10% in out-domain scenarios <ref:1910.10655#pg5>.

Dev: End-to-end Model Design The authors propose an end-to-end model that takes the raw waveform as input and passes it through feature extraction, voice activity detection, and domain classification branches <ref:1910.10655#pg3>.

Rosa: The architecture involves SincNet convolutional layers for feature extraction, a stack of two bi-directional LSTMs followed by three feed-forward layers for the voice activity detection branch, and a unidirectional LSTM followed by max-pooling and one feed-forward layer for the domain classification branch <ref:1910.10655#pg3>.

Paper summary: Rosa: So this paper is about this thing called End-to-end Domain-Adversarial Voice Activity Detection. The main idea they're pushing here is designing a neural network that takes the raw audio waveform directly and figures out voice activity detection from that, instead of using traditional methods like handcrafted features.

Dev: Yeah, it claims this end-to-end model does better than existing variants when tested on the DIHARD dataset. Specifically, it outperforms a version where they just swap out those trainable filters for standard cepstral coefficients <ref:1910.10655#pg3>.

Taro: But they don't stop there; their second big contribution is making this model robust to domain mismatch. They add a domain classification branch to the network so it learns features that aren't tied to a specific recording environment.

Rosa: Exactly, and they use a gradient reversal layer between the voice activity detection part and the domain classification part. This forces the network to extract features that are independent of which specific domain you're testing it on <ref:1910.10655#pg4>.

Dev: It seems like they structure it with three main branches: feature extraction, voice activity detection, and domain classification. The upper branch is trained to minimize a cross-entropy loss using standard back-propagation <ref:1910.10655#pg3>.

Taro: And the domain classification branch minimizes a mean squared error loss where each domain label is one-hot encoded <ref:1910.10655#pg4>. The whole setup is designed to push the system toward finding those domain-independent features.

Rosa: So, what does this mean for us in practical terms? It suggests that by training the model with this adversarial setup, we can get better performance when we move to domains we haven't seen during training <ref:1910.10655#pg5>.

Dev: The numbers they show are pretty compelling; the domain-adversarial approach brings a relative improvement of more than ten percent in out-domain scenarios <ref:1910.10655#pg5>. That's a significant jump over the standard models that ignore domain information.

Conclusion: Rosa: So, wrapping this up, the paper "End-to-end Domain-Adversarial Voice Activity Detection" is about combining trainable filters with recurrent layers to handle voice activity directly from raw waveforms <ref:1910.10655#pg1>.

Dev: The core of the work is that adding that domain classification branch and using the gradient reversal layer helps extract features that generalize across different recording domains <ref:1910.10655#pg4>.

Taro: When you look at the results, it seems like muting or ignoring the specific domain information actually works for extracting robust features that generalize well to new, unseen domains <ref:1910.10655#pg3>.

Rosa: It really suggests that this method could reduce the need for having tons of labeled data for every single environment we want our system to work in <ref:1910.10655#pg3>.

Dev: And because it improves performance compared to standard voice activity detection models that don't use domain information, this strategy might help improve downstream tasks like speaker diarization or speech recognition <ref:1910.10655#pg3>.

Taro: It’s a fully reproducible open-source pipeline they provide, which is great for testing and adapting it to other datasets <ref:1910.10655#pg3>.

Rosa: Overall, this paper shows that the end-to-end approach combined with domain adversarial learning is a viable path toward more robust voice activity detection systems <ref:1910.10655#pg3>.

Marvin Lavechin, Marie-Philippe Gill, Ruben Bousbib, Herve Bredin, Leibny Paola Garcia-Perera

Cognitive Machine Learning team, Ecole Normale Supérie/INRIA, PSL, Paris, France · Ecole de Technologie Supérie, Université du QuĜbec, Montreal, Canada · LIMSI, CNRS, Univ. Paris-Sud, Universite Paris-Saclay

eess.AS

Submitted: 2019-10-23

Updated: 2020-05-26

Comments: submitted to Interspeech 2020

Journal ref: Proc. Interspeech 2020, 3685-3689

DOI: 10.21437/Interspeech.2020-2285

Code: https://github.com/hbredin/DomainAdversarialVoiceActivityDetection

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: Voice activity detection is addressed by designing an end-to-end neural network combining trainable filters and recurrent layers to tackle voice activity detection directly from the waveform…

Key concepts

End-to-end Model Design
The proposed architecture takes raw audio waveforms as input and processes them through three main branches: feature extraction using SincNet convolutional layers, voice activity detection via two bi-directional LSTMs, and domain classification. The upper branch is trained to minimize cross-entropy loss.
Domain Adversarial Training Mechanism
This technique involves training a separate domain classification branch to minimize a mean squared error loss against the main network. A gradient reversal layer is used to update the main network weights, compelling it to extract features that are invariant or independent of the specific domain they are tested on.
SincNet Convolutional Layers
These layers are employed in the feature extraction branch of the model. SincNet is a type of convolutional layer specifically designed for audio processing, helping the network learn meaningful representations directly from the raw waveform data before passing them to the detection and classification parts.
Domain-Independent Features
The goal of this adversarial training is to make sure the features extracted by the model are not specific to a particular recording environment or domain. By minimizing domain loss, the network learns representations that generalize better when applied to new, previously unseen domains.

Terminology

Summary

Voice activity detection is addressed by designing an end-to-end neural network combining trainable filters and recurrent layers to tackle voice activity detection directly from the waveform <ref:1910. p. 0

Where the paper says it

First, we design a neural network combining trainable filters and recurrent layers to tackle voice activity detection directly from the waveform. Experiments on the challenging DIHARD dataset show that the proposed end-to-end model reaches state-of-the-art performance and outperforms a variant where trainable filters are replaced by standard cepstral coefficients.

10655#pg3>. The proposed domain-adversarial approach shows that it brings a relative improvement of more than 10% in out-domain scenarios <ref:1910. p. 0
Where the paper says it

In the in-domain scenario where the training and test sets cover the exact same domains, we show that the domain-adversarial approach does not degrade performance of the proposed end-to-end model. In the out-domain scenario where the test domain is different from training domains, it brings a relative improvement of more than 10%.

10655#pg5>.

End-to-end Model Design

The authors propose an end-to-end model that takes the raw waveform as input and passes it through feature extraction, voice activity detection, and domain classification branches <ref:1910.10655#pg3>. The architecture involves SincNet convolutional layers for feature extraction, a stack of two bi-directional LSTMs followed by three feed-forward layers for the voice activity detection branch, and a unidirectional LSTM followed by max-pooling and one feed-forward layer for the domain classification branch <ref:1910. p. 1

Where the paper says it

The voice activity detection branch is made of a stack of two bi-directional LSTMs, followed by three feed-forward layers. The domain classification branch is made of one unidirectional LSTM, followed by max-pooling along the time axis and one feed-forward layer outputting the probability distribution over the domains.

10655#pg3>. The upper branch is trained to minimize the cross-entropy loss using standard gradient back-propagation <ref:1910. p. 1
Where the paper says it

ture extraction and sequence labeling steps. The upper branch is trained to minimize the cross-entropy loss, using standard gradient back-propagation to update the weights of the feature extraction network θf and the voice activity detection network θv (using abusive notation to highlight function composition): Lv(θf , θv) = − 1 N XN i=1 XT t=1 yit · log θv (θf (xi))t (1) At test time, audio files are processed using overlapping sliding sub-sequences of the same length as the one used in training.

10655#pg3>.

Domain Adversarial Training Mechanism

The domain classification branch is trained in conjunction with the rest of the network to minimize a mean squared error loss, MSE, where each domain label is one-hot encoded <ref:1910.10655#pg4>. Instead of simply summing the two losses, a gradient reversal layer is added in front of the domain classification branch to update the weights as follows: θv ← θv − ∂Lv/∂θv, θd ← θd − ∂Ld/∂θd, and θf ← θf − λ∂Ld/∂θf <ref:1910. p. 1

Where the paper says it

Instead of simply summing the two losses Lv and Ld as in a standard multi-task learning scenario, a gradient reversal layer [2] is added in front of the domain classification branch. The backpropagation update rules become: θv ← θv − ∂Lv ∂θv θd ← θd − ∂Ld ∂θd θf ← θf − ∂Lv ∂θv −λ ∂Ld ∂θd where is the learning rate and λ is a scaling factor controlling the importance of the domain classification loss Ld with respect to the main voice activity detection loss Lv.

10655#pg4>. This mechanism forces the network to extract domain-independent features <ref:1910. p. 0
Where the paper says it

Such as ours, their architecture relies on a first branch responsible for solving the main task, and a secondary branch responsible for classifying the domain. The latter goes through a gradient reversal layer, forcing the network to extract domain-independent features.

10655#pg4>.

Performance Gains on Unseen Data

The domain-adversarial multi-task learning strategy greatly improves performances compared to the standard VAD model which does not use the domain information when applied on unseen domains. Specifically, lines A and D show that the performances are always better when models are tested on target domains that have been seen during training <ref:1910. p. 2

Where the paper says it

Domain-adversarial voice activity detection Returning to the speech activity detection task, Table 2 shows performances of our domain-adversarial architecture on target domains that have been seen, or not, during the training phase. Unsurprisingly, lines A and D show that the performances are always better when models are tested on target domains that have been seen during training.

10655#pg5>. The models trained adversarially on the domain classification task show a 12% relative improvement compared to the single-task voice activity detection in the leave-one-out setup for λ = 1 and λ = 10 <ref:1910. p. 3
Where the paper says it

13 Average relative improvement (in %) Figure 3: Relative improvement (in %) in terms of detection error rate for different values of λ compared to the single-task voice activity detection in the leave-one-out setup. In Figure 3, we show relative improvement, aggregated across domains, in terms of detection error rate for different values of λ, the weight associated to the domain classification loss.

10655#pg5>.

Reproducibility and Open Source

The authors provide a fully reproducible open-source pipeline that can be easily adapted to other datasets. All the code has been implemented using pyannote. p. 3

Where the paper says it

Best results are obtained for λ = 1 and λ = 10 that decrease, on average, the detection error rate by 11. Reproducible research All the code has been implemented (and integrated into) using pyannote.

audio and a Github page provides instructions to reproduce results, along with additional results such as a per-domain analysis as well as ready-to-use pretrained models. p. 3
Where the paper says it

audio [19], a python toolkit to build neural networks for the speaker diarization task. A Github page1 provides instructions to reproduce results, along with additional results such as a per-domain analysis as well as ready-to-use pretrained models.

The paper concludes that muting the domain information appears viable for extracting robust features that generalize well to new unseen domains. p. 3
Where the paper says it

Furthermore, when applied on unseen domains, the domain-adversarial multi-task learning strategy greatly improves the performances compared to the standard VAD model which does not use the domain information. Therefore, on the voice activity detection task, it seems that the strategy of muting the domain information appears as viable for extracting robust features that generalize well to new unseen domains.

Conclusion

The paper explores the learning of filters using the SincNet model in conjunction with domain-adversarial neural networks for explicitly extracting domain-independent features. The end-to-end voice activity detection leads to a significant improvement compared to models based on handcrafted features <ref:1910. p. 3

Where the paper says it

Conclusions This paper explores the learning of filters using the SincNet model [11], in conjunction with the use of domainadversarial neural networks [12] for explicitly extracting domain-independent features. We show that end-to-end voice activity detection leads to a significant improvement compared to models based on handcrafted features.

10655#pg3>. Furthermore, the domain-adversarial multi-task learning strategy greatly improves the performances compared to the standard VAD model which does not use the domain information. p. 3
Where the paper says it

We show that end-to-end voice activity detection leads to a significant improvement compared to models based on handcrafted features. Furthermore, when applied on unseen domains, the domain-adversarial multi-task learning strategy greatly improves the performances compared to the standard VAD model which does not use the domain information.

Therefore, on the voice activity detection task, it seems that the strategy of muting the domain information appears as viable for extracting robust features that generalize well to new unseen domains. p. 3
Where the paper says it

Furthermore, when applied on unseen domains, the domain-adversarial multi-task learning strategy greatly improves the performances compared to the standard VAD model which does not use the domain information. Therefore, on the voice activity detection task, it seems that the strategy of muting the domain information appears as viable for extracting robust features that generalize well to new unseen domains.

The method potentially reduces the need for labelled data and can improve performances on downstream tasks such as the speaker diarization or the speech recognition task that require robust voice activity detection systems. p. 3
Where the paper says it

Therefore, on the voice activity detection task, it seems that the strategy of muting the domain information appears as viable for extracting robust features that generalize well to new unseen domains. This method potentially reduces the need for labelled data and can improve performances on downstream tasks such as the speaker diarization or the speech recognition task that require robust voice activity detection systems.

The paper provides a fully reproducible open-source pipeline that can be easily adapted to other datasets as well as ready-to-use pretrained models. p. 3
Where the paper says it

This method potentially reduces the need for labelled data and can improve performances on downstream tasks such as the speaker diarization or the speech recognition task that require robust voice activity detection systems. Finally, we provide a fully reproducible open-source pipeline that can be easily adapted to other datasets as well as ready-to-use pretrained models.

--- Page 2 ---

Lv(θf, θv) = −

1

N

XN

i=1

XT

t=1

yit· log θv (θf (xi))t <ref:1910.10655#pg3>

--- Page 3 ---

Ld(θf, θd) = 1

N

XN

i=1

kdi − θd (θf (xi))k 2/2 <ref:1910.10655#pg4>

--- Page 4 ---

Table 2: Comparison of regular and domain-adversarial voice activity detection. Results are calculated on the test set for λ = 1. p. 3

Where the paper says it

Table 2: Comparison of regular and domain-adversarial voice activity detection. Results are calculated on the test set for λ = 1.

p. 3
Where the paper says it

Table 2: Comparison of regular and domain-adversarial voice activity detection. Results are calculated on the test set for λ = 1.

p. 3
Where the paper says it

Table 2: Comparison of regular and domain-adversarial voice activity detection. Results are calculated on the test set for λ = 1.

p. 3
Where the paper says it

Table 2: Comparison of regular and domain-adversarial voice activity detection. Results are calculated on the test set for λ = 1.

Line A corresponds to 11 models trained, tuned and tested separately on each of the 11 domains of the dataset <ref:1910. p. 3
Where the paper says it

Results are calculated on the test set for λ = 1. Line A corresponds to 11 models trained, tuned and tested separately on each of the 11 domains of the dataset.

p. 3
Where the paper says it

Results are calculated on the test set for λ = 1. Line A corresponds to 11 models trained, tuned and tested separately on each of the 11 domains of the dataset.

p. 3
Where the paper says it

Results are calculated on the test set for λ = 1. Line A corresponds to 11 models trained, tuned and tested separately on each of the 11 domains of the dataset.

p. 3
Where the paper says it

Results are calculated on the test set for λ = 1. Line A corresponds to 11 models trained, tuned and tested separately on each of the 11 domains of the dataset.

10655#pg5>

--- Page 5 ---

Figure 3: Relative improvement (in %) in terms of detection error rate for different values of λ compared to the single-task voice activity detection in the leave-one-out setup <ref:1910.10655#pg5>

--- Page 7 ---

Table 2: Comparison of regular and domain-adversarial voice activity detection. Results are calculated on the test set for λ = 1. Line A corresponds to 11 models trained, tuned and tested separately on each of the 11 domains of the dataset <ref:1910.10655#pg5>

--- Page 7 ---

Figure 3: Relative improvement (in %) in terms of detection error rate for different values of λ compared to the single-task voice activity detection in the leave-one-out setup <ref:1910.10655#pg5>

--- Page 8 ---

Table 2: Comparison of regular and domain-adversarial voice activity detection. Results are calculated on the test set for λ = 1. Line A corresponds to 11 models trained, tuned and tested separately on each of the 11 domains of the dataset <ref:1910.10655#pg5>

--- Page 8 ---

Figure 3: Relative improvement (in %) in terms of detection error rate for different values of λ compared to the single-task voice activity detection in the leave-one-out setup <ref:1910.10655#pg5>

--- Page 8 ---

Table 2: Comparison of regular and domain-adversarial voice activity detection. Results are calculated on the test set for λ = 1. Line A corresponds to 11 models trained, tuned and tested separately on each of the 11 domains of the dataset <ref:1910.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. End-to-end waveform processing for Voice Activity Detection (VAD) can be implemented by combining trainable filters and recurrent layers to tackle VAD directly from the raw audio stream, as shown in the proposed architecture. This end-to-end model reaches state-of-the-art performance on challenging datasets like DIHARD.

  2. Domain classification branches can be added to the network and trained in an adversarial manner to make the VAD model robust to domain mismatch. This strategy allows models to extract domain independent features by forcing them against a domain classification loss, which can lead to a relative improvement of more than 10% in out-domain scenarios.

  3. The system can be fully reproducible and open-source using a pipeline implemented with pyannote.audio, allowing it to be easily adapted to other datasets for downstream tasks like speaker diarization or speech recognition.

Abstract

Voice activity detection is the task of detecting speech regions in a given audio stream or recording. First, we design a neural network combining trainable filters and recurrent layers to tackle voice activity detection directly from the waveform. Experiments on the challenging DIHARD dataset show that the proposed end-to-end model reaches state-of-the-art performance and outperforms a variant where trainable filters are replaced by standard cepstral coefficients. Our second contribution aims at making the proposed voice activity detection model robust to domain mismatch. To that end, a domain classification branch is added to the network and trained in an adversarial manner. The same DIHARD dataset, drawn from 11 different domains is used for evaluation under two scenarios. In the in-domain scenario where the training and test sets cover the exact same domains, we show that the domain-adversarial approach does not degrade performance of the proposed end-to-end model. In the out-domain scenario where the test domain is different from training domains, it brings a relative improvement of more than 10%. Finally, our last contribution is the provision of a fully reproducible open-source pipeline than can be easily adapted to other datasets.

Sources

Related papers