Self-Supervised Speech Representations for Cross-Speaker Dysarthria Detection During Awake Craniotomy

The gist The results show that cross-speaker performance is influenced more strongly by the speech representation than by classifier choice, indicating that reliable patient-specific speech isolation and strong pretrained representations are more important than increased classifier complexity in low-resource intra-operative settings (Page 1).

How it works

The study presents a systematic component-wise evaluation of a pipeline for distinguishing dysarthric from no-trouble speech in the DATABRASE corpus of awake-craniotomy recordings (Page 1). This pipeline incorporates several key steps to enhance cross-speaker robustness, including speaker diarization to isolate patient speech, a multi-view representation combining handcrafted acoustic descriptors with multilayer wav2vec 2.0 embeddings, speaker-conditional normalization and transferabilitybased feature selection, and a cascaded classifier comprising a gradient-boosted first stage and a neural second stage (Page 1).

The framework is designed to address the constraints of the intraoperative environment through sequential processing stages (Page 1). The pipeline involves six stages, starting from raw-audio conditioning to speaker-grouped evaluation (Page 5). Key components include HiFiGAN pitch augmentation using a vocoder that improves prosodic diversity while retaining dysarthric cues (Page 4), speaker diarization using pyannote.audio to separate patient speech from clinician speech (Page 6), and the use of complementary feature families such as conventional acoustic descriptors, conversational features from diarization, and multi-layer self-supervised embeddings (Page 6).

Data Augmentation and Feature Extraction

To mitigate the severe class imbalance in the cohort, augmentation is applied using a HiFi-GAN neural vocoder to expand training folds with label-preserving synthetic clips (Page 4). This augmentation involves drawing pitch offsets uniformly from ±2 semitones, which stays clinically plausible while remaining labelpreserving (Page 4).

Feature extraction is performed on three distinct groups of features organized by the region on which they can be measured validly (Page 6). The first group covers spectral and temporal properties of the clip as a whole, including cepstral coefficients and pause statistics (Page 6). The second group covers source- and articulator-level properties, which are computed on patient frames only, such as prosodic measures and glottal-source descriptors (Page 6). The third group is derived from the diarization output, encoding the dynamics of the intra-operative exchange like response latency (Page 6).

Per-fold Preprocessing and Representation

Per-fold preprocessing involves three operations applied to conventional descriptors only (Page 6). These include speaker-conditional centering, which expresses each conventional descriptor relative to the patient’s own baseline, and a transferability filter that retains features whose sign of Spearman correlation with the label agrees across at least 80% of training speakers (Page 6). The self-supervised representation is derived by applying a frozen French-pretrained W2V2 encoder to the patient frames, retaining layers 6, 9, and the final layer to create zSSL = z6 z9 zfinal (Page 6).

Classification Models and Evaluation

The classification framework evaluates three single-stage classifiers—XGBoost, Multilayer perceptron (MLP), and Bidirectional GRU (BiGRU)—along with two proposed cascades that couple a gradient-boosted stage with a neural one (Page 7). The Cascade-MLP is particularly noted for its performance gains under the stricter LOSO protocol, attaining 0.6669 AUC against the single-stage MLP's 0.6117 (Page 8).

Evaluation protocols include Speaker-independent (SID), Speaker-dependent (SD), and Leave-One-Speaker-Out (LOSO) evaluation (Page 8). The results demonstrate that the difference between SD and SID evaluation quantifies the cost of transferring across speakers rather than calibrating to one (Page 9).

Conclusion

The work concludes that cross-speaker performance is governed principally by the representation rather than the classifier (Page 12). Enriching the representation from conventional descriptors to a multilayer self-supervised fusion raises AUC by 18.2% to 26.1%, an order of magnitude larger, with diarization-conditioned extraction adding a further consistent gain (Page 12). The priority should be clean patient-attributable measurement and strong pretrained representations rather than more complex classifiers for low-resource intra-operative settings (Page 12).

The main limitation is the small speaker cohort, and the most promising directions for further work are the acquisition of a larger and more diverse speaker set and the fine-tuning, rather than freezing of the self-supervised encoder once cohort size makes this safe against overfitting to speaker identity (Page 12). The binary dysarthria versus no-trouble target also collapses severity levels and does not address whether the dysarthric signature varies with the cortical or subcortical stimulation site, suggesting a multiclass extension would add clinical granularity beyond that of prior AC work (Page 12). Finally, generalization beyond DATABRASE remains untested, and validation on independent AC cohorts at different centers is the natural next step before stronger deployment claims can be made (Page 12).

--- Page 1 ---

PREPRINT 1

Self-Supervised Speech Representations for Cross-Speaker Dysarthria Detection During Awake Craniotomy Chinmayi Kanthila, Nassib Abdallah, Harrison Misy, Celine Panheleux, Vanessa Saliou, Romuald Seizeur, and Guillaume Dardenne Abstract—Detecting intra-operative speech impairment during awake craniotomy is essential for preserving language function. However, automated detection remains challenging because operating-room recordings contain substantial acoustic interference, clinically relevant speech events are rare, and available cohorts are small and heterogeneous. This study presents a systematic component-wise evaluation of a pipeline for distinguishing dysarthric from no-trouble speech in the DATABRASE corpus of awake-craniotomy recordings (Page 1). The pipeline incorporates speaker diarization to isolate patient speech, a multi-view representation combining handcrafted acoustic descriptors with multilayer wav2vec 2.0 embeddings, speaker-conditional normalization and transferabilitybased feature selection to improve cross-speaker robustness, and a cascaded classifier comprising a gradient-boosted first stage and a neural second stage (Page 1). Evaluation was conducted under strict speaker-independent conditions using leave-one-speakerout cross-validation (Page 1). The results show that cross-speaker performance is influenced more strongly by the speech representation than by classifier choice. The AUCs of three classifiers differed by no more than 4.7%, whereas replacing conventional acoustic descriptors with the multilayer self-supervised representation produced AUC improvements of 18.2%–26.1%. Diarizationconditioned feature extraction and the proposed classifier cascade provided additional consistent gains (Page 1). These findings indicate that reliable patient-specific speech isolation and strong pretrained representations are more important than increased classifier complexity in low-resource intra-operative settings (Page 1). They also quantify the potential performance gains that may be achieved through patient-specific preoperative calibration (Page 1). Index Terms—Awake craniotomy, dysarthria detection, selfsupervised speech representations, speaker diarization, crossspeaker generalization, intraoperative monitoring.

--- Page 2 ---

PREPRINT 2

II. BACKGROUND AND RELATED WORK Automated detection of dysarthric speech has developed almost entirely on chronic motor-speech disorders recorded under controlled conditions (Page 2). Existing literature mainly focuses on what is methodologically achievable, including which acoustic properties carry the impairment, which representations expose them, and which evaluation protocols provide an honest estimate of performance on a new patient (Page 2). A. Automated dysarthria detection in general conditions Dysarthria is a group of motor-speech disorders arising from impaired control of the articulators (Page 2). Most automated work uses the UA-Speech or TORGO corpora [6], [7], which contain speakers with chronic dysarthria of graded severity recorded under controlled conditions (Page 2). Early systems described these recordings with interpretable acoustic descriptors. Prabhakera and Alku [8] classified dysarthric speech using glottal-source parameters with a support vector machine, reporting 94.29% under leave-onespeaker-out (LOSO) evaluation, and showing that sourcelevel measures alone carry substantial diagnostic information (Page 2). Prosodic and voice-quality descriptors have been used in the same way, selected for their correspondence to identifiable production mechanisms [9], [10] (Page 2). Millet and Zeghidour [11] instead learned the front end directly from the waveform, obtaining roughly a 10% gain over fixed features and demonstrating that hand-designed descriptors do not exhaust the available signal, though at the cost of clinical interpretability (Page 2).

1. Improved cross-speaker generalization by prioritizing patient-specific calibration over classifier complexity, as cross-speaker performance is influenced more strongly by the speech representation than by classifier choice. This implies that instead of focusing on optimizing the gradient-boosted or neural stage alone, research efforts should concentrate on enhancing the multilayer self-supervised representation to capture speaker-agnostic speech structure.

2. Enhanced detection sensitivity through diarization-conditioned feature extraction, as this isolates the frames on which contamination-sensitive features are measured and yields conversational descriptors of the clinician–patient exchange. This allows the system to leverage information from clinician conduct, such as response latency and follow-up behavior, which provides context correlated with impairment.

3. Increased discriminative power by employing a multi-layer self-supervised representation, specifically retaining layers 6, 9, and the final layer of wav2vec 2.0 embeddings to capture complementary information: phonetic detail emerges in midnetwork layers, phonological structure in deeper layers, and task- and speaker-conditional information in the final layers. This fusion allows the model to capture the simultaneous disturbance of articulatory precision, phonological realization, and intelligibility caused by dysarthria.

4. Improved robustness against noise and contamination by using per-fold preprocessing steps that include Speaker-conditional centering, which expresses each descriptor relative to the patient’s own baseline, and a transferability filter that retains features only if their correlation with the label is consistent across speakers, as defined by the criterion: Sj ≥ 0.8.