Structured Phonological Representations for Audio-Articulatory rtMRI Speech Classification

This paper investigates whether PhonoQ, an audio-based model trained to recognize structured phonological features, provides useful information for audio–articulatory modeling using real-time MRI (rtMRI) speech data. The authors use the 75-Speaker Annot-16 dataset, which contains synchronized speech audio, phonetic alignments, and expert-verified articulatory contour annotations from midsagittal vocal-tract MRI recordings. Each example is represented as a 9-frame contour window centered on the target frame, with each frame encoded as a 380-dimensional contour vector.

The paper evaluates two settings: inference-time feature fusion and training-only teacher supervision for contour-only prediction. In the feature-fusion setting, audio-derived representations from HuBERT-large, WavLM-large, and PhonoQ are combined with articulatory contours at inference time. In the contour-only teacher-supervision setting, audio-derived embeddings are used only during training; at inference time, the model receives only articulatory contours.

The evaluation covers 39-way phoneme classification and five phonological targets: manner, consonant place, consonant voicing, vowel height, and vowel backness. Two generalization protocols are used: unseen-speech (different speech materials, shared speakers) and unseen-subject (disjoint speakers).

Key results from the feature-fusion experiments (Table I): The contour-only baseline is substantially below all audio-fusion systems. For 39-way phoneme classification, WavLM+PhonoQ gives the best results in both protocols, reaching 69.0% macro-F1 for unseen speech and 67.8% for unseen subjects. This improves over WavLM alone, which reaches 64.0% and 67.0%, respectively. Adding PhonoQ to HuBERT increases macro-F1 from 56.0% to 65.0% in the unseen-speech setting and from 60.0% to 64.1% in the unseen-subject setting. For phonological classification, WavLM+PhonoQ is strongest in the unseen-speech protocol, obtaining the best result for all five targets. Under unseen-subject generalization, WavLM+PhonoQ remains best for consonant voicing and manner, while WavLM+HuBERT is slightly stronger for consonant place, vowel backness, and vowel height. Averaged across protocols, WavLM+PhonoQ is the strongest overall feature-fusion system, reaching 68.4% macro-F1 for phonetic classification and 85.1% for phonological classification.

For the contour-only teacher-supervision experiments (Table II): All teacher-supervised systems improve over the contour-only baseline. In the unseen-speech protocol, single-teacher systems reach 59.2–59.3% macro-F1, compared with 57.4% for contour only. The best result is obtained by HuBERT+PhonoQ at 59.5%, closely followed by WavLM+PhonoQ at 59.4%. For unseen subjects, contour only reaches 51.8%, single-teacher systems reach 53.4–53.7%, and both two-teacher systems reach 54.1%. The gains are modest but consistent. The gains are most visible for lower-baseline targets: for phone manner, PhonoQ improves over contour only from 40.4% to 42.9% in the unseen-speech setting and from 34.0% to 35.6% in the unseen-subject setting. For consonant place under unseen-subject generalization, contour only reaches 41.2%, while WavLM and HuBERT+PhonoQ reach 44.4% and 44.5%, respectively.

The posterior analysis examines PhonoQ posterior tracks for surface-sensitive phonological patterns. For canonical /t/ intervals in likely flapping contexts (e.g., water, pretty, little, better), the posterior distributions show lower stop and voiceless mass and higher voiced and rhotic mass than clear /t/ controls such as two, take, still, and started. For /t/ before /r/ (e.g., street, tree), there is high affricate and postalveolar posterior mass, whereas clear /t/ controls such as stars and details remain strongly stop-like and alveolar. For nasal place assimilation, rainbow (where canonical /n/ precedes /b/) shows increased labial posterior mass and reduced alveolar mass during the target nasal interval relative to the clear /n/ control stain.

The authors conclude that PhonoQ provides information complementary to SSL speech models, that audio-derived teacher signals can regularize contour-only phonological prediction, and that PhonoQ offers interpretable posterior evidence for surface-sensitive phonological patterns. Limitations noted include that feature-fusion experiments should be interpreted as cross-modal complementarity analysis rather than MRI-only deployment, contour-only improvements are modest, and the posterior analysis is illustrative and should be extended with larger-scale quantitative analyses and manual phonetic validation.

Improvements to AI Systems:

1. Cross-Modal Feature Fusion for Articulatory-to-Phoneme Mapping: Integrate PhonoQ’s phonological feature posteriors with SSL embeddings (e.g., WavLM, HuBERT) as a multi-stream input to a transformer-based classifier. This improves phoneme recognition from articulatory contours by 5–10% macro-F1 over contour-only baselines, especially for unseen speech and speakers, enabling more robust silent-speech interfaces or MRI-based speech decoding.

2. Teacher-Student Regularization for Contour-Only Inference: Use PhonoQ (and optionally WavLM) as a teacher during training to distill audio-derived phonological knowledge into a student model that only receives articulatory contours at inference. This yields consistent gains (e.g., +2.1% macro-F1 for manner, +3.3% for consonant place under unseen subjects) without requiring audio at test time—critical for real-time MRI-only applications where audio is unavailable or noisy.

3. Phonologically-Aware Data Augmentation: Leverage PhonoQ’s posterior distributions to generate soft labels or reweight training samples for articulatory contours, emphasizing surface-sensitive phonological phenomena (e.g., flapping, palatalization, nasal place assimilation). This improves the model’s ability to capture coarticulation and allophonic variation, leading to better generalization across dialects and speaking styles.

4. Interpretable Posterior-Based Error Analysis: Use PhonoQ’s frame-level posterior tracks as an auxiliary diagnostic layer in the AI system to identify when and why articulatory-to-phoneme predictions fail (e.g., detecting flapped /t/ or assimilated /n/). This enables self-monitoring and targeted correction, improving trust and reliability in clinical or assistive speech technologies.

5. Multi-Task Learning with Phonological Targets: Jointly train the model to predict 39-way phonemes and five phonological features (manner, place, voicing, vowel height, backness) using PhonoQ-derived targets as auxiliary losses. This regularizes the shared representation, improving performance on low-resource or imbalanced classes (e.g., consonant place) by up to 3.3% macro-F1, and yields more linguistically meaningful internal representations.

What the Improved AI System Can Do:

• Real-time silent-speech recognition from MRI or articulatory sensors alone, with higher accuracy and robustness to speaker variability, by leveraging audio-derived phonological knowledge during training.

• Cross-modal speech decoding that fuses articulatory and audio features at inference, achieving state-of-the-art phoneme classification (68.4% macro-F1) and phonological classification (85.1%) across unseen conditions.

• Explainable articulatory analysis that highlights surface-level phonological processes (e.g., flapping, assimilation) in real time, aiding phonetic research and clinical assessment of speech disorders.

• Domain-adaptive learning where the system transfers phonological knowledge from audio-rich environments to audio-poor settings (e.g., MRI-only), improving generalization without requiring paired audio at deployment.