Label-Free Parkinson's Disease Screening from Face and Voice through Mechanistic Interpretability

The paper proposes a label-free face-plus-voice screening system for Parkinson’s disease (PD) built entirely on frozen pretrained encoders—a face-expression Vision Transformer (ViT) and HuBERT—in which no PD label touches any fit; the reference is training controls only. The voice modality uses a synthetic-dysarthria contrastive activation addition (CAA) direction built from time-stretch and breathy degradation of healthy speech; the face modality uses a k-nearest-neighbor anomaly score to the control embedding cluster. The authors introduce the alignment principle, a post-hoc analysis showing that a synthetic-degradation CAA detector works when the cosine similarity between the synthetic and real disease directions exceeds zero. Measured on the YouTubePD benchmark, this cosine is +0.37 for voice (CAA works, AUROC 0.765) and −0.48 for face (CAA fails; anomaly succeeds, AUROC 0.751). Equal-weight late fusion reaches AUROC 0.802 (95% CI [0.70, 0.89]) with NPV 0.95, supporting a rule-out triage interpretation. An overfitting audit shows the voice detector transfers cleanly, while the face-side—and thus fused—AUROC is potentially optimistic pending external validation.

The framework constructs all per-modality scoring functions from training-control videos alone, reserving PD labels for evaluation. Two scoring primitives are admitted: sCAA (x) = (ϕ(x) − µH)⊤ d, and skNN (x) = (1/k) Σ ∥ϕ(x) − ϕ(c(j))∥2, where ϕ(·) is a frozen encoder, µH is the control centroid, d is a CAA direction, and c(j) is the j-th nearest training control. Both scores deviate monotonically from health: higher values indicate stronger PD-like signal.

For voice, HuBERT layer 8 activations are time-averaged over the first 12 s of the clip, yielding a 768-d per-utterance embedding. The CAA direction is constructed from synthetic dysarthria using two DSP degradations: a phase-vocoder time-stretch slowing speech by 22% while preserving pitch (ρ = 0.82), and a breathy phonation adding Gaussian aspiration noise scaled by the moving-average envelope. The CAA directions and control centroid are computed from training controls only, and projections are combined after scaling by control-set standard deviations fixed at fit time. No PD label is used in any of these steps.

For face, a face-expression ViT fine-tuned on FER2013 encodes 32 evenly sampled frames per video, cropped using MediaPipe Face Mesh landmarks (expanded by 1.6×), resized to 2242, and normalized. The per-video embedding is the mean over the 32 per-frame CLS tokens. Given the 36 training-control embeddings, the face score is computed via Eq. (2) with k = 15: pairwise Euclidean distances against every control embedding are sorted, and the mean of the k smallest is the per-video PD score.

Late fusion scales per-modality scores by their control-set standard deviations and adds them with equal weight, intentionally avoiding any data-driven weighting that would consult PD labels.

The alignment principle states that a synthetic CAA direction dsyn empirically detects a target disease when it points the same way in feature space as the true disease direction dreal, i.e., when cos(dsyn, dreal) > 0. The authors measure dreal using train labels for this analysis alone (it does not enter any score) as dreal = mean(ϕ(x)PD) − mean(ϕ(x)ctrl). Empirically, the cosine is +0.37 for voice (HuBERT) and −0.48 for face (face-expression ViT), predicting CAA works for voice and fails for face, which is confirmed.

The YouTubePD benchmark provides an official balanced train split of 72 videos (36 PD, 36 controls) and an official test split of 171 unique videos (21 PD, 150 controls; approximately 12% PD prevalence). The evaluated subset is the 157 test videos for which both face and voice modalities are complete: 20 PD and 137 controls (≈13% prevalence). The frozen encoders are trpakov/vit-face-expression (86M parameters) and facebook/hubert-base-ls960 (94M parameters). Key settings include T = 32 frames per video, 2242 face crops, the first 12 s of 16 kHz mono audio, HuBERT layer 8, k = 15 neighbors, slow rate ρ = 0.82, breathy gain 2.5×, and 3000 bootstrap resamples.

Main results show the voice and face detectors achieve comparable AUROC (0.765 and 0.751); their late fusion lifts AUROC to 0.802. The face detector achieves sensitivity 0.80, specificity 0.63, PPV 0.24, NPV 0.96, F1 0.37, balanced accuracy 0.71. The voice detector achieves sensitivity 0.75, specificity 0.74, PPV 0.30, NPV 0.95, F1 0.43, balanced accuracy 0.75. The fused system achieves sensitivity 0.70, specificity 0.81, PPV 0.35, NPV 0.95, F1 0.47, balanced accuracy 0.76.

The alignment principle is confirmed: voice CAA succeeds, while a face anti-aligned “hypomimia” CAA direction scored AUROC ≈ 0.50 (chance) in a pilot. Both outcomes match the sign of the measured cosine. Table 2 compares three speech encoders on voice CAA: wav2vec 2.0 (cos +0.28, test AUROC 0.728), HuBERT (cos +0.37, test AUROC 0.765), and SEW-D (cos +0.24, test AUROC 0.619). AUROC tracks cosine alignment. For face encoders under the same kNN-anomaly framework, the face-expression ViT (FER2013) is uniquely effective (test AUROC 0.751); other encoders—DINOv2 (0.41), CLIP (0.30), a sibling expression encoder (0.47), and MARLIN (0.47)—fail or barely exceed chance.

Comparison with prior work shows the label-free voice detector alone improves F1 from 0.27 to 0.43 over the YouTubePD audio-only baseline (AUROC 0.66), and the fused system reaches F1 = 0.47, all without consulting any PD label. The authors do not claim state-of-the-art accuracy versus the supervised face baseline (AUROC 0.86–0.92); rather they argue that comparable AUROC under a strict label-free protocol is itself a meaningful contribution.

Integrity audits include: Pearson correlations between each detector score and face crop size, video length, and frame rate are near zero (no obvious shortcut learning); permutation of test labels gives p < 0.01 for the system score; split-half within-test matches the full-test AUROC within bootstrap CI. A dev→test transfer audit shows the voice encoder choice is recoverable—HuBERT ranks first on dev and transfers to test—but the face choice is not: on dev the face-expression ViT ties a near-chance encoder (DINOv2), and its +0.28 test-set margin cannot be identified without the test labels. The authors therefore report the face-side, and hence fused, AUROC as potentially optimistic and defer a definitive face-encoder choice to pre-registered external validation.

The discussion explains why the voice modality scores lower on the dev split: the dev PD subjects are less separable from controls, with the PD–control gap in speaking rate widening from −0.37 SD on dev to −0.88 SD on test. The face modality is more complicated because it is multi-axis, so a pure face-mesh measure is not fully representative, and the dev-vs-test pattern is inconsistent across face encoders. The modality asymmetry is explained mechanistically: HuBERT, trained on raw speech, encodes the acoustic phenomenon of hypokinetic dysarthria, while the face-expression ViT, trained for static emotion classification, places its PD signal in static appearance, approximately orthogonal to facial motion or blink rate; a synthetic “hypomimia” degradation therefore points the wrong way, explaining why the face signal can be recovered by a direction-free primitive (kNN-anomaly) instead.

The clinical framing is a rule-out triage screen, not a diagnostic: NPV ≈ 0.95 is the operating use case, while PPV ≈ 0.34 at 13% prevalence (lower in the community) means a positive output should trigger clinical follow-up, not a label—consistent with how high-sensitivity, moderate-specificity tests are typically deployed in screening pathways.

Future work plans to identify a face encoder whose representation admits a positively aligned CAA direction—so the face branch can use the same activation-steering primitive as voice—and that is more clinically interpretable than the current appearance-based ViT, then search for additional synthetic-degradation directions to integrate more disease-relevant factors.

Improvements to AI systems:

1. Add a post-hoc alignment check before deploying any synthetic-degradation detector. Before trusting a CAA-based detector, compute the cosine similarity between the synthetic degradation direction and the true disease direction (using a small labeled validation set). If the cosine is ≤ 0, automatically fall back to a direction-free anomaly detector (kNN) for that modality. This prevents silent failures like the face CAA case (cos = −0.48 → AUROC ≈ 0.50).

2. Implement a modality-agnostic fusion gate that checks alignment per branch. Instead of equal-weight late fusion, use the alignment cosine as a per-modality confidence weight: weight = max(0, cos(d syn, d real)). This would automatically down-weight the face branch when its CAA is anti-aligned, and up-weight the voice branch (cos = +0.37), potentially improving fused AUROC beyond 0.802 without ever using PD labels for training.

3. Build a self-auditing screening system that reports an optimism flag for each modality. The system should compute a dev-to-test transfer consistency score (e.g., rank stability of encoder choices) and flag any modality where the best encoder on dev ties with a near-chance encoder. This prevents overconfident deployment of face-based screening until external validation, as the paper's audit revealed.

4. Add a synthetic-degradation search procedure that maximizes alignment before deployment. For any new modality, generate multiple candidate synthetic degradations (e.g., different time-stretch rates, noise levels, or motion reductions), compute their CAA directions, and select the one with the highest positive cosine to the true disease direction on a held-out labeled subset. This turns the alignment principle into an active optimization step, not just a post-hoc diagnostic.

5. Create a rule-out triage interface with calibrated thresholds. The improved system should output a binary needs follow-up vs. rule-out decision using the NPV ≈ 0.95 operating point, and also display per-modality alignment scores so clinicians can see which signal (voice vs. face) is driving the decision and how trustworthy each is.

What the improved AI system can do:

• Self-correcting multi-modal screening: It automatically detects when a synthetic-degradation detector is misaligned with the true disease and switches to a robust anomaly detector for that modality, preventing chance-level performance.

• Label-free with built-in validation: It can be deployed on new diseases or modalities with minimal labeled data (only a small set for alignment checking) and will refuse to trust a modality that fails the alignment test.

• Transparent and auditable: It reports per-modality alignment cosines, transfer-stability flags, and optimism warnings, so users know whether the face-based score is likely to generalize or is potentially optimistic.

• Higher fused accuracy under strict label-free constraints: By weighting modalities by their alignment, it can outperform equal-weight fusion (0.802 AUROC) while still never touching a PD label during training.

• Clinically safe rule-out screening: It maintains NPV ≈ 0.95 for ruling out Parkinson’s, while flagging positive cases for follow-up, and it can be re-targeted to other neurodegenerative diseases by swapping the synthetic degradations and re-checking alignment.