Interpretable Embeddings of Speech Enhance and Explain Brain Encoding Performance of Audio Models
Researchers tested how well speech foundation models (SFMs) align with brain activity by comparing their representations against six interpretable feature families and three state-of-the-art SFMs. Findings show SFMs learn simple features early on, but they develop high-level semantics in deeper layers that cannot be explained by basic acoustic features. Combining these interpretable features with deep layers improves the accuracy of brain encoding models.