Interpreting Brain Responses to Language with Sparse Features from Language Models

summary

The gist

A central goal of cognitive neuroscience is to characterize the features that are represented by human language cortex…

In short

The study introduced Augmented Sparse Encoding Models to link artificial language models (LMs) to brain activity. By replacing dense LM features with sparse autoencoder features and adding surprisal, researchers found that these sparse features accurately predict neural responses and reveal shared, general linguistic representations between AI and humans.

Key concepts

Augmented Sparse Encoding Models
This framework modifies standard LM encoding by projecting dense language features into a sparse, hierarchical autoencoder basis. It also explicitly adds 'surprisal'—a measure of how difficult a sentence is to predict—to provide interpretable features for brain analysis.
Surprisal
Surprisal is a mathematical measure, similar to negative log probability, that quantifies the processing difficulty of a linguistic stimulus. It tells researchers how much effort or surprise an LM (or human) experiences when encountering a specific word or sentence.
Matryoshka SAE
This specific autoencoder architecture organizes language features into a hierarchical structure. Early layers capture general, widely-applicable variance in language representations, while later layers capture more granular, specific details. This hierarchy helps reveal shared organizing principles between LM and brain data.

Terminology used across episodes

This episode discusses

The paper

Interpreting Brain Responses to Language with Sparse Features from Language Models · Read on arXiv

Brown University · University of Minnesota · Harvard University

A central goal of cognitive neuroscience is to characterize the features that are represented by human language cortex. Artificial language models (LMs) have emerged as a powerful tool to address this challenge, but studies relating biological and artificial representations are often criticized as relating one black box to another. The present work introduces Augmented Sparse Encoding Models, an encoding framework that replaces dense LM hidden states with hierarchically-organized sparse autoencoder (SAE) features, while explicitly including surprisal as a predictor. Using this approach, we (i) produce interpretations of neural responses and (ii) test whether model-brain alignment reflects primary or idiosyncratic variation in LM representations. Using a high-field 7T fMRI dataset of eight participants listening to 200 linguistically diverse sentences, we first validate our modeling framework by recovering previous interpretations of voxel populations tuned to processing difficulty and meaning abstractness. We then interpret a previously-uncharacterized (but reliable) voxel population and find that it is tuned to people-related content. Next, we show that the fronto-temporal human language network is predicted by a common set of features across its constituent regions, but find that frontal regions are relatively well-explained by surprisal alone, even in the absence of LM-based features. Finally, we show that brain responses are best explained by the features that tend to capture the most general information encoded in LM representations, suggesting a nontrivial correspondence between brain and LM language representation.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Interpreting Brain Responses to Language with Sparse Features from Language Models".

Jane: Here's the gist of the paper. A central goal of cognitive neuroscience is to characterize the features that are represented by human language cortex; Artificial language models (LMs) have emerged as a powerful tool to address this challenge, but studies relating biological and artificial representations are often criticized as relating one black box to another The present work introduces Augmented Sparse Encoding Models, an encoding framework that replaces dense LM hidden states with hierarchically organized sparse autoencoder (SAE) features, while explicitly including surprisal as a predictor Using this approach, we (i) produce interpretations of neural responses and (ii) test whether model-brain alignment reflects primary or idiosyncratic variation in LM representations Model Framework and Contributions The authors propose Augmented Sparse Encoding Models, which introduce two key changes to standard LM encoding models First, they project dense residual-stream LM features into a sparse, hierarchically organized SAE basis Second, they augment that basis with an explicit feature that captures processing difficulty (surprisal) The authors make the following contributions using Augmented Sparse Encoding Models: 1 SAE features can predict voxel responses to language as accurately as dense LM residual-stream features, while also providing interpretations that affirm and extend prior neuroscience findings on processing difficulty-tuned and content-tuned voxels 2 We discover and interpret previously uncharacterized voxel populations 3 We show how different regions of the human language network respond differentially to processing difficulty vs.

Tom: content features, finding that some frontal brain regions are explained well by processing difficulty, whereas temporal regions draw more on LM-derived “content” features 4 We show that brain responses to language are predicted by primary, general features of LM representations rather than idiosyncratic ones—revealing a deeper correspondence between artificial and biological language representations than implied by measures of encoding accuracy alone Validation and Voxel Subtype Analysis The framework is validated using a high-field 7T fMRI dataset of eight participants listening to 200 linguistically diverse sentences The authors first recover previous interpretations of voxel populations tuned to processing difficulty and meaning abstractness They interpret a previously uncharacterized (but reliable) voxel population and find that it is tuned to people-related content Next, they show that the fronto-temporal human language network is predicted by a common set of features across its constituent regions, but find that frontal regions are relatively well-explained by surprisal alone, even in the absence of LM-based features Finally, they show that brain responses during language processing are not merely predictable from an arbitrary set of LM features; rather, brain responses are best explained by the features that tend to capture the most general information encoded in LM representations Predictivity Across Voxel Subtypes The study analyzes four PC-derived voxel subtypes: Hard-to-Process, Easy-to-Process, Abstract, and Concrete Finding 1 shows that SAE features tend to predict voxel responses to language stimuli nearly as well as dense LM residual features In particular, features in the Matryoshka SAE basis achieve very similar performance as features in the dense residual LM basis, across all four voxel subtypes Finding 2 demonstrates a dissociation between processing difficulty-driven and content-driven voxel regimes: voxels in the Hard-to-Process subtype are captured by surprisal alone, with no significant improvement from SAE or residual stream features, in contrast to Abstract and Concrete subtypes which benefit significantly from the inclusion of SAE or residual-stream features above and beyond surprisal Characterization of Content-Driven Features Finding 3 indicates that distinct features refine interpretations of content-driven voxels, showing that Abstract and Concrete voxels are consistently supported by different feature subspaces For the "Ghost" voxel subtype, a coherent subpopulation is identified which is tuned to people-centered content This subpopulation is driven by features related to sentences that include “people-specific” features, such as discussions of relationships/emotions (Feat.

Jane: 40+), descriptions of people taking concrete actions (Feat.

Tom: 94+), or pronouns (Feat.

Paper summary: Jane: To get into the details of "Interpreting Brain Responses to Language with Sparse Features from Language Models," they are using this new encoding framework called Augmented Sparse Encoding Models. They introduce two key changes compared to standard language model encoding models. First, they take the dense residual-stream features from the language model and project them into a sparse, hierarchically organized sparse autoencoder basis. Second, they augment that basis by adding an explicit feature that captures processing difficulty, which is surprisal.

Lu: The purpose of this augmentation is twofold: to produce interpretations of voxel response tuning and to test whether the resulting model-brain alignment points toward primary or idiosyncratic variation in the language model representations. They are essentially trying to understand *why* a certain brain area responds the way it does to language stimuli, not just that it responds at all.

Tom: So they’re moving away from just seeing if models predict neural responses, which is what standard studies have done, toward figuring out the actual properties of those features that drive alignment between the model and the brain. That's a big step in linking artificial and biological representations.

Meng: I wonder how much this sparse feature basis actually predicts voxel responses compared to the dense residual stream features before we get into what those specific results are showing about primary versus idiosyncratic variation. We need to know if it’s just as good, or if the sparsity introduces a trade-off in predictive power.

Lalam: The paper claims that these SAE features can predict voxel responses to language nearly as well as the dense residual stream features, and they also provide interpretations that support prior neuroscience findings about voxels tuned to processing difficulty and content.

Jane: And they found that this approach allows them to discover previously uncharacterized voxel populations, which is pretty exciting because it means we're finding brain regions we hadn't even looked at before.

Tom: Finding those new populations is important, but what about the bigger picture? They also show how different regions in the human language network respond differently depending on whether you’re looking at processing difficulty or content features.

Lu: They found that some frontal brain regions are explained well by processing difficulty alone, while temporal regions seem to rely more on content-linked features derived from the language model. This suggests a kind of functional specialization in how different brain areas handle linguistic information.

Meng: That dissociation between frontal and temporal responses is interesting because it implies that the mechanism for processing meaning isn't uniform across all language centers; it’s specialized, right?

Jane: Exactly. And they show that brain responses to language are best predicted by the most general, widely applicable features of the language model representations rather than some very specific, idiosyncratic ones.

Lalam: So the core idea is that brain alignment with human responses relies on these more general features of the language model, which is a deeper correspondence than just looking at encoding accuracy.

Conclusion: Tom: So to wrap up this paper, "Interpreting Brain Responses to Language with Sparse Features from Language Models," the authors argue that their Augmented Sparse Encoding Models offer a way to move beyond treating both models and brains as black boxes. They introduce this framework by using sparse autoencoder features and adding surprisal as a predictor.

Jane: The main implication here is that tuning within the language network isn't uniform; different regions have different sensitivities to processing difficulty versus content features, which is something they’ve identified. They also found a new way to characterize "ghost" voxels tuned to people-related content.

Lu: So what this means for the field is that we can start characterizing linguistic representations in the language network and interpret those "ghost" voxels tuned to social content. It suggests that shared tuning exists, but it’s not uniform across all areas of the brain.

Meng: If you’re thinking about deployment, knowing which general features of an LM are driving the brain response instead of just whatever arbitrary feature set is best for prediction is valuable because it grounds the interpretation in something more fundamental to the model's structure.

Lalam: This work provides a foundation for investigating how linguistic meaning is represented and organized in the human brain by connecting those general, widely-applicable LM features to neural activity.

Tom: It shows that even though LMs and brains use very different methods to arrive at representations, there’s some partial overlap in the features they use to organize internal language representations. That's a significant connection for anyone trying to build better AI systems that understand language.

Jane: And they showed that the strongest alignment comes from the set of features that most generically characterize an LM’s representation, not just on having many fine-grained features contributing to the high-dimensional space. That’s a key insight into what makes language representations work in this context.

More episodes

← Home