Interpreting Brain Responses to Language with Sparse Features from Language Models

arXiv:2606.06857 · cs.CL · Submitted 2026-06-05 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Interpreting Brain Responses to Language with Sparse Features from Language Models".

Jane: Here's the gist of the paper. A central goal of cognitive neuroscience is to characterize the features that are represented by human language cortex; Artificial language models (LMs) have emerged as a powerful tool to address this challenge, but studies relating biological and artificial representations are often criticized as relating one black box to another The present work introduces Augmented Sparse Encoding Models, an encoding framework that replaces dense LM hidden states with hierarchically organized sparse autoencoder (SAE) features, while explicitly including surprisal as a predictor Using this approach, we (i) produce interpretations of neural responses and (ii) test whether model-brain alignment reflects primary or idiosyncratic variation in LM representations Model Framework and Contributions The authors propose Augmented Sparse Encoding Models, which introduce two key changes to standard LM encoding models First, they project dense residual-stream LM features into a sparse, hierarchically organized SAE basis Second, they augment that basis with an explicit feature that captures processing difficulty (surprisal) The authors make the following contributions using Augmented Sparse Encoding Models: 1 SAE features can predict voxel responses to language as accurately as dense LM residual-stream features, while also providing interpretations that affirm and extend prior neuroscience findings on processing difficulty-tuned and content-tuned voxels 2 We discover and interpret previously uncharacterized voxel populations 3 We show how different regions of the human language network respond differentially to processing difficulty vs.

Tom: content features, finding that some frontal brain regions are explained well by processing difficulty, whereas temporal regions draw more on LM-derived “content” features 4 We show that brain responses to language are predicted by primary, general features of LM representations rather than idiosyncratic ones—revealing a deeper correspondence between artificial and biological language representations than implied by measures of encoding accuracy alone Validation and Voxel Subtype Analysis The framework is validated using a high-field 7T fMRI dataset of eight participants listening to 200 linguistically diverse sentences The authors first recover previous interpretations of voxel populations tuned to processing difficulty and meaning abstractness They interpret a previously uncharacterized (but reliable) voxel population and find that it is tuned to people-related content Next, they show that the fronto-temporal human language network is predicted by a common set of features across its constituent regions, but find that frontal regions are relatively well-explained by surprisal alone, even in the absence of LM-based features Finally, they show that brain responses during language processing are not merely predictable from an arbitrary set of LM features; rather, brain responses are best explained by the features that tend to capture the most general information encoded in LM representations Predictivity Across Voxel Subtypes The study analyzes four PC-derived voxel subtypes: Hard-to-Process, Easy-to-Process, Abstract, and Concrete Finding 1 shows that SAE features tend to predict voxel responses to language stimuli nearly as well as dense LM residual features In particular, features in the Matryoshka SAE basis achieve very similar performance as features in the dense residual LM basis, across all four voxel subtypes Finding 2 demonstrates a dissociation between processing difficulty-driven and content-driven voxel regimes: voxels in the Hard-to-Process subtype are captured by surprisal alone, with no significant improvement from SAE or residual stream features, in contrast to Abstract and Concrete subtypes which benefit significantly from the inclusion of SAE or residual-stream features above and beyond surprisal Characterization of Content-Driven Features Finding 3 indicates that distinct features refine interpretations of content-driven voxels, showing that Abstract and Concrete voxels are consistently supported by different feature subspaces For the "Ghost" voxel subtype, a coherent subpopulation is identified which is tuned to people-centered content This subpopulation is driven by features related to sentences that include “people-specific” features, such as discussions of relationships/emotions (Feat.

Jane: 40+), descriptions of people taking concrete actions (Feat.

Tom: 94+), or pronouns (Feat.

Paper summary: Jane: To get into the details of "Interpreting Brain Responses to Language with Sparse Features from Language Models," they are using this new encoding framework called Augmented Sparse Encoding Models. They introduce two key changes compared to standard language model encoding models. First, they take the dense residual-stream features from the language model and project them into a sparse, hierarchically organized sparse autoencoder basis. Second, they augment that basis by adding an explicit feature that captures processing difficulty, which is surprisal.

Lu: The purpose of this augmentation is twofold: to produce interpretations of voxel response tuning and to test whether the resulting model-brain alignment points toward primary or idiosyncratic variation in the language model representations. They are essentially trying to understand *why* a certain brain area responds the way it does to language stimuli, not just that it responds at all.

Tom: So they’re moving away from just seeing if models predict neural responses, which is what standard studies have done, toward figuring out the actual properties of those features that drive alignment between the model and the brain. That's a big step in linking artificial and biological representations.

Meng: I wonder how much this sparse feature basis actually predicts voxel responses compared to the dense residual stream features before we get into what those specific results are showing about primary versus idiosyncratic variation. We need to know if it’s just as good, or if the sparsity introduces a trade-off in predictive power.

Lalam: The paper claims that these SAE features can predict voxel responses to language nearly as well as the dense residual stream features, and they also provide interpretations that support prior neuroscience findings about voxels tuned to processing difficulty and content.

Jane: And they found that this approach allows them to discover previously uncharacterized voxel populations, which is pretty exciting because it means we're finding brain regions we hadn't even looked at before.

Tom: Finding those new populations is important, but what about the bigger picture? They also show how different regions in the human language network respond differently depending on whether you’re looking at processing difficulty or content features.

Lu: They found that some frontal brain regions are explained well by processing difficulty alone, while temporal regions seem to rely more on content-linked features derived from the language model. This suggests a kind of functional specialization in how different brain areas handle linguistic information.

Meng: That dissociation between frontal and temporal responses is interesting because it implies that the mechanism for processing meaning isn't uniform across all language centers; it’s specialized, right?

Jane: Exactly. And they show that brain responses to language are best predicted by the most general, widely applicable features of the language model representations rather than some very specific, idiosyncratic ones.

Lalam: So the core idea is that brain alignment with human responses relies on these more general features of the language model, which is a deeper correspondence than just looking at encoding accuracy.

Conclusion: Tom: So to wrap up this paper, "Interpreting Brain Responses to Language with Sparse Features from Language Models," the authors argue that their Augmented Sparse Encoding Models offer a way to move beyond treating both models and brains as black boxes. They introduce this framework by using sparse autoencoder features and adding surprisal as a predictor.

Jane: The main implication here is that tuning within the language network isn't uniform; different regions have different sensitivities to processing difficulty versus content features, which is something they’ve identified. They also found a new way to characterize "ghost" voxels tuned to people-related content.

Lu: So what this means for the field is that we can start characterizing linguistic representations in the language network and interpret those "ghost" voxels tuned to social content. It suggests that shared tuning exists, but it’s not uniform across all areas of the brain.

Meng: If you’re thinking about deployment, knowing which general features of an LM are driving the brain response instead of just whatever arbitrary feature set is best for prediction is valuable because it grounds the interpretation in something more fundamental to the model's structure.

Lalam: This work provides a foundation for investigating how linguistic meaning is represented and organized in the human brain by connecting those general, widely-applicable LM features to neural activity.

Tom: It shows that even though LMs and brains use very different methods to arrive at representations, there’s some partial overlap in the features they use to organize internal language representations. That's a significant connection for anyone trying to build better AI systems that understand language.

Jane: And they showed that the strongest alignment comes from the set of features that most generically characterize an LM’s representation, not just on having many fine-grained features contributing to the high-dimensional space. That’s a key insight into what makes language representations work in this context.

Brown University · University of Minnesota · Harvard University

cs.CL

Submitted: 2026-06-05

Updated: 2026-11-11

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: A central goal of cognitive neuroscience is to characterize the features that are represented by human language cortex…

Key concepts

Augmented Sparse Encoding Models
This framework modifies standard LM encoding by projecting dense language features into a sparse, hierarchical autoencoder basis. It also explicitly adds 'surprisal'—a measure of how difficult a sentence is to predict—to provide interpretable features for brain analysis.
Surprisal
Surprisal is a mathematical measure, similar to negative log probability, that quantifies the processing difficulty of a linguistic stimulus. It tells researchers how much effort or surprise an LM (or human) experiences when encountering a specific word or sentence.
Matryoshka SAE
This specific autoencoder architecture organizes language features into a hierarchical structure. Early layers capture general, widely-applicable variance in language representations, while later layers capture more granular, specific details. This hierarchy helps reveal shared organizing principles between LM and brain data.

Terminology

Summary

A central goal of cognitive neuroscience is to characterize the features that are represented by human language cortex; Artificial language models (LMs) have emerged as a powerful tool to address this challenge, but studies relating biological and artificial representations are often criticized as relating one black box to another The present work introduces Augmented Sparse Encoding Models, an encoding framework that replaces dense LM hidden states with hierarchically organized sparse autoencoder (SAE) features, while explicitly including surprisal as a predictor Using this approach, we (i) produce interpretations of neural responses and (ii) test whether model-brain alignment reflects primary or idiosyncratic variation in LM representations

Model Framework and Contributions

The authors propose Augmented Sparse Encoding Models, which introduce two key changes to standard LM encoding models First, they project dense residual-stream LM features into a sparse, hierarchically organized SAE basis Second, they augment that basis with an explicit feature that captures processing difficulty (surprisal) The authors make the following contributions using Augmented Sparse Encoding Models:

1 SAE features can predict voxel responses to language as accurately as dense LM residual-stream features, while also providing interpretations that affirm and extend prior neuroscience findings on processing difficulty-tuned and content-tuned voxels

2 We discover and interpret previously uncharacterized voxel populations

3 We show how different regions of the human language network respond differentially to processing difficulty vs. content features, finding that some frontal brain regions are explained well by processing difficulty, whereas temporal regions draw more on LM-derived “content” features

4 We show that brain responses to language are predicted by primary, general features of LM representations rather than idiosyncratic ones—revealing a deeper correspondence between artificial and biological language representations than implied by measures of encoding accuracy alone

Validation and Voxel Subtype Analysis

The framework is validated using a high-field 7T fMRI dataset of eight participants listening to 200 linguistically diverse sentences The authors first recover previous interpretations of voxel populations tuned to processing difficulty and meaning abstractness They interpret a previously uncharacterized (but reliable) voxel population and find that it is tuned to people-related content Next, they show that the fronto-temporal human language network is predicted by a common set of features across its constituent regions, but find that frontal regions are relatively well-explained by surprisal alone, even in the absence of LM-based features Finally, they show that brain responses during language processing are not merely predictable from an arbitrary set of LM features; rather, brain responses are best explained by the features that tend to capture the most general information encoded in LM representations

Predictivity Across Voxel Subtypes

The study analyzes four PC-derived voxel subtypes: Hard-to-Process, Easy-to-Process, Abstract, and Concrete Finding 1 shows that SAE features tend to predict voxel responses to language stimuli nearly as well as dense LM residual features In particular, features in the Matryoshka SAE basis achieve very similar performance as features in the dense residual LM basis, across all four voxel subtypes Finding 2 demonstrates a dissociation between processing difficulty-driven and content-driven voxel regimes: voxels in the Hard-to-Process subtype are captured by surprisal alone, with no significant improvement from SAE or residual stream features, in contrast to Abstract and Concrete subtypes which benefit significantly from the inclusion of SAE or residual-stream features above and beyond surprisal

Characterization of Content-Driven Features

Finding 3 indicates that distinct features refine interpretations of content-driven voxels, showing that Abstract and Concrete voxels are consistently supported by different feature subspaces For the Ghost voxel subtype, a coherent subpopulation is identified which is tuned to people-centered content This subpopulation is driven by features related to sentences that include “people-specific” features, such as discussions of relationships/emotions (Feat. 40+), descriptions of people taking concrete actions (Feat. p. 7

Where the paper says it

We find that the features used to predict this voxel subpopulation are typically driven by sentences that include “people-specific” features, such as discussions of relationships/emotions (Feat. #40+), descriptions of people taking concrete actions (Feat.

94+), or pronouns (Feat. p. 7
Where the paper says it

#40+), descriptions of people taking concrete actions (Feat. #94+), or pronouns (Feat.

49+) Interestingly, this subpopulation appears to respond to sentences centered on social relations and person-referential events, which are orthogonal to processing difficulty or content dimensions

General Feature Dominance in the Language Network

In Study 3, the authors analyze the five canonical left-lateralized language regions (fROIs) using Augmented Sparse Encoding Models Finding 1 shows that frontal fROIs are relatively well-captured by surprisal alone, particularly the middle frontal gyrus (MFG), which does not show a significant gain from SAE/residual-stream features beyond surprisal In contrast, temporal fROIs show a significant boost from feature-based models above and beyond surprisal, indicating stronger sensitivity to content-linked structure during sentence comprehension Finding 3 concludes that human language network voxels are preferentially explained by general, widely applicable LM features Specifically, the language-network encoding models preferentially load on the most general Matryoshka features (feature indices < 128), suggesting that brain-alignment with the human language network relies on the LM’s general, widely-applicable feature dimensions rather than fine-grained, idiosyncratic features

Conclusion and Future Directions

Augmented Sparse Encoding Models move beyond prior approaches that treat both models and brains as black boxes The authors conclude that tuning within the language network is shared but not uniform, with regions differing in their sensitivity to processing difficulty versus content Implications include characterizing linguistic representations in the language network, interpreting the “Ghost” voxels tuned to people-related content, and determining which properties of LM representations mediate brain alignment with human language responses Future work suggests designing stimuli to drive targeted voxel populations for investigating neural control in interpretable voxel subspaces

Methodological Details

The analysis pipeline for each voxel involves a two-stage process: feature selection using LASSO regression followed by refitting with Ridge regression, always including the surprisal feature in the Ridge regression The authors employ two SAE architectures, JumpReLU and Matryoshka, with Matryoshka introducing additional structure to learn more disentangled features Cross-validated predictivity is estimated using a two-stage processing pipeline for each voxel The analysis also replicates primary quantitative results using an independent 3T dataset, showing qualitative similarity to the main 7T dataset

Feature Hierarchy and Sharing

The Matryoshka SAE imposes an ordered feature hierarchy where early latent features account for general, widely-applicable variance in LM representations, and later features are progressively more granular The analysis of feature prevalence across participants reveals four regimes of shared features The two most prevalent features are Feat. 79 and Feat. p. 6

Where the paper says it

of these features broadly align with existing interpretations (Tuckute et al. For example, Abstract voxels are driven by features related to emotions (Feat.

44, which are content-linked and token-count linked, respectively The results suggest that the entire language network is organized around a shared, dominant feature set consisting of both content- and form-linked features General features are most predictive across language fROIs, and the most general Matryoshka bin enables better predictivity than the union of all other feature bins The strongest alignment is carried by the multiplicity of fine-grained features that contribute to an LM’s high-dimensional feature space, but brain-predictive features are concentrated among the set of features that most generically characterize LM representations

Limitations and Specific Findings

Limitations include the interpretations being limited to the SAE's feature basis, and some SAE features remaining opaque The authors note that more general Matryoshka SAE features predict voxel tuning because voxels are a coarse-grained measure of neural activity Furthermore, the Ghost analysis shows that even the best performing feature sets only achieve a Normalized Predictivity of approximately.1 for Ghost voxels The authors find that social tuning during comprehension of short sentences appears partly adjacent to the core language network and partly interspersed with temporal language areas The paper provides a foundation for further investigations into feature tuning during language processing with larger sets of diverse stimuli

Anatomical Localization of Novel Subpopulations

The people-centered voxel subpopulation, identified through the Ghost analysis, was located largely outside traditional frontal and temporal areas for language The largest proportion of voxels were located in the angular gyrus area (43.8%), outside the five “core” fronto-temporal language fROIs analyzed in Study 6 This subpopulation also included some medial frontal voxels, which have been described as implicated in social cognition and theory-of-mind tasks The identification and interpretation of this subpopulation demonstrates how Augmented Sparse Encoding Models can recover not only broad organizing dimensions, but also representational structure shared across a subset of individuals

Feature Granularity and Alignment

The analysis connects SAE feature granularity to work in the visual domain, suggesting that hierarchical SAE bases can increase model–brain alignment when superposition obscures shared latent structure The strongest alignment is not carried by the multiplicity of fine-grained features that contribute to an LM’s high-dimensional feature space, but rather by the set of features that most generically characterize LM representations This suggests partial overlap in the features that organize internal language representations in the human language network and LMs, even though these systems presumably having very different means of arriving at these representations The claim about general SAE dimensions aligns with Antonello et al., 2021, who showed that low-dimensional structure in LM representations is reflected in brain responses The paper extends this work by identifying the relevant dimensions in an interpretable sparse feature basis

Replication and Data Handling

The replication using the 3T brain dataset shows that MFG exhibits the smallest difference between surprisal vs. other feature sets, and AntTemp exhibits the largest difference The detailed information about language fROI voxel selection for generalization analysis involved sub-sampling to manage computational cost, including deduplication of voxels with identical brain response profiles within each participant The final 2,296 voxels are distributed across participants and fROIs as reported in Table 5

Summary of Findings

The paper demonstrates that small sets of features can be used to predict brain responses to language and separate processing difficulty-related signal from content Within the language network, tuning is shared but not uniform, with regions differing in their sensitivity to processing difficulty versus content The study provides a step towards characterizing how linguistic meaning is represented and organized in the human brain

The gist

A central goal of cognitive neuroscience is to characterize the features that are represented by human language cortex; Artificial language models (LMs) have emerged as a powerful tool to address this challenge, but studies relating biological and artificial representations are often criticized as relating one black box to another

Pruning redundant reasoning tokens

A central goal of cognitive neuroscience is to characterize the features that are represented by human language cortex; Artificial language models (LMs) have emerged as a powerful tool to address this challenge, but studies relating biological and artificial representations are often criticized as relating one black box to another

How the training signal is built

LM encoding models are defined by two components: (i) a feature extractor that transforms language stimuli into a feature vector, and (ii) a linear readout that maps that feature vector onto neural responses to the same linguistic stimuli In this work, we introduce two changes to the typical framework to make the resulting LM encoding model more interpretable First, we employ pretrained SAEs to transform dense, difficult-to-interpret residual-stream LM representations into a much larger, sparser, and (ideally) more interpretable feature basis Our second change is the inclusion of a dedicated feature that encodes the average surprisal (i.e., negative log probability) of a sentence, as computed by an LM

Where the gains come from

Finding 1 shows that SAE features tend to predict voxel responses to language stimuli nearly as well as dense LM residual features Finding 2 demonstrates a dissociation between processing difficulty-driven and content-driven voxel regimes, with Abstract and Concrete voxels benefiting significantly from the inclusion of SAE or residual-stream features above and beyond surprisal The strongest alignment is carried by the multiplicity of fine-grained features that contribute to an LM’s high-dimensional feature space, but brain-predictive features are concentrated among the set of features that most generically characterize LM representations This finding suggests partial overlap in the features that organize internal language representations in the human language network and LMs, even though these systems presumably having very different means of arriving at these representations

The Matryoshka SAE imposes an ordered feature hierarchy where early latent features account for general, widely-applicable variance in LM representations, and later features are progressively more granular The strongest alignment is not carried by the multiplicity of fine-grained features that contribute to an LM’s high-dimensional feature space, but rather by the set of features that most generically characterize LM representations This suggests partial overlap in the features that organize internal language representations in the human language network and LMs, even though these systems presumably having very different means of arriving at these representations

LM encoding models are defined by two components: (i) a feature extractor that transforms language stimuli into a feature vector, and (ii) a linear readout that maps that feature vector onto neural responses to the same linguistic stimuli In this work, we introduce two changes to the typical framework to make the resulting LM encoding model more interpretable First, we employ pretrained SAEs to transform dense, difficult-to-interpret residual-stream LM representations into a much larger, sparser, and (ideally) more interpretable feature basis Our second change is the inclusion of a dedicated feature that encodes the average surprisal (i.e.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improve interpretation of neural responses by using Augmented Sparse Encoding Models to distinguish between processing difficulty-driven and content-driven features, as brain responses are not merely predictable from an arbitrary set of LM features. Rather, brain responses are best explained by the features that tend to capture the most general information encoded in LM representations.

  2. Enable discovery of previously uncharacterized voxel populations by using the hierarchical structure of sparse autoencoders to identify coherent feature-defined subpopulations, such as a subset of the Ghost voxels form a coherent feature-defined subpopulation, which are tuned to people-centered content.

  3. Refine understanding of language network organization by characterizing how different regions respond differentially to processing difficulty versus content features, showing that some frontal brain regions are explained well by processing difficulty, whereas temporal regions draw more on LM-derived “content” features.

  4. Enhance model-brain alignment by focusing on primary, general features rather than idiosyncratic ones, as the study shows that brain responses to language are predicted by primary, general features of LM representations rather than idiosynractic ones—revealing a deeper correspondence between artificial and biological language representations.

Abstract

A central goal of cognitive neuroscience is to characterize the features that are represented by human language cortex. Artificial language models (LMs) have emerged as a powerful tool to address this challenge, but studies relating biological and artificial representations are often criticized as relating one black box to another. The present work introduces Augmented Sparse Encoding Models, an encoding framework that replaces dense LM hidden states with hierarchically-organized sparse autoencoder (SAE) features, while explicitly including surprisal as a predictor. Using this approach, we (i) produce interpretations of neural responses and (ii) test whether model-brain alignment reflects primary or idiosyncratic variation in LM representations. Using a high-field 7T fMRI dataset of eight participants listening to 200 linguistically diverse sentences, we first validate our modeling framework by recovering previous interpretations of voxel populations tuned to processing difficulty and meaning abstractness. We then interpret a previously-uncharacterized (but reliable) voxel population and find that it is tuned to people-related content. Next, we show that the fronto-temporal human language network is predicted by a common set of features across its constituent regions, but find that frontal regions are relatively well-explained by surprisal alone, even in the absence of LM-based features. Finally, we show that brain responses are best explained by the features that tend to capture the most general information encoded in LM representations, suggesting a nontrivial correspondence between brain and LM language representation.

Sources

Related papers