Interpreting Brain Responses to Language with Sparse Features from Language Models
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Interpreting Brain Responses to Language with Sparse Features from Language Models".
Jane: Here's the gist of the paper. A central goal of cognitive neuroscience is to characterize the features that are represented by human language cortex; Artificial language models (LMs) have emerged as a powerful tool to address this challenge, but studies relating biological and artificial representations are often criticized as relating one black box to another The present work introduces Augmented Sparse Encoding Models, an encoding framework that replaces dense LM hidden states with hierarchically organized sparse autoencoder (SAE) features, while explicitly including surprisal as a predictor Using this approach, we (i) produce interpretations of neural responses and (ii) test whether model-brain alignment reflects primary or idiosyncratic variation in LM representations Model Framework and Contributions The authors propose Augmented Sparse Encoding Models, which introduce two key changes to standard LM encoding models First, they project dense residual-stream LM features into a sparse, hierarchically organized SAE basis Second, they augment that basis with an explicit feature that captures processing difficulty (surprisal) The authors make the following contributions using Augmented Sparse Encoding Models: 1 SAE features can predict voxel responses to language as accurately as dense LM residual-stream features, while also providing interpretations that affirm and extend prior neuroscience findings on processing difficulty-tuned and content-tuned voxels 2 We discover and interpret previously uncharacterized voxel populations 3 We show how different regions of the human language network respond differentially to processing difficulty vs.
Tom: content features, finding that some frontal brain regions are explained well by processing difficulty, whereas temporal regions draw more on LM-derived “content” features 4 We show that brain responses to language are predicted by primary, general features of LM representations rather than idiosyncratic ones—revealing a deeper correspondence between artificial and biological language representations than implied by measures of encoding accuracy alone Validation and Voxel Subtype Analysis The framework is validated using a high-field 7T fMRI dataset of eight participants listening to 200 linguistically diverse sentences The authors first recover previous interpretations of voxel populations tuned to processing difficulty and meaning abstractness They interpret a previously uncharacterized (but reliable) voxel population and find that it is tuned to people-related content Next, they show that the fronto-temporal human language network is predicted by a common set of features across its constituent regions, but find that frontal regions are relatively well-explained by surprisal alone, even in the absence of LM-based features Finally, they show that brain responses during language processing are not merely predictable from an arbitrary set of LM features; rather, brain responses are best explained by the features that tend to capture the most general information encoded in LM representations Predictivity Across Voxel Subtypes The study analyzes four PC-derived voxel subtypes: Hard-to-Process, Easy-to-Process, Abstract, and Concrete Finding 1 shows that SAE features tend to predict voxel responses to language stimuli nearly as well as dense LM residual features In particular, features in the Matryoshka SAE basis achieve very similar performance as features in the dense residual LM basis, across all four voxel subtypes Finding 2 demonstrates a dissociation between processing difficulty-driven and content-driven voxel regimes: voxels in the Hard-to-Process subtype are captured by surprisal alone, with no significant improvement from SAE or residual stream features, in contrast to Abstract and Concrete subtypes which benefit significantly from the inclusion of SAE or residual-stream features above and beyond surprisal Characterization of Content-Driven Features Finding 3 indicates that distinct features refine interpretations of content-driven voxels, showing that Abstract and Concrete voxels are consistently supported by different feature subspaces For the "Ghost" voxel subtype, a coherent subpopulation is identified which is tuned to people-centered content This subpopulation is driven by features related to sentences that include “people-specific” features, such as discussions of relationships/emotions (Feat.
Jane: 40+), descriptions of people taking concrete actions (Feat.
Tom: 94+), or pronouns (Feat.
Paper summary: Jane: To get into the details of "Interpreting Brain Responses to Language with Sparse Features from Language Models," they are using this new encoding framework called Augmented Sparse Encoding Models. They introduce two key changes compared to standard language model encoding models. First, they take the dense residual-stream features from the language model and project them into a sparse, hierarchically organized sparse autoencoder basis. Second, they augment that basis by adding an explicit feature that captures processing difficulty, which is surprisal.
Lu: The purpose of this augmentation is twofold: to produce interpretations of voxel response tuning and to test whether the resulting model-brain alignment points toward primary or idiosyncratic variation in the language model representations. They are essentially trying to understand *why* a certain brain area responds the way it does to language stimuli, not just that it responds at all.
Tom: So they’re moving away from just seeing if models predict neural responses, which is what standard studies have done, toward figuring out the actual properties of those features that drive alignment between the model and the brain. That's a big step in linking artificial and biological representations.
Meng: I wonder how much this sparse feature basis actually predicts voxel responses compared to the dense residual stream features before we get into what those specific results are showing about primary versus idiosyncratic variation. We need to know if it’s just as good, or if the sparsity introduces a trade-off in predictive power.
Lalam: The paper claims that these SAE features can predict voxel responses to language nearly as well as the dense residual stream features, and they also provide interpretations that support prior neuroscience findings about voxels tuned to processing difficulty and content.
Jane: And they found that this approach allows them to discover previously uncharacterized voxel populations, which is pretty exciting because it means we're finding brain regions we hadn't even looked at before.
Tom: Finding those new populations is important, but what about the bigger picture? They also show how different regions in the human language network respond differently depending on whether you’re looking at processing difficulty or content features.
Lu: They found that some frontal brain regions are explained well by processing difficulty alone, while temporal regions seem to rely more on content-linked features derived from the language model. This suggests a kind of functional specialization in how different brain areas handle linguistic information.
Meng: That dissociation between frontal and temporal responses is interesting because it implies that the mechanism for processing meaning isn't uniform across all language centers; it’s specialized, right?
Jane: Exactly. And they show that brain responses to language are best predicted by the most general, widely applicable features of the language model representations rather than some very specific, idiosyncratic ones.
Lalam: So the core idea is that brain alignment with human responses relies on these more general features of the language model, which is a deeper correspondence than just looking at encoding accuracy.
Conclusion: Tom: So to wrap up this paper, "Interpreting Brain Responses to Language with Sparse Features from Language Models," the authors argue that their Augmented Sparse Encoding Models offer a way to move beyond treating both models and brains as black boxes. They introduce this framework by using sparse autoencoder features and adding surprisal as a predictor.
Jane: The main implication here is that tuning within the language network isn't uniform; different regions have different sensitivities to processing difficulty versus content features, which is something they’ve identified. They also found a new way to characterize "ghost" voxels tuned to people-related content.
Lu: So what this means for the field is that we can start characterizing linguistic representations in the language network and interpret those "ghost" voxels tuned to social content. It suggests that shared tuning exists, but it’s not uniform across all areas of the brain.
Meng: If you’re thinking about deployment, knowing which general features of an LM are driving the brain response instead of just whatever arbitrary feature set is best for prediction is valuable because it grounds the interpretation in something more fundamental to the model's structure.
Lalam: This work provides a foundation for investigating how linguistic meaning is represented and organized in the human brain by connecting those general, widely-applicable LM features to neural activity.
Tom: It shows that even though LMs and brains use very different methods to arrive at representations, there’s some partial overlap in the features they use to organize internal language representations. That's a significant connection for anyone trying to build better AI systems that understand language.
Jane: And they showed that the strongest alignment comes from the set of features that most generically characterize an LM’s representation, not just on having many fine-grained features contributing to the high-dimensional space. That’s a key insight into what makes language representations work in this context.
Brown University · University of Minnesota · Harvard University
cs.CL
Submitted: 2026-06-05
Updated: 2026-11-11
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 92/100
The gist: A central goal of cognitive neuroscience is to characterize the features that are represented by human language cortex…
Key concepts
- Augmented Sparse Encoding Models
- This framework modifies standard LM encoding by projecting dense language features into a sparse, hierarchical autoencoder basis. It also explicitly adds 'surprisal'—a measure of how difficult a sentence is to predict—to provide interpretable features for brain analysis.
- Surprisal
- Surprisal is a mathematical measure, similar to negative log probability, that quantifies the processing difficulty of a linguistic stimulus. It tells researchers how much effort or surprise an LM (or human) experiences when encountering a specific word or sentence.
- Matryoshka SAE
- This specific autoencoder architecture organizes language features into a hierarchical structure. Early layers capture general, widely-applicable variance in language representations, while later layers capture more granular, specific details. This hierarchy helps reveal shared organizing principles between LM and brain data.
Terminology
Summary
A central goal of cognitive neuroscience is to characterize the features that are represented by human language cortex; Artificial language models (LMs) have emerged as a powerful tool to address this challenge, but studies relating biological and artificial representations are often criticized as relating one black box to another The present work introduces Augmented Sparse Encoding Models, an encoding framework that replaces dense LM hidden states with hierarchically organized sparse autoencoder (SAE) features, while explicitly including surprisal as a predictor Using this approach, we (i) produce interpretations of neural responses and (ii) test whether model-brain alignment reflects primary or idiosyncratic variation in LM representations
Model Framework and Contributions
The authors propose Augmented Sparse Encoding Models, which introduce two key changes to standard LM encoding models First, they project dense residual-stream LM features into a sparse, hierarchically organized SAE basis Second, they augment that basis with an explicit feature that captures processing difficulty (surprisal) The authors make the following contributions using Augmented Sparse Encoding Models:
1 SAE features can predict voxel responses to language as accurately as dense LM residual-stream features, while also providing interpretations that affirm and extend prior neuroscience findings on processing difficulty-tuned and content-tuned voxels
2 We discover and interpret previously uncharacterized voxel populations
3 We show how different regions of the human language network respond differentially to processing difficulty vs. content features, finding that some frontal brain regions are explained well by processing difficulty, whereas temporal regions draw more on LM-derived “content” features
4 We show that brain responses to language are predicted by primary, general features of LM representations rather than idiosyncratic ones—revealing a deeper correspondence between artificial and biological language representations than implied by measures of encoding accuracy alone
Validation and Voxel Subtype Analysis
The framework is validated using a high-field 7T fMRI dataset of eight participants listening to 200 linguistically diverse sentences The authors first recover previous interpretations of voxel populations tuned to processing difficulty and meaning abstractness They interpret a previously uncharacterized (but reliable) voxel population and find that it is tuned to people-related content Next, they show that the fronto-temporal human language network is predicted by a common set of features across its constituent regions, but find that frontal regions are relatively well-explained by surprisal alone, even in the absence of LM-based features Finally, they show that brain responses during language processing are not merely predictable from an arbitrary set of LM features; rather, brain responses are best explained by the features that tend to capture the most general information encoded in LM representations
Predictivity Across Voxel Subtypes
The study analyzes four PC-derived voxel subtypes: Hard-to-Process, Easy-to-Process, Abstract, and Concrete Finding 1 shows that SAE features tend to predict voxel responses to language stimuli nearly as well as dense LM residual features In particular, features in the Matryoshka SAE basis achieve very similar performance as features in the dense residual LM basis, across all four voxel subtypes Finding 2 demonstrates a dissociation between processing difficulty-driven and content-driven voxel regimes: voxels in the Hard-to-Process subtype are captured by surprisal alone, with no significant improvement from SAE or residual stream features, in contrast to Abstract and Concrete subtypes which benefit significantly from the inclusion of SAE or residual-stream features above and beyond surprisal
Characterization of Content-Driven Features
Finding 3 indicates that distinct features refine interpretations of content-driven voxels, showing that Abstract and Concrete voxels are consistently supported by different feature subspaces For the We find that the features used to predict this voxel subpopulation are
typically driven by sentences that include “people-specific” features, such as discussions of relationships/emotions (Feat. #40+), descriptions of people taking concrete actions (Feat. #40+), descriptions of people taking concrete actions (Feat. #94+), or
pronouns (Feat.Ghost
voxel subtype, a coherent subpopulation is identified which is tuned to people-centered content This subpopulation is driven by features related to sentences that include “people-specific” features, such as discussions of relationships/emotions (Feat. 40+), descriptions of people taking concrete actions (Feat. p. 7Where the paper says it
Where the paper says it
General Feature Dominance in the Language Network
In Study 3, the authors analyze the five canonical left-lateralized language regions (fROIs) using Augmented Sparse Encoding Models Finding 1 shows that frontal fROIs are relatively well-captured by surprisal alone, particularly the middle frontal gyrus (MFG), which does not show a significant gain from SAE/residual-stream features beyond surprisal In contrast, temporal fROIs show a significant boost from feature-based models above and beyond surprisal, indicating stronger sensitivity to content-linked structure during sentence comprehension Finding 3 concludes that human language network voxels are preferentially explained by general, widely applicable LM features Specifically, the language-network encoding models preferentially load on the most general Matryoshka features (feature indices < 128), suggesting that brain-alignment with the human language network relies on the LM’s general, widely-applicable feature dimensions rather than fine-grained, idiosyncratic features
Conclusion and Future Directions
Augmented Sparse Encoding Models move beyond prior approaches that treat both models and brains as black boxes The authors conclude that tuning within the language network is shared but not uniform, with regions differing in their sensitivity to processing difficulty versus content Implications include characterizing linguistic representations in the language network, interpreting the “Ghost” voxels tuned to people-related content, and determining which properties of LM representations mediate brain alignment with human language responses Future work suggests designing stimuli to drive targeted voxel populations for investigating neural control in interpretable voxel subspaces
Methodological Details
The analysis pipeline for each voxel involves a two-stage process: feature selection using LASSO regression followed by refitting with Ridge regression, always including the surprisal feature in the Ridge regression The authors employ two SAE architectures, JumpReLU and Matryoshka, with Matryoshka introducing additional structure to learn more disentangled features Cross-validated predictivity is estimated using a two-stage processing pipeline for each voxel The analysis also replicates primary quantitative results using an independent 3T dataset, showing qualitative similarity to the main 7T dataset
Feature Hierarchy and Sharing
The Matryoshka SAE imposes an ordered feature hierarchy where early latent features account for general, widely-applicable variance in LM representations, and later features are progressively more granular The analysis of feature prevalence across participants reveals four regimes of shared features The two most prevalent features are Feat. 79 and Feat. p. 6 of these features broadly align with existing interpretations (Tuckute et al. For example,
Abstract voxels are driven by features related to emotions (Feat.Where the paper says it
Limitations and Specific Findings
Limitations include the interpretations being limited to the SAE's feature basis, and some SAE features remaining opaque The authors note that more general Matryoshka SAE features predict voxel tuning because voxels are a coarse-grained measure of neural activity Furthermore, the Ghost analysis shows that even the best performing feature sets only achieve a Normalized Predictivity of approximately.1 for Ghost voxels The authors find that social tuning during comprehension of short sentences appears partly adjacent to the core language network and partly interspersed with temporal language areas The paper provides a foundation for further investigations into feature tuning during language processing with larger sets of diverse stimuli
Anatomical Localization of Novel Subpopulations
The people-centered voxel subpopulation, identified through the Ghost analysis, was located largely outside traditional frontal and temporal areas for language The largest proportion of voxels were located in the angular gyrus area (43.8%), outside the five “core” fronto-temporal language fROIs analyzed in Study 6 This subpopulation also included some medial frontal voxels, which have been described as implicated in social cognition and theory-of-mind tasks The identification and interpretation of this subpopulation demonstrates how Augmented Sparse Encoding Models can recover not only broad organizing dimensions, but also representational structure shared across a subset of individuals
Feature Granularity and Alignment
The analysis connects SAE feature granularity to work in the visual domain, suggesting that hierarchical SAE bases can increase model–brain alignment when superposition obscures shared latent structure The strongest alignment is not carried by the multiplicity of fine-grained features that contribute to an LM’s high-dimensional feature space, but rather by the set of features that most generically characterize LM representations This suggests partial overlap in the features that organize internal language representations in the human language network and LMs, even though these systems presumably having very different means of arriving at these representations The claim about general SAE dimensions aligns with Antonello et al., 2021, who showed that low-dimensional structure in LM representations is reflected in brain responses The paper extends this work by identifying the relevant dimensions in an interpretable sparse feature basis
Replication and Data Handling
The replication using the 3T brain dataset shows that MFG exhibits the smallest difference between surprisal vs. other feature sets, and AntTemp exhibits the largest difference The detailed information about language fROI voxel selection for generalization analysis involved sub-sampling to manage computational cost, including deduplication of voxels with identical brain response profiles within each participant The final 2,296 voxels are distributed across participants and fROIs as reported in Table 5
Summary of Findings
The paper demonstrates that small sets of features can be used to predict brain responses to language and separate processing difficulty-related signal from content Within the language network, tuning is shared but not uniform, with regions differing in their sensitivity to processing difficulty versus content The study provides a step towards characterizing how linguistic meaning is represented and organized in the human brain
The gist
A central goal of cognitive neuroscience is to characterize the features that are represented by human language cortex; Artificial language models (LMs) have emerged as a powerful tool to address this challenge, but studies relating biological and artificial representations are often criticized as relating one black box to another
Pruning redundant reasoning tokens
A central goal of cognitive neuroscience is to characterize the features that are represented by human language cortex; Artificial language models (LMs) have emerged as a powerful tool to address this challenge, but studies relating biological and artificial representations are often criticized as relating one black box to another
How the training signal is built
LM encoding models are defined by two components: (i) a feature extractor that transforms language stimuli into a feature vector, and (ii) a linear readout that maps that feature vector onto neural responses to the same linguistic stimuli In this work, we introduce two changes to the typical framework to make the resulting LM encoding model more interpretable First, we employ pretrained SAEs to transform dense, difficult-to-interpret residual-stream LM representations into a much larger, sparser, and (ideally) more interpretable feature basis Our second change is the inclusion of a dedicated feature that encodes the average surprisal (i.e., negative log probability) of a sentence, as computed by an LM
Where the gains come from
Finding 1 shows that SAE features tend to predict voxel responses to language stimuli nearly as well as dense LM residual features Finding 2 demonstrates a dissociation between processing difficulty-driven and content-driven voxel regimes, with Abstract and Concrete voxels benefiting significantly from the inclusion of SAE or residual-stream features above and beyond surprisal The strongest alignment is carried by the multiplicity of fine-grained features that contribute to an LM’s high-dimensional feature space, but brain-predictive features are concentrated among the set of features that most generically characterize LM representations This finding suggests partial overlap in the features that organize internal language representations in the human language network and LMs, even though these systems presumably having very different means of arriving at these representations
The Matryoshka SAE imposes an ordered feature hierarchy where early latent features account for general, widely-applicable variance in LM representations, and later features are progressively more granular The strongest alignment is not carried by the multiplicity of fine-grained features that contribute to an LM’s high-dimensional feature space, but rather by the set of features that most generically characterize LM representations This suggests partial overlap in the features that organize internal language representations in the human language network and LMs, even though these systems presumably having very different means of arriving at these representations
LM encoding models are defined by two components: (i) a feature extractor that transforms language stimuli into a feature vector, and (ii) a linear readout that maps that feature vector onto neural responses to the same linguistic stimuli In this work, we introduce two changes to the typical framework to make the resulting LM encoding model more interpretable First, we employ pretrained SAEs to transform dense, difficult-to-interpret residual-stream LM representations into a much larger, sparser, and (ideally) more interpretable feature basis Our second change is the inclusion of a dedicated feature that encodes the average surprisal (i.e.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improve interpretation of neural responses by using Augmented Sparse Encoding Models to distinguish between processing difficulty-driven and content-driven features, as
brain responses are not merely predictable from an arbitrary set of LM features. Rather, brain responses are best explained by the features that tend to capture the most general information encoded in LM representations.
-
Enable discovery of previously uncharacterized voxel populations by using the hierarchical structure of sparse autoencoders to identify coherent feature-defined subpopulations, such as
a subset of the Ghost voxels form a coherent feature-defined subpopulation,
which are tuned to people-centered content. -
Refine understanding of language network organization by characterizing how different regions respond differentially to processing difficulty versus content features, showing that
some frontal brain regions are explained well by processing difficulty, whereas temporal regions draw more on LM-derived “content” features.
-
Enhance model-brain alignment by focusing on primary, general features rather than idiosyncratic ones, as the study shows that
brain responses to language are predicted by primary, general features of LM representations rather than idiosynractic ones—revealing a deeper correspondence between artificial and biological language representations.
Abstract
A central goal of cognitive neuroscience is to characterize the features that are represented by human language cortex. Artificial language models (LMs) have emerged as a powerful tool to address this challenge, but studies relating biological and artificial representations are often criticized as relating one black box to another. The present work introduces Augmented Sparse Encoding Models, an encoding framework that replaces dense LM hidden states with hierarchically-organized sparse autoencoder (SAE) features, while explicitly including surprisal as a predictor. Using this approach, we (i) produce interpretations of neural responses and (ii) test whether model-brain alignment reflects primary or idiosyncratic variation in LM representations. Using a high-field 7T fMRI dataset of eight participants listening to 200 linguistically diverse sentences, we first validate our modeling framework by recovering previous interpretations of voxel populations tuned to processing difficulty and meaning abstractness. We then interpret a previously-uncharacterized (but reliable) voxel population and find that it is tuned to people-related content. Next, we show that the fronto-temporal human language network is predicted by a common set of features across its constituent regions, but find that frontal regions are relatively well-explained by surprisal alone, even in the absence of LM-based features. Finally, we show that brain responses are best explained by the features that tend to capture the most general information encoded in LM representations, suggesting a nontrivial correspondence between brain and LM language representation.
Sources
- Generative causal testing to bridge data-driven models and scientific theories in language neuroscience
- Sparse Autoencoders Map Brain-LLM Alignment onto Cortical Semantic Topography
- Superposition disentanglement of neural representations reveals hidden alignment
- From Atoms to Trees: Building a Structured Feature Forest with Hierarchical Sparse Autoencoders
- Sparse Autoencoders Bridge The Deep Learning Model and The Brain
- Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders
- Gemma 2: Improving Open Language Models at a Practical Size
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data