Prototypes of the Mind: A Unified Framework for Probing the Visual Brain
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.
Ines: I'm Ines.
Marcus: And I'm Marcus, here with Yuki.
Ines: We're looking at the paper: "Prototypes of the Mind".
Marcus: Here's the gist of the paper. Deep learning has driven substantial progress in modeling fMRI responses, yet most existing approaches are optimized for a single objective—such as neural encoding, stimulus decoding, or post hoc interpretation—making it difficult to obtain a coherent account of neural representations Our approach replaces black-box features with a sparse, interpretable representation composed of visual prototypes, and links these prototypes to voxel responses through a linear readout This design makes the model transparent at both levels: prototype activations expose the visual concepts present in the stimulus, while the learned linear weights characterize each voxel’s selectivity Together, these results establish prototype-based representations as a unified and interpretable foundation for studying visual representations in the human brain Framework Architecture The method introduces a two-stage framework that connects visual perception with neural activity through interpretable prototypes Stage 1 involves learning a dictionary of visual prototypes that factorize foundation model features into sparse combinations, capturing distinct semantics and spatial structure This stage uses a causal encoder with spatial downsampling to reduce feature map resolution and then factors the encoded features H as weighted combinations of these prototypes The resulting prototype-decomposed representation Hˆ = AP provides a sparse, interpretable decomposition of the visual scene in terms of prototypes and their spatial distribution Stage 2 learns a lightweight mapping from spatially distributed prototype activations to fMRI voxel responses using a learned spatial attention mechanism and a single linear layer This yields an interpretable weight matrix W where each row encodes the sensitivity of voxel v to each of the P prototypes Prototype Interpretation and Selectivity The learned prototypes are shown to correspond to interpretable visual concepts rather than arbitrary directions in feature space These prototypes capture diverse visual concepts, such as “airplanes”, “buildings”, “faces”, and “zebra” The linear mapping allows each voxel to be associated with a small set of visual concepts using a linear projection Analysis of prototype profiles for category-selective regions, such as face-selective areas (e.g., Occipital Face Area), shows that prototypes corresponding to faces preferentially contribute to the corresponding ROIs Furthermore, the analysis reveals that a face-selective prototype produces elevated weights concentrated in face-selective cortex, particularly the occipital face area (OFA) and nearby ventral regions Causal Probing Capabilities A key advantage is that individual prototypes can be selectively manipulated, enabling targeted manipulation of visual features for testing causal hypotheses about selectivity in the brain This capability is demonstrated through three types of permutations: Removal, Addition, and Spatial Manipulation For face stimuli, setting prototype activations to zero removes faces selectively from the scene while other objects remain largely intact Increasing the activation of face prototypes induces the appearance of face-like features in previously non-face stimuli Spatial manipulation allows shifting the locations of faces without significantly affecting content in unaltered regions Close-loop experiments show that targeted permutations significantly increase predicted responses in corresponding brain ROIs compared to random permutations Performance and Robustness The prototype representations consistently improve prediction across all three visual backbones and all eight ROIs With DINOv3-ViT, the mean correlation across ROIs increases from 0.338 to 0.394 The prototype representation captures substantial neural response information through a simple linear mapping and improves prediction over the original backbone features across different models The method is relatively robust to the selection of the number of prototypes P and sparsity constraint k, suggesting that increasing P or loosening k leads to only a minor drop in performance Broader Implications This work demonstrates how interpretable representation learning can turn foundation models from black-box predictors into structured tools for probing biological intelligence The framework unifies analyses that are typically studied separately, including neural encoding, categorical selectivity, cortical organization, spatial attribution, and controlled stimulus manipulation The results show how specific concepts map onto cortical responses by visualizing the weights of the linear projection matrix W This may help generate hypotheses about stimulus–response relationships in specific brain regions and guide the design of targeted neuroscience experiments The prototypes generalize well across subjects, with substantial overlap observed for face- and body-selective regions Limitations The authors note that the method may be less effective for interpreting synthetic or atypical stimuli because vision foundation models are largely trained with naturalistic images Additionally, the prototypes do not necessarily correspond to biological prototypes implemented by the brain The PCA control baseline fails to characterize selectivity across visual regions because PCA features do not exhibit clear category-selective patterns The method is evaluated in established brain areas, suggesting room for improvement when applied to less understood regions Experimental Details The study uses the MSCOCO dataset for learning prototypes and the processed NSD dataset for fMRI data The fMRI encoding model is trained on a subject basis, with different training sets created for each subject by splitting the official training set into 80%/10%/10% The diffusion model used in causal intervention analysis is trained from scratch using only visual data, specifically the JiT diffusion model The protocol involves training the encoding model on a subject basis and training the prototype mapping to predict responses for multiple ROIs This ensures that the analysis is less prone to potential biases compared to models pretrained on language data The computational resources utilized include two RTX 4090 GPUs for the main model and one H100 and two RTX 6000s for the diffusion model References
1: Talia Konkle and George A.
Ines: Alvarez.
Marcus: A self-supervised domain-general learning framework for human ventral stream representation.
Ines: Nature Communications, 13(1):491, 2022
2: Aria Y.
Title and authors: Ines: We're starting with this paper called "Prototypes of the Mind: A Unified Framework for Probing the Visual Brain". The title itself suggests they're building a unified way to probe visual representations in the human brain.
Marcus: It’s about using these visual prototypes, which are learned from foundation models like DINOv3, to create an interpretable bridge between those models and fMRI data.
Yuki: I wonder what that means practically for us as population geneticists; can this framework help us understand how visual concepts map onto the brain across different people?
Ines: It suggests they are using a causal-attention encoder in stage one to factorize the features from the foundation model into a sparse combination of learnable prototypes, which captures both distinct semantics and spatial structure.
Marcus: Then in stage two, they train a linear mapping between those prototype activations and the fMRI responses using a learned spatial attention mechanism and just a single linear layer.
Yuki: So it’s connecting the visual concepts directly to voxel responses through this learned mapping, which sounds like a way to visualize cortical organization.
Ines: It's about making the model transparent at both levels, showing both what drives neural activity and where those driving concepts are located in the image. This is a big shift from just looking at one side of the problem.
The paper's summary: Marcus: So to summarize this framework, they factorize rich features from vision foundation models, like DINOv3, into spatially grounded prototype units that capture recurring visual patterns in natural images. These activations are sparse and keep the spatial layout intact.
Ines: And then they use a causal-attention decoder in stage one to learn these prototypes, which decompose the foundation model features into sparse components that represent distinct semantics.
Yuki: The key takeaway here is that this results in a prototype-decomposed representation, which allows each brain voxel to be associated with only a small set of visual concepts using a linear projection.
Marcus: Stage two then takes those spatially distributed prototype activations and learns a lightweight mapping to fMRI responses, resulting in a weight matrix where each row encodes the sensitivity of that specific voxel to every one of the learned prototypes.
Ines: They show that these learned prototypes actually correspond to interpretable visual concepts, like "airplanes" or "faces," rather than just random directions in feature space.
Yuki: This is significant because it moves us from abstract feature vectors to something we can actually name and map onto the visual world. It gives structure to the representation itself.
The paper's improvements: Ines: They highlight a few key improvements over previous methods. First, they show that prototypes corresponding to specific concepts, like faces, preferentially contribute to the corresponding regions of interest in brain analysis.
Marcus: They also reveal that a face-selective prototype produces elevated weights concentrated in face-selective cortex, specifically the occipital face area and nearby ventral regions. That gives us a concrete map of where those concepts are being processed.
Yuki: I'm interested in the causal probing capabilities they show, because it means we can manipulate these prototypes to test hypotheses about selectivity in the brain directly.
Ines: They demonstrate this through three types of permutations: removal, addition, and spatial manipulation. For example, setting prototype activations to zero removes faces selectively from the scene while other objects stay largely intact.
Marcus: And they found that targeted permutations significantly increase predicted responses in corresponding brain regions compared to random ones. That’s a strong signal for causal inference in this context.
Yuki: If we can manipulate the stimulus at the prototype level and see a predictable change in brain response, that gives us a tool to test hypotheses about how specific visual concepts drive neural activity across different subjects.
Conclusion: Ines: So wrapping up, this paper on "Prototypes of the Mind: A Unified Framework for Probing the Visual Brain" shows how to unify vision foundation models with brain activity using a sparse, interpretable prototype representation.
Marcus: The main implication is that we can now identify which visual concepts drive neural responses and where those concepts appear in the image, while also characterizing voxel selectivity through that learned weight matrix.
Yuki: For me, it’s about having a structured way to link stimulus features to cortical organization, giving us tools to test hypotheses about how specific visual concepts map onto brain regions across different people.
Ines: It turns foundation models from black-box predictors into tools for studying biological intelligence by making the representations transparent at both the concept and voxel level.
Marcus: It’s a framework that supports analysis, stimulus reconstruction, and controlled manipulation of visual content because of that strong generative property in the representation space.
Yuki: I think it’s a step toward understanding how specific concepts map onto cortical responses by visualizing those linear projection weights W, which could guide experimental design for targeted neuroscience.
Shi Chen, Seoyoung Ahn, Doris Tsao
Department of Neuroscience, University of California Berkeley · College of AI Convergence, Hankuk University of Foreign Studies
q-bio.NC, cs.AI
Submitted: 2026-11-11
Updated: 2026-11-11
Comments: NeurIPS 2026
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 91/100
The gist: Deep learning has driven substantial progress in modeling fMRI responses, yet most existing approaches are optimized for a single objective—such as neural encoding, stimulus decoding…
Key concepts
- Visual Prototypes
- These are sparse, learned representations of distinct visual concepts (e.g., 'zebra', 'building'). Instead of using raw image data, the model decomposes any visual scene into a small set of these meaningful concepts. This makes the representation transparent because each prototype corresponds to a recognizable visual idea.
- Linear Readout
- This is the simple mathematical step that connects the learned visual prototypes to actual brain activity in voxels. It acts like a filter, determining how much each specific visual concept (prototype) influences the response of every single brain location (voxel). This mapping is linear and easy to interpret.
- Causal Probing
- This capability lets researchers test hypotheses about cause and effect in the brain. By selectively turning on or off a specific prototype, like 'faces,' scientists can see exactly how much that concept contributes to the predicted brain response in a particular area, allowing for controlled manipulation of visual stimuli.
Terminology
Summary
Deep learning has driven substantial progress in modeling fMRI responses, yet most existing approaches are optimized for a single objective—such as neural encoding, stimulus decoding, or post hoc interpretation—making it difficult to obtain a coherent account of neural representations Our approach replaces black-box features with a sparse, interpretable representation composed of visual prototypes, and links these prototypes to voxel responses through a linear readout This design makes the model transparent at both levels: prototype activations expose the visual concepts present in the stimulus, while the learned linear weights characterize each voxel’s selectivity Together, these results establish prototype-based representations as a unified and interpretable foundation for studying visual representations in the human brain
Framework Architecture
The method introduces a two-stage framework that connects visual perception with neural activity through interpretable prototypes Stage 1 involves learning a dictionary of visual prototypes that factorize foundation model features into sparse combinations, capturing distinct semantics and spatial structure This stage uses a causal encoder with spatial downsampling to reduce feature map resolution and then factors the encoded features H as weighted combinations of these prototypes The resulting prototype-decomposed representation Hˆ = AP provides a sparse, interpretable decomposition of the visual scene in terms of prototypes and their spatial distribution Stage 2 learns a lightweight mapping from spatially distributed prototype activations to fMRI voxel responses using a learned spatial attention mechanism and a single linear layer This yields an interpretable weight matrix W where each row encodes the sensitivity of voxel v to each of the P prototypes
Prototype Interpretation and Selectivity
The learned prototypes are shown to correspond to interpretable visual concepts rather than arbitrary directions in feature space These prototypes capture diverse visual concepts, such as “airplanes”, “buildings”, “faces”, and “zebra” The linear mapping allows each voxel to be associated with a small set of visual concepts using a linear projection Analysis of prototype profiles for category-selective regions, such as face-selective areas (e.g., Occipital Face Area), shows that prototypes corresponding to faces preferentially contribute to the corresponding ROIs Furthermore, the analysis reveals that a face-selective prototype produces elevated weights concentrated in face-selective cortex, particularly the occipital face area (OFA) and nearby ventral regions
Causal Probing Capabilities
A key advantage is that individual prototypes can be selectively manipulated, enabling targeted manipulation of visual features for testing causal hypotheses about selectivity in the brain This capability is demonstrated through three types of permutations: Removal, Addition, and Spatial Manipulation For face stimuli, setting prototype activations to zero removes faces selectively from the scene while other objects remain largely intact Increasing the activation of face prototypes induces the appearance of face-like features in previously non-face stimuli Spatial manipulation allows shifting the locations of faces without significantly affecting content in unaltered regions Close-loop experiments show that targeted permutations significantly increase predicted responses in corresponding brain ROIs compared to random permutations
Performance and Robustness
The prototype representations consistently improve prediction across all three visual backbones and all eight ROIs With DINOv3-ViT, the mean correlation across ROIs increases from 0.338 to 0.394 The prototype representation captures substantial neural response information through a simple linear mapping and improves prediction over the original backbone features across different models The method is relatively robust to the selection of the number of prototypes P and sparsity constraint k, suggesting that increasing P or loosening k leads to only a minor drop in performance
Broader Implications
This work demonstrates how interpretable representation learning can turn foundation models from black-box predictors into structured tools for probing biological intelligence The framework unifies analyses that are typically studied separately, including neural encoding, categorical selectivity, cortical organization, spatial attribution, and controlled stimulus manipulation The results show how specific concepts map onto cortical responses by visualizing the weights of the linear projection matrix W This may help generate hypotheses about stimulus–response relationships in specific brain regions and guide the design of targeted neuroscience experiments The prototypes generalize well across subjects, with substantial overlap observed for face- and body-selective regions
Limitations
The authors note that the method may be less effective for interpreting synthetic or atypical stimuli because vision foundation models are largely trained with naturalistic images Additionally, the prototypes do not necessarily correspond to biological prototypes implemented by the brain The PCA control baseline fails to characterize selectivity across visual regions because PCA features do not exhibit clear category-selective patterns The method is evaluated in established brain areas, suggesting room for improvement when applied to less understood regions
Experimental Details
The study uses the MSCOCO dataset for learning prototypes and the processed NSD dataset for fMRI data The fMRI encoding model is trained on a subject basis, with different training sets created for each subject by splitting the official training set into 80%/10%/10% The diffusion model used in causal intervention analysis is trained from scratch using only visual data, specifically the JiT diffusion model The protocol involves training the encoding model on a subject basis and training the prototype mapping to predict responses for multiple ROIs This ensures that the analysis is less prone to potential biases compared to models pretrained on language data The computational resources utilized include two RTX 4090 GPUs for the main model and one H100 and two RTX 6000s for the diffusion model
References
[1] Talia Konkle and George A. Alvarez. A self-supervised domain-general learning framework for human ventral stream representation. p. 9 References
[1] Talia Konkle and George A. A self-supervised domain-general learning framework for human
ventral stream representation.Where the paper says it
[2] Aria Y. Wang, Kendrick Kay, Thomas Naselaris, Michael J. p. 9 [2] Aria Y. Wang, Kendrick Kay, Thomas Naselaris, Michael J. Wang, Kendrick Kay, Thomas Naselaris, Michael J. Tarr, and Leila Wehbe. Tarr, and Leila Wehbe. Better models of
human high-level visual cortex emerge from natural language supervision with a large and diverse dataset.Where the paper says it
Where the paper says it
Where the paper says it
[3] Huzheng Yang, James Gee, and Jianbo Shi. Brain decodes deep nets. p. 9 [3] Huzheng Yang, James Gee, and Jianbo Shi. Brain decodes deep nets.Where the paper says it
[4] Hossein Adeli, Minni Sun, and Nikolaus Kriegeskorte. Transformer brain encoders explain human high-level visual responses. p. 9 [4] Hossein Adeli, Minni Sun, and Nikolaus Kriegeskorte. Transformer brain encoders explain human
high-level visual responses.Where the paper says it
[5] Colin Conwell, Jacob S. Prince, Kendrick N. p. 9 [5] Colin Conwell, Jacob S. Prince, Kendrick N. Prince, Kendrick N. Kay, George A. Kay, George A. Alvarez, and Talia Konkle. Alvarez, and Talia Konkle. A large-scale
examination of inductive biases shaping high-level visual representation in brains and machines.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[6] Andrew Luo, Margaret Marie Henderson, Leila Wehbe, and Michael J. p. 9 Nature
Communications, 15(1):9383, 2024. [6] Andrew Luo, Margaret Marie Henderson, Leila Wehbe, and Michael J. [6] Andrew Luo, Margaret Marie Henderson, Leila Wehbe, and Michael J. Brain diffusion for visual
exploration: Cortical discovery using large scale generative models.Where the paper says it
Where the paper says it
[7] Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. p. 9 [7] Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from
human brain activity.Where the paper says it
[8] Pinyuan Feng, Hossein Adeli, Wenxuan Guo, Fan L. Cheng, Ethan Hwang, and Nikolaus Kriegeskorte. p. 9 [8] Pinyuan Feng, Hossein Adeli, Wenxuan Guo, Fan L. Cheng, Ethan Hwang, and Nikolaus Kriegeskorte. Cheng, Ethan Hwang, and Nikolaus Kriegeskorte. Towards interpretable visual decoding with attention to brain representations.Where the paper says it
Where the paper says it
[9] Mu Nan, Muquan Yu, Weijian Mai, Jacob S. Prince, Hossein Adeli, Rui Zhang, Jiahang Cao, Benjamin Becker, John A. p. 9 In CVPR, pages 23030–23040, 2024. [4] Hossein Adeli, Minni Sun, and Nikolaus Kriegeskorte. Nature
Communications, 15(1):9383, 2024. [6] Andrew Luo, Margaret Marie Henderson, Leila Wehbe, and Michael J. Nature
Communications, 15(1):9383, 2024. [6] Andrew Luo, Margaret Marie Henderson, Leila Wehbe, and Michael J. Wang, Kendrick Kay, Thomas Naselaris, Michael J. Tarr, and Leila Wehbe. Prototypes of the Mind: A Unified Framework for
Probing the Visual Brain
Shi Chen1 Seoyoung Ahn2 Doris Tsao1
1Department of Neuroscience, UC Berkeley
2College of AI Convergence, Hankuk University of Foreign Studies
Abstract
Deep learning has driven substantial progress in modeling fMRI responses, yet most
existing approaches are optimized for a single objective—such as neural encoding,
stimulus decoding, or post hoc interpretation—making it difficult to obtain a
coherent account of neural representations. We introduce a unified framework that
connects vision foundation models to brain activity while supporting neural analysis
from multiple complementary perspectives.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
[10] Paul Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, ethan cohen, Aidan Dempster, Nathalie Verlinde, Elad Yundler, David Weisberg, Kenneth Norman, and Tanishq Abraham. Reconstructing the mind's eye: fmri-to-image with contrastive learning and diffusion priors. In NeurIPS 36
[11] Roman Beliy, Amit Zalcher, Jonathan Kogman, Navve Wasserman, and Michal Irani. Brain-it: Image reconstruction from fmri via brain-interaction transformer. p. 9 [7] Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from
human brain activity.Where the paper says it
[12] Alicia Zeng and Jack Gallant. Disentangling superpositions: Interpretable brain encoding model with sparse concept atoms. p. 2 (2) In stage-2, we train a linear mapping between the activation of prototypes and
fMRI responses, enabling brain understanding from complementary perspectives. decomposes foundation model features into sparse, interpretable components.Where the paper says it
[13] Navve Wasserman, Matias Cosarinsky, Yuval Golbari, Aude Oliva, Antonio Torralba, Tamar Rott Shaham, and Michal Irani. Brainexplore: Large-scale discovery of interpretable visual representations in the human brain. p. 9 [6] Andrew Luo, Margaret Marie Henderson, Leila Wehbe, and Michael J. Brain diffusion for visual
exploration: Cortical discovery using large scale generative models.Where the paper says it
[14] Gabriel Sarch, Michael Tarr, Katerina Fragkiadaki, and Leila Wehbe. Brain dissection: fmri-trained networks reveal spatial selectivity in the processing of natural images. p. 0 Moving beyond black-box features learned by deep networks, several recent studies develop interpretable representations that reveal
categorical [12, 13] and spatial [14, 15] selectivity. Collectively, these approaches have produced valuable tools for probing the functional organization of the brain.Where the paper says it
[15] Andrew F. Luo, Jacob Yeung, Rushikesh Zawar, Shaurya Dewan, Margaret M. p. 9 Nature
Communications, 15(1):9383, 2024. [6] Andrew Luo, Margaret Marie Henderson, Leila Wehbe, and Michael J. Our central contribution is to transform dense features from vision foundation models into
sparse, spatially grounded prototypes that capture coherent visual concepts and can be linked to fMRI
responses through a transparent linear mapping. This design turns an opaque visual representation
into an interpretable interface between images and cortex: prototypes describe what concepts are
present and where they occur, while voxel weights reveal which concepts drive each brain region.Where the paper says it
Where the paper says it
[16] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jegou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. p. 2 decomposes foundation model features into sparse, interpretable components. Using a self-supervised
learning paradigm that emphasizes contextual relationships among visual regions, the model supports
a rich interpretation of both the concepts present in a scene and their spatial organization.Where the paper says it
[17] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. p. 2 decomposes foundation model features into sparse, interpretable components. Using a self-supervised
learning paradigm that emphasizes contextual relationships among visual regions, the model supports
a rich interpretation of both the concepts present in a scene and their spatial organization.Where the paper says it
[18] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. p. 0 Prototypes of the Mind: A Unified Framework for
Probing the Visual Brain
Shi Chen1 Seoyoung Ahn2 Doris Tsao1
1Department of Neuroscience, UC Berkeley
2College of AI Convergence, Hankuk University of Foreign Studies
Abstract
Deep learning has driven substantial progress in modeling fMRI responses, yet most
existing approaches are optimized for a single objective—such as neural encoding,
stimulus decoding, or post hoc interpretation—making it difficult to obtain a
coherent account of neural representations. We introduce a unified framework that
connects vision foundation models to brain activity while supporting neural analysis
from multiple complementary perspectives.Where the paper says it
[19] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan Su. This looks like that: Deep learning for interpretable image recognition. p. 0 Prototypes of the Mind: A Unified Framework for
Probing the Visual Brain
Shi Chen1 Seoyoung Ahn2 Doris Tsao1
1Department of Neuroscience, UC Berkeley
2College of AI Convergence, Hankuk University of Foreign Studies
Abstract
Deep learning has driven substantial progress in modeling fMRI responses, yet most
existing approaches are optimized for a single objective—such as neural encoding,
stimulus decoding, or post hoc interpretation—making it difficult to obtain a
coherent account of neural representations. We introduce a unified framework that
connects vision foundation models to brain activity while supporting neural analysis
from multiple complementary perspectives.Where the paper says it
[20] Pang Wei Koh, Thao Nguyen, Yewen Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. p. 5 Specifically, we extract prototype activation maps A of
images from MSCOCO validation set, and identify the representative images with strong activation
(averaged across all patches) for each prototype (see Appendix P for details). To enhance consistency
among the final representatives, we also include an optional step that computes pairwise cosine
similarities between CLIP [35] embeddings of these candidates and applies spectral clustering [36]
to select 20 images that are both highly activated and representative of a shared visual concept (see
Appendix E for comparison between visualization with and without clustering).Where the paper says it
[21] Oriane Siméoni et al. Dinov3. In arXiv 2025
[23] Thomas Naselaris, Kendrick N. Kay, Shinji Nishimoto, and Jack L. p. 9 In NeurIPS, 2023. [7] Yu Takagi and Shinji Nishimoto. Prototypes of the Mind: A Unified Framework for
Probing the Visual Brain
Shi Chen1 Seoyoung Ahn2 Doris Tsao1
1Department of Neuroscience, UC Berkeley
2College of AI Convergence, Hankuk University of Foreign Studies
Abstract
Deep learning has driven substantial progress in modeling fMRI responses, yet most
existing approaches are optimized for a single objective—such as neural encoding,
stimulus decoding, or post hoc interpretation—making it difficult to obtain a
coherent account of neural representations. We introduce a unified framework that
connects vision foundation models to brain activity while supporting neural analysis
from multiple complementary perspectives.Where the paper says it
Where the paper says it
[24] Zijin Gu et al. Neurogen: Activation optimized image synthesis for discovery neuroscience. p. 0 Prototypes of the Mind: A Unified Framework for
Probing the Visual Brain
Shi Chen1 Seoyoung Ahn2 Doris Tsao1
1Department of Neuroscience, UC Berkeley
2College of AI Convergence, Hankuk University of Foreign Studies
Abstract
Deep learning has driven substantial progress in modeling fMRI responses, yet most
existing approaches are optimized for a single objective—such as neural encoding,
stimulus decoding, or post hoc interpretation—making it difficult to obtain a
coherent account of neural representations. We introduce a unified framework that
connects vision foundation models to brain activity while supporting neural analysis
from multiple complementary perspectives.Where the paper says it
[25] Carlos R. Ponce et al. Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences. p. 1 Early studies used optimizationbased approaches to generate stimuli that maximally activate neurons or brain regions [25, 26]. More
recent methods combine fMRI decoding with powerful generative models such as diffusion models
[7, 10, 11], enabling reconstruction of naturalistic images from measured neural activity.Where the paper says it
[26] N. Apurva Ratan Murty et al. Computational models of category-selective brain regions enable high-throughput tests of selectivity. p. 0 Advances in deep learning, together with the increasing availability of large-scale neural datasets,
have transformed Functional Magnetic Resonance Imaging (fMRI) analysis by enabling increasingly
effective computational models of brain responses. One major line of work develops encoding models
[1, 2, 3, 4, 5] that seek to predict fMRI responses from visual stimuli with high accuracy.Where the paper says it
[27] A. T. Gifford et al. The algonauts project 2023 challenge: How the human brain makes sense of natural scenes. p. 0 1 Introduction
Understanding how the human brain represents complex visual scenes remains a fundamental
challenge at the intersection of neuroscience and computer vision. Neural measurements are sparse
and noisy, while natural scenes are incredibly rich and complex.Where the paper says it
[28] Lijun Yu et al. Language model beats diffusion - tokenizer is key to visual generation. p. 7 7 Prototype Manipulations Enable Controlled Stimulus Interventions
A key advantage of our method is that individual prototypes can be selectively manipulated, enabling
targeted manipulation of visual features for testing causal hypotheses about selectivity in the brain. We
demonstrate this unique capability by training a diffusion model (see Appendix O) that reconstructs
the original image based on its prototype activation map (i.Where the paper says it
[29] Ashish Vaswani et al. Attention is all you need. p. 3 Specifically, we rasterize the
visual patches, and task our model to predict features of the next patches based on preceding ones:
Lproto =
1
M − 1
M
X−1
m=1
∥yˆm − ym+1∥
2
, (7)
where yˆm is the prediction at position m and ym+1 is the target at position m+1. This objective
encourages the model to capture the spatial structure without the need of additional annotations.Where the paper says it
[30] Alec Radford et al. Improving language understanding by generative pre-training. p. 3 The reconstructed representation Hˆ is passed through
a decoder composed of a single causal self-attention block followed by a linear readout head:
Yˆ = gread
fdec(Hˆ )
∈ R
M×D′
, (6)
Note that the readout projects each decoded patch into a higher-dimensional space (D′=4D=3072),
to accommodate the 2×2 spatial downsampling within the encoder. To preserve the discriminative power of the learned prototypes, we adopt an autoregressive training
paradigm used in large language [30] and multimodal [32] models.Where the paper says it
[31] Robert Huben et al. Sparse autoencoders find highly interpretable features in language models. p. 3 For each encoded patch hm ∈ R
D, the prototype weights sm are computed from
its cosine similarities with all prototypes:
sm = hmP
⊤ ∈ R
P
. (3)
With the overarching goal of developing an interpretable representation, we draw inspirations from
sparse autoencoder for analyzing language models [31] and further impose sparsity constraints on
prototype selection.Where the paper says it
[32] Sharut Gupta et al. Better together: Leveraging unpaired multimodal data for stronger unimodal models. p. 9 Tarr, and Leila Wehbe. Better models of
human high-level visual cortex emerge from natural language supervision with a large and diverse dataset.Where the paper says it
[33] Tsung-Yi Lin et al. Microsoft COCO: Common objects in context. p. 5 As shown in Figure 2, individual prototypes consistently capture coherent visual concepts. Some
prototypes specialize in inanimate objects (“airplanes”, “buildings”), while others correspond to
animate entities (“faces”, “zebra”) or activities (“tennis”) or abstract scene-level concepts (“crowds”).Where the paper says it
[34] Emily J. Allen et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. p. 9 Better models of
human high-level visual cortex emerge from natural language supervision with a large and diverse dataset. Nature Machine Intelligence, 5(12):1415–1426, 2023.Where the paper says it
[35] Alec Radford et al. Learning transferable visual models from natural language supervision. p. 9 Tarr, and Leila Wehbe. Better models of
human high-level visual cortex emerge from natural language supervision with a large and diverse dataset.Where the paper says it
[36] Andrew Y. Ng, Michael I. p. 9 [2] Aria Y. Wang, Kendrick Kay, Thomas Naselaris, Michael J. Specifically, we extract prototype activation maps A of
images from MSCOCO validation set, and identify the representative images with strong activation
(averaged across all patches) for each prototype (see Appendix P for details). To enhance consistency
among the final representatives, we also include an optional step that computes pairwise cosine
similarities between CLIP [35] embeddings of these candidates and applies spectral clustering [36]
to select 20 images that are both highly activated and representative of a shared visual concept (see
Appendix E for comparison between visualization with and without clustering).Where the paper says it
Where the paper says it
[37] Alexander G. Huth et al. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. p. 1 as an interpretable bridge between vision foundation models [16, 17, 18] and the brain, grounding
neural analysis in the semantic structure learned from large-scale visual data. At the same time, by
linking brain responses to a representational space with strong generative properties, it can support
more advanced applications that go beyond passive analysis, including stimulus reconstruction and
controlled manipulation of visual content.Where the paper says it
[38] Robin Rombach et al. High-resolution image synthesis with latent diffusion models. p. 9 [7] Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from
human brain activity.Where the paper says it
[39] David Mayo et al. Brainbits: How much of the brain are generative reconstruction methods using? p. 1 Early studies used optimizationbased approaches to generate stimuli that maximally activate neurons or brain regions [25, 26]. More
recent methods combine fMRI decoding with powerful generative models such as diffusion models
[7, 10, 11], enabling reconstruction of naturalistic images from measured neural activity.Where the paper says it
[40] Harold Hotelling. Analysis of a complex of statistical variables into principal components. p. 1 Realizing this goal requires a representation that is semantically interpretable, structurally organized,
and flexible enough to support both analysis and intervention. Guided by the intuition that complex
visual scenes can be decomposed into distinct visual concepts [19, 20], we factorize rich features
from vision foundation models (e.Where the paper says it
[41] Tianhong Li and Kaiming He. Back to basics: Let denoising generative models denoise. p. 1 as an interpretable bridge between vision foundation models [16, 17, 18] and the brain, grounding
neural analysis in the semantic structure learned from large-scale visual data. At the same time, by
linking brain responses to a representational space with strong generative properties, it can support
more advanced applications that go beyond passive analysis, including stimulus reconstruction and
controlled manipulation of visual content.Where the paper says it
[42] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. p. 1 Early studies used optimizationbased approaches to generate stimuli that maximally activate neurons or brain regions [25, 26]. More
recent methods combine fMRI decoding with powerful generative models such as diffusion models
[7, 10, 11], enabling reconstruction of naturalistic images from measured neural activity.Where the paper says it
[43] Jia Deng et al. Imagenet: A large-scale hierarchical image database. p. 1 Recent studies increasingly leverage features from deep networks
or multimodal foundation models [1, 2, 3, 4, 5], which capture hierarchical visual representations
aligned with cortical organization. While these DNN-based approaches are effective for predicting
neural activity and characterizing voxel selectivity, they typically operate as black-box predictors,
offering limited interpretability about which specific image properties drive the responses.Where the paper says it
[44] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. p. 2 , DINOv3, N = 16 × 16 is the number of spatial patches and D = 768
is the feature dimension), we first apply a learnable layer [28] to reduce the feature map resolution
from N = 16 × 16 to M = 8 × 8. This step both lowers computational cost and facilitates model
optimization:
Z = SpatialDown(X) ∈ R
M×D.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
A unified framework connects vision foundation models to brain activity by replacing black-box features with a sparse, interpretable representation composed of visual prototypes, allowing for analysis from multiple complementary perspectives such as
prototype activations expose the visual concepts present in the stimulus
andthe learned linear weights characterize each voxel’s selectivity.
-
The improved system can identify
which visual concepts drive neural responses and where they appear in the image,
revealcortical organization through voxel tuning in prototype space,
and perform controlled stimulus synthesis to probe cortical selectivity through targeted interventions. -
The system supports causal probing by enabling
targeted stimulus interventions on interpretable prototypes (e.g., manipulating face and body prototypes),
allowing researchers to test hypotheses aboutthe relationship between neural activity and visual concepts.
-
The system allows for a direct characterization of voxel selectivity via the learned weight matrix W, where
each row wv in R P of W encodes the sensitivity of voxel v to each of the P prototypes,
providing adirect and interpretable characterization of voxel selectivity.
-
The system can reveal spatial selectivity by deriving
spatial attention maps from the prototype activations associated with each ROI
and showing that it revealswhere those features are located within the scene
by highlighting which prototypes contribute most strongly to predicted activity in specific brain areas.
Abstract
Deep learning has driven substantial progress in modeling fMRI responses, yet most existing approaches are optimized for a single objective---such as neural encoding, stimulus decoding, or post hoc interpretation---making it difficult to obtain a coherent account of neural representations. We introduce a unified framework that connects vision foundation models to brain activity while supporting neural analysis from multiple complementary perspectives. Our approach replaces black-box features with a sparse, interpretable representation composed of visual prototypes, and links these prototypes to voxel responses through a linear readout. This design makes the model transparent at both levels: prototype activations expose the visual concepts present in the stimulus, while the learned linear weights characterize each voxel's selectivity. Using this interface, we identify which visual concepts drive neural responses and where they appear in the image, reveal cortical organization through voxel tuning in prototype space, and perform controlled stimulus synthesis to probe cortical selectivity through targeted interventions. Together, these results establish prototype-based representations as a unified and interpretable foundation for studying visual representations in the human brain.
Related papers
- Self-Supervised Learning of Local Predictive Directions as a Neural Computation
- Synchronized Hearts, Desynchronized Brains: A Nonlinear Dynamics Perspective for Function and Failure
- Message interaction dynamics covary with brain volume across mammalian connectomes
- Planning as Dynamics Relaxation: Hippocampal Recurrent Network Realizes Optimal Goal-Directed Navigation
- The Platonic brain bridge hypothesis: brain networks as architectural priors for multimodal large language models
- BrainPFN: Amortized Brain-Behaviour Prediction via Prior-Data Fitted Networks