Prototypes of the Mind: A Unified Framework for Probing the Visual Brain

arXiv:2611.15046 · q-bio.NC, cs.AI · Submitted 2026-11-11 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.

Ines: I'm Ines.

Marcus: And I'm Marcus, here with Yuki.

Ines: We're looking at the paper: "Prototypes of the Mind".

Marcus: Here's the gist of the paper. Deep learning has driven substantial progress in modeling fMRI responses, yet most existing approaches are optimized for a single objective—such as neural encoding, stimulus decoding, or post hoc interpretation—making it difficult to obtain a coherent account of neural representations Our approach replaces black-box features with a sparse, interpretable representation composed of visual prototypes, and links these prototypes to voxel responses through a linear readout This design makes the model transparent at both levels: prototype activations expose the visual concepts present in the stimulus, while the learned linear weights characterize each voxel’s selectivity Together, these results establish prototype-based representations as a unified and interpretable foundation for studying visual representations in the human brain Framework Architecture The method introduces a two-stage framework that connects visual perception with neural activity through interpretable prototypes Stage 1 involves learning a dictionary of visual prototypes that factorize foundation model features into sparse combinations, capturing distinct semantics and spatial structure This stage uses a causal encoder with spatial downsampling to reduce feature map resolution and then factors the encoded features H as weighted combinations of these prototypes The resulting prototype-decomposed representation Hˆ = AP provides a sparse, interpretable decomposition of the visual scene in terms of prototypes and their spatial distribution Stage 2 learns a lightweight mapping from spatially distributed prototype activations to fMRI voxel responses using a learned spatial attention mechanism and a single linear layer This yields an interpretable weight matrix W where each row encodes the sensitivity of voxel v to each of the P prototypes Prototype Interpretation and Selectivity The learned prototypes are shown to correspond to interpretable visual concepts rather than arbitrary directions in feature space These prototypes capture diverse visual concepts, such as “airplanes”, “buildings”, “faces”, and “zebra” The linear mapping allows each voxel to be associated with a small set of visual concepts using a linear projection Analysis of prototype profiles for category-selective regions, such as face-selective areas (e.g., Occipital Face Area), shows that prototypes corresponding to faces preferentially contribute to the corresponding ROIs Furthermore, the analysis reveals that a face-selective prototype produces elevated weights concentrated in face-selective cortex, particularly the occipital face area (OFA) and nearby ventral regions Causal Probing Capabilities A key advantage is that individual prototypes can be selectively manipulated, enabling targeted manipulation of visual features for testing causal hypotheses about selectivity in the brain This capability is demonstrated through three types of permutations: Removal, Addition, and Spatial Manipulation For face stimuli, setting prototype activations to zero removes faces selectively from the scene while other objects remain largely intact Increasing the activation of face prototypes induces the appearance of face-like features in previously non-face stimuli Spatial manipulation allows shifting the locations of faces without significantly affecting content in unaltered regions Close-loop experiments show that targeted permutations significantly increase predicted responses in corresponding brain ROIs compared to random permutations Performance and Robustness The prototype representations consistently improve prediction across all three visual backbones and all eight ROIs With DINOv3-ViT, the mean correlation across ROIs increases from 0.338 to 0.394 The prototype representation captures substantial neural response information through a simple linear mapping and improves prediction over the original backbone features across different models The method is relatively robust to the selection of the number of prototypes P and sparsity constraint k, suggesting that increasing P or loosening k leads to only a minor drop in performance Broader Implications This work demonstrates how interpretable representation learning can turn foundation models from black-box predictors into structured tools for probing biological intelligence The framework unifies analyses that are typically studied separately, including neural encoding, categorical selectivity, cortical organization, spatial attribution, and controlled stimulus manipulation The results show how specific concepts map onto cortical responses by visualizing the weights of the linear projection matrix W This may help generate hypotheses about stimulus–response relationships in specific brain regions and guide the design of targeted neuroscience experiments The prototypes generalize well across subjects, with substantial overlap observed for face- and body-selective regions Limitations The authors note that the method may be less effective for interpreting synthetic or atypical stimuli because vision foundation models are largely trained with naturalistic images Additionally, the prototypes do not necessarily correspond to biological prototypes implemented by the brain The PCA control baseline fails to characterize selectivity across visual regions because PCA features do not exhibit clear category-selective patterns The method is evaluated in established brain areas, suggesting room for improvement when applied to less understood regions Experimental Details The study uses the MSCOCO dataset for learning prototypes and the processed NSD dataset for fMRI data The fMRI encoding model is trained on a subject basis, with different training sets created for each subject by splitting the official training set into 80%/10%/10% The diffusion model used in causal intervention analysis is trained from scratch using only visual data, specifically the JiT diffusion model The protocol involves training the encoding model on a subject basis and training the prototype mapping to predict responses for multiple ROIs This ensures that the analysis is less prone to potential biases compared to models pretrained on language data The computational resources utilized include two RTX 4090 GPUs for the main model and one H100 and two RTX 6000s for the diffusion model References

1: Talia Konkle and George A.

Ines: Alvarez.

Marcus: A self-supervised domain-general learning framework for human ventral stream representation.

Ines: Nature Communications, 13(1):491, 2022

2: Aria Y.

Title and authors: Ines: We're starting with this paper called "Prototypes of the Mind: A Unified Framework for Probing the Visual Brain". The title itself suggests they're building a unified way to probe visual representations in the human brain.

Marcus: It’s about using these visual prototypes, which are learned from foundation models like DINOv3, to create an interpretable bridge between those models and fMRI data.

Yuki: I wonder what that means practically for us as population geneticists; can this framework help us understand how visual concepts map onto the brain across different people?

Ines: It suggests they are using a causal-attention encoder in stage one to factorize the features from the foundation model into a sparse combination of learnable prototypes, which captures both distinct semantics and spatial structure.

Marcus: Then in stage two, they train a linear mapping between those prototype activations and the fMRI responses using a learned spatial attention mechanism and just a single linear layer.

Yuki: So it’s connecting the visual concepts directly to voxel responses through this learned mapping, which sounds like a way to visualize cortical organization.

Ines: It's about making the model transparent at both levels, showing both what drives neural activity and where those driving concepts are located in the image. This is a big shift from just looking at one side of the problem.

The paper's summary: Marcus: So to summarize this framework, they factorize rich features from vision foundation models, like DINOv3, into spatially grounded prototype units that capture recurring visual patterns in natural images. These activations are sparse and keep the spatial layout intact.

Ines: And then they use a causal-attention decoder in stage one to learn these prototypes, which decompose the foundation model features into sparse components that represent distinct semantics.

Yuki: The key takeaway here is that this results in a prototype-decomposed representation, which allows each brain voxel to be associated with only a small set of visual concepts using a linear projection.

Marcus: Stage two then takes those spatially distributed prototype activations and learns a lightweight mapping to fMRI responses, resulting in a weight matrix where each row encodes the sensitivity of that specific voxel to every one of the learned prototypes.

Ines: They show that these learned prototypes actually correspond to interpretable visual concepts, like "airplanes" or "faces," rather than just random directions in feature space.

Yuki: This is significant because it moves us from abstract feature vectors to something we can actually name and map onto the visual world. It gives structure to the representation itself.

The paper's improvements: Ines: They highlight a few key improvements over previous methods. First, they show that prototypes corresponding to specific concepts, like faces, preferentially contribute to the corresponding regions of interest in brain analysis.

Marcus: They also reveal that a face-selective prototype produces elevated weights concentrated in face-selective cortex, specifically the occipital face area and nearby ventral regions. That gives us a concrete map of where those concepts are being processed.

Yuki: I'm interested in the causal probing capabilities they show, because it means we can manipulate these prototypes to test hypotheses about selectivity in the brain directly.

Ines: They demonstrate this through three types of permutations: removal, addition, and spatial manipulation. For example, setting prototype activations to zero removes faces selectively from the scene while other objects stay largely intact.

Marcus: And they found that targeted permutations significantly increase predicted responses in corresponding brain regions compared to random ones. That’s a strong signal for causal inference in this context.

Yuki: If we can manipulate the stimulus at the prototype level and see a predictable change in brain response, that gives us a tool to test hypotheses about how specific visual concepts drive neural activity across different subjects.

Conclusion: Ines: So wrapping up, this paper on "Prototypes of the Mind: A Unified Framework for Probing the Visual Brain" shows how to unify vision foundation models with brain activity using a sparse, interpretable prototype representation.

Marcus: The main implication is that we can now identify which visual concepts drive neural responses and where those concepts appear in the image, while also characterizing voxel selectivity through that learned weight matrix.

Yuki: For me, it’s about having a structured way to link stimulus features to cortical organization, giving us tools to test hypotheses about how specific visual concepts map onto brain regions across different people.

Ines: It turns foundation models from black-box predictors into tools for studying biological intelligence by making the representations transparent at both the concept and voxel level.

Marcus: It’s a framework that supports analysis, stimulus reconstruction, and controlled manipulation of visual content because of that strong generative property in the representation space.

Yuki: I think it’s a step toward understanding how specific concepts map onto cortical responses by visualizing those linear projection weights W, which could guide experimental design for targeted neuroscience.

Shi Chen, Seoyoung Ahn, Doris Tsao

Department of Neuroscience, University of California Berkeley · College of AI Convergence, Hankuk University of Foreign Studies

q-bio.NC, cs.AI

Submitted: 2026-11-11

Updated: 2026-11-11

Comments: NeurIPS 2026

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: Deep learning has driven substantial progress in modeling fMRI responses, yet most existing approaches are optimized for a single objective—such as neural encoding, stimulus decoding…

Key concepts

Visual Prototypes
These are sparse, learned representations of distinct visual concepts (e.g., 'zebra', 'building'). Instead of using raw image data, the model decomposes any visual scene into a small set of these meaningful concepts. This makes the representation transparent because each prototype corresponds to a recognizable visual idea.
Linear Readout
This is the simple mathematical step that connects the learned visual prototypes to actual brain activity in voxels. It acts like a filter, determining how much each specific visual concept (prototype) influences the response of every single brain location (voxel). This mapping is linear and easy to interpret.
Causal Probing
This capability lets researchers test hypotheses about cause and effect in the brain. By selectively turning on or off a specific prototype, like 'faces,' scientists can see exactly how much that concept contributes to the predicted brain response in a particular area, allowing for controlled manipulation of visual stimuli.

Terminology

Summary

Deep learning has driven substantial progress in modeling fMRI responses, yet most existing approaches are optimized for a single objective—such as neural encoding, stimulus decoding, or post hoc interpretation—making it difficult to obtain a coherent account of neural representations Our approach replaces black-box features with a sparse, interpretable representation composed of visual prototypes, and links these prototypes to voxel responses through a linear readout This design makes the model transparent at both levels: prototype activations expose the visual concepts present in the stimulus, while the learned linear weights characterize each voxel’s selectivity Together, these results establish prototype-based representations as a unified and interpretable foundation for studying visual representations in the human brain

Framework Architecture

The method introduces a two-stage framework that connects visual perception with neural activity through interpretable prototypes Stage 1 involves learning a dictionary of visual prototypes that factorize foundation model features into sparse combinations, capturing distinct semantics and spatial structure This stage uses a causal encoder with spatial downsampling to reduce feature map resolution and then factors the encoded features H as weighted combinations of these prototypes The resulting prototype-decomposed representation Hˆ = AP provides a sparse, interpretable decomposition of the visual scene in terms of prototypes and their spatial distribution Stage 2 learns a lightweight mapping from spatially distributed prototype activations to fMRI voxel responses using a learned spatial attention mechanism and a single linear layer This yields an interpretable weight matrix W where each row encodes the sensitivity of voxel v to each of the P prototypes

Prototype Interpretation and Selectivity

The learned prototypes are shown to correspond to interpretable visual concepts rather than arbitrary directions in feature space These prototypes capture diverse visual concepts, such as “airplanes”, “buildings”, “faces”, and “zebra” The linear mapping allows each voxel to be associated with a small set of visual concepts using a linear projection Analysis of prototype profiles for category-selective regions, such as face-selective areas (e.g., Occipital Face Area), shows that prototypes corresponding to faces preferentially contribute to the corresponding ROIs Furthermore, the analysis reveals that a face-selective prototype produces elevated weights concentrated in face-selective cortex, particularly the occipital face area (OFA) and nearby ventral regions

Causal Probing Capabilities

A key advantage is that individual prototypes can be selectively manipulated, enabling targeted manipulation of visual features for testing causal hypotheses about selectivity in the brain This capability is demonstrated through three types of permutations: Removal, Addition, and Spatial Manipulation For face stimuli, setting prototype activations to zero removes faces selectively from the scene while other objects remain largely intact Increasing the activation of face prototypes induces the appearance of face-like features in previously non-face stimuli Spatial manipulation allows shifting the locations of faces without significantly affecting content in unaltered regions Close-loop experiments show that targeted permutations significantly increase predicted responses in corresponding brain ROIs compared to random permutations

Performance and Robustness

The prototype representations consistently improve prediction across all three visual backbones and all eight ROIs With DINOv3-ViT, the mean correlation across ROIs increases from 0.338 to 0.394 The prototype representation captures substantial neural response information through a simple linear mapping and improves prediction over the original backbone features across different models The method is relatively robust to the selection of the number of prototypes P and sparsity constraint k, suggesting that increasing P or loosening k leads to only a minor drop in performance

Broader Implications

This work demonstrates how interpretable representation learning can turn foundation models from black-box predictors into structured tools for probing biological intelligence The framework unifies analyses that are typically studied separately, including neural encoding, categorical selectivity, cortical organization, spatial attribution, and controlled stimulus manipulation The results show how specific concepts map onto cortical responses by visualizing the weights of the linear projection matrix W This may help generate hypotheses about stimulus–response relationships in specific brain regions and guide the design of targeted neuroscience experiments The prototypes generalize well across subjects, with substantial overlap observed for face- and body-selective regions

Limitations

The authors note that the method may be less effective for interpreting synthetic or atypical stimuli because vision foundation models are largely trained with naturalistic images Additionally, the prototypes do not necessarily correspond to biological prototypes implemented by the brain The PCA control baseline fails to characterize selectivity across visual regions because PCA features do not exhibit clear category-selective patterns The method is evaluated in established brain areas, suggesting room for improvement when applied to less understood regions

Experimental Details

The study uses the MSCOCO dataset for learning prototypes and the processed NSD dataset for fMRI data The fMRI encoding model is trained on a subject basis, with different training sets created for each subject by splitting the official training set into 80%/10%/10% The diffusion model used in causal intervention analysis is trained from scratch using only visual data, specifically the JiT diffusion model The protocol involves training the encoding model on a subject basis and training the prototype mapping to predict responses for multiple ROIs This ensures that the analysis is less prone to potential biases compared to models pretrained on language data The computational resources utilized include two RTX 4090 GPUs for the main model and one H100 and two RTX 6000s for the diffusion model

References

[1] Talia Konkle and George A. Alvarez. A self-supervised domain-general learning framework for human ventral stream representation. p. 9

Where the paper says it

References [1] Talia Konkle and George A. A self-supervised domain-general learning framework for human ventral stream representation.

Nature Communications, 13(1):491, 2022

[2] Aria Y. Wang, Kendrick Kay, Thomas Naselaris, Michael J. p. 9

Where the paper says it

[2] Aria Y. Wang, Kendrick Kay, Thomas Naselaris, Michael J.

Tarr, and Leila Wehbe. p. 9
Where the paper says it

Wang, Kendrick Kay, Thomas Naselaris, Michael J. Tarr, and Leila Wehbe.

Better models of human high-level visual cortex emerge from natural language supervision with a large and diverse dataset. p. 9
Where the paper says it

Tarr, and Leila Wehbe. Better models of human high-level visual cortex emerge from natural language supervision with a large and diverse dataset.

Nature Machine Intelligence, 5(12):1415–1426, 2023

[3] Huzheng Yang, James Gee, and Jianbo Shi. Brain decodes deep nets. p. 9

Where the paper says it

[3] Huzheng Yang, James Gee, and Jianbo Shi. Brain decodes deep nets.

In CVPR, pages 23030–23040, 2024

[4] Hossein Adeli, Minni Sun, and Nikolaus Kriegeskorte. Transformer brain encoders explain human high-level visual responses. p. 9

Where the paper says it

[4] Hossein Adeli, Minni Sun, and Nikolaus Kriegeskorte. Transformer brain encoders explain human high-level visual responses.

NeurIPS, 2025

[5] Colin Conwell, Jacob S. Prince, Kendrick N. p. 9

Where the paper says it

[5] Colin Conwell, Jacob S. Prince, Kendrick N.

Kay, George A. p. 9
Where the paper says it

Prince, Kendrick N. Kay, George A.

Alvarez, and Talia Konkle. p. 9
Where the paper says it

Kay, George A. Alvarez, and Talia Konkle.

A large-scale examination of inductive biases shaping high-level visual representation in brains and machines. p. 9
Where the paper says it

Alvarez, and Talia Konkle. A large-scale examination of inductive biases shaping high-level visual representation in brains and machines.

Nature Communications, 15(1):9383, 2024

[6] Andrew Luo, Margaret Marie Henderson, Leila Wehbe, and Michael J. p. 9

Where the paper says it

Nature Communications, 15(1):9383, 2024. [6] Andrew Luo, Margaret Marie Henderson, Leila Wehbe, and Michael J.

Tarr. Brain diffusion for visual exploration: Cortical discovery using large scale generative models. p. 9
Where the paper says it

[6] Andrew Luo, Margaret Marie Henderson, Leila Wehbe, and Michael J. Brain diffusion for visual exploration: Cortical discovery using large scale generative models.

In NeurIPS, 2023

[7] Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. p. 9

Where the paper says it

[7] Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity.

In CVPR, pages 14453–14463, 2023

[8] Pinyuan Feng, Hossein Adeli, Wenxuan Guo, Fan L. Cheng, Ethan Hwang, and Nikolaus Kriegeskorte. p. 9

Where the paper says it

[8] Pinyuan Feng, Hossein Adeli, Wenxuan Guo, Fan L. Cheng, Ethan Hwang, and Nikolaus Kriegeskorte.

Towards interpretable visual decoding with attention to brain representations. p. 9
Where the paper says it

Cheng, Ethan Hwang, and Nikolaus Kriegeskorte. Towards interpretable visual decoding with attention to brain representations.

In ICLR, 2026

[9] Mu Nan, Muquan Yu, Weijian Mai, Jacob S. Prince, Hossein Adeli, Rui Zhang, Jiahang Cao, Benjamin Becker, John A. p. 9

Where the paper says it

In CVPR, pages 23030–23040, 2024. [4] Hossein Adeli, Minni Sun, and Nikolaus Kriegeskorte.

Pyles, Margaret M. p. 9
Where the paper says it

Nature Communications, 15(1):9383, 2024. [6] Andrew Luo, Margaret Marie Henderson, Leila Wehbe, and Michael J.

Henderson, Chunfeng Song, Nikolaus Kriegeskorte, Michael J. p. 9
Where the paper says it

Nature Communications, 15(1):9383, 2024. [6] Andrew Luo, Margaret Marie Henderson, Leila Wehbe, and Michael J.

Tarr, Xiaoqing Hu, and Andrew F. p. 9
Where the paper says it

Wang, Kendrick Kay, Thomas Naselaris, Michael J. Tarr, and Leila Wehbe.

Luo. Meta-learning in-context enables training-free cross subject brain decoding. p. 0
Where the paper says it

Prototypes of the Mind: A Unified Framework for Probing the Visual Brain Shi Chen1 Seoyoung Ahn2 Doris Tsao1 1Department of Neuroscience, UC Berkeley 2College of AI Convergence, Hankuk University of Foreign Studies Abstract Deep learning has driven substantial progress in modeling fMRI responses, yet most existing approaches are optimized for a single objective—such as neural encoding, stimulus decoding, or post hoc interpretation—making it difficult to obtain a coherent account of neural representations. We introduce a unified framework that connects vision foundation models to brain activity while supporting neural analysis from multiple complementary perspectives.

In CVPR, 2026

[10] Paul Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, ethan cohen, Aidan Dempster, Nathalie Verlinde, Elad Yundler, David Weisberg, Kenneth Norman, and Tanishq Abraham. Reconstructing the mind's eye: fmri-to-image with contrastive learning and diffusion priors. In NeurIPS 36

[11] Roman Beliy, Amit Zalcher, Jonathan Kogman, Navve Wasserman, and Michal Irani. Brain-it: Image reconstruction from fmri via brain-interaction transformer. p. 9

Where the paper says it

[7] Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity.

In ICLR, 2026

[12] Alicia Zeng and Jack Gallant. Disentangling superpositions: Interpretable brain encoding model with sparse concept atoms. p. 2

Where the paper says it

(2) In stage-2, we train a linear mapping between the activation of prototypes and fMRI responses, enabling brain understanding from complementary perspectives. decomposes foundation model features into sparse, interpretable components.

NeurIPS, 2025

[13] Navve Wasserman, Matias Cosarinsky, Yuval Golbari, Aude Oliva, Antonio Torralba, Tamar Rott Shaham, and Michal Irani. Brainexplore: Large-scale discovery of interpretable visual representations in the human brain. p. 9

Where the paper says it

[6] Andrew Luo, Margaret Marie Henderson, Leila Wehbe, and Michael J. Brain diffusion for visual exploration: Cortical discovery using large scale generative models.

arXiv, 2025

[14] Gabriel Sarch, Michael Tarr, Katerina Fragkiadaki, and Leila Wehbe. Brain dissection: fmri-trained networks reveal spatial selectivity in the processing of natural images. p. 0

Where the paper says it

Moving beyond black-box features learned by deep networks, several recent studies develop interpretable representations that reveal categorical [12, 13] and spatial [14, 15] selectivity. Collectively, these approaches have produced valuable tools for probing the functional organization of the brain.

In NeurIPS 36

[15] Andrew F. Luo, Jacob Yeung, Rushikesh Zawar, Shaurya Dewan, Margaret M. p. 9

Where the paper says it

Nature Communications, 15(1):9383, 2024. [6] Andrew Luo, Margaret Marie Henderson, Leila Wehbe, and Michael J.

Henderson, Leila Wehbe, and Michael J. Tarr. Brain mapping with dense features: Grounding cortical semantic selectivity in natural images with vision transformers. p. 9
Where the paper says it

Our central contribution is to transform dense features from vision foundation models into sparse, spatially grounded prototypes that capture coherent visual concepts and can be linked to fMRI responses through a transparent linear mapping. This design turns an opaque visual representation into an interpretable interface between images and cortex: prototypes describe what concepts are present and where they occur, while voxel weights reveal which concepts drive each brain region.

In ICLR 2025

[16] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jegou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. p. 2

Where the paper says it

decomposes foundation model features into sparse, interpretable components. Using a self-supervised learning paradigm that emphasizes contextual relationships among visual regions, the model supports a rich interpretation of both the concepts present in a scene and their spatial organization.

In ICCV 2021

[17] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. p. 2

Where the paper says it

decomposes foundation model features into sparse, interpretable components. Using a self-supervised learning paradigm that emphasizes contextual relationships among visual regions, the model supports a rich interpretation of both the concepts present in a scene and their spatial organization.

In CVPR 2023

[18] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. p. 0

Where the paper says it

Prototypes of the Mind: A Unified Framework for Probing the Visual Brain Shi Chen1 Seoyoung Ahn2 Doris Tsao1 1Department of Neuroscience, UC Berkeley 2College of AI Convergence, Hankuk University of Foreign Studies Abstract Deep learning has driven substantial progress in modeling fMRI responses, yet most existing approaches are optimized for a single objective—such as neural encoding, stimulus decoding, or post hoc interpretation—making it difficult to obtain a coherent account of neural representations. We introduce a unified framework that connects vision foundation models to brain activity while supporting neural analysis from multiple complementary perspectives.

In CVPR 2022

[19] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan Su. This looks like that: Deep learning for interpretable image recognition. p. 0

Where the paper says it

Prototypes of the Mind: A Unified Framework for Probing the Visual Brain Shi Chen1 Seoyoung Ahn2 Doris Tsao1 1Department of Neuroscience, UC Berkeley 2College of AI Convergence, Hankuk University of Foreign Studies Abstract Deep learning has driven substantial progress in modeling fMRI responses, yet most existing approaches are optimized for a single objective—such as neural encoding, stimulus decoding, or post hoc interpretation—making it difficult to obtain a coherent account of neural representations. We introduce a unified framework that connects vision foundation models to brain activity while supporting neural analysis from multiple complementary perspectives.

In NeurIPS 2019

[20] Pang Wei Koh, Thao Nguyen, Yewen Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. p. 5

Where the paper says it

Specifically, we extract prototype activation maps A of images from MSCOCO validation set, and identify the representative images with strong activation (averaged across all patches) for each prototype (see Appendix P for details). To enhance consistency among the final representatives, we also include an optional step that computes pairwise cosine similarities between CLIP [35] embeddings of these candidates and applies spectral clustering [36] to select 20 images that are both highly activated and representative of a shared visual concept (see Appendix E for comparison between visualization with and without clustering).

In ICML 2020

[21] Oriane Siméoni et al. Dinov3. In arXiv 2025

[23] Thomas Naselaris, Kendrick N. Kay, Shinji Nishimoto, and Jack L. p. 9

Where the paper says it

In NeurIPS, 2023. [7] Yu Takagi and Shinji Nishimoto.

Gallant. Encoding and decoding in fmri. p. 0
Where the paper says it

Prototypes of the Mind: A Unified Framework for Probing the Visual Brain Shi Chen1 Seoyoung Ahn2 Doris Tsao1 1Department of Neuroscience, UC Berkeley 2College of AI Convergence, Hankuk University of Foreign Studies Abstract Deep learning has driven substantial progress in modeling fMRI responses, yet most existing approaches are optimized for a single objective—such as neural encoding, stimulus decoding, or post hoc interpretation—making it difficult to obtain a coherent account of neural representations. We introduce a unified framework that connects vision foundation models to brain activity while supporting neural analysis from multiple complementary perspectives.

NeuroImage 56(2):400–410, 2011

[24] Zijin Gu et al. Neurogen: Activation optimized image synthesis for discovery neuroscience. p. 0

Where the paper says it

Prototypes of the Mind: A Unified Framework for Probing the Visual Brain Shi Chen1 Seoyoung Ahn2 Doris Tsao1 1Department of Neuroscience, UC Berkeley 2College of AI Convergence, Hankuk University of Foreign Studies Abstract Deep learning has driven substantial progress in modeling fMRI responses, yet most existing approaches are optimized for a single objective—such as neural encoding, stimulus decoding, or post hoc interpretation—making it difficult to obtain a coherent account of neural representations. We introduce a unified framework that connects vision foundation models to brain activity while supporting neural analysis from multiple complementary perspectives.

NeuroImage 247:118812, 2022

[25] Carlos R. Ponce et al. Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences. p. 1

Where the paper says it

Early studies used optimizationbased approaches to generate stimuli that maximally activate neurons or brain regions [25, 26]. More recent methods combine fMRI decoding with powerful generative models such as diffusion models [7, 10, 11], enabling reconstruction of naturalistic images from measured neural activity.

Cell, 177(4):999–1009.e10, 2019

[26] N. Apurva Ratan Murty et al. Computational models of category-selective brain regions enable high-throughput tests of selectivity. p. 0

Where the paper says it

Advances in deep learning, together with the increasing availability of large-scale neural datasets, have transformed Functional Magnetic Resonance Imaging (fMRI) analysis by enabling increasingly effective computational models of brain responses. One major line of work develops encoding models [1, 2, 3, 4, 5] that seek to predict fMRI responses from visual stimuli with high accuracy.

Nature Communications, 12(1):5540, 2021

[27] A. T. Gifford et al. The algonauts project 2023 challenge: How the human brain makes sense of natural scenes. p. 0

Where the paper says it

1 Introduction Understanding how the human brain represents complex visual scenes remains a fundamental challenge at the intersection of neuroscience and computer vision. Neural measurements are sparse and noisy, while natural scenes are incredibly rich and complex.

In arXiv 2023

[28] Lijun Yu et al. Language model beats diffusion - tokenizer is key to visual generation. p. 7

Where the paper says it

7 Prototype Manipulations Enable Controlled Stimulus Interventions A key advantage of our method is that individual prototypes can be selectively manipulated, enabling targeted manipulation of visual features for testing causal hypotheses about selectivity in the brain. We demonstrate this unique capability by training a diffusion model (see Appendix O) that reconstructs the original image based on its prototype activation map (i.

In ICLR 2024

[29] Ashish Vaswani et al. Attention is all you need. p. 3

Where the paper says it

Specifically, we rasterize the visual patches, and task our model to predict features of the next patches based on preceding ones: Lproto = 1 M − 1 M X−1 m=1 ∥yˆm − ym+1∥ 2 , (7) where yˆm is the prediction at position m and ym+1 is the target at position m+1. This objective encourages the model to capture the spatial structure without the need of additional annotations.

In NeurIPS, page 6000–6010, 2017

[30] Alec Radford et al. Improving language understanding by generative pre-training. p. 3

Where the paper says it

The reconstructed representation Hˆ is passed through a decoder composed of a single causal self-attention block followed by a linear readout head: Yˆ = gread fdec(Hˆ ) ∈ R M×D′ , (6) Note that the readout projects each decoded patch into a higher-dimensional space (D′=4D=3072), to accommodate the 2×2 spatial downsampling within the encoder. To preserve the discriminative power of the learned prototypes, we adopt an autoregressive training paradigm used in large language [30] and multimodal [32] models.

OpenAI, 2018

[31] Robert Huben et al. Sparse autoencoders find highly interpretable features in language models. p. 3

Where the paper says it

For each encoded patch hm ∈ R D, the prototype weights sm are computed from its cosine similarities with all prototypes: sm = hmP ⊤ ∈ R P . (3) With the overarching goal of developing an interpretable representation, we draw inspirations from sparse autoencoder for analyzing language models [31] and further impose sparsity constraints on prototype selection.

In ICLR 2024

[32] Sharut Gupta et al. Better together: Leveraging unpaired multimodal data for stronger unimodal models. p. 9

Where the paper says it

Tarr, and Leila Wehbe. Better models of human high-level visual cortex emerge from natural language supervision with a large and diverse dataset.

In ICLR 2026

[33] Tsung-Yi Lin et al. Microsoft COCO: Common objects in context. p. 5

Where the paper says it

As shown in Figure 2, individual prototypes consistently capture coherent visual concepts. Some prototypes specialize in inanimate objects (“airplanes”, “buildings”), while others correspond to animate entities (“faces”, “zebra”) or activities (“tennis”) or abstract scene-level concepts (“crowds”).

In ECCV 2014

[34] Emily J. Allen et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. p. 9

Where the paper says it

Better models of human high-level visual cortex emerge from natural language supervision with a large and diverse dataset. Nature Machine Intelligence, 5(12):1415–1426, 2023.

Nature Neuroscience, 25(1):116–126, 2022

[35] Alec Radford et al. Learning transferable visual models from natural language supervision. p. 9

Where the paper says it

Tarr, and Leila Wehbe. Better models of human high-level visual cortex emerge from natural language supervision with a large and diverse dataset.

In ICML volume 139, pages 8748–8763, 2021

[36] Andrew Y. Ng, Michael I. p. 9

Where the paper says it

[2] Aria Y. Wang, Kendrick Kay, Thomas Naselaris, Michael J.

Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. p. 5
Where the paper says it

Specifically, we extract prototype activation maps A of images from MSCOCO validation set, and identify the representative images with strong activation (averaged across all patches) for each prototype (see Appendix P for details). To enhance consistency among the final representatives, we also include an optional step that computes pairwise cosine similarities between CLIP [35] embeddings of these candidates and applies spectral clustering [36] to select 20 images that are both highly activated and representative of a shared visual concept (see Appendix E for comparison between visualization with and without clustering).

In NeurIPS volume 14, pages 849–856, 2002

[37] Alexander G. Huth et al. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. p. 1

Where the paper says it

as an interpretable bridge between vision foundation models [16, 17, 18] and the brain, grounding neural analysis in the semantic structure learned from large-scale visual data. At the same time, by linking brain responses to a representational space with strong generative properties, it can support more advanced applications that go beyond passive analysis, including stimulus reconstruction and controlled manipulation of visual content.

Neuron, 76(6):1210–1224, 2012

[38] Robin Rombach et al. High-resolution image synthesis with latent diffusion models. p. 9

Where the paper says it

[7] Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity.

In CVPR 2022

[39] David Mayo et al. Brainbits: How much of the brain are generative reconstruction methods using? p. 1

Where the paper says it

Early studies used optimizationbased approaches to generate stimuli that maximally activate neurons or brain regions [25, 26]. More recent methods combine fMRI decoding with powerful generative models such as diffusion models [7, 10, 11], enabling reconstruction of naturalistic images from measured neural activity.

In NeurIPS volume 37, pages 54396–54420, 2024

[40] Harold Hotelling. Analysis of a complex of statistical variables into principal components. p. 1

Where the paper says it

Realizing this goal requires a representation that is semantically interpretable, structurally organized, and flexible enough to support both analysis and intervention. Guided by the intuition that complex visual scenes can be decomposed into distinct visual concepts [19, 20], we factorize rich features from vision foundation models (e.

Journal of Educational Psychology, 24(6):417, 1933

[41] Tianhong Li and Kaiming He. Back to basics: Let denoising generative models denoise. p. 1

Where the paper says it

as an interpretable bridge between vision foundation models [16, 17, 18] and the brain, grounding neural analysis in the semantic structure learned from large-scale visual data. At the same time, by linking brain responses to a representational space with strong generative properties, it can support more advanced applications that go beyond passive analysis, including stimulus reconstruction and controlled manipulation of visual content.

In arXiv 2026

[42] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. p. 1

Where the paper says it

Early studies used optimizationbased approaches to generate stimuli that maximally activate neurons or brain regions [25, 26]. More recent methods combine fMRI decoding with powerful generative models such as diffusion models [7, 10, 11], enabling reconstruction of naturalistic images from measured neural activity.

In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021

[43] Jia Deng et al. Imagenet: A large-scale hierarchical image database. p. 1

Where the paper says it

Recent studies increasingly leverage features from deep networks or multimodal foundation models [1, 2, 3, 4, 5], which capture hierarchical visual representations aligned with cortical organization. While these DNN-based approaches are effective for predicting neural activity and characterizing voxel selectivity, they typically operate as black-box predictors, offering limited interpretability about which specific image properties drive the responses.

In CVPR 2009

[44] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. p. 2

Where the paper says it

, DINOv3, N = 16 × 16 is the number of spatial patches and D = 768 is the feature dimension), we first apply a learnable layer [28] to reduce the feature map resolution from N = 16 × 16 to M = 8 × 8. This step both lowers computational cost and facilitates model optimization: Z = SpatialDown(X) ∈ R M×D.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. A unified framework connects vision foundation models to brain activity by replacing black-box features with a sparse, interpretable representation composed of visual prototypes, allowing for analysis from multiple complementary perspectives such as prototype activations expose the visual concepts present in the stimulus and the learned linear weights characterize each voxel’s selectivity.

  2. The improved system can identify which visual concepts drive neural responses and where they appear in the image, reveal cortical organization through voxel tuning in prototype space, and perform controlled stimulus synthesis to probe cortical selectivity through targeted interventions.

  3. The system supports causal probing by enabling targeted stimulus interventions on interpretable prototypes (e.g., manipulating face and body prototypes), allowing researchers to test hypotheses about the relationship between neural activity and visual concepts.

  4. The system allows for a direct characterization of voxel selectivity via the learned weight matrix W, where each row wv in R P of W encodes the sensitivity of voxel v to each of the P prototypes, providing a direct and interpretable characterization of voxel selectivity.

  5. The system can reveal spatial selectivity by deriving spatial attention maps from the prototype activations associated with each ROI and showing that it reveals where those features are located within the scene by highlighting which prototypes contribute most strongly to predicted activity in specific brain areas.

Abstract

Deep learning has driven substantial progress in modeling fMRI responses, yet most existing approaches are optimized for a single objective---such as neural encoding, stimulus decoding, or post hoc interpretation---making it difficult to obtain a coherent account of neural representations. We introduce a unified framework that connects vision foundation models to brain activity while supporting neural analysis from multiple complementary perspectives. Our approach replaces black-box features with a sparse, interpretable representation composed of visual prototypes, and links these prototypes to voxel responses through a linear readout. This design makes the model transparent at both levels: prototype activations expose the visual concepts present in the stimulus, while the learned linear weights characterize each voxel's selectivity. Using this interface, we identify which visual concepts drive neural responses and where they appear in the image, reveal cortical organization through voxel tuning in prototype space, and perform controlled stimulus synthesis to probe cortical selectivity through targeted interventions. Together, these results establish prototype-based representations as a unified and interpretable foundation for studying visual representations in the human brain.

Related papers