Prototypes of the Mind: A Unified Framework for Probing the Visual Brain

summary

The gist

Deep learning has driven substantial progress in modeling fMRI responses, yet most existing approaches are optimized for a single objective—such as neural encoding, stimulus decoding…

In short

The approach replaces complex neural features with sparse visual prototypes—like 'faces' or 'airplanes'—to create an interpretable link between images and brain activity. This allows researchers to see which specific visual concepts drive brain responses, map these concepts onto cortical regions, and perform targeted experiments by manipulating these prototypes.

Key concepts

Visual Prototypes
These are sparse, learned representations of distinct visual concepts (e.g., 'zebra', 'building'). Instead of using raw image data, the model decomposes any visual scene into a small set of these meaningful concepts. This makes the representation transparent because each prototype corresponds to a recognizable visual idea.
Linear Readout
This is the simple mathematical step that connects the learned visual prototypes to actual brain activity in voxels. It acts like a filter, determining how much each specific visual concept (prototype) influences the response of every single brain location (voxel). This mapping is linear and easy to interpret.
Causal Probing
This capability lets researchers test hypotheses about cause and effect in the brain. By selectively turning on or off a specific prototype, like 'faces,' scientists can see exactly how much that concept contributes to the predicted brain response in a particular area, allowing for controlled manipulation of visual stimuli.

Terminology used across episodes

This episode discusses

The paper

Prototypes of the Mind: A Unified Framework for Probing the Visual Brain · Read on arXiv

Shi Chen, Seoyoung Ahn, Doris Tsao

Department of Neuroscience, University of California Berkeley · College of AI Convergence, Hankuk University of Foreign Studies

Deep learning has driven substantial progress in modeling fMRI responses, yet most existing approaches are optimized for a single objective---such as neural encoding, stimulus decoding, or post hoc interpretation---making it difficult to obtain a coherent account of neural representations. We introduce a unified framework that connects vision foundation models to brain activity while supporting neural analysis from multiple complementary perspectives. Our approach replaces black-box features with a sparse, interpretable representation composed of visual prototypes, and links these prototypes to voxel responses through a linear readout. This design makes the model transparent at both levels: prototype activations expose the visual concepts present in the stimulus, while the learned linear weights characterize each voxel's selectivity. Using this interface, we identify which visual concepts drive neural responses and where they appear in the image, reveal cortical organization through voxel tuning in prototype space, and perform controlled stimulus synthesis to probe cortical selectivity through targeted interventions. Together, these results establish prototype-based representations as a unified and interpretable foundation for studying visual representations in the human brain.

Transcript

Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.

Ines: I'm Ines.

Marcus: And I'm Marcus, here with Yuki.

Ines: We're looking at the paper: "Prototypes of the Mind".

Marcus: Here's the gist of the paper. Deep learning has driven substantial progress in modeling fMRI responses, yet most existing approaches are optimized for a single objective—such as neural encoding, stimulus decoding, or post hoc interpretation—making it difficult to obtain a coherent account of neural representations Our approach replaces black-box features with a sparse, interpretable representation composed of visual prototypes, and links these prototypes to voxel responses through a linear readout This design makes the model transparent at both levels: prototype activations expose the visual concepts present in the stimulus, while the learned linear weights characterize each voxel’s selectivity Together, these results establish prototype-based representations as a unified and interpretable foundation for studying visual representations in the human brain Framework Architecture The method introduces a two-stage framework that connects visual perception with neural activity through interpretable prototypes Stage 1 involves learning a dictionary of visual prototypes that factorize foundation model features into sparse combinations, capturing distinct semantics and spatial structure This stage uses a causal encoder with spatial downsampling to reduce feature map resolution and then factors the encoded features H as weighted combinations of these prototypes The resulting prototype-decomposed representation Hˆ = AP provides a sparse, interpretable decomposition of the visual scene in terms of prototypes and their spatial distribution Stage 2 learns a lightweight mapping from spatially distributed prototype activations to fMRI voxel responses using a learned spatial attention mechanism and a single linear layer This yields an interpretable weight matrix W where each row encodes the sensitivity of voxel v to each of the P prototypes Prototype Interpretation and Selectivity The learned prototypes are shown to correspond to interpretable visual concepts rather than arbitrary directions in feature space These prototypes capture diverse visual concepts, such as “airplanes”, “buildings”, “faces”, and “zebra” The linear mapping allows each voxel to be associated with a small set of visual concepts using a linear projection Analysis of prototype profiles for category-selective regions, such as face-selective areas (e.g., Occipital Face Area), shows that prototypes corresponding to faces preferentially contribute to the corresponding ROIs Furthermore, the analysis reveals that a face-selective prototype produces elevated weights concentrated in face-selective cortex, particularly the occipital face area (OFA) and nearby ventral regions Causal Probing Capabilities A key advantage is that individual prototypes can be selectively manipulated, enabling targeted manipulation of visual features for testing causal hypotheses about selectivity in the brain This capability is demonstrated through three types of permutations: Removal, Addition, and Spatial Manipulation For face stimuli, setting prototype activations to zero removes faces selectively from the scene while other objects remain largely intact Increasing the activation of face prototypes induces the appearance of face-like features in previously non-face stimuli Spatial manipulation allows shifting the locations of faces without significantly affecting content in unaltered regions Close-loop experiments show that targeted permutations significantly increase predicted responses in corresponding brain ROIs compared to random permutations Performance and Robustness The prototype representations consistently improve prediction across all three visual backbones and all eight ROIs With DINOv3-ViT, the mean correlation across ROIs increases from 0.338 to 0.394 The prototype representation captures substantial neural response information through a simple linear mapping and improves prediction over the original backbone features across different models The method is relatively robust to the selection of the number of prototypes P and sparsity constraint k, suggesting that increasing P or loosening k leads to only a minor drop in performance Broader Implications This work demonstrates how interpretable representation learning can turn foundation models from black-box predictors into structured tools for probing biological intelligence The framework unifies analyses that are typically studied separately, including neural encoding, categorical selectivity, cortical organization, spatial attribution, and controlled stimulus manipulation The results show how specific concepts map onto cortical responses by visualizing the weights of the linear projection matrix W This may help generate hypotheses about stimulus–response relationships in specific brain regions and guide the design of targeted neuroscience experiments The prototypes generalize well across subjects, with substantial overlap observed for face- and body-selective regions Limitations The authors note that the method may be less effective for interpreting synthetic or atypical stimuli because vision foundation models are largely trained with naturalistic images Additionally, the prototypes do not necessarily correspond to biological prototypes implemented by the brain The PCA control baseline fails to characterize selectivity across visual regions because PCA features do not exhibit clear category-selective patterns The method is evaluated in established brain areas, suggesting room for improvement when applied to less understood regions Experimental Details The study uses the MSCOCO dataset for learning prototypes and the processed NSD dataset for fMRI data The fMRI encoding model is trained on a subject basis, with different training sets created for each subject by splitting the official training set into 80%/10%/10% The diffusion model used in causal intervention analysis is trained from scratch using only visual data, specifically the JiT diffusion model The protocol involves training the encoding model on a subject basis and training the prototype mapping to predict responses for multiple ROIs This ensures that the analysis is less prone to potential biases compared to models pretrained on language data The computational resources utilized include two RTX 4090 GPUs for the main model and one H100 and two RTX 6000s for the diffusion model References

1: Talia Konkle and George A.

Ines: Alvarez.

Marcus: A self-supervised domain-general learning framework for human ventral stream representation.

Ines: Nature Communications, 13(1):491, 2022

2: Aria Y.

Title and authors: Ines: We're starting with this paper called "Prototypes of the Mind: A Unified Framework for Probing the Visual Brain". The title itself suggests they're building a unified way to probe visual representations in the human brain.

Marcus: It’s about using these visual prototypes, which are learned from foundation models like DINOv3, to create an interpretable bridge between those models and fMRI data.

Yuki: I wonder what that means practically for us as population geneticists; can this framework help us understand how visual concepts map onto the brain across different people?

Ines: It suggests they are using a causal-attention encoder in stage one to factorize the features from the foundation model into a sparse combination of learnable prototypes, which captures both distinct semantics and spatial structure.

Marcus: Then in stage two, they train a linear mapping between those prototype activations and the fMRI responses using a learned spatial attention mechanism and just a single linear layer.

Yuki: So it’s connecting the visual concepts directly to voxel responses through this learned mapping, which sounds like a way to visualize cortical organization.

Ines: It's about making the model transparent at both levels, showing both what drives neural activity and where those driving concepts are located in the image. This is a big shift from just looking at one side of the problem.

The paper's summary: Marcus: So to summarize this framework, they factorize rich features from vision foundation models, like DINOv3, into spatially grounded prototype units that capture recurring visual patterns in natural images. These activations are sparse and keep the spatial layout intact.

Ines: And then they use a causal-attention decoder in stage one to learn these prototypes, which decompose the foundation model features into sparse components that represent distinct semantics.

Yuki: The key takeaway here is that this results in a prototype-decomposed representation, which allows each brain voxel to be associated with only a small set of visual concepts using a linear projection.

Marcus: Stage two then takes those spatially distributed prototype activations and learns a lightweight mapping to fMRI responses, resulting in a weight matrix where each row encodes the sensitivity of that specific voxel to every one of the learned prototypes.

Ines: They show that these learned prototypes actually correspond to interpretable visual concepts, like "airplanes" or "faces," rather than just random directions in feature space.

Yuki: This is significant because it moves us from abstract feature vectors to something we can actually name and map onto the visual world. It gives structure to the representation itself.

The paper's improvements: Ines: They highlight a few key improvements over previous methods. First, they show that prototypes corresponding to specific concepts, like faces, preferentially contribute to the corresponding regions of interest in brain analysis.

Marcus: They also reveal that a face-selective prototype produces elevated weights concentrated in face-selective cortex, specifically the occipital face area and nearby ventral regions. That gives us a concrete map of where those concepts are being processed.

Yuki: I'm interested in the causal probing capabilities they show, because it means we can manipulate these prototypes to test hypotheses about selectivity in the brain directly.

Ines: They demonstrate this through three types of permutations: removal, addition, and spatial manipulation. For example, setting prototype activations to zero removes faces selectively from the scene while other objects stay largely intact.

Marcus: And they found that targeted permutations significantly increase predicted responses in corresponding brain regions compared to random ones. That’s a strong signal for causal inference in this context.

Yuki: If we can manipulate the stimulus at the prototype level and see a predictable change in brain response, that gives us a tool to test hypotheses about how specific visual concepts drive neural activity across different subjects.

Conclusion: Ines: So wrapping up, this paper on "Prototypes of the Mind: A Unified Framework for Probing the Visual Brain" shows how to unify vision foundation models with brain activity using a sparse, interpretable prototype representation.

Marcus: The main implication is that we can now identify which visual concepts drive neural responses and where those concepts appear in the image, while also characterizing voxel selectivity through that learned weight matrix.

Yuki: For me, it’s about having a structured way to link stimulus features to cortical organization, giving us tools to test hypotheses about how specific visual concepts map onto brain regions across different people.

Ines: It turns foundation models from black-box predictors into tools for studying biological intelligence by making the representations transparent at both the concept and voxel level.

Marcus: It’s a framework that supports analysis, stimulus reconstruction, and controlled manipulation of visual content because of that strong generative property in the representation space.

Yuki: I think it’s a step toward understanding how specific concepts map onto cortical responses by visualizing those linear projection weights W, which could guide experimental design for targeted neuroscience.

More episodes

← Home