MindVLM: Neural-Grounded Visual Captioning via Subject-Aware Semantic Evidence Selection
summary
The gist
Translating visual brain activity into natural language offers a powerful window into human cognition and neural semantic representation Most prior brain-captioning approaches project fMRI signals…
In short
MindVLM maps fMRI brain activity into personalized visual tokens and uses a frozen vision-language model (VLM) to generate descriptions of what an individual perceives from an image. It improves caption quality by aligning brain signals with visual tokens using contrastive learning, offering a computationally efficient way to decode subjective visual experiences.
Key concepts
- Neural-Grounded Visual Captioning
- This task aims to create a module that generates text descriptions of an image conditioned on the corresponding fMRI response. It focuses on linking brain activity directly to visual evidence rather than just predicting text, aiming for interpretable and personalized descriptions based on individual perception.
- Subject-Aware Visual Tokens
- These are tokens derived from fMRI signals that represent a specific person's unique neural encoding of a visual stimulus. The model aligns these subject-specific brain tokens with visual tokens extracted by a VLM, allowing the system to capture how each individual processes and attends to visual information.
- Semantic Loss
- This loss function measures how much an object in an image loses its pixel information when cropped. It is used in training to ensure that the neural representations derived from fMRI are semantically grounded in the actual visual content, helping the model focus on perceptually relevant parts of the brain activity.
- Paradigm Efficiency
- MindVLM is computationally efficient because its training only requires aligning neural and visual features through contrastive learning, avoiding complex LLM optimization during training. This allows it to operate offline with minimal GPU memory usage compared to other models.
Terminology used across episodes
This episode discusses
- MindVLM: Neural-Grounded Visual Captioning via Subject-Aware Semantic Evidence Selection · Paper Radio
- Brain Captioning: Decoding human brain activity into images and text
- Improving visual image reconstruction from human brain activity using latent diffusion models via multiple decoded inputs
- LLaMA: Open and Efficient Foundation Language Models
- Exploring The Visual Feature Space for Multimodal Neural Decoding
- Optimal Transport for Brain-Image Alignment: Unveiling Redundancy and Synergy in Neural Information Processing
The paper
MindVLM: Neural-Grounded Visual Captioning via Subject-Aware Semantic Evidence Selection · Read on arXiv
Zixiang Yin, Yu-Ping Wang, Zhengming Ding
Tulane University
Translating visual brain activity into natural language offers a powerful window into human cognition and neural semantic representation. Most prior brain-captioning approaches project fMRI signals into language space using paired supervision. However, the widely used Natural Scenes Dataset (NSD) was built based only on image-fMRI pairs, while existing methods rely on MS-COCO captions as indirect textual supervision. Critically, during fMRI acquisition, participants observe only cropped image regions, whereas the associated MS-COCO captions describe full scenes. This mismatch creates a fundamental gap between neural representations and textual supervision, introducing an inherent semantic loss in NSD-based brain captioning. To mitigate this limitation, we reformulate brain captioning as Neural-Grounded Visual Captioning, where language is generated directly from visually grounded neural evidence rather than text supervision. We propose MindVLM, a framework that maps fMRI signals into subject-aware visual tokens and leverages a frozen vision-language model (VLM) to produce interpretable and personalized descriptions. Instead of point-wise regression, MindVLM focuses on extracting semantically relevant visual cues from neural activity, naturally accommodating partial observability and inter-subject variability. Extensive experiments on NSD show that MindVLM maintains, and often improves, caption quality across multiple VLM backbones, including LLaVA-v1.5, Qwen2.5-VL, and Gemma3, beyond intrinsic model priors. Despite requiring neither text supervision nor an LLM during training and operating under an image-grounded inference paradigm distinct from conventional captioning methods, MindVLM remains competitive with supervised approaches, suggesting that subject-specific neural grounding provides an efficient, scalable, and interpretable path toward faithful brain-to-language generation.
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "MindVLM: Neural-Grounded Visual Captioning via Subject-Aware Semantic Evidence Selection".
Jane: Here's the gist of the paper. Translating visual brain activity into natural language offers a powerful window into human cognition and neural semantic representation Most prior brain-captioning approaches project fMRI signals into language space using paired supervision This work proposes MindVLM, a framework that maps fMRI signals into subject-aware visual tokens and leverages a frozen vision language model (VLM) to produce interpretable and personalized descriptions MindVLM consistently improves caption quality across multiple VLM backbones, including LLaVA-v1.5, Qwen2.5-VL, and Gemma3, beyond intrinsic model priors Problem Formulation The goal of Neural-Grounded Visual Captioning is to design a module M that generates a textual description of a perceived image conditioned on its corresponding fMRI response This task is defined as learning a mapping: M: R H×W×3 × R L → T, (1) The framework is driven by three key insights: (1) Semantic Loss, (2) Personalization, and (3) Visual Grounding Semantic loss is defined as the fraction of objects in an image that lose more than 50% of their total pixels during cropping
1: How the training signal is built The framework constructs a shared latent space in which neural tokens derived from fMRI are aligned with visual tokens During training, a brain encoder is trained to align fMRI responses with the visual token representations extracted by a frozen VLM image encoder This alignment is enforced through a CLIP-style symmetric contrastive learning objective Lalign = 1/2 (Lvox→img + Limg→vox) The final optimization object Lalign is formulated as the average of the two losses Where the gains come from The performance gains stem from identifying which visual tokens are supported by neural activity, which is controlled by the hyperparameter q The analysis shows that both BLEU-3 and SPICE exhibit a general increase with q, confirming that selecting more fMRI-visual correspondences tends to improve decoding quality This behavior aligns with the notion of redundant or weakly informative voxels in visual cortex Furthermore, captions generated from these subject-specific tokens capture unique visual information encoded by each individual, highlighting the well-documented inter-individual variability in high-level visual cortex
4: Subject-Specific Reconstruction The model reconstructs what each person actually perceived by leveraging subject-specific cortical encoding Figure 5 illustrates how the same visual stimulus leads to distinct, subject-specific perceptual interpretations that ultimately shape the generated captions For instance, subj05 retains the largest proportion of tokens (only 61% masked), with preserved regions concentrated around the action and utensil This indicates a higher-fidelity perceptual representation Paradigm Efficiency MindVLM is significantly more computationally efficient than competing models because its training objective involves only neural-visual contrastive alignment, requiring no LLM during optimization MindVLM can operate entirely in an offline setting, where nearly no frozen parameters are required The results highlight a substantial computational advantage, as MindVLM-Online and Offline require only 10% and 1% of the GPU memory used by MindLLM, respectively MindVLM consistently improves over the base VLM across three architectures, and remains competitive with text-supervised brain captioning methods on semantic metrics despite requiring no text supervision These results indicate the promise of neural-guided visual grounding as an efficient, interpretable, and cognitively faithful pathway for personalized decoding Pruning redundant reasoning tokens The authors explicitly state that MindVLM is designed for brain captioning under partial visual observability This paradigm fundamentally departs from conventional fMRI-to-text generation by reframing brain decoding from text regression to semantic evidence grounding The comparison is intended to characterize the semantic quality of neural grounding rather than to establish ranking How the training signal is built The framework first extracts local neural features and local visual features, then aggregates them into global representations to perform batch-level contrastive matching This pooling operation mirrors the population-level coding observed in the visual cortex
21: To enforce a global semantic consistency between brain and image representations, we project neural and visual global features into a shared embedding space of dimension d using two linear heads Where the gains come from The pairwise similarity matrix A captures the fine-grained neural-visual correspondence, where each Ai,j measures how strongly the i-th neural token (a localized cortical response) is associated with the j-th visual token The selected visual tokens and the common text prompt “Please describe the image as simply as possible.” are then passed to a frozen VLM, which interprets subject-aware fMRI-activated visual tokens to generate individualized textual descriptions of the perceived content This demonstrates that MindVLM decodes fMRI signals not by directly regressing to text, but by inferring which visual evidence the subject attended to and using the VLM to translate that evidence into language Pruning redundant reasoning tokens The authors emphasize that accurate text supervision and semantically aligned fMRI-image pairs are indispensable for reliable fMRI-to-text decoding MindVLM trades lexical precision for semantic faithfulness–a trade-off that favors the underlying goal of brain captioning as a window into individual perception The no-semantic-loss group achieves approximately 0.47 ROUGE and 0.28 SPICE at q = 95, compared to around 0.45 ROUGE and 0.
Paper summary: Tom: So we're looking at this paper called MindVLM: Neural-Grounded Visual Captioning via Subject-Aware Semantic Evidence Selection, and the main idea is translating visual brain activity into natural language.
Jane: That sounds really cool for understanding how people actually see things, right? So what's the core problem they're trying to solve with this work?
Tom: The authors point out that most existing brain captioning methods project fMRI signals into language space using paired supervision, but there’s a big mismatch. They say the Natural Scenes Dataset was built on image-fMRI pairs, while the captions used for training come from full scenes.
Lu: So they argue this creates what they call semantic loss because participants only see cropped image regions during the fMRI scan, but the captions describe the whole scene, which means there's a fundamental gap between what's in their brain and what they are being trained on >
Meng: That makes sense from a deployment standpoint. If your model is trained on incomplete information, it’s going to struggle when you actually show it something complete later >
Lalam: Exactly. The paper says directly using those full scene captions as ground truth supervision risks pushing models to hallucinate visual content that wasn't actually observed in the neural signals >
Jane: So MindVLM proposes a different way: instead of relying on that text supervision, they reformulate brain captioning as Neural-Grounded Visual Captioning, where language is generated directly from visually grounded neural evidence >
Tom: Right. The thesis is that they propose a framework that maps fMRI signals into subject-aware visual tokens and then uses a frozen vision language model to produce descriptions that are both interpretable and personalized >
Lu: That sounds like they’re trying to bypass the need for text supervision entirely by focusing on what the brain actually showed, not what the old text told it to expect >
Meng: I like that focus on grounding. If you can link a specific brain pattern to a specific visual token, that feels much more robust than just matching fuzzy language >
Paper summary: Jane: So they call this mapping learning: M maps fMRI signals into subject-aware visual tokens and then uses a frozen vision language model to generate personalized descriptions >
Tom: That’s the setup. The way they build the training signal is through this alignment process where a brain encoder is trained to align fMRI responses with visual tokens extracted by a frozen vision language model image encoder >
Lu: They enforce this alignment using what they call a CLIP-style symmetric contrastive learning objective, which is half of the loss from mapping visual tokens to voxels and half mapping voxels back to images >
Meng: That sounds like it’s trying to create a shared latent space between the brain activity and the visual world representations >
Lalam: And they mention that this training process aims to discover which visual tokens are supported by neural activity, which they control with a hyperparameter q >
Jane: So what do we gain from tuning that q? The paper says performance gains come from identifying which visual tokens are supported by neural activity, and increasing the number of fMRI-visual correspondences tends to improve decoding quality >
Tom: That's interesting because it suggests that selecting more of those fMRI-visual correspondences helps decode better, even if some of those voxels might be redundant or weakly informative in the visual cortex >
Lu: It aligns with what they find in the visual cortex, where there’s a lot of inter-individual variability, and these subject-specific tokens capture that unique information encoded by each person >
Meng: So it’s not just about getting a better caption; it's about getting a caption that is more faithful to *that specific person's* perception of the image >
Jane: And they show this in their reconstruction phase, where the model reconstructs what each person actually perceived by using subject-specific cortical encoding >
Tom: They give an example where one subject, subj05, retains a large proportion of tokens, only sixty-one percent masked, with preserved regions concentrated around the action and utensil >
Paper summary: Lu: That suggests a higher fidelity perceptual representation for that individual because the encoding is more specific to their experience >
Meng: From an engineering standpoint, I'm interested in how efficient this is. They claim it’s significantly more computationally efficient than competing models because their training objective only involves neural-visual contrastive alignment, requiring no LLM during optimization >
Lalam: And they quantify that advantage: MindVLM-Online and Offline require only ten percent and one percent of the GPU memory used by MindLLM, respectively >
Jane: So it’s not just theoretically better for capturing personalized perception; it’s also much cheaper to run once trained compared to other ways of doing this >
Tom: It seems like they're trading some lexical precision for semantic faithfulness, which the authors acknowledge is a trade-off that favors the goal of brain captioning as a window into individual perception >
Lu: They explicitly state that MindVLM is designed for brain captioning under partial visual observability, which means it’s departing from conventional fMRI-to-text generation by reframing decoding from text regression to semantic evidence grounding >
Meng: That grounding idea is key. It’s not trying to regress directly to text; it's inferring which visual evidence the subject attended to and then using the vision language model just for translation >
Jane: So, what does this mean practically? If we use this approach, what kind of results can we expect when we look at how people see things?
Tom: The conclusion is that MindVLM consistently improves over the base vision language model across three different architectures like LLaVA-v1 point 5, Qwen2 point 5-VL, and Gemma3 > <ref:2610.12673#pg1>
Lu: And importantly, it remains competitive with fully text-supervised methods on semantic metrics like SPICE and CIDEr even though it’s relying on image-grounded inference instead of direct text regression >
Meng: So the main thing is that you get strong semantic fidelity without needing massive amounts of text supervision to train the whole thing >
Jane: It sounds like the implication here is that we can get a more efficient, interpretable, and cognitively faithful pathway for personalized decoding using neural-guided visual grounding >
Conclusion: Tom: So, we're wrapping up this look at MindVLM, which is called "Neural-Grounded Visual Captioning via Subject-Aware Semantic Evidence Selection." This whole idea is about using brain activity to generate descriptions of images that are really personalized for each individual.
Jane: Right. To put it simply, the paper is saying we can skip relying on text supervision—that old way of training models—and instead focus on what the brain actually showed when it saw a picture.
Lu: Exactly. They build this system by mapping fMRI signals into tokens that are specific to each person's brain and then using a frozen vision language model to translate those tokens into captions. It’s about grounding the language in neural evidence rather than just guessing from text prompts.
Meng: From a deployment view, what I find interesting is how efficient they make this process. They claim this method requires way less computational power than other models because the main training step only involves aligning brain activity with visual tokens, not needing to run a massive language model during that optimization phase.
Lalam: And LLaVA says that it keeps performing well across different vision language model backbones like Qwen2 point five-VL and Gemma3, even when we look at semantic metrics. It suggests this approach is robust because it captures the unique visual information each person encodes in their own visual cortex, which is super variable between people.
Jane: So the main point here for a listener not reading papers is that MindVLM offers a pathway to creating descriptions that are both interpretable—you can see what brain activity led to which part of the caption—and personalized because it reflects an individual's unique perceptual encoding.
Tom: The authors really drive home the idea that this isn't just about getting better scores on a benchmark; it’s about creating a more cognitively faithful way to decode perception. They emphasize trading some lexical precision for that semantic faithfulness, which they argue is the right trade-off here for brain captioning.
Lu: And they show this through reconstruction experiments where they keep tokens that are specific to how one person perceived something—like an action or an utensil—which proves the subject-specific encoding is actually capturing high-fidelity visual information.
Meng: I think what’s important for us shipping models is that it’s much more efficient to run once you have this setup, as the memory usage drops significantly compared to running a full large language model during inference.
Jane: It sounds like the implication is that we might move toward AI systems where understanding perception isn't just about matching words, but about understanding the underlying neural evidence each person provides.
Tom: This opens up a whole new direction for how we think about AI—moving from text regression to semantic evidence grounding in visual tasks. Next up, we'll look at what they did when they tried to prune some of those redundant reasoning tokens.
More episodes
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines
- 2601.15318-On the closest balanced game