RABBiT: Rapidly adaptive BOLD foundation model via brain-tuning for accurate zero-shot and few-shot prediction of speech-elicited responses in the brain

summary

The gist

RABBiT introduces Rapidly adaptive BOLD foundation model via BraIn-Tuning, a compact audio-to-fMRI encoder designed for accurate zero- and few-shot prediction of speech-elicited responses in the…

In short

RABBiT is a compact audio-to-fMRI encoder designed to predict brain activity from speech with zero or few examples. It achieves state-of-the-art accuracy by using learned region-specific attention and decomposing brain responses into shared population patterns and subject-specific variations. This allows for accurate predictions across many participants without extensive per-person training.

Key concepts

Temporal Brain Transformer (TBT)
This component constructs explicit representations for different regions of interest (ROIs) by focusing on specific parts of the speech signal. It learns how to pay attention to relevant parts of the audio stream, replacing older methods that used fixed temporal pooling and fragmented voxel-wise processing.
Shared–Idiosyncratic Decomposition (SID)
SID models each brain region's response as a combination of two parts: a low-rank shared component representing common human brain responses, and a low-rank idiosyncratic deviation representing unique differences for an individual. This structure enables good zero-shot prediction and efficient few-shot adaptation.
Brain-Tuning with LoRA
The speech backbone is adapted using LoRA adapters of rank 8. This technique allows the model to fine-tune its understanding of speech representations efficiently, enabling it to capture both general population patterns and specific subject differences with minimal trainable parameters.

Terminology used across episodes

This episode discusses

The paper

RABBiT: Rapidly adaptive BOLD foundation model via brain-tuning for accurate zero-shot and few-shot prediction of speech-elicited responses in the brain · Read on arXiv

Omer Moussa, Mariya Toneva

Max Planck Institute for Software Systems

Language understanding in the brain is context-dependent, varying across experimental stimuli and individuals, which makes it difficult to build computational models that generalize across both. This calls for a foundation model of language-evoked brain activity that can capture shared structure while adapting efficiently to new participants and inputs. We introduce RABBiT (Rapidly Adaptive BOLD foundation model via BraIn-Tuning), a compact audio-to-fMRI encoder designed for accurate zero- and few-shot prediction. A comprehensive evaluation on 324 participants across multiple unseen fMRI datasets shows that RABBiT enables accurate zero-shot prediction of fMRI responses to natural speech across auditory and language-selective regions, surpassing the SOTA foundation model for fMRI and predictions based on group averages. With as little as 10 minutes of participant-specific data, RABBiT further improves performance via parameter-efficient tuning, substantially outperforming per-participant linear models. RABBiT's performance is driven by two key innovations: (1) learned region-specific attention, and (2) a decomposition of brain responses into shared and subject-specific components, combined with a brain-tuned speech backbone. In addition to supporting strong predictive accuracy, the structured, region-specific representations that RABBiT learns enable interpretability. By eliminating the need for extensive per-participant data and model fitting, RABBiT enables scalable population-level analyses of language in the human brain. We make the code available at https://github.com/bridge-ai-neuro/rabbit.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "RABBiT: Rapidly adaptive BOLD foundation model via brain-tuning for accurate zero-shot and few-shot prediction of speech-elicited responses in the brain".

Paper summary: Jane: So, to wrap up, we're talking about this paper called "RABBiT: Rapidly adaptive BOLD foundation model via brain-tuning for accurate zero-shot and few-shot prediction of speech-elicited responses in the brain" and its authors >

Lu: The core idea is that language understanding in the brain isn't fixed; it changes based on context, so we need a model that captures shared structure but adapts to new inputs efficiently >

Tom: RABBiT is this compact audio-to-fMRI encoder that lets you predict brain activity for speech with zero or few shots, outperforming existing models and group averages across many participants >

Meng: For deployment, the crucial part is the Shared–Idiosyncratic Decomposition which lets it handle new subjects with just a little bit of data and way fewer parameters to update than traditional methods >

Jane: It implies that we can do much more scalable population-level studies of how language works in the brain without needing to collect huge amounts of specific data for every single person >

Lu: They show this model can reproduce canonical language network localizers without any re-training, suggesting it’s a useful tool for in-silico neuroscience of speech and language >

Tom: So it's about building a brain model that learns the general principles of neural response while having the flexibility to handle individual differences quickly >

Conclusion: Tom: So we’re wrapping up this look at RABBiT, which stands for Rapidly Adaptive BOLD foundation model via Brain-Tuning.

Jane: It’s a compact audio to fMRI encoder designed for predicting brain responses to speech, and it does that with zero or few shots.

Tom: Right, the title itself is pretty descriptive: rapid adaptation through brain tuning. I like that because it sounds like it actually has a mechanism for learning on the fly.

Lu: It’s about taking a foundation model of brain activity and making it adaptable to new people without needing massive amounts of their specific data during training.

Meng: From my side, the promise is parameter efficiency; if you only need a few minutes of participant data, this model can adapt much faster than retraining a whole system from scratch.

Jane: So the implication here is that we can start doing population studies on how language works in the brain without needing huge datasets for every single new person.

Tom: Exactly. It shifts the focus from collecting more individual data to building better, more flexible AI tools for neuroscience research.

Lu: It’s about capturing that shared structure of language responses while letting it flexibly adjust to individual brain differences efficiently through those learned decomposition components.

Meng: What I worry about is the real-world deployment cost, but the paper suggests the adaptation happens with just a small set of trainable parameters, which makes it much more deployable for quick testing.

Jane: The results showed it actually outperformed both other foundation models and predictions based on just averaging groups together.

Tom: That's a big deal because group averages are often a poor way to understand individual variation in the brain’s response to speech.

Lu: And the structure they learned, that split into shared and subject-specific parts, is really what makes the zero shot prediction work so well across all those unseen participants.

Meng: So, it moves us closer to having tools that can model complex brain functions where we only need a tiny bit of new data to get a good personalized result.

Jane: It definitely opens up a new path for how we analyze language in the human brain using these kinds of models.

Tom: The next thing we’re looking at is how this specific method compares to other ways people are trying to model neural networks, so we can see where RABBiT fits into the bigger picture.

More episodes

← Home