RABBiT: Rapidly adaptive BOLD foundation model via brain-tuning for accurate zero-shot and few-shot prediction of speech-elicited responses in the brain
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "RABBiT: Rapidly adaptive BOLD foundation model via brain-tuning for accurate zero-shot and few-shot prediction of speech-elicited responses in the brain".
Paper summary: Jane: So, to wrap up, we're talking about this paper called "RABBiT: Rapidly adaptive BOLD foundation model via brain-tuning for accurate zero-shot and few-shot prediction of speech-elicited responses in the brain" and its authors >
Lu: The core idea is that language understanding in the brain isn't fixed; it changes based on context, so we need a model that captures shared structure but adapts to new inputs efficiently >
Tom: RABBiT is this compact audio-to-fMRI encoder that lets you predict brain activity for speech with zero or few shots, outperforming existing models and group averages across many participants >
Meng: For deployment, the crucial part is the Shared–Idiosyncratic Decomposition which lets it handle new subjects with just a little bit of data and way fewer parameters to update than traditional methods >
Jane: It implies that we can do much more scalable population-level studies of how language works in the brain without needing to collect huge amounts of specific data for every single person >
Lu: They show this model can reproduce canonical language network localizers without any re-training, suggesting it’s a useful tool for in-silico neuroscience of speech and language >
Tom: So it's about building a brain model that learns the general principles of neural response while having the flexibility to handle individual differences quickly >
Conclusion: Tom: So we’re wrapping up this look at RABBiT, which stands for Rapidly Adaptive BOLD foundation model via Brain-Tuning.
Jane: It’s a compact audio to fMRI encoder designed for predicting brain responses to speech, and it does that with zero or few shots.
Tom: Right, the title itself is pretty descriptive: rapid adaptation through brain tuning. I like that because it sounds like it actually has a mechanism for learning on the fly.
Lu: It’s about taking a foundation model of brain activity and making it adaptable to new people without needing massive amounts of their specific data during training.
Meng: From my side, the promise is parameter efficiency; if you only need a few minutes of participant data, this model can adapt much faster than retraining a whole system from scratch.
Jane: So the implication here is that we can start doing population studies on how language works in the brain without needing huge datasets for every single new person.
Tom: Exactly. It shifts the focus from collecting more individual data to building better, more flexible AI tools for neuroscience research.
Lu: It’s about capturing that shared structure of language responses while letting it flexibly adjust to individual brain differences efficiently through those learned decomposition components.
Meng: What I worry about is the real-world deployment cost, but the paper suggests the adaptation happens with just a small set of trainable parameters, which makes it much more deployable for quick testing.
Jane: The results showed it actually outperformed both other foundation models and predictions based on just averaging groups together.
Tom: That's a big deal because group averages are often a poor way to understand individual variation in the brain’s response to speech.
Lu: And the structure they learned, that split into shared and subject-specific parts, is really what makes the zero shot prediction work so well across all those unseen participants.
Meng: So, it moves us closer to having tools that can model complex brain functions where we only need a tiny bit of new data to get a good personalized result.
Jane: It definitely opens up a new path for how we analyze language in the human brain using these kinds of models.
Tom: The next thing we’re looking at is how this specific method compares to other ways people are trying to model neural networks, so we can see where RABBiT fits into the bigger picture.
Omer Moussa, Mariya Toneva
Max Planck Institute for Software Systems
cs.CL
Submitted: 2026-07-06
Updated: 2026-11-18
Comments: Accepted at NeurIPS 2026
Code: https://github.com/bridge-aineuro/rabbit
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 92/100
The gist: RABBiT introduces Rapidly adaptive BOLD foundation model via BraIn-Tuning, a compact audio-to-fMRI encoder designed for accurate zero- and few-shot prediction of speech-elicited responses in the…
Key concepts
- Temporal Brain Transformer (TBT)
- This component constructs explicit representations for different regions of interest (ROIs) by focusing on specific parts of the speech signal. It learns how to pay attention to relevant parts of the audio stream, replacing older methods that used fixed temporal pooling and fragmented voxel-wise processing.
- Shared–Idiosyncratic Decomposition (SID)
- SID models each brain region's response as a combination of two parts: a low-rank shared component representing common human brain responses, and a low-rank idiosyncratic deviation representing unique differences for an individual. This structure enables good zero-shot prediction and efficient few-shot adaptation.
- Brain-Tuning with LoRA
- The speech backbone is adapted using LoRA adapters of rank 8. This technique allows the model to fine-tune its understanding of speech representations efficiently, enabling it to capture both general population patterns and specific subject differences with minimal trainable parameters.
Terminology
Summary
RABBiT introduces Rapidly adaptive BOLD foundation model via BraIn-Tuning, a compact audio-to-fMRI encoder designed for accurate zero- and few-shot prediction of speech-elicited responses in the brain The paper demonstrates that RABBiT enables accurate zero-shot prediction of fMRI responses to natural speech across auditory and language-selective regions, surpassing the SOTA foundation model for fMRI and predictions based on group averages With as little as 10 minutes of participantspecific data, RABBiT further improves performance via parameter-efficient tuning, substantially outperforming per-participant linear models. p. 0 A comprehensive evaluation on 324
participants across multiple unseen fMRI datasets shows that RABBiT enables
accurate zero-shot prediction of fMRI responses to natural speech across auditory
and language-selective regions, surpassing the SOTA foundation model for fMRI
and predictions based on group averages. With as little as 10 minutes of participantspecific data, RABBiT further improves performance via parameter-efficient tuning,
substantially outperforming per-participant linear models. With as little as 10 minutes of participantspecific data, RABBiT further improves performance via parameter-efficient tuning,
substantially outperforming per-participant linear models. RABBiT’s performance
is driven by two key innovations: (1) learned region-specific attention, and (2) a
decomposition of brain responses into shared and subject-specific components,
combined with a brain-tuned speech backbone.Where the paper says it
Where the paper says it
Model Architecture Components
RABBiT utilizes several key components to achieve its dual capability of zero-shot prediction and few-shot adaptation. The architecture involves a speech model, a Temporal Brain Transformer (TBT), and the Shared–Idiosyncratic Decomposition (SID). p. 1 A novel
Temporal Brain Transformer (TBT) then constructs explicit ROI-level representations by learning
region-specific attention over the speech stream, replacing fixed temporal pooling and fragmented
voxel-wise processing. Finally, a novel Shared–Idiosyncratic Decomposition (SID) maps each ROI
representation to cortical activity through a low-rank shared component and a low-rank participantspecific deviation. RABBiT first brain-tunes a self-supervised speech backbone
using LoRA adapters, adapting speech representations directly for cortical prediction. A novel
Temporal Brain Transformer (TBT) then constructs explicit ROI-level representations by learning
region-specific attention over the speech stream, replacing fixed temporal pooling and fragmented
voxel-wise processing. A novel
Temporal Brain Transformer (TBT) then constructs explicit ROI-level representations by learning
region-specific attention over the speech stream, replacing fixed temporal pooling and fragmented
voxel-wise processing. Finally, a novel Shared–Idiosyncratic Decomposition (SID) maps each ROI
representation to cortical activity through a low-rank shared component and a low-rank participantspecific deviation.Where the paper says it
Where the paper says it
Where the paper says it
Training Signal Construction
The training process involves jointly optimizing the SID parameters, the temporal brain transformer, and LoRA-based speech adapters. The loss function combines a per-ROI z-scored MSE and a Pearson-correlation term. p. 5 We train jointly on S=6 participants from Friends, in fsaverage6 space (Sec. The loss
combines a per-ROI z-scored MSE and a Pearson-correlation term. We initialize the bases from the same empirical split that motivated the model: the shared bases from
PCA of the mean training response, and the deviation bases from SVD of each subject’s residual from
that mean. This starts training from cross-subject structure in the shared term and residual subject
structure in the deviation term; initialization details are given in App. Finally, a novel Shared–Idiosyncratic Decomposition (SID) maps each ROI
representation to cortical activity through a low-rank shared component and a low-rank participantspecific deviation. The shared component enables accurate zero-shot population prediction, while
the deviation pathway supports efficient few-shot adaptation for a new participant with only a small
number of trainable parameters. Following recent work [21, 23], we therefore brain-tune the speech backbone
during RABBiT training using LoRA adapters [15]. Brain-tuning is integrated directly into the encoder: gradients propagate from the SID readout
through the temporal brain transformer into the LoRA adapters, adapting speech representations for
both shared population prediction and subject-specific deviations.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the Gains Come From
The performance gains are attributed to the learned region-specific attention, the SID decomposition, and brain-tuning. The SID decomposition models each ROI response as a sum of a low-rank shared component and a lowrank subject-specific deviation. p. 3 Motivated by this structure, we introduce a novel Shared–Idiosyncratic Decomposition (SID)
in which each ROI response is modeled as the sum of a low-rank shared component and a lowrank subject-specific deviation. The shared component captures response structure that transfers
across listeners, while the deviation component models individual variability without requiring a full
participant-specific voxel-wise readout. Finally, a novel Shared–Idiosyncratic Decomposition (SID) maps each ROI
representation to cortical activity through a low-rank shared component and a low-rank participantspecific deviation. The shared component enables accurate zero-shot population prediction, while
the deviation pathway supports efficient few-shot adaptation for a new participant with only a small
number of trainable parameters.Where the paper says it
Where the paper says it
Evaluation and Adaptability
RABBiT is evaluated on two held-out naturalistic listening cohorts totaling 324 unseen participants to test zero-shot prediction. In few-shot adaptation, only the low-rank per-ROI deviation parameters are updated, resulting in only ∼115K coefficients to update. p. 8 3 Few-shot Evaluation
We further test whether RABBiT can improve over zero-shot prediction for a new participant using
limited fMRI calibration data. For each participant, we freeze the speech backbone, temporal brain
transformer, and shared SID pathway, and update only the low-rank per-ROI deviation parameters
(∼115K parameters; Sec. It can also reproduce canonical language network localizers without
any re-training, indicating its utility as a model organism for in-silico neuroscience of
language and speech. (v) We show that few-shot SID adaptation significantly beats voxel-wise ridge encoders trained
on the same calibration data. (v) We show that few-shot SID adaptation significantly beats voxel-wise ridge encoders trained
on the same calibration data. The largest gains occur in higher-order language ROIs that
vary most across participants, confirming the model’s central intuition.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Brain tuning of speech backbones using LoRA adapters allows for parameter-efficient adaptation while preserving a shared backbone across participants, avoiding both full model retraining and reliance on fixed pretrained features. This enables rapid improvement in speech representations for cortical prediction during training with minimal additional parameters.
-
The Temporal Brain Transformer (TBT) constructs explicit ROI-level representations by learning region-specific attention over the speech stream, replacing fixed temporal pooling or fragmented voxel-wise processing. This allows the model to learn
learned ROI query tokens
that route temporal context specifically for each region, leading to interpretable cortical hierarchies. -
The Shared–Idiosyncratic Decomposition (SID) maps each ROI representation into a low-rank shared component and a low-rank participant-specific deviation, preserving zero-shot accuracy while enabling efficient few-shot adaptation from limited data. This structure allows for
efficient adaptation to new participants with only a small number of trainable parameters.
-
The model can perform accurate zero-shot prediction of fMRI responses to natural speech across auditory and language regions by utilizing the shared component, as RABBiT's predictions on auditory ROIs
already saturate near the inter-subject consistency.
-
The system can adapt efficiently to new participants with only about 10 minutes of paired audio calibration data by updating only the per-ROI deviation coefficients, which are
only ~115K parameters to update
in the few-shot regime. -
RABBiT enables scalable population-level analyses of language in the human brain by eliminating the need for extensive per-participant data and model fitting, functioning as a
foundation model of language-evoked brain activity.
-
The learned ROI query embeddings provide a direct view into the temporal brain transformer, allowing researchers to test relationships among regions through a greedy walk that
recovers a coherent trajectory without any hierarchy or spatial supervision,
revealing an interpretable cortical structure. -
The system can localize the language network by reproducing canonical left-lateralized language responses in controlled stimuli, showing that after naturalistic brain tuning, RABBiT
recovers the spatial signature of the language network without ever being trained on a controlled localizer.
Abstract
Language understanding in the brain is context-dependent, varying across experimental stimuli and individuals, which makes it difficult to build computational models that generalize across both. This calls for a foundation model of language-evoked brain activity that can capture shared structure while adapting efficiently to new participants and inputs. We introduce RABBiT (Rapidly Adaptive BOLD foundation model via BraIn-Tuning), a compact audio-to-fMRI encoder designed for accurate zero- and few-shot prediction. A comprehensive evaluation on 324 participants across multiple unseen fMRI datasets shows that RABBiT enables accurate zero-shot prediction of fMRI responses to natural speech across auditory and language-selective regions, surpassing the SOTA foundation model for fMRI and predictions based on group averages. With as little as 10 minutes of participant-specific data, RABBiT further improves performance via parameter-efficient tuning, substantially outperforming per-participant linear models. RABBiT's performance is driven by two key innovations: (1) learned region-specific attention, and (2) a decomposition of brain responses into shared and subject-specific components, combined with a brain-tuned speech backbone. In addition to supporting strong predictive accuracy, the structured, region-specific representations that RABBiT learns enable interpretability. By eliminating the need for extensive per-participant data and model fitting, RABBiT enables scalable population-level analyses of language in the human brain. We make the code available at https://github.com/bridge-ai-neuro/rabbit.
Sources
- The Wisdom of a Crowd of Brains: A Universal Brain Encoder
- The Algonauts Project 2025 Challenge: How the Human Brain Makes Sense of Multimodal Movies
- Brain-tuned Speech Models Better Reflect Speech Processing Stages in the Brain
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data