RABBiT: Rapidly adaptive BOLD foundation model via brain-tuning for accurate zero-shot and few-shot prediction of speech-elicited responses in the brain

arXiv:2607.05171 · cs.CL · Submitted 2026-07-06 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "RABBiT: Rapidly adaptive BOLD foundation model via brain-tuning for accurate zero-shot and few-shot prediction of speech-elicited responses in the brain".

Paper summary: Jane: So, to wrap up, we're talking about this paper called "RABBiT: Rapidly adaptive BOLD foundation model via brain-tuning for accurate zero-shot and few-shot prediction of speech-elicited responses in the brain" and its authors >

Lu: The core idea is that language understanding in the brain isn't fixed; it changes based on context, so we need a model that captures shared structure but adapts to new inputs efficiently >

Tom: RABBiT is this compact audio-to-fMRI encoder that lets you predict brain activity for speech with zero or few shots, outperforming existing models and group averages across many participants >

Meng: For deployment, the crucial part is the Shared–Idiosyncratic Decomposition which lets it handle new subjects with just a little bit of data and way fewer parameters to update than traditional methods >

Jane: It implies that we can do much more scalable population-level studies of how language works in the brain without needing to collect huge amounts of specific data for every single person >

Lu: They show this model can reproduce canonical language network localizers without any re-training, suggesting it’s a useful tool for in-silico neuroscience of speech and language >

Tom: So it's about building a brain model that learns the general principles of neural response while having the flexibility to handle individual differences quickly >

Conclusion: Tom: So we’re wrapping up this look at RABBiT, which stands for Rapidly Adaptive BOLD foundation model via Brain-Tuning.

Jane: It’s a compact audio to fMRI encoder designed for predicting brain responses to speech, and it does that with zero or few shots.

Tom: Right, the title itself is pretty descriptive: rapid adaptation through brain tuning. I like that because it sounds like it actually has a mechanism for learning on the fly.

Lu: It’s about taking a foundation model of brain activity and making it adaptable to new people without needing massive amounts of their specific data during training.

Meng: From my side, the promise is parameter efficiency; if you only need a few minutes of participant data, this model can adapt much faster than retraining a whole system from scratch.

Jane: So the implication here is that we can start doing population studies on how language works in the brain without needing huge datasets for every single new person.

Tom: Exactly. It shifts the focus from collecting more individual data to building better, more flexible AI tools for neuroscience research.

Lu: It’s about capturing that shared structure of language responses while letting it flexibly adjust to individual brain differences efficiently through those learned decomposition components.

Meng: What I worry about is the real-world deployment cost, but the paper suggests the adaptation happens with just a small set of trainable parameters, which makes it much more deployable for quick testing.

Jane: The results showed it actually outperformed both other foundation models and predictions based on just averaging groups together.

Tom: That's a big deal because group averages are often a poor way to understand individual variation in the brain’s response to speech.

Lu: And the structure they learned, that split into shared and subject-specific parts, is really what makes the zero shot prediction work so well across all those unseen participants.

Meng: So, it moves us closer to having tools that can model complex brain functions where we only need a tiny bit of new data to get a good personalized result.

Jane: It definitely opens up a new path for how we analyze language in the human brain using these kinds of models.

Tom: The next thing we’re looking at is how this specific method compares to other ways people are trying to model neural networks, so we can see where RABBiT fits into the bigger picture.

Omer Moussa, Mariya Toneva

Max Planck Institute for Software Systems

cs.CL

Submitted: 2026-07-06

Updated: 2026-11-18

Comments: Accepted at NeurIPS 2026

Code: https://github.com/bridge-aineuro/rabbit

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: RABBiT introduces Rapidly adaptive BOLD foundation model via BraIn-Tuning, a compact audio-to-fMRI encoder designed for accurate zero- and few-shot prediction of speech-elicited responses in the…

Key concepts

Temporal Brain Transformer (TBT)
This component constructs explicit representations for different regions of interest (ROIs) by focusing on specific parts of the speech signal. It learns how to pay attention to relevant parts of the audio stream, replacing older methods that used fixed temporal pooling and fragmented voxel-wise processing.
Shared–Idiosyncratic Decomposition (SID)
SID models each brain region's response as a combination of two parts: a low-rank shared component representing common human brain responses, and a low-rank idiosyncratic deviation representing unique differences for an individual. This structure enables good zero-shot prediction and efficient few-shot adaptation.
Brain-Tuning with LoRA
The speech backbone is adapted using LoRA adapters of rank 8. This technique allows the model to fine-tune its understanding of speech representations efficiently, enabling it to capture both general population patterns and specific subject differences with minimal trainable parameters.

Terminology

Summary

RABBiT introduces Rapidly adaptive BOLD foundation model via BraIn-Tuning, a compact audio-to-fMRI encoder designed for accurate zero- and few-shot prediction of speech-elicited responses in the brain The paper demonstrates that RABBiT enables accurate zero-shot prediction of fMRI responses to natural speech across auditory and language-selective regions, surpassing the SOTA foundation model for fMRI and predictions based on group averages With as little as 10 minutes of participantspecific data, RABBiT further improves performance via parameter-efficient tuning, substantially outperforming per-participant linear models. p. 0

Where the paper says it

A comprehensive evaluation on 324 participants across multiple unseen fMRI datasets shows that RABBiT enables accurate zero-shot prediction of fMRI responses to natural speech across auditory and language-selective regions, surpassing the SOTA foundation model for fMRI and predictions based on group averages. With as little as 10 minutes of participantspecific data, RABBiT further improves performance via parameter-efficient tuning, substantially outperforming per-participant linear models.

RABBiT’s performance is driven by two key innovations: (1) learned region-specific attention, and (2) a decomposition of brain responses into shared and subject-specific components, combined with a brain-tuned speech backbone. p. 0
Where the paper says it

With as little as 10 minutes of participantspecific data, RABBiT further improves performance via parameter-efficient tuning, substantially outperforming per-participant linear models. RABBiT’s performance is driven by two key innovations: (1) learned region-specific attention, and (2) a decomposition of brain responses into shared and subject-specific components, combined with a brain-tuned speech backbone.

Model Architecture Components

RABBiT utilizes several key components to achieve its dual capability of zero-shot prediction and few-shot adaptation. The architecture involves a speech model, a Temporal Brain Transformer (TBT), and the Shared–Idiosyncratic Decomposition (SID). p. 1

Where the paper says it

A novel Temporal Brain Transformer (TBT) then constructs explicit ROI-level representations by learning region-specific attention over the speech stream, replacing fixed temporal pooling and fragmented voxel-wise processing. Finally, a novel Shared–Idiosyncratic Decomposition (SID) maps each ROI representation to cortical activity through a low-rank shared component and a low-rank participantspecific deviation.

The TBT constructs explicit ROI-level representations by learning region-specific attention over the speech stream, replacing fixed temporal pooling and fragmented voxel-wise processing. p. 1
Where the paper says it

RABBiT first brain-tunes a self-supervised speech backbone using LoRA adapters, adapting speech representations directly for cortical prediction. A novel Temporal Brain Transformer (TBT) then constructs explicit ROI-level representations by learning region-specific attention over the speech stream, replacing fixed temporal pooling and fragmented voxel-wise processing.

The SID maps each ROI representation to cortical activity through a low-rank shared component and a low-rank participantspecific deviation. p. 1
Where the paper says it

A novel Temporal Brain Transformer (TBT) then constructs explicit ROI-level representations by learning region-specific attention over the speech stream, replacing fixed temporal pooling and fragmented voxel-wise processing. Finally, a novel Shared–Idiosyncratic Decomposition (SID) maps each ROI representation to cortical activity through a low-rank shared component and a low-rank participantspecific deviation.

Training Signal Construction

The training process involves jointly optimizing the SID parameters, the temporal brain transformer, and LoRA-based speech adapters. The loss function combines a per-ROI z-scored MSE and a Pearson-correlation term. p. 5

Where the paper says it

We train jointly on S=6 participants from Friends, in fsaverage6 space (Sec. The loss combines a per-ROI z-scored MSE and a Pearson-correlation term.

The anchor term (Lanchor) is defined in App. p. 3
Where the paper says it

We initialize the bases from the same empirical split that motivated the model: the shared bases from PCA of the mean training response, and the deviation bases from SVD of each subject’s residual from that mean. This starts training from cross-subject structure in the shared term and residual subject structure in the deviation term; initialization details are given in App.

B.2, which is used to regularize the adaptation pathway. p. 1
Where the paper says it

Finally, a novel Shared–Idiosyncratic Decomposition (SID) maps each ROI representation to cortical activity through a low-rank shared component and a low-rank participantspecific deviation. The shared component enables accurate zero-shot population prediction, while the deviation pathway supports efficient few-shot adaptation for a new participant with only a small number of trainable parameters.

The speech backbone is brain-tuned using LoRA adapters of rank 8, which adapt speech representations for both shared population prediction and subject-specific deviations. p. 4
Where the paper says it

Following recent work [21, 23], we therefore brain-tune the speech backbone during RABBiT training using LoRA adapters [15]. Brain-tuning is integrated directly into the encoder: gradients propagate from the SID readout through the temporal brain transformer into the LoRA adapters, adapting speech representations for both shared population prediction and subject-specific deviations.

Where the Gains Come From

The performance gains are attributed to the learned region-specific attention, the SID decomposition, and brain-tuning. The SID decomposition models each ROI response as a sum of a low-rank shared component and a lowrank subject-specific deviation. p. 3

Where the paper says it

Motivated by this structure, we introduce a novel Shared–Idiosyncratic Decomposition (SID) in which each ROI response is modeled as the sum of a low-rank shared component and a lowrank subject-specific deviation. The shared component captures response structure that transfers across listeners, while the deviation component models individual variability without requiring a full participant-specific voxel-wise readout.

This structure enables accurate zero-shot population prediction through the shared component, while the deviation pathway supports efficient few-shot adaptation for a new participant with only a small number of trainable parameters. p. 1
Where the paper says it

Finally, a novel Shared–Idiosyncratic Decomposition (SID) maps each ROI representation to cortical activity through a low-rank shared component and a low-rank participantspecific deviation. The shared component enables accurate zero-shot population prediction, while the deviation pathway supports efficient few-shot adaptation for a new participant with only a small number of trainable parameters.

Evaluation and Adaptability

RABBiT is evaluated on two held-out naturalistic listening cohorts totaling 324 unseen participants to test zero-shot prediction. In few-shot adaptation, only the low-rank per-ROI deviation parameters are updated, resulting in only ∼115K coefficients to update. p. 8

Where the paper says it

3 Few-shot Evaluation We further test whether RABBiT can improve over zero-shot prediction for a new participant using limited fMRI calibration data. For each participant, we freeze the speech backbone, temporal brain transformer, and shared SID pathway, and update only the low-rank per-ROI deviation parameters (∼115K parameters; Sec.

Few-shot adaptation significantly beats voxel-wise ridge encoders trained on the same calibration data. p. 2
Where the paper says it

It can also reproduce canonical language network localizers without any re-training, indicating its utility as a model organism for in-silico neuroscience of language and speech. (v) We show that few-shot SID adaptation significantly beats voxel-wise ridge encoders trained on the same calibration data.

The largest gains occur in higher-order language ROIs that vary most across participants, confirming the model’s central intuition. p. 2
Where the paper says it

(v) We show that few-shot SID adaptation significantly beats voxel-wise ridge encoders trained on the same calibration data. The largest gains occur in higher-order language ROIs that vary most across participants, confirming the model’s central intuition.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Brain tuning of speech backbones using LoRA adapters allows for parameter-efficient adaptation while preserving a shared backbone across participants, avoiding both full model retraining and reliance on fixed pretrained features. This enables rapid improvement in speech representations for cortical prediction during training with minimal additional parameters.

  2. The Temporal Brain Transformer (TBT) constructs explicit ROI-level representations by learning region-specific attention over the speech stream, replacing fixed temporal pooling or fragmented voxel-wise processing. This allows the model to learn learned ROI query tokens that route temporal context specifically for each region, leading to interpretable cortical hierarchies.

  3. The Shared–Idiosyncratic Decomposition (SID) maps each ROI representation into a low-rank shared component and a low-rank participant-specific deviation, preserving zero-shot accuracy while enabling efficient few-shot adaptation from limited data. This structure allows for efficient adaptation to new participants with only a small number of trainable parameters.

  4. The model can perform accurate zero-shot prediction of fMRI responses to natural speech across auditory and language regions by utilizing the shared component, as RABBiT's predictions on auditory ROIs already saturate near the inter-subject consistency.

  5. The system can adapt efficiently to new participants with only about 10 minutes of paired audio calibration data by updating only the per-ROI deviation coefficients, which are only ~115K parameters to update in the few-shot regime.

  6. RABBiT enables scalable population-level analyses of language in the human brain by eliminating the need for extensive per-participant data and model fitting, functioning as a foundation model of language-evoked brain activity.

  7. The learned ROI query embeddings provide a direct view into the temporal brain transformer, allowing researchers to test relationships among regions through a greedy walk that recovers a coherent trajectory without any hierarchy or spatial supervision, revealing an interpretable cortical structure.

  8. The system can localize the language network by reproducing canonical left-lateralized language responses in controlled stimuli, showing that after naturalistic brain tuning, RABBiT recovers the spatial signature of the language network without ever being trained on a controlled localizer.

Abstract

Language understanding in the brain is context-dependent, varying across experimental stimuli and individuals, which makes it difficult to build computational models that generalize across both. This calls for a foundation model of language-evoked brain activity that can capture shared structure while adapting efficiently to new participants and inputs. We introduce RABBiT (Rapidly Adaptive BOLD foundation model via BraIn-Tuning), a compact audio-to-fMRI encoder designed for accurate zero- and few-shot prediction. A comprehensive evaluation on 324 participants across multiple unseen fMRI datasets shows that RABBiT enables accurate zero-shot prediction of fMRI responses to natural speech across auditory and language-selective regions, surpassing the SOTA foundation model for fMRI and predictions based on group averages. With as little as 10 minutes of participant-specific data, RABBiT further improves performance via parameter-efficient tuning, substantially outperforming per-participant linear models. RABBiT's performance is driven by two key innovations: (1) learned region-specific attention, and (2) a decomposition of brain responses into shared and subject-specific components, combined with a brain-tuned speech backbone. In addition to supporting strong predictive accuracy, the structured, region-specific representations that RABBiT learns enable interpretability. By eliminating the need for extensive per-participant data and model fitting, RABBiT enables scalable population-level analyses of language in the human brain. We make the code available at https://github.com/bridge-ai-neuro/rabbit.

Sources

Related papers