BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction
summary
The gist
Naturalistic fMRI characterizes the brain as a dynamic predictive engine driven by continuous sensory streams; BrainVista…
In short
BrainVista models brain dynamics as a predictive engine using a multimodal autoregressive framework. It uses Networkwise Tokenizers to capture functional networks and a Spatial Mixer Head to manage cross-network information flow. The system forecasts future cortical states from past history and sensory inputs, achieving state-of-the-art performance on naturalistic fMRI benchmarks.
Key concepts
- Multimodal Autoregressive Framework
- This is a modeling approach where the system predicts the next part of a sequence (like brain activity) based on all previous parts and current external inputs (like video or audio). It treats brain states as tokens in a sequence, allowing it to forecast future conditions by learning the causal transitions between them.
- Networkwise Tokenizers
- These are specialized components designed to compress complex fMRI data from different brain regions into compact 'circuit tokens.' They act like lightweight encoders that learn how local brain networks reconstruct the original fMRI signal, helping the model understand functional heterogeneity across different areas.
- Spatial Mixer Head
- This mechanism is introduced to explicitly control how information flows between different brain networks. Instead of relying on implicit global attention, this head learns to manage and mix signals across these distinct functional units while respecting their original boundaries, improving cross-network modeling.
Terminology used across episodes
This episode discusses
- BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction · Paper Radio
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- CineBrain: A Large-Scale Multi-Modal Brain Dataset During Naturalistic Audiovisual Narrative Processing
- The Algonauts Project 2025 Challenge: How the Human Brain Makes Sense of Multimodal Movies
- The Llama 3 Herd of Models · Paper Radio
- BrainCodec: Neural fMRI codec for the decoding of cognitive brain states
- SLIM-Brain: A Data- and Training-Efficient Foundation Model for fMRI Data Analysis
- A Brain Graph Foundation Model: Pre-Training and Prompt-Tuning across Broad Atlases and Disorders
- Hierarchical Characterization of Brain Dynamics via State Space-based Vector Quantization
- Improving Multimodal Brain Encoding Model with Dynamic Subject-awareness Routing
The paper
BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction · Read on arXiv
University of Sydney · University of New South Wales · CSIRO
Transcript
Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.
Ines: I'm Ines.
Marcus: And I'm Marcus, here with Yuki.
Ines: We're looking at the paper: "BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction".
Marcus: Here's the gist of the paper. Naturalistic fMRI characterizes the brain as a dynamic predictive engine driven by continuous sensory streams; BrainVista, a multimodal autoregressive framework designed to model the causal evolution of brain states, addresses this challenge by incorporating Networkwise Tokenizers and a Spatial Mixer Head The Core Framework BrainVista is introduced as a multimodal autoregressive framework designed to model the causal evolution of brain states It incorporates Networkwise Tokenizers to disentangle system-specific dynamics and a Spatial Mixer Head that captures inter-network information flow without compromising functional boundaries The framework is designed to model naturalistic fMRI responses from stimuli in a consistent and scalable manner Modeling Causal Dynamics The modeling of naturalistic fMRI is cast as a time-aligned multimodal next-token forecasting under a past-only constraint, where open-loop rollout iterates a learned one-step transition The autoregressive objective parameterizes the one-step transition to forecast the next cortical state from past brain history together with an exogenous stimulus stream that is assumed observed at inference An H-step rollout is obtained by recursively applying the same transition to previously predicted states Multimodal Input Construction Stimulus tokens are formed by encoding video, audio, and text with frozen encoders and aggregating them onto the fMRI grid These high-rate tokens are then downsampled onto the fMRI grid for causal prediction A shared stimulus token is formed by concatenating modality tokens and projecting them to a common feature space, denoted as τt s Circuit Tokenization and Mixing To address cortical functional heterogeneity, parcel-wise fMRI is compressed into compact circuit tokens via network-wise tokenizer–detokenizer pairs These tokenizers are lightweight twolayer MLPs trained to self-reconstruct fMRI signals within each functional network The whole-brain circuit token is formed by concatenating these network tokens to form the whole-brain circuit token τt f A Spatial Mixer Head is introduced to explicitly control cross-network mixing instead of relying on implicit global attention Temporal Conditioning and Masking To ensure strict temporal validity in multimodal conditioning, a Stimulus-to-Brain (S2B) masking mechanism is introduced to enforce a constraint to prevent temporal leakage and keep conditioning temporally consistent across training and inference This mask constrains selfattention so that circuit-token queries at time ti attend only to past indices tj ≤ ti and never to future-aligned stimuli Experimental Validation The framework is validated on three public naturalistic fMRI benchmarks, Algonauts 2025, CineBrain, and HAD BrainVista achieves state-of-the-art fMRI encoding performance In longhorizon rollout settings, the model yields substantial improvements over baselines, increasing pattern correlation by 36.0% and 33.3% on relative to the strongest baseline Algonauts 2025 and CineBrain, respectively The results show that BrainVista achieves the best or tied-best performance despite differing ROI parcellations and coordinate spaces Key Findings on Robustness The improvements persist under ρ, suggesting that the gains are not driven by a small set of extreme values and that the predicted dynamics better preserve the relative temporal ordering of neural responses The Spatial Mixer Head achieves the best pcorr across horizons, and its margin grows with H (e.g., ∼ +17.8% at H=20 versus removing the mixer) The combined setting yields the best correlation fidelity and identification accuracy, and the gap typically widens at longer horizons The full multimodal setting consistently performs best, while dropping any modality degrades pcorr, with the largest decline observed when video is removed The results also clarify trade-offs of past-only multimodal forecasting Ablation Insights Network-wise tokenization yields the most consistent gains across datasets and ROI definitions, improving both correlation fidelity and identification under open-loop long-horizon rollout The network-level pcorr breakdown shows that Visual and Default modes achieve the highest pattern fidelity (Visual: 0.3275–0.3512; Default: 0.3015–0.3237) The attention-based coupling visualization from the Spatial Mixer exhibits similarly structured, non-uniform connectivity across subjects, suggesting that the qualitative behavior observed in the main paper is not driven by a single subject-specific artifact The full multimodal setting consistently performs best, while dropping any modality degrades pcorr BrainVista achieves gains over baselines on three benchmarks, with the clearest improvements under long-horizon open-loop rollout As the rollout horizon grows, performance inevitably degrades due to compounding error, but BrainVista mitigates drift via temporally consistent conditioning and structured cross-network computation Future work will explore stronger tokenization/decoding objectives and calibration under distribution shifts to further improve long-horizon stability and cross-dataset generalization References Assran, M.
Ines: et al.
Marcus: V-JEPA 2: Self-supervised video models enable understanding, prediction and planning Bao, G., Zhang, Q., Gong, Z., Wu, Z., and Miao, D.
Ines: MindSimulator: Exploring brain concept localization via synthetic fMRI Brown, T.
Paper summary: Ines: So, we're looking at this paper called BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction. The core idea here is that the brain acts like a dynamic predictive engine, constantly driven by these continuous sensory streams we get from things like video or audio.
Marcus: Exactly. But modeling how these brain states evolve causally in a realistic way is tough because there's this massive timescale mismatch between what the inputs are—the high-frequency sensory stuff—and the slow BOLD signals from fMRI.
Yuki: And that's where the authors of BrainVista claim they're solving it by introducing this multimodal autoregressive framework to model the causal evolution of brain states. What they claim is that they can do this in a consistent and scalable way, even with different types of brain regions.
Ines: They achieve this by using Networkwise Tokenizers to separate the system specific dynamics and a Spatial Mixer Head that lets them capture how information flows between different functional networks without messing up those boundaries. It’s a big structural move.
Marcus: And they also have this new Stimulus-to-Brain masking mechanism which is designed to keep the high frequency sensory stimuli synchronized with the hemodynamically filtered signals, allowing for strict history only causal conditioning. That seems pretty tight for controlling temporal leakage.
Yuki: It sounds like they are trying to enforce causality across these mismatched scales, which is a really hard problem in neuroimaging modeling. But what's the actual payoff for understanding how this works?
Ines: The validation on three public naturalistic fMRI benchmarks—Algonauts two thousand twenty-five CineBrain, and HAD BrainVista—shows that BrainVista achieves state of the art encoding performance <ref:2602.04512#pg2>. In long horizon rollout settings, they see substantial improvements over baselines.
Marcus: They claim a pattern correlation increase of thirty six point zero percent on Algonauts two thousand twenty-five and thirty three point three percent on CineBrain relative to the strongest baseline, and this holds even when they change the ROI parcellations and coordinate spaces <ref:2602.04512#pg2>. That suggests it's robust.
Paper summary: Yuki: From a population geneticist's view, if it works across different brain maps, that means we're seeing something about the underlying dynamic structure of neural activity that isn't just tied to one specific anatomical labeling. What does this imply about how different populations might process the same sensory input?
Ines: The results also point to robustness under rho, which suggests these gains aren't just from a few extreme data points. Plus, the Spatial Mixer Head achieves the best correlation across horizons, and that margin grows with H up to about seventeen point eight percent at H equals ten versus when you remove the mixer.
Marcus: It’s interesting because it shows that coupling those network tokens explicitly seems to be more effective than relying on implicit global attention mechanisms. The combined setting—full multimodal input—consistently performs best, and dropping any single modality, like video or audio, degrades the pattern correlation; losing video causes the largest drop.
Yuki: So what this means for us in general is that to really capture the brain's predictive engine model accurately across different contexts, you need to feed it all of it in. It’s not enough to just focus on one sense or one region of interest.
Ines: And looking at the ablation insights, Networkwise tokenization seems to give the most consistent gains across different datasets and ROI definitions for both correlation fidelity and identification under open loop long horizon rollout. Visual and Default modes show the highest pattern fidelity, with visual being roughly zero point three two seven five to zero point three five one two.
Marcus: That’s a concrete number showing where the strongest information is coming from in terms of pattern preservation across different brain regions. And I see that the attention based coupling visualization from the Spatial Mixer shows similar structured connectivity across subjects, which suggests what they're seeing qualitatively isn't just an artifact of one specific person's data.
Yuki: That’s important because it hints at a shared underlying principle in how these networks interact dynamically, rather than just individual noise. It makes the model feel more grounded in some kind of population-level structure, even when working on individual fMRI data.
Paper summary: Ines: Overall, the BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction framework is showing strong gains over existing baselines on three benchmarks when using long horizon open loop rollout. As the rollout horizon gets longer, performance naturally degrades because of compounding error, but BrainVista manages to mitigate that drift through temporally consistent conditioning and structured cross network computation.
Marcus: The authors are clear about the trade-offs in their conclusion; they acknowledge that while their method handles drift better than some baselines, the performance still drops as time goes on. They are also flagging that extending this approach to naturalistic fMRI is non trivial because of that core hurdle—maintaining strict causality across those mismatched timescales between high frequency visual stimuli and the slow BOLD signals (Lahner et al., two thousand twenty-four; Chang et al <ref:2602.04512#pg2>., two thousand twenty-two) <ref:2602.04512#pg2>.
Yuki: So for anyone listening, this means we’re seeing a path forward where we can use these powerful autoregressive models to simulate brain dynamics in a way that respects the actual flow of information over time, not just static correlations. It moves us closer to understanding the brain as a true predictive system.
Ines: And looking at the future work, they plan to explore stronger tokenization decoding objectives and calibration under distribution shifts, which should further improve long horizon stability and cross dataset generalization. That’s where the real push for making these models truly generalizable will happen.
Marcus: I think the implication for large scale cohorts is that if we can get this kind of robust modeling, it could help us interpret how different groups of people process complex multimodal information in a way that respects their specific neural dynamics rather than just averaging everything out.
Yuki: It’s about moving beyond just seeing what correlations exist and actually building a model that predicts the sequence of events. That’s what this work is aiming for with BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction, page one of that work reads: "BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction Xuanhua Yin1 Runkai Zhao1* Lina Yao2,3 Weidong Cai1* one The University of Sydney, Australia two University of New South Wales, Australia three CSIRO’s Data61 xuanhua <ref:2602.04512#pg1,BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction Xuanhua Yin1>.yin@sydney.edu.au runkai.zhao@sydney.edu.au lina.yao@unsw.edu.au tom.cai@sydney.edu"
Conclusion: Ines: So, BrainVista is essentially taking that idea of the brain being a constantly running prediction engine—driven by all these sensory inputs—and trying to map out how those states evolve over time in a way that respects causality.
Marcus: It’s about using this multimodal autoregressive framework to model the causal evolution of brain states, and they did it using Networkwise Tokenizers and a Spatial Mixer Head.
Yuki: The authors are tackling the challenge of naturalistic fMRI by treating it like forecasting the next token in a sequence, incorporating video, audio, and text together.
Ines: They frame the modeling as time-aligned multimodal next-token forecasting under a past-only constraint, meaning you can only condition on what has already happened to predict what comes next.
Marcus: The results are pretty strong on the public benchmarks like Algonauts two thousand twenty-five and CineBrain, where BrainVista achieves state of the art encoding performance.
Yuki: They show substantial improvements in long horizon rollout settings, increasing pattern correlation by about thirty six point zero percent on one dataset and thirty three point three percent on another.
Ines: It’s interesting because they show these gains persist even when you change the ROI parcellations and the coordinate spaces used in the experiments.
Marcus: They also found that their spatial mixer head gives them the best correlation across different time horizons, with its margin growing with how long you look ahead, up to about seventeen point eight percent at a horizon of ten steps.
Yuki: This suggests that the way they structure those cross-network interactions is really capturing something fundamental about how these different brain areas talk to each other.
Ines: So what this means is that we’re seeing a model that can handle the complexity of multimodal data and still maintain a causal view of time in a dynamic system.
Marcus: It suggests that if we can build these kinds of models, they could help us interpret how different populations process complex information, respecting their specific neural dynamics.
Yuki: And they’re already looking ahead to future work on calibration under distribution shifts, which is a big step toward making these models truly generalizable across different people.
More episodes
- 2610.33881-Sensorimotor behavior from an untrained brain-and-nerve-cord connectome
- 2610.40344-Morphological Features of Murine Pulmonary Arterial Networks for Diagnosing Vascular Disease
- Daily Summary for 2026-11-19
- 2609.10193-Are You Learning Biological Signal or Shortcuts? Auditing and Mitigating Bias in Protein-Protein Interaction Datasets
- 2610.38236-Generative discovery of chemical reaction networks
- 2507.11176-An Interpretable AI framework Quantifying Traditional Chinese Medicine Principles Towards Enhancing and Integrating with Modern Biomedicine
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2504.03590-Towards a Unified Framework for Determining Conformational Ensembles of Disordered Proteins
- 2603.15217-A multiscale discrete-to-continuum framework for structured population models