BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.
Ines: I'm Ines.
Marcus: And I'm Marcus, here with Yuki.
Ines: We're looking at the paper: "BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction".
Marcus: Here's the gist of the paper. Naturalistic fMRI characterizes the brain as a dynamic predictive engine driven by continuous sensory streams; BrainVista, a multimodal autoregressive framework designed to model the causal evolution of brain states, addresses this challenge by incorporating Networkwise Tokenizers and a Spatial Mixer Head The Core Framework BrainVista is introduced as a multimodal autoregressive framework designed to model the causal evolution of brain states It incorporates Networkwise Tokenizers to disentangle system-specific dynamics and a Spatial Mixer Head that captures inter-network information flow without compromising functional boundaries The framework is designed to model naturalistic fMRI responses from stimuli in a consistent and scalable manner Modeling Causal Dynamics The modeling of naturalistic fMRI is cast as a time-aligned multimodal next-token forecasting under a past-only constraint, where open-loop rollout iterates a learned one-step transition The autoregressive objective parameterizes the one-step transition to forecast the next cortical state from past brain history together with an exogenous stimulus stream that is assumed observed at inference An H-step rollout is obtained by recursively applying the same transition to previously predicted states Multimodal Input Construction Stimulus tokens are formed by encoding video, audio, and text with frozen encoders and aggregating them onto the fMRI grid These high-rate tokens are then downsampled onto the fMRI grid for causal prediction A shared stimulus token is formed by concatenating modality tokens and projecting them to a common feature space, denoted as τt s Circuit Tokenization and Mixing To address cortical functional heterogeneity, parcel-wise fMRI is compressed into compact circuit tokens via network-wise tokenizer–detokenizer pairs These tokenizers are lightweight twolayer MLPs trained to self-reconstruct fMRI signals within each functional network The whole-brain circuit token is formed by concatenating these network tokens to form the whole-brain circuit token τt f A Spatial Mixer Head is introduced to explicitly control cross-network mixing instead of relying on implicit global attention Temporal Conditioning and Masking To ensure strict temporal validity in multimodal conditioning, a Stimulus-to-Brain (S2B) masking mechanism is introduced to enforce a constraint to prevent temporal leakage and keep conditioning temporally consistent across training and inference This mask constrains selfattention so that circuit-token queries at time ti attend only to past indices tj ≤ ti and never to future-aligned stimuli Experimental Validation The framework is validated on three public naturalistic fMRI benchmarks, Algonauts 2025, CineBrain, and HAD BrainVista achieves state-of-the-art fMRI encoding performance In longhorizon rollout settings, the model yields substantial improvements over baselines, increasing pattern correlation by 36.0% and 33.3% on relative to the strongest baseline Algonauts 2025 and CineBrain, respectively The results show that BrainVista achieves the best or tied-best performance despite differing ROI parcellations and coordinate spaces Key Findings on Robustness The improvements persist under ρ, suggesting that the gains are not driven by a small set of extreme values and that the predicted dynamics better preserve the relative temporal ordering of neural responses The Spatial Mixer Head achieves the best pcorr across horizons, and its margin grows with H (e.g., ∼ +17.8% at H=20 versus removing the mixer) The combined setting yields the best correlation fidelity and identification accuracy, and the gap typically widens at longer horizons The full multimodal setting consistently performs best, while dropping any modality degrades pcorr, with the largest decline observed when video is removed The results also clarify trade-offs of past-only multimodal forecasting Ablation Insights Network-wise tokenization yields the most consistent gains across datasets and ROI definitions, improving both correlation fidelity and identification under open-loop long-horizon rollout The network-level pcorr breakdown shows that Visual and Default modes achieve the highest pattern fidelity (Visual: 0.3275–0.3512; Default: 0.3015–0.3237) The attention-based coupling visualization from the Spatial Mixer exhibits similarly structured, non-uniform connectivity across subjects, suggesting that the qualitative behavior observed in the main paper is not driven by a single subject-specific artifact The full multimodal setting consistently performs best, while dropping any modality degrades pcorr BrainVista achieves gains over baselines on three benchmarks, with the clearest improvements under long-horizon open-loop rollout As the rollout horizon grows, performance inevitably degrades due to compounding error, but BrainVista mitigates drift via temporally consistent conditioning and structured cross-network computation Future work will explore stronger tokenization/decoding objectives and calibration under distribution shifts to further improve long-horizon stability and cross-dataset generalization References Assran, M.
Ines: et al.
Marcus: V-JEPA 2: Self-supervised video models enable understanding, prediction and planning Bao, G., Zhang, Q., Gong, Z., Wu, Z., and Miao, D.
Ines: MindSimulator: Exploring brain concept localization via synthetic fMRI Brown, T.
Paper summary: Ines: So, we're looking at this paper called BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction. The core idea here is that the brain acts like a dynamic predictive engine, constantly driven by these continuous sensory streams we get from things like video or audio.
Marcus: Exactly. But modeling how these brain states evolve causally in a realistic way is tough because there's this massive timescale mismatch between what the inputs are—the high-frequency sensory stuff—and the slow BOLD signals from fMRI.
Yuki: And that's where the authors of BrainVista claim they're solving it by introducing this multimodal autoregressive framework to model the causal evolution of brain states. What they claim is that they can do this in a consistent and scalable way, even with different types of brain regions.
Ines: They achieve this by using Networkwise Tokenizers to separate the system specific dynamics and a Spatial Mixer Head that lets them capture how information flows between different functional networks without messing up those boundaries. It’s a big structural move.
Marcus: And they also have this new Stimulus-to-Brain masking mechanism which is designed to keep the high frequency sensory stimuli synchronized with the hemodynamically filtered signals, allowing for strict history only causal conditioning. That seems pretty tight for controlling temporal leakage.
Yuki: It sounds like they are trying to enforce causality across these mismatched scales, which is a really hard problem in neuroimaging modeling. But what's the actual payoff for understanding how this works?
Ines: The validation on three public naturalistic fMRI benchmarks—Algonauts two thousand twenty-five CineBrain, and HAD BrainVista—shows that BrainVista achieves state of the art encoding performance <ref:2602.04512#pg2>. In long horizon rollout settings, they see substantial improvements over baselines.
Marcus: They claim a pattern correlation increase of thirty six point zero percent on Algonauts two thousand twenty-five and thirty three point three percent on CineBrain relative to the strongest baseline, and this holds even when they change the ROI parcellations and coordinate spaces <ref:2602.04512#pg2>. That suggests it's robust.
Paper summary: Yuki: From a population geneticist's view, if it works across different brain maps, that means we're seeing something about the underlying dynamic structure of neural activity that isn't just tied to one specific anatomical labeling. What does this imply about how different populations might process the same sensory input?
Ines: The results also point to robustness under rho, which suggests these gains aren't just from a few extreme data points. Plus, the Spatial Mixer Head achieves the best correlation across horizons, and that margin grows with H up to about seventeen point eight percent at H equals ten versus when you remove the mixer.
Marcus: It’s interesting because it shows that coupling those network tokens explicitly seems to be more effective than relying on implicit global attention mechanisms. The combined setting—full multimodal input—consistently performs best, and dropping any single modality, like video or audio, degrades the pattern correlation; losing video causes the largest drop.
Yuki: So what this means for us in general is that to really capture the brain's predictive engine model accurately across different contexts, you need to feed it all of it in. It’s not enough to just focus on one sense or one region of interest.
Ines: And looking at the ablation insights, Networkwise tokenization seems to give the most consistent gains across different datasets and ROI definitions for both correlation fidelity and identification under open loop long horizon rollout. Visual and Default modes show the highest pattern fidelity, with visual being roughly zero point three two seven five to zero point three five one two.
Marcus: That’s a concrete number showing where the strongest information is coming from in terms of pattern preservation across different brain regions. And I see that the attention based coupling visualization from the Spatial Mixer shows similar structured connectivity across subjects, which suggests what they're seeing qualitatively isn't just an artifact of one specific person's data.
Yuki: That’s important because it hints at a shared underlying principle in how these networks interact dynamically, rather than just individual noise. It makes the model feel more grounded in some kind of population-level structure, even when working on individual fMRI data.
Paper summary: Ines: Overall, the BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction framework is showing strong gains over existing baselines on three benchmarks when using long horizon open loop rollout. As the rollout horizon gets longer, performance naturally degrades because of compounding error, but BrainVista manages to mitigate that drift through temporally consistent conditioning and structured cross network computation.
Marcus: The authors are clear about the trade-offs in their conclusion; they acknowledge that while their method handles drift better than some baselines, the performance still drops as time goes on. They are also flagging that extending this approach to naturalistic fMRI is non trivial because of that core hurdle—maintaining strict causality across those mismatched timescales between high frequency visual stimuli and the slow BOLD signals (Lahner et al., two thousand twenty-four; Chang et al <ref:2602.04512#pg2>., two thousand twenty-two) <ref:2602.04512#pg2>.
Yuki: So for anyone listening, this means we’re seeing a path forward where we can use these powerful autoregressive models to simulate brain dynamics in a way that respects the actual flow of information over time, not just static correlations. It moves us closer to understanding the brain as a true predictive system.
Ines: And looking at the future work, they plan to explore stronger tokenization decoding objectives and calibration under distribution shifts, which should further improve long horizon stability and cross dataset generalization. That’s where the real push for making these models truly generalizable will happen.
Marcus: I think the implication for large scale cohorts is that if we can get this kind of robust modeling, it could help us interpret how different groups of people process complex multimodal information in a way that respects their specific neural dynamics rather than just averaging everything out.
Yuki: It’s about moving beyond just seeing what correlations exist and actually building a model that predicts the sequence of events. That’s what this work is aiming for with BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction, page one of that work reads: "BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction Xuanhua Yin1 Runkai Zhao1* Lina Yao2,3 Weidong Cai1* one The University of Sydney, Australia two University of New South Wales, Australia three CSIRO’s Data61 xuanhua <ref:2602.04512#pg1,BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction Xuanhua Yin1>.yin@sydney.edu.au runkai.zhao@sydney.edu.au lina.yao@unsw.edu.au tom.cai@sydney.edu"
Conclusion: Ines: So, BrainVista is essentially taking that idea of the brain being a constantly running prediction engine—driven by all these sensory inputs—and trying to map out how those states evolve over time in a way that respects causality.
Marcus: It’s about using this multimodal autoregressive framework to model the causal evolution of brain states, and they did it using Networkwise Tokenizers and a Spatial Mixer Head.
Yuki: The authors are tackling the challenge of naturalistic fMRI by treating it like forecasting the next token in a sequence, incorporating video, audio, and text together.
Ines: They frame the modeling as time-aligned multimodal next-token forecasting under a past-only constraint, meaning you can only condition on what has already happened to predict what comes next.
Marcus: The results are pretty strong on the public benchmarks like Algonauts two thousand twenty-five and CineBrain, where BrainVista achieves state of the art encoding performance.
Yuki: They show substantial improvements in long horizon rollout settings, increasing pattern correlation by about thirty six point zero percent on one dataset and thirty three point three percent on another.
Ines: It’s interesting because they show these gains persist even when you change the ROI parcellations and the coordinate spaces used in the experiments.
Marcus: They also found that their spatial mixer head gives them the best correlation across different time horizons, with its margin growing with how long you look ahead, up to about seventeen point eight percent at a horizon of ten steps.
Yuki: This suggests that the way they structure those cross-network interactions is really capturing something fundamental about how these different brain areas talk to each other.
Ines: So what this means is that we’re seeing a model that can handle the complexity of multimodal data and still maintain a causal view of time in a dynamic system.
Marcus: It suggests that if we can build these kinds of models, they could help us interpret how different populations process complex information, respecting their specific neural dynamics.
Yuki: And they’re already looking ahead to future work on calibration under distribution shifts, which is a big step toward making these models truly generalizable across different people.
University of Sydney · University of New South Wales · CSIRO
q-bio.NC, cs.AI
Submitted: 2026-02-04
Updated: 2026-11-10
Comments: Accepted by NeurIPS 2026. 46 pages, 8 figures, 58 tables
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 86/100
The gist: Naturalistic fMRI characterizes the brain as a dynamic predictive engine driven by continuous sensory streams; BrainVista…
Key concepts
- Multimodal Autoregressive Framework
- This is a modeling approach where the system predicts the next part of a sequence (like brain activity) based on all previous parts and current external inputs (like video or audio). It treats brain states as tokens in a sequence, allowing it to forecast future conditions by learning the causal transitions between them.
- Networkwise Tokenizers
- These are specialized components designed to compress complex fMRI data from different brain regions into compact 'circuit tokens.' They act like lightweight encoders that learn how local brain networks reconstruct the original fMRI signal, helping the model understand functional heterogeneity across different areas.
- Spatial Mixer Head
- This mechanism is introduced to explicitly control how information flows between different brain networks. Instead of relying on implicit global attention, this head learns to manage and mix signals across these distinct functional units while respecting their original boundaries, improving cross-network modeling.
Terminology
Summary
Naturalistic fMRI characterizes the brain as a dynamic predictive engine driven by continuous sensory streams; BrainVista, a multimodal autoregressive framework designed to model the causal evolution of brain states, addresses this challenge by incorporating Networkwise Tokenizers and a Spatial Mixer Head
The Core Framework
BrainVista is introduced as a multimodal autoregressive framework designed to model the causal evolution of brain states It incorporates Networkwise Tokenizers to disentangle system-specific dynamics and a Spatial Mixer Head that captures inter-network information flow without compromising functional boundaries The framework is designed to model naturalistic fMRI responses from stimuli in a consistent and scalable manner
Modeling Causal Dynamics
The modeling of naturalistic fMRI is cast as a time-aligned multimodal next-token forecasting under a past-only constraint, where open-loop rollout iterates a learned one-step transition The autoregressive objective parameterizes the one-step transition to forecast the next cortical state from past brain history together with an exogenous stimulus stream that is assumed observed at inference An H-step rollout is obtained by recursively applying the same transition to previously predicted states
Multimodal Input Construction
Stimulus tokens are formed by encoding video, audio, and text with frozen encoders and aggregating them onto the fMRI grid These high-rate tokens are then downsampled onto the fMRI grid for causal prediction A shared stimulus token is formed by concatenating modality tokens and projecting them to a common feature space, denoted as τt s
Circuit Tokenization and Mixing
To address cortical functional heterogeneity, parcel-wise fMRI is compressed into compact circuit tokens via network-wise tokenizer–detokenizer pairs These tokenizers are lightweight twolayer MLPs trained to self-reconstruct fMRI signals within each functional network The whole-brain circuit token is formed by concatenating these network tokens to form the whole-brain circuit token τt f A Spatial Mixer Head is introduced to explicitly control cross-network mixing instead of relying on implicit global attention
Temporal Conditioning and Masking
To ensure strict temporal validity in multimodal conditioning, a Stimulus-to-Brain (S2B) masking mechanism is introduced to enforce a constraint to prevent temporal leakage and keep conditioning temporally consistent across training and inference This mask constrains selfattention so that circuit-token queries at time ti attend only to past indices tj ≤ ti and never to future-aligned stimuli
Experimental Validation
The framework is validated on three public naturalistic fMRI benchmarks, Algonauts 2025, CineBrain, and HAD BrainVista achieves state-of-the-art fMRI encoding performance In longhorizon rollout settings, the model yields substantial improvements over baselines, increasing pattern correlation by 36.0% and 33.3% on relative to the strongest baseline Algonauts 2025 and CineBrain, respectively The results show that BrainVista achieves the best or tied-best performance despite differing ROI parcellations and coordinate spaces
Key Findings on Robustness
The improvements persist under ρ, suggesting that the gains are not driven by a small set of extreme values and that the predicted dynamics better preserve the relative temporal ordering of neural responses The Spatial Mixer Head achieves the best pcorr across horizons, and its margin grows with H (e.g., ∼ +17.8% at H=20 versus removing the mixer) The combined setting yields the best correlation fidelity and identification accuracy, and the gap typically widens at longer horizons The full multimodal setting consistently performs best, while dropping any modality degrades pcorr, with the largest decline observed when video is removed The results also clarify trade-offs of past-only multimodal forecasting
Ablation Insights
Network-wise tokenization yields the most consistent gains across datasets and ROI definitions, improving both correlation fidelity and identification under open-loop long-horizon rollout The network-level pcorr breakdown shows that Visual and Default modes achieve the highest pattern fidelity (Visual: 0.3275–0.3512; Default: 0. p. 33015–0.3237) The attention-based coupling visualization from the Spatial Mixer exhibits similarly structured, non-uniform connectivity across subjects, suggesting that the qualitative behavior observed in the main paper is not driven by a single subject-specific artifact The full multimodal setting consistently performs best, while dropping any modality degrades pcorr
Conclusion and Future Directions
BrainVista achieves gains over baselines on three benchmarks, with the clearest improvements under long-horizon open-loop rollout As the rollout horizon grows, performance inevitably degrades due to compounding error, but BrainVista mitigates drift via temporally consistent conditioning and structured cross-network computation Future work will explore stronger tokenization/decoding objectives and calibration under distribution shifts to further improve long-horizon stability and cross-dataset generalization
References
Assran, M. et al. V-JEPA 2: Self-supervised video models enable understanding, prediction and planning Bao, G. p. 8, Zhang, Q. p. 8, Gong, Z., Wu, Z., and Miao, D. p. 8 MindSimulator: Exploring brain concept localization via synthetic fMRI Brown, T. p. 8 B., Mann, B., Ryder, N. p. 8, Subbiah, M. p. 8, Kaplan, J. p. 8 D., et al. Language models are few-shot learners Chang, C. p. 8 H. C., Nastase, S. p. 8 A., and Hasson, U. p. 6 p. 6 p. 6 Information flow across the cortical timescale hierarchy during narrative construction Chung, Y. p. 8, Zhang, Y. p. 8 p. 8, Han, W., Chiu, C., Qin, J., Pang, R., and Wu, Y. 2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training Clark, A. p. 8 Whatever next? p. 8 predictive brains, situated agents, and the future of cognitive science Darvishi Bayazi, M. p. 8 J., Ghonia, H. p. 8, Riachi, R. p. 8, Aristimunha, B. p. 8, Khorasani, A. p. 8, Arefin, M. p. 8 R., Darabi, A. p. 8, Dumas, G. p. 8, and Rish, I. p. 8 General-purpose brain foundation models for time-series neuroimaging data d’Ascoli, S. p. 8, Rapin, J. p. 8, Benchetrit, Y. p. 8, Banville, H. p. 8, and King, J. p. 8-R. TRIBE: TRImodal brain encoder for wholebrain fMRI response prediction Demirtas¸, M. p. 8, Burt, J. B., Helmer, M. p. 8, Ji, J. L., Adkinson, B. p. 8 D., Glasser, M. p. 1 F., Van Essen, D. p. 8 C., Sotiropoulos, S. p. 8 N., Anticevic, A. p. 8, and Murray, J. p. 6 D. Hierarchical heterogeneity across human cortex shapes large-scale neural dynamics Dong, Z. p. 8, Li, R., Wu, Y., Nguyen, T. p. 8 T., Chong, J. p. 8 p. 8 S. X., Ji, F., Tong, N. R. J., Chen, C. L. H., and Zhou, J. p. 1 p. 1 H. BrainJEPA: Brain dynamics foundation model with gradient positioning and spatiotemporal masking Dong, Z. p. 8, Li, R., Chong, J. S. X., Dehestani, N. p. 8, Teng, Y., Lin, Y., Li, Z., Zhang, Y., Xie, Y., Ooi, L. Q. R., Yeo, B. T. T., and Zhou, J. H. Brain Harmony: A multimodal foundation model unifying morphology and function into 1D tokens Esteban, O. p. 8 et al. fMRIPrep: A robust preprocessing pipeline for functional MRI Finn, E. p. 8 S., Shen, X., Scheinost, D. p. 8, Rosenberg, M. p. 8 D., Huang, J. p. 8, Chun, M. M., Papademetris, X. p. 8, and Constable, R. p. 8 T. Functional connectome fingerprinting: Identifying individuals using patterns of brain connectivity Friston, K. p. 8 A theory of cortical responses Gao, J. p. 8, Liu, Y., Yang, B., Feng, J., and Fu, Y. CineBrain: A large-scale multi-modal brain dataset during naturalistic audiovisual narrative processing Gifford, A. p. 8 T., Bersch, D. p. 8, St-Laurent, M. p. 8, Pinsard, B. p. 8 p. 8, Boyle, J. p. 8, Bellec, L. p. 8, Oliva, A. p. 8, Roig, G., and Cichy, R. p. 8 M. The Algonauts Project 2025 challenge: How the human brain makes sense of multimodal movies Glasser, M. p. 8 F., Coalson, T. p. 8 S., Robinson, E. p. 8 C., Hacker, C. p. 8 D., Harwell, J. p. 8, Yacoub, E. p. 8, Ugurbil, K. p. 8, Andersson, J. p. 8, Beckmann, C. p. 8 F., Jenkinson, M. p. 8, Smith, S. p. 8 M., and Van Essen, D. p. 8 C. A multi-modal parcellation of human cerebral cortex Granger, C. p. 8 W. J. Investigating causal relations by econometric models and cross-spectral methods Gratton, C. p. 8, Laumann, T. p. 9 O., Nielsen, A. p. 9 N., Greene, D. p. 9 J., Gordon, E. p. 9 M., Gilmore, A. p. 9 W., Nelson, S. p. 9 M., Coalson, R. p. 8 S., Snyder, A. p. 9 Z., Schlaggar, B. p. 9 L., Dosenbach, N. p. 9 U. F., and Petersen, S. p. 9 E.: Functional brain networks are dominated by stable group and individual factors Hasson, U. p. 9, Yang, E., Vallines, I. p. 9, Heeger, D. p. 9 p. 9 J., and Rubin, N. p. 9 A hierarchy of temporal receptive windows in human cortex Honey, C. p. 9 J., Thesen, T. p. 9, Donner, T. p. 9 H., Silbert, L. p. 9 p. 9 J., Carlson, C. p. 9 E., Devinsky, O. p. 9, Doyle, W. p. 9 K., Rubin, N. p. 9, Heeger, D. J., and Hasson, U.: Slow cortical dynamics and the accumulation of information over long timescales Lahner, B. p. 9, Dwivedi, K. p. 9, Iamshchinina, P. p. 9, et al. Modeling short visual events through the BOLD moments video fmri dataset and metadata Lerner, Y. p. 9, Honey, C. p. 6 J., Silbert, L. J., and Hasson, U.: Topographic mapping of a hierarchy of temporal receptive windows using a narrated story Li, Y. p. 9, Xu, L., Zuo, L., Chang, Y. p. 1, Ding, Z., Anderson, A. p. 9 W., Schilling, K. p. 9 G., Gore, J. C., and Gao, Y. p. 1 BrainVAE: Exploring the role of white matter BOLD in preclinical Alzheimer’s disease classification Liu, Y. p. 9, Hu, T., Zhang, H. p. 8, Wu, H., Wang, S., Ma, L., and Long, M. p. 1 p. 1 p. 1 itransformer: Inverted transformers are effective for time series forecasting Liu, Y. p. 9, Qin, G., Huang, X. p. 8, Wang, J., and Long, M. TIMERXL: Long-context transformers for unified time series forecasting Logothetis, N. p. 9 K., Pauls, J. p. 9, Augath, M. p. 9, Trinath, T. p. 9, and Oeltermann, A. p. 9 Neurophysiological investigation of the basis of the fMRI signal Mai, W. p. 9, Wu, J., Zhu, Y., Yao, Z., Zhou, D., Luo, A. F., Zheng, Q. p. 9, Ouyang, W. p. 9, and Song, C. p. 9 SynBrain: Enhancing visual-to-fMRI synthesis via probabilistic representation learning Murray, J. p. 9 D., Bernacchia, A. p. 9, Roy, N. A., Constantinidis, C. p. 9, Romo, R., and Wang, X. p. 1-J. A hierarchy of intrinsic timescales across primate cortex Nie, Y. p. 9, Nguyen, N. p. 8 H., Sinthong, P. p. 9, and Kalagnanam, J. p. 9 R. A time series is worth 64 words: Long-term forecasting with transformers Nishimura, Y. p. 9, Sawayama, M. p. 9, Yamashita, A. p. 9, Nakayama, H. p. 9, and Amano, K. p. 9 BrainCodec: Neural fMRI codec for the decoding of cognitive brain states Ortega Caro, J. p. 9, Fonseca, A. p. 9 H. d. O., Rizvi, S. p. 9 A., Rosati, M. p. 9, et al. BrainLM: A foundation model for brain activity recordings Paugam, F. p. 9, Pinsard, B., Lajoie, G. p. 9, and Bellec, P. p. 8 A benchmark of individual auto-regressive models in a massive fMRI dataset Qiu, W. p. 9, Huang, Z. p. 8, Hu, H., Feng, A., Yan, Y., and Ying, R. p. 9 Mindllm: A subject-agnostic and versatile model for fmri-to-text decoding Schaefer, A. p. 9 et al. Local-Global Parcellation of the human cerebral cortex from intrinsic functional connectivity Scotti, P. p. 9, Banerjee, A. p. 9, Goode, J. p. 9, Shabalin, S. p. 9, Nguyen, A. p. 8, Cohen, E. p. 9, Dempster, A. p. 9, Verlinde, N. p. 9, Yundler, E. p. 9, Weisberg, D. p. 9, Norman, K. p. 9, and Abraham, T. p. 9 Reconstructing the mind’s eye: fmri-to-image with contrastive learning and diffusion priors Takagi, Y. p. 9 and Nishimoto, S. p. 2 High-resolution image reconstruction with latent diffusion models from human brain activity Wang, E. p. 9 Y., Fahey, P. G., Ding, Z., Papadopoulos, S., Ponder, K., Weis, M. A., Chang, A. p. 1, Muhammad, T., Patel, S., Ding, Z., Tran, D., Fu, J.: Foundation model of neural activity predicts response to new stimulus types Wang, M. p. 9, Xia, J., Ye, W., Liu, E., Peng, K., Feng, J., Liu, Q., and Wen, H. SLIM-Brain: A data- and trainingefficient foundation model for fMRI data analysis Wei, X. p. 8, Zhao, K., Jiao, Y., He, L., and Zhang, Y. p. 8 A brain graph foundation model: Pre-training and prompttuning for any atlas and disorder Wu, H. p. 8, Hu, T., Liu, Y., Zhou, H., Wang, J., and Long, M. Timesnet: Temporal 2d-variation modeling for general time series analysis Xia, W. p. 8, Charette, R. d., Oztireli, C., and Xue, J.-H. UMBRAE: Unified multimodal brain decoding Xia, W. p. 2, Zhang, X. p. 8, Hu, D., Cheng, J., Yu, X., Yin, W., Wu, Z., Wang, L., Lin, W., and Li, G. Triplet longitudinal masked autoencoder for predicting individualized functional connectome development during infancy Yang, Y. p. 9 and Wolfers, T. p. 1 Hierarchical characterization of brain dynamics via state space-based vector quantization Yeo, B. p. 6 T. T., et al.: The organization of the human cerebral cortex estimated by intrinsic functional connectivity Yin, X. p. 9, Zhao, R., and Cai, W. p. 0: Improving multimodal brain encoding model with dynamic subject-awareness routing Zhao, Y. p. 2, Nie, D., Chen, G., Wu, X., Zhang, D. p. 8, and Wen, X. TARDRL: Task-aware reconstruction for dynamic representation learning of fMRI Zhou, M. p. 2, Gong, Z., Dai, Y., Wen, Y., Liu, Y., Zhen, Z., et al. A large-scale fMRI dataset for human action recognition
Table 1 shows the main results under open-loop rollout with a rollout horizon of 20 steps on Algonauts 2025, CineBrain, and HAD BrainVista achieves the best or tied-best performance despite differing ROI parcellations and coordinate spaces Table 7 reports the Yeo-7 network-level pcorr values corresponding to the maps in Fig. 4(b) under H=10 rollout Across both datasets, performance decreases as the rollout horizon increases, reflecting compounding error under open-loop prediction BrainVista remains best at every horizon and its pcorr margin does not collapse, supporting reduced long-horizon drift under causal rollout Table 8 reports cross-subject generalizability on the remaining two test subjects (Sub-02 and Sub-05) to complement the main-paper table The overall pattern is consistent with the main results: BrainVista shows strong gains over the baseline across training subjects, and the improvements persist under heterogeneous train–test combinations rather than being driven by a single favorable subject pairing Table 7 reports the Yeo-7 network-level pcorr values under H=10 rollout Visual and Default modes achieve the highest pattern fidelity (Visual: 0. p. 5
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Multimodal causal forecasting for naturalistic brain dynamics: BrainVista can model BOLD evolution as a
time-aligned multimodal autoregressive prediction task,
allowing it to simulate future neural trajectories byconditioning on past brain tokens and aligned stimuli to recursively simulate future trajectories.
-
Strict temporal conditioning: The system implements a
Stimulus-to-Brain (S2B) masking mechanism to synchronize high-frequency sensory stimuli with hemodynamically filtered signals, enabling strict, history-only causal conditioning.
This preventstemporal leakage
and ensures that each prediction isconditioned only on the stimulus history available up to the current time step.
-
Network-aware representation learning: By designing
Networkwise Tokenizers as MLP autoencoders trained to self-reconstruct fMRI signals within each functional network,
the system preserves functional specificity and supportscross-subject transfer
by producing compact tokens for each specialized network. -
Explicit cross-network modeling: The introduction of a
Spatial Mixer Head
allows the model toexplicitly control cross-network mixing, instead of relying on implicit global attention that can over-smooth cross-ROI spatial activation patterns and reduce network-specific contrast.
This enables the modeling ofdirectional cross-network communication without blurring functional boundaries.
-
Improved long horizon stability: The framework is designed for open-loop rollout, where the model
yield[s] substantial improvements over baselines, increasing pattern correlation by 36.0% and 33.3% on relative to the strongest baseline Algonauts 2025 and CineBrain, respectively
in longhorizon settings. This suggests it maintains stability under error accumulation throughtemporally consistent conditioning.
-
Enhanced cross-subject generalization: Ablation studies show that BrainVista
improves transfer over TRIBE across all source–target pairings,
indicating that the model encourages representations that aresystemspecific yet more subject-invariant
by reducing overfitting to subject noise patterns.
Sources
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- CineBrain: A Large-Scale Multi-Modal Brain Dataset During Naturalistic Audiovisual Narrative Processing
- The Algonauts Project 2025 Challenge: How the Human Brain Makes Sense of Multimodal Movies
- The Llama 3 Herd of Models
- BrainCodec: Neural fMRI codec for the decoding of cognitive brain states
- SLIM-Brain: A Data- and Training-Efficient Foundation Model for fMRI Data Analysis
- A Brain Graph Foundation Model: Pre-Training and Prompt-Tuning across Broad Atlases and Disorders
- Hierarchical Characterization of Brain Dynamics via State Space-based Vector Quantization
- Improving Multimodal Brain Encoding Model with Dynamic Subject-awareness Routing
Related papers
- Self-Supervised Learning of Local Predictive Directions as a Neural Computation
- Synchronized Hearts, Desynchronized Brains: A Nonlinear Dynamics Perspective for Function and Failure
- Message interaction dynamics covary with brain volume across mammalian connectomes
- Planning as Dynamics Relaxation: Hippocampal Recurrent Network Realizes Optimal Goal-Directed Navigation
- The Platonic brain bridge hypothesis: brain networks as architectural priors for multimodal large language models
- BrainPFN: Amortized Brain-Behaviour Prediction via Prior-Data Fitted Networks