Human and Machine Annotations Predict Brain Responses to Naturalistic Movies Across Development

arXiv:2611.13637 · q-bio.NC · Submitted 2026-11-10 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.

Ines: We're looking at the paper: "Human and Machine Annotations Predict Brain Responses to Naturalistic Movies Across Development".

Marcus: Here's the gist of the paper. Naturalistic movie watching provides a powerful framework for studying dynamic brain function, but linking movie content to neural responses typically relies on labor-intensive human annotation <ref:2611.13637#pg2>.

Ines: MLLM-generated annotations capture meaningful aspects of human judgments and neural responses, supporting MLLMs as a scalable complement to, rather than a replacement for, human annotation in naturalistic neuroimaging <ref:2611.13637#pg3>.

Marcus: Investigating Human-Machine Correspondence The study compared MLLM-generated theory-of-mind (ToM) and valence ratings with human ratings across three animated movies to assess behavioral agreement and the consistency of human–machine neural correspondence <ref:2611.13637#pg5>.

Ines: The findings indicated that MLLM ratings partially tracked human judgments, but their neural correspondence differed by construct and model <ref:2611.13637#pg5>.

Title and authors: Yuki: So it’s not a perfect match across the board then? What does this mean for our understanding of how the brain processes these kinds of social and emotional cues when we only have AI labels as a starting point?

Ines: It suggests that while MLLMs capture some meaningful aspects of human judgment, they aren't just perfectly mirroring what a human feels or thinks. The study found that theory of mind showed the most consistent overlap between human and machine ratings in certain areas of the visual cortex.

Marcus: That’s interesting because it tells us where the AI is doing a better job capturing something specific—in this case, how people are thinking about others—compared to general emotional responses like valence.

Ines: Exactly. But for valence, the neural correspondence varied a lot depending on the movie itself, which means you can't just assume an AI label will always map cleanly onto a specific brain region for every emotion.

Yuki: So, if we want to study how kids or young people process these complex social scenes without exhausting human annotators, this paper suggests MLLMs are a useful tool but not the final word.

The paper's summary: Ines: The paper sets up a multimodal encoding framework that combines three types of information to predict brain activity from fMRI data. They used low-level visual features from something like AlexNet, language-semantic representations from the descriptions generated by the MLLM, and social-cognitive features like theory of mind and valence ratings.

Marcus: That fusion of visual input, language meaning, and those direct social judgments into a single representation was key to seeing how it predicted brain responses across different groups. They used a banded ridge regression encoding model for that prediction.

Yuki: I want to know how the results show up developmentally since that was a big focus in their setup. How did the features change as people got older?

Ines: The analysis showed that the low-level visual features had the strongest and most widespread predictive power across brain regions, but those language descriptions were more focused, concentrating their prediction in dorsal attention and visual areas.

Marcus: And there was a clear developmental pattern there too; for example, the encoding of the description feature actually increased with age in dorsal attention and visual networks. They found this increase was strongest in the oldest group.

The paper's improvements: Yuki: So what does this suggest we should do next, beyond just using MLLMs as a supplement? What are the specific improvements they suggest for future research?

Ines: They point out that while the MLLM ratings tracked human ones closely overall, there’s still a lot of room for improvement in understanding continuous socio-affective judgments. They suggest using richer, more multidimensional social features instead of just a small set of scalar ratings like valence and ToM.

Marcus: And they also look at extending this framework to more advanced multimodal models like InternVL3 or GLM-4 point 5V, which suggests the method itself can be adapted for newer, more complex AI systems.

Ines: Another big suggestion is that this entire framework could be extended to neurodevelopmental and clinical populations, specifically mentioning individuals with autism spectrum disorder, which shows they are thinking about how this applies beyond just healthy development.

Conclusion: Marcus: So, to wrap up, the paper on "Human and Machine Annotations Predict Brain Responses to Naturalistic Movies Across Development" confirms that MLLMs can provide scalable annotations for naturalistic fMRI data by tracking human judgments, although we have to be careful because the neural signals they capture vary depending on whether you are looking at theory of mind or valence.

Ines: It’s a solid finding that the machine-generated features explained meaningful neural responses whose spatial profile shifted with age, showing feature-specific developmental differences in how we process information during movie viewing.

Yuki: From my perspective, it supports the idea that narrative processing isn't just one thing; it’s a complex interplay between visual input and those higher-order social and cognitive systems that mature over time.

Marcus: Yeah, so the implication is that MLLMs are a scalable way to get more data on these complex processes without needing massive human annotation efforts for every single movie.

Ines: That’s the core idea: using MLLMs as a complement, not a replacement, for human work in naturalistic neuroimaging. We'll keep an eye on how they apply this to those other big multimodal models like InternVL3 later on.

Yiran Huang, Zeyong Zhang, Rounak Rahman, Bharat Biswal, Xin Di, Mengjia Xu

New Jersey Institute of Technology

q-bio.NC

Submitted: 2026-11-10

Updated: 2026-11-10

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 79/100

The gist: Naturalistic movie watching provides a powerful framework for studying dynamic brain function, but linking movie content to neural responses typically relies on labor-intensive human annotation…

Summary

Naturalistic movie watching provides a powerful framework for studying dynamic brain function, but linking movie content to neural responses typically relies on labor-intensive human annotation <ref:2611. p. 013637#pg2>. MLLM-generated annotations capture meaningful aspects of human judgments and neural responses, supporting MLLMs as a scalable complement to, rather than a replacement for, human annotation in naturalistic neuroimaging <ref:2611. p. 113637#pg3>.

Investigating Human-Machine Correspondence

The study compared MLLM-generated theory-of-mind (ToM) and valence ratings with human ratings across three animated movies to assess behavioral agreement and the consistency of human–machine neural correspondence <ref:2611.13637#pg5>. The findings indicated that MLLM ratings partially tracked human judgments, but their neural correspondence differed by construct and model <ref:2611. p. 013637#pg5>. Specifically, ToM showed the most consistent human–machine overlap in the visual cortex, whereas valence correspondence was largely movie dependent <ref:2611. p. 013637#pg5>.

Encoding Model Construction

A multimodal encoding framework was developed integrating three complementary feature spaces: (1) low-level visual features extracted from a pre-trained deep convolutional network (AlexNet) <ref:2611.13637#pg5>, (2) language-semantic representations derived from MLLM-generated scene descriptions <ref:2611. p. 313637#pg6>, and (3) social-cognitive features including ToM, valence, and contextual variables such as the number of onscreen people and face presence <ref:2611. p. 313637#pg5>. These feature streams were fused into a multimodal representation which was then used to predict subject-specific fMRI BOLD responses using a banded ridge regression encoding model <ref:2611. p. 413637#pg6>.

Neural Correlates and Developmental Effects

Analysis of ROI-wise correlations revealed that visual features yielded the strongest and most widespread prediction, while scene descriptions were more spatially selective, concentrated in dorsal-attention and visual regions <ref:2611.13637#pg5>. Age-group comparisons suggested feature-specific developmental differences, such as scene-description encoding being higher in the oldest group in dorsal-attention cortex, whereas visual-feature encoding peaked in salience/ventral-attention networks in the middle group <ref:2611. p. 113637#pg3>.

Feature Space Sensitivity and Age Variation

Feature correlations showed clear movie-specific structure, with low-level social features being strongly coupled with visual and description features in The Present and Despicable Me, but weaker in Partly Cloudy <ref:2611.13637#pg9>. Furthermore, the encoding of the description feature increased with age in dorsal attention and visual networks, being strongest in the oldest participants <ref:2611. p. 013637#pg5>. This developmental increase was robust to the choice of chunk length used for movieboundary-aware cross-validation <ref:2611. p. 913637#pg6>.

Conclusion on MLLM Utility

Overall, MLLM ratings tracked human ratings closely, and the machine-generated features explained meaningful neural responses whose spatial profile varied with age, indicating feature-specific developmental differences <ref:2611.13637#pg21>. Together, these results suggest that MLLMs can serve as a scalable complement to human annotation for studying perceptual, semantic, and social processing during naturalistic viewing <ref:2611. p. 213637#pg26>.

Human and Machine Correspondence in Ratings and Neural Responses

The selected machine-generated time series showed partial tracking of human judgments over the full movie trajectory, with a correlation of r = 0.740 for valence ratings on Despicable Me <ref:2611. p. 313637#pg13>. For ToM, the most consistent human–machine correspondence across both models was observed in bilateral lateral occipitotemporal cortex near MT+, rather than in early visual cortex <ref:2611. p. 013637#pg21>. Valence correspondence was more stimulus-dependent than ToM correspondence, with human ratings being far less consistent across movies than model ratings <ref:2611. p. 013637#pg5>.

Neural Encoding of MLLM-Derived Representations

The description feature space showed that it captured perceptual as well as semantic aspects of the movie rather than a purely higher-order semantic representation, being concentrated in dorsal attention and visual regions <ref:2611.13637#pg24>. Increasing the dimensionality of the description features from 64 to 128 dimensions was associated with higher encoding performance and a broader cortical distribution, including control and frontal regions <ref:2611. p. 013637#pg6>.

Developmental Differences in Feature Encoding

The low-level visual feature showed no age-related difference within the visual network, but increased from the young age group into the middle one in the salience, control, and limbic networks <ref:2611.13637#pg5>. This pattern suggests that it is not the early visual representations themselves that change, but rather the downstream integration of that visual information into attentional, executive, and affective systems that keeps maturing into adolescence <ref:2611. p. 313637#pg25>.

Considerations and Future Directions

Future work could extend this framework to broader multimodal models such as InternVL3 [76] and GLM-4.5V [77], and use richer, more multidimensional social features rather than a small set of scalar ratings <ref:2611. p. 113637#pg25>. An important future direction is to extend this framework to neurodevelopmental and clinical populations, such as individuals with autism spectrum disorder <ref:2611.13637#pg25>.

Declarations

The authors declare that they have no competing interests <ref:2611.13637#pg6>. This work was supported by the DOE SEA-CROGS project (DE-SC0023191) and the AFOSR project (FA955024-1-0231) to Y. p. 3H., Z.Z., and M.X.; the U.S. National Institutes of Health (R15MH125332) to X.D.; and the New Jersey Governor’s Council for Medical Research and Treatment of Autism (CAUT25BRP005) to X. p. 0D.<ref:2611.13637#pg6>. The authors acknowledge George B. p. 0 Hanna for assistance with behavioral data collection and the computing resources provided by the High Performance Computing (HPC) facility at NJIT<ref:2611.13637#pg6>.

References

[1] Hasson, U., Nir, Y., Levy, I., Fuhrmann, G., Malach, R.: Intersubject synchronization of cortical activity during natural vision. p. 2 science 303(5664), 1634–1640 (2004)<ref:2611. p. 013637#pg28>.

[2] Nastase, S.A., Liu, Y.-F., Hillman, H., Zadbood, A., Hasenfratz, L., Keshavarzian, N., Chen, J., Honey, C.J., Yeshurun, Y., Regev, M., et al.: The “narratives” fmri dataset for evaluating models of naturalistic language comprehension. p. 0 Scientific data 8(1), 250 (2021)<ref:2611. p. 013637#pg28>.

[3] Jindal, P., Khandelwal, A., Gupta, M., Raju, B.S., Oota, S.R., Chakraborty, T.: How does longer temporal context enhance multimodal narrative video processing in the brain? p. 1 arXiv preprint arXiv:2602. p. 007570 (2026)<ref:2611. p. 013637#pg28>.

[4] Di, X., Zhang, Z., Xu, T., Biswal, B. p. 0 p. 0B.: Dynamic and stationary brain connectivity during movie watching as revealed by functional mri. p. 1 Brain Structure and Function 227(7), 2299–2312 (2022)<ref:2611. p. 013637#pg5>.

[5] Huang, Y., Nouranizadeh, A., Ahrends, C., Xu, M.: BrainATCL: Adaptive temporal brain connectivity learning for functional link prediction and age estimation. p. 1 In: Proceedings of The 9th International Conference on Medical Imaging with Deep Learning, PMLR, pp. 3947–3970 (2026)<ref:2611. p. 013637#pg6>.

[6] Huth, A.G., De Heer, W.A., Griffiths, T.L., Theunissen, F.E., Gallant, J.L.: Natural speech reveals the semantic maps that tile human cerebral cortex. p. 0 Nature 532(7600), 453–458 (2016)<ref:2611. p. 013637#pg29>.

[7] Saxe, R., Kanwisher, N.: People thinking about thinking people: the role of the temporo-parietal junction in ”theory of mind”. p. 4 In: Social Neuroscience, pp. p. 1 171–182. Psychology Press, New York (2013)<ref:2611. p. 013637#pg29>.

[8] Frith, C.D., Frith, U.: The neural basis of mentalizing. p. 0 Neuron 50(4), 531–534 (2006)<ref:2611. p. 013637#pg29>.

[9] Richardson, H., Lisandrelli, G., Riobueno-Naylor, A., Saxe, R.: Development of the social brain from age three to twelve years. p. 3 Nature communications 9(1), 1027 (2018)<ref:2611. p. 013637#pg29>.

[10] Russell, J.A.: A circumplex model of affect. p. 0 Journal of personality and social psychology 39(6), 1161 (1980)<ref:2611. p. 013637#pg29>.

[11] Lindquist, K.A., Wager, T.D., Kober, H., Bliss-Moreau, E., Barrett, L.F.: The brain basis of emotion: a meta-analytic review. p. 0 Behavioral and brain sciences 35(3), 121–143 (2012)<ref:2611. p. 313637#pg29>.

[12] Cowen, A.S., Keltner, D.: Self-report captures 27 distinct categories of emotion bridged by continuous gradients. p. 2 Proceedings of the national academy of sciences 114(38), 7900–7909 (2017)<ref:2611. p. 013637#pg29>.

[13] Bradley, M.M., Lang, P.J.: Measuring emotion: the self-assessment manikin and the semantic differential. p. 1 Journal of behavior therapy and experimental psychiatry 25(1), 49–59 (1994)<ref:2611. p. 013637#pg29>.

[14] Jacoby, N., Bruneau, E., Koster-Hale, J., Saxe, R.: Localizing pain matrix and theory of mind networks with both verbal and non-verbal stimuli. p. 1 NeuroImage 126, 39–48 (2016)<ref:2611. p. 013637#pg29>.

[15] Skerry, A.E., Saxe, R.: Neural representations of emotion are organized around abstract event features. p. 0 Current biology 25(15), 1945–1954 (2015)<ref:2611. p. 013637#pg29>.

[16] Wellman, H.M., Cross, D. p. 8, Watson, J.: Meta-analysis of theory-of-mind development: The truth about false belief. p. 1 Child development 72(3), 655–684 (2001)<ref:2611. p. 013637#pg29>.

[17] Strachan, J.W., Albergo, D., Borghini, G., Pansardi, O., Scaliti, E., Gupta, S., Saxena, K., Rufo, A., Panzeri, S., Manzi, G.: Testing theory of mind in large language models and humans. p. 1 Nature human behaviour 8(7), 1285–1295 (2024)<ref:2611. p. 013637#pg31>.

[18] Gilardi, F., Alizadeh, M., Kubli, M.: Chatgpt outperforms crowd workers for text-annotation tasks. p. 2 Proceedings of the National Academy of Sciences 120(30), 2305016120 (2023)<ref:2611. p. 013637#pg5>.

[19] Boji´c, L., Zagovora, O., Zelenkauskaite, A., Vukovi´c, V., Cabarkapa, M., Vesel- ˇ jevi´c Jerkovi´c, S., Jovanˇcevi´c, A.: Comparing large language models and human annotators in latent content analysis of sentiment, political leaning, emotional intensity and sarcasm. p. 1 Scientific reports 15(1), 11477 (2025)<ref:2611. p. 013637#pg31>.

[20] Alrasheed, H., Alghihab, A., Pentland, A., Alghowinem, S.: Evaluating the capacity of large language models to interpret emotions in images. p. 2 Plos one 20(6), 0324127 (2025)<ref:2611. p. 013637#pg31>.

[21] Santavirta, S., Wu, Y., Suominen, L., Nummenmaa, L.: Gpt-4v shows humanlike social perceptual capabilities at phenomenological and neural levels. p. 2 Imaging Neuroscience 3, (2025)<ref:2611. p. 013637#pg31>.

[22] Ying, L., Wu, J., Wang, Y.S., Aarya, S., Buschoff, L.M.S., Chen, H., Collins, K.M., Varda, A., Fu, S., Houlihan, S.D., et al.: Coggym: Towards large-scale comparative evaluation of human and machine cognition. p. 2 arXiv preprint arXiv:2609. p. 021259 (2026)<ref:2611. p. 013637#pg31>.

[23] Kay, K.N., Naselaris, T., Prenger, R.J., Gallant, J.L.: Identifying natural images from human brain activity. p. 2 Nature 452(7185), 352–355 (2008)<ref:2611. p. 013637#pg5>.

[24] Nishimoto, S., Vu, A.T., Naselaris, T., Benjamini, Y. p. 8, Yu, B., Gallant, J.L.: Reconstructing visual experiences from brain activity evoked by natural movies. p. 2 Current biology 21(19), 1641–1646 (2011)<ref:2611. p. 013637#pg29>.

[25] Sohn, W., Di, X., Liang, Z., Zhang, Z., Biswal, B.B.: Explorations of using a convolutional neural network to understand brain activations during movie watching. p. 3 Psychoradiology 4, 021 (2024)<ref:2611. p. 013637#pg31>.

[26] Huth, A.G., Nishimoto, S., Vu, A.T., Gallant, J.L.: A continuous semantic space describes the representation of thousands of object and action categories across the human brain. p. 2 Neuron 76(6), 1210–1224 (2012)<ref:2611. p. 013637#pg31>.

[27] Lee Masson, H., Chang, L., Isik, L.: Multidimensional neural representations of social features during movie viewing. p. 2 Social Cognitive and Affective Neuroscience 19(1), 030 (2024)<ref:2611. p. 113637#pg29>.

[28] d’Ascoli, S., Rapin, J., Benchetrit, Y., Brooks, T., Begany, K., Raugel, J., Banville, H., King, J.-R.: A foundation model of vision, audition, and language for in-silico neuroscience. p. 1 arXiv preprint arXiv:2605. p. 004326 (2026)<ref:2611. p. 013637#pg29>.

[29] Schad, D.C., Dixit, S., Keck, J., Studenyak, V., Shpilevoi, A., Bicanski, A.: Vibe: Video-input brain encoder for fmri response modeling. p. 7 arXiv preprint arXiv:2507. p. 017958 (2025)<ref:2611. p. 013637#pg29>.

[30] Tang, J., Du, M., Vo, V., Lal, V., Huth, A.: Brain encoding models based on multimodal transformers can transfer across language and vision. p. 2 Advances in neural information processing systems 36, 29654–29666 (2023)<ref:2611. p. 113637#pg29>.

[31] Small, H., Masson, H.L., Mostofsky, S., Isik, L.: Vision and language representations in multimodal ai models and human social brain regions during natural movie viewing. p. 2 In: UniReps: 2nd Edition of the Workshop on Unifying Representations in Neural Models (2024)<ref:2611. p. 213637#pg31>.

[32] Small, H., Lee Masson, H., Wodka, E., Mostofsky, S., Isik, L.: Ubiquitous cortical sensitivity to visual information during naturalistic, audiovisual movie viewing (2025)<ref:2611. p. 213637#pg29>.

[33] Li, M.: Mapping the movie-watching brain with ai-derived semantics. p. 0 Imaging Neuroscience 4, (2026)<ref:2611.13637#pg29>.

[34] Garcia, K., McMahon, E., Conwell, C., Bonner, M., Isik, L.: Modeling dynamic social vision highlights gaps between deep learning and humans. p. 1 In: International Conference on Learning Representations, vol. p. 1 2025, pp. 71337–71362 (2025)<ref:2611. p. 013637#pg5>.

[35] You, Z., Nie, S., Zhang, X., ZHOU, J., Lu, Z., Wen, J.-R., Li, C.: Llada-v: Large language diffusion models with visual instruction tuning. p. 6 In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. p. 2 10093–10105 (2026)<ref:2611. p. 013637#pg6>.

[36] Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. p. 8 arXiv preprint arXiv:2511. p. 021631 (2025)<ref:2611. p. 013637#pg6>.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. MLLMs can provide scalable socio-affective and semantic annotations for naturalistic fMRI data by generating theory of mind (ToM) and valence ratings, which partially track human judgments, offering a scalable complement to human annotation.

  2. A multimodal encoding framework integrating low-level visual features from AlexNet, language-semantic representations from MLLM descriptions, and social-cognitive features (ToM and valence) can be used to predict subject-specific fMRI BOLD responses.

  3. The system can distinguish feature spaces by their spatial profiles; for instance, description features are concentrated in dorsal attention and visual regions, while low-level visual features show widespread predictive performance across cortex.

  4. The encoding performance of the description feature is developmentally sensitive, with it increasing with age in dorsal attention and visual networks, suggesting that an automatically produced description of the ongoing narrative is sufficient to recover age-related change in higher-order cortex.

  5. The system can capture stimulus-dependent neural correspondence by analyzing cross-movie ROI correlations, revealing that valence correspondence was more stimulus-dependent than ToM correspondence.

Sources

Related papers