Human and Machine Annotations Predict Brain Responses to Naturalistic Movies Across Development

summary

The gist

Naturalistic movie watching provides a powerful framework for studying dynamic brain function, but linking movie content to neural responses typically relies on labor-intensive human annotation…

This episode discusses

The paper

Human and Machine Annotations Predict Brain Responses to Naturalistic Movies Across Development · Read on arXiv

Yiran Huang, Zeyong Zhang, Rounak Rahman, Bharat Biswal, Xin Di, Mengjia Xu

New Jersey Institute of Technology

Transcript

Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.

Ines: We're looking at the paper: "Human and Machine Annotations Predict Brain Responses to Naturalistic Movies Across Development".

Marcus: Here's the gist of the paper. Naturalistic movie watching provides a powerful framework for studying dynamic brain function, but linking movie content to neural responses typically relies on labor-intensive human annotation <ref:2611.13637#pg2>.

Ines: MLLM-generated annotations capture meaningful aspects of human judgments and neural responses, supporting MLLMs as a scalable complement to, rather than a replacement for, human annotation in naturalistic neuroimaging <ref:2611.13637#pg3>.

Marcus: Investigating Human-Machine Correspondence The study compared MLLM-generated theory-of-mind (ToM) and valence ratings with human ratings across three animated movies to assess behavioral agreement and the consistency of human–machine neural correspondence <ref:2611.13637#pg5>.

Ines: The findings indicated that MLLM ratings partially tracked human judgments, but their neural correspondence differed by construct and model <ref:2611.13637#pg5>.

Title and authors: Yuki: So it’s not a perfect match across the board then? What does this mean for our understanding of how the brain processes these kinds of social and emotional cues when we only have AI labels as a starting point?

Ines: It suggests that while MLLMs capture some meaningful aspects of human judgment, they aren't just perfectly mirroring what a human feels or thinks. The study found that theory of mind showed the most consistent overlap between human and machine ratings in certain areas of the visual cortex.

Marcus: That’s interesting because it tells us where the AI is doing a better job capturing something specific—in this case, how people are thinking about others—compared to general emotional responses like valence.

Ines: Exactly. But for valence, the neural correspondence varied a lot depending on the movie itself, which means you can't just assume an AI label will always map cleanly onto a specific brain region for every emotion.

Yuki: So, if we want to study how kids or young people process these complex social scenes without exhausting human annotators, this paper suggests MLLMs are a useful tool but not the final word.

The paper's summary: Ines: The paper sets up a multimodal encoding framework that combines three types of information to predict brain activity from fMRI data. They used low-level visual features from something like AlexNet, language-semantic representations from the descriptions generated by the MLLM, and social-cognitive features like theory of mind and valence ratings.

Marcus: That fusion of visual input, language meaning, and those direct social judgments into a single representation was key to seeing how it predicted brain responses across different groups. They used a banded ridge regression encoding model for that prediction.

Yuki: I want to know how the results show up developmentally since that was a big focus in their setup. How did the features change as people got older?

Ines: The analysis showed that the low-level visual features had the strongest and most widespread predictive power across brain regions, but those language descriptions were more focused, concentrating their prediction in dorsal attention and visual areas.

Marcus: And there was a clear developmental pattern there too; for example, the encoding of the description feature actually increased with age in dorsal attention and visual networks. They found this increase was strongest in the oldest group.

The paper's improvements: Yuki: So what does this suggest we should do next, beyond just using MLLMs as a supplement? What are the specific improvements they suggest for future research?

Ines: They point out that while the MLLM ratings tracked human ones closely overall, there’s still a lot of room for improvement in understanding continuous socio-affective judgments. They suggest using richer, more multidimensional social features instead of just a small set of scalar ratings like valence and ToM.

Marcus: And they also look at extending this framework to more advanced multimodal models like InternVL3 or GLM-4 point 5V, which suggests the method itself can be adapted for newer, more complex AI systems.

Ines: Another big suggestion is that this entire framework could be extended to neurodevelopmental and clinical populations, specifically mentioning individuals with autism spectrum disorder, which shows they are thinking about how this applies beyond just healthy development.

Conclusion: Marcus: So, to wrap up, the paper on "Human and Machine Annotations Predict Brain Responses to Naturalistic Movies Across Development" confirms that MLLMs can provide scalable annotations for naturalistic fMRI data by tracking human judgments, although we have to be careful because the neural signals they capture vary depending on whether you are looking at theory of mind or valence.

Ines: It’s a solid finding that the machine-generated features explained meaningful neural responses whose spatial profile shifted with age, showing feature-specific developmental differences in how we process information during movie viewing.

Yuki: From my perspective, it supports the idea that narrative processing isn't just one thing; it’s a complex interplay between visual input and those higher-order social and cognitive systems that mature over time.

Marcus: Yeah, so the implication is that MLLMs are a scalable way to get more data on these complex processes without needing massive human annotation efforts for every single movie.

Ines: That’s the core idea: using MLLMs as a complement, not a replacement, for human work in naturalistic neuroimaging. We'll keep an eye on how they apply this to those other big multimodal models like InternVL3 later on.

More episodes

← Home