Foundation Models as Modular Tools for fMRI Analysis

summary

The gist

Foundation models (FMs) for the brain are rapidly growing in number and each captures a different aspect of the same data The two FMs win on different questions…

In short

The study investigated how combining multiple foundation models (FMs) for brain data improves answers to complex scientific questions. By treating each FM as an independent tool server, a fixed language agent can dynamically select and use the best model for different parts of a task, leading to significant quality gains. The key challenge remains creating an intelligent system to orchestrate which model to use when multiple models disagree.

Key concepts

Foundation Models (FMs) for the Brain
These are large AI models trained on brain data that each focus on a different aspect of the same information. They capture unique views, like one focusing on connections and another on spatial locations. The paper shows that using several of these specialized models together provides a more complete understanding than any single model alone.
Modular Tool Server Route
Instead of retraining every model when adding a new one, the authors propose a modular approach where each FM acts as an independent tool server. A central language agent calls these servers iteratively. This allows researchers to add new models simply by adding a server, without needing extensive retraining or complex alignment processes.
Interface Crossing
This refers to the moment when an AI agent has to switch between the different ways two or more models name or describe brain regions. When this happens, the agent's ability to select and use tools improves significantly. This highlights that how models communicate their findings is as important as what they know.
Orchestration
This is the main unsolved problem: creating a policy for an agent to decide which specific FM tool to use for each part of a question. The goal is to build a system that can intelligently switch between models and use their outputs effectively, treating disagreements between models as useful signals.

Terminology used across episodes

This episode discusses

The paper

Foundation Models as Modular Tools for fMRI Analysis · Read on arXiv

Brookhaven National Laboratory

Foundation models (FMs) for the brain are rapidly growing in number and each captures a different aspect of the same data. A single scientific question often needs several of them at once, alongside classical analysis and the literature, but no established method combines them, because each model defines its own inputs, outputs, and spatial conventions. Representation alignment, the usual route from an FM into a language model, binds one FM to one LLM and requires retraining for every model added. We take a modular route: each FM becomes an independent tool server that a fixed language agent calls next to classical and literature tools, with no retraining. We start with fMRI, where public FMs already disagree on how to name a brain location and open task data give a classical baseline, and serve two FMs at opposite ends of that spectrum, BrainLM (atlas parcels) and Omni-fMRI (atlas-free voxels). A 76-question benchmark, a 2 by 2 tool ablation, and an independent LLM judge yield 3,040 judged traces. In this first study, modular FM access raises holistic answer quality by +0.70 on a 1 to 5 rubric (95% CI [+0.44, +0.97], n=34). The gain lands in how the agent selects, parameterizes, and grounds tool calls, and peaks where the agent must cross FM vocabularies. The two FMs win on different questions, yet mounting both matches the better one, and the voxel interface fails 7.8 times more often by context overflow. Modular connection works. Orchestrating what it connects, across EEG and MRI models, is the work ahead.

Transcript

Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.

Ines: We're looking at the paper: "Foundation Models as Modular Tools for fMRI Analysis".

Marcus: Here's the gist of the paper. Foundation models (FMs) for the brain are rapidly growing in number and each captures a different aspect of the same data The two FMs win on different questions, yet mounting both matches the better one Modular connection works.

Ines: Pruning redundant reasoning tokens Foundation models (FMs) for the brain are rapidly growing in number and each captures a different aspect of the same data A single scientific question often needs several of them at once, alongside classical analysis and the literature, but no established method combines them, because each model defines its own inputs, outputs, and spatial conventions.

Marcus: Representation alignment binds one FM to one LLM and requires retraining for every model added.

Ines: How the training signal is built The authors propose a modular route where each FM becomes an independent tool server that a fixed language agent calls next to classical and literature tools, with no retraining.

Paper summary: Marcus: So wrapping up this paper on "Foundation Models as Modular Tools for fMRI Analysis," the authors are arguing that the modular approach, where foundation models act as independent tool servers called by a language agent, is better than trying to align them all into one system.

Ines: They found this modularity test with seventy-six questions showed an improvement in answer quality, up to zero point seven on a five-point rubric <ref:2611.06926#pg1>. This gain happens because the agent gets better at choosing and grounding its tool calls, particularly when it has to cross those different model vocabularies.

Yuki: It means that for researchers looking at brain data, we might not need one all-encompassing foundation model; we can leverage specialized tools that each understand a different way of describing the brain’s structure.

Marcus: The paper also points out the importance of using the correct discovery calls for each server, like making sure you call BrainLM's specific region listing tool instead of a general one, because that's where many errors happen when models use mismatched vocabularies.

Ines: So, in simple terms, this work suggests that as foundation models get more specialized for different aspects of brain data, the way we connect them through a modular agent is a viable path forward without needing massive retraining efforts.

Yuki: It shifts the focus from making one giant model perfect to building a smart system that knows how to use several specialized models effectively.

Conclusion: Ines: So, to wrap up this talk on "Foundation Models as Modular Tools for fMRI Analysis," the authors are essentially saying that you don't need one single foundation model to get all your brain data insights anymore.

Marcus: Right, they're proposing a modular system where different models handle different parts of the analysis—like structure versus function—and an agent picks the right tool for each question.

Ines: They found that this setup actually boosts the overall quality of the answers by about zero point seven on a five-point scale. That gain comes from how well the agent learns to choose and use those different tools when they don't all speak the same language.

Marcus: It’s about how much better it gets at figuring out which model to call and what settings to use for that specific brain question. They pointed out that this improvement is biggest where the agent has to switch between model vocabularies, like translating a brain location name from one system to another.

Ines: That’s actually interesting because it suggests we might not need one giant, all-encompassing model trained on everything if we can just connect these specialized tools together modularly.

Marcus: The main thing they flag is orchestration—figuring out the best policy for picking the right model for each question and how to use disagreements between models as a signal. That's the next big hurdle they see.

Ines: And it opens up possibilities for applying this idea beyond just fMRI, to EEG or structural data too, where having that interface property of a model matters just as much as what it knows about the data itself.

Marcus: So, while this modular approach shows promise for improving answer quality in a test set, the real challenge now is building that smart policy to manage all those different tools effectively. We need to figure out how the agent actually makes those decisions in practice.

More episodes

← Home