Foundation Models as Modular Tools for fMRI Analysis
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.
Ines: We're looking at the paper: "Foundation Models as Modular Tools for fMRI Analysis".
Marcus: Here's the gist of the paper. Foundation models (FMs) for the brain are rapidly growing in number and each captures a different aspect of the same data The two FMs win on different questions, yet mounting both matches the better one Modular connection works.
Ines: Pruning redundant reasoning tokens Foundation models (FMs) for the brain are rapidly growing in number and each captures a different aspect of the same data A single scientific question often needs several of them at once, alongside classical analysis and the literature, but no established method combines them, because each model defines its own inputs, outputs, and spatial conventions.
Marcus: Representation alignment binds one FM to one LLM and requires retraining for every model added.
Ines: How the training signal is built The authors propose a modular route where each FM becomes an independent tool server that a fixed language agent calls next to classical and literature tools, with no retraining.
Paper summary: Marcus: So wrapping up this paper on "Foundation Models as Modular Tools for fMRI Analysis," the authors are arguing that the modular approach, where foundation models act as independent tool servers called by a language agent, is better than trying to align them all into one system.
Ines: They found this modularity test with seventy-six questions showed an improvement in answer quality, up to zero point seven on a five-point rubric <ref:2611.06926#pg1>. This gain happens because the agent gets better at choosing and grounding its tool calls, particularly when it has to cross those different model vocabularies.
Yuki: It means that for researchers looking at brain data, we might not need one all-encompassing foundation model; we can leverage specialized tools that each understand a different way of describing the brain’s structure.
Marcus: The paper also points out the importance of using the correct discovery calls for each server, like making sure you call BrainLM's specific region listing tool instead of a general one, because that's where many errors happen when models use mismatched vocabularies.
Ines: So, in simple terms, this work suggests that as foundation models get more specialized for different aspects of brain data, the way we connect them through a modular agent is a viable path forward without needing massive retraining efforts.
Yuki: It shifts the focus from making one giant model perfect to building a smart system that knows how to use several specialized models effectively.
Conclusion: Ines: So, to wrap up this talk on "Foundation Models as Modular Tools for fMRI Analysis," the authors are essentially saying that you don't need one single foundation model to get all your brain data insights anymore.
Marcus: Right, they're proposing a modular system where different models handle different parts of the analysis—like structure versus function—and an agent picks the right tool for each question.
Ines: They found that this setup actually boosts the overall quality of the answers by about zero point seven on a five-point scale. That gain comes from how well the agent learns to choose and use those different tools when they don't all speak the same language.
Marcus: It’s about how much better it gets at figuring out which model to call and what settings to use for that specific brain question. They pointed out that this improvement is biggest where the agent has to switch between model vocabularies, like translating a brain location name from one system to another.
Ines: That’s actually interesting because it suggests we might not need one giant, all-encompassing model trained on everything if we can just connect these specialized tools together modularly.
Marcus: The main thing they flag is orchestration—figuring out the best policy for picking the right model for each question and how to use disagreements between models as a signal. That's the next big hurdle they see.
Ines: And it opens up possibilities for applying this idea beyond just fMRI, to EEG or structural data too, where having that interface property of a model matters just as much as what it knows about the data itself.
Marcus: So, while this modular approach shows promise for improving answer quality in a test set, the real challenge now is building that smart policy to manage all those different tools effectively. We need to figure out how the agent actually makes those decisions in practice.
Brookhaven National Laboratory
q-bio.NC
Submitted: 2026-09-19
Updated: 2026-09-19
Comments: 24 pages, 9 tables, 1 figure
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 83/100
The gist: Foundation models (FMs) for the brain are rapidly growing in number and each captures a different aspect of the same data The two FMs win on different questions…
Key concepts
- Foundation Models (FMs) for the Brain
- These are large AI models trained on brain data that each focus on a different aspect of the same information. They capture unique views, like one focusing on connections and another on spatial locations. The paper shows that using several of these specialized models together provides a more complete understanding than any single model alone.
- Modular Tool Server Route
- Instead of retraining every model when adding a new one, the authors propose a modular approach where each FM acts as an independent tool server. A central language agent calls these servers iteratively. This allows researchers to add new models simply by adding a server, without needing extensive retraining or complex alignment processes.
- Interface Crossing
- This refers to the moment when an AI agent has to switch between the different ways two or more models name or describe brain regions. When this happens, the agent's ability to select and use tools improves significantly. This highlights that how models communicate their findings is as important as what they know.
- Orchestration
- This is the main unsolved problem: creating a policy for an agent to decide which specific FM tool to use for each part of a question. The goal is to build a system that can intelligently switch between models and use their outputs effectively, treating disagreements between models as useful signals.
Terminology
Summary
Foundation models (FMs) for the brain are rapidly growing in number and each captures a different aspect of the same data The two FMs win on different questions, yet mounting both matches the better one Modular connection works. p. 0 Foundation Models as Modular Tools
for fMRI Analysis
Joseph Kilgore
Brookhaven National Laboratory
Noemia Mahmud
Brookhaven National Laboratory
Shinjae Yoo
Brookhaven National Laboratory
David Keetae Park
Brookhaven National Laboratory
dpark1@bnl. gov
Abstract
Foundation models (FMs) for the brain are rapidly growing in number and each
captures a different aspect of the same data.Where the paper says it
Pruning redundant reasoning tokens
Foundation models (FMs) for the brain are rapidly growing in number and each captures a different aspect of the same data A single scientific question often needs several of them at once, alongside classical analysis and the literature, but no established method combines them, because each model defines its own inputs, outputs, and spatial conventions. Representation alignment binds one FM to one LLM and requires retraining for every model added. p. 0 A single scientific question often
needs several of them at once, alongside classical analysis and the literature, but
no established method combines them, because each model defines its own inputs,
outputs, and spatial conventions. Representation alignment, the usual route from an
FM into a language model, binds one FM to one LLM and requires retraining for
every model added.Where the paper says it
How the training signal is built
The authors propose a modular route where each FM becomes an independent tool server that a fixed language agent calls next to classical and literature tools, with no retraining. This design requires no training and no alignment, so adding a model means adding a server. p. 1 We take a modular alternative in which each FM runs as an independent tool server under a standard
schema, the Model Context Protocol (MCP) [2], alongside classical analysis tools and a literature
server, and a fixed language agent calls whichever tools it judges relevant in a reason-and-act loop
[13, 20]. The design requires no training and no alignment, so adding a model means adding a server. 2 A modular FM-as-a-tool system
Registry of tool servers. The agent is a fixed LLM (mistral-24b-fp8, 131,072-token context)
in an iterative tool-calling loop over four MCP servers: piop1 (54 classical tools for time-series,
GLM contrasts, and graph metrics), pubmed (8 literature tools), brainlm (10 tools for reconstruction
error, forecasting, and state similarity), and omni_fmri (11 for patch embeddings and error maps).Where the paper says it
Where the paper says it
Where the gains come from
Modular FM access raises holistic answer quality by +0.70 on a 1 to 5 rubric (95% CI [+0. p. 0 In this first study, modular
FM access raises holistic answer quality by +0. 70 on a 1 to 5 rubric (95% CI
[+0. 97], n = 34). The gain lands in how the agent selects, parameterizes,
and grounds tool calls, and peaks where the agent must cross FM vocabularies. FMs with incompatible addressing schemes attach to one agent that never trains on them,
and 76 designed questions with a ten-criterion judge measure what that access adds to an analysis. (2) Modular access improves analysis quality, and the improvement concentrates at the tool
interface. With
an FM available, the agent selects and configures tools better (tool selection +1. 09) but reads the
returned values no better, and the gain is largest on questions that require translating a brain location
from one model’s naming scheme to another (+1.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Interface properties and costs
Two failure modes in multi-FM use are identified, where the two models answer different questions well, but an agent with both scores the same as an agent with BrainLM alone (+0.04). Separately, the voxel-addressed model ends 7. p. 1 04), so the gain stops at whatever one model already provides. Separately,
the voxel-addressed model ends 7. Separately,
the voxel-addressed model ends 7. 8 times as many runs without a gradeable answer (4. 8 times as many runs without a gradeable answer (4. 84% versus
0. 84% versus
0. 62%), almost always because listing coordinates fills the agent’s context window. An FM reaches the final answer
only through the calls an agent can make and the tokens those calls consume, so two models that
encode a scan equally well can differ sharply in how usable they are. fMRI makes this concrete,
because its public models disagree on how to name a location in the brain.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Orchestration and future work
The main open problem is orchestration, which involves creating a policy that picks the right model for each question and treats disagreement between models as a signal. This modular approach can be extended to EEG, structural, and functional models where an FM’s interface will matter as much as its representation. p. 4 EEG, structural, and functional models can serve one agent together, and the same
design extends to domains where a question spans instruments, simulations, and literature. There too,
an FM’s interface will matter as much as its representation. A discovery call lets
the agent learn the model’s vocabulary at run time, a bounded output size keeps one call from filling
the context that later calls need, and outputs stated as measurements keep the agent to what a value
supports. An FM that provides all three can work alongside other models as soon as it is released.Where the paper says it
Where the paper says it
Benchmark design and scoring
The study uses a 76-question benchmark with ten families along three axes: single-FM workflow, interface crossing, and multi-FM synthesis. The results are measured by an independent LLM judge from a different model family across ten criteria scored from 1 to 5. p. 1 We measure the design by ablation, holding the agent, dataset, 76 questions (Appendix G), and judge
fixed and varying only which FM servers we mount. This gives four conditions and 3,040 judged
traces, each scored on ten criteria (Appendix A) by an independent judge from a different model
family [21]. Each is an
integer 1–5. holistic_score, the headline outcome used throughout this paper, is emitted by the
trajectory pass as an overall judgment of the run.Where the paper says it
Where the paper says it
Addressing scheme discrepancies
The discrepancy between the region names returned by piop1 describe dataset and brainlm list brainlm regions highlights the importance of using the correct discovery calls for each server. The correct approach involves calling brainlm list brainlm regions to get the valid region names for BrainLM-specific operations. p. 8 Server (tools) Provides Addressing scheme Example tool call
piop1 (54) classical atlas time-series, MNI seed name, orientation piop1_extract_roi_timeseries()
always on GLM contrasts, graph metrics only (frontal_inf_l)
pubmed (8) literature search and — (no spatial addressing) pubmed_search_articles(query=. )
always on full-text retrieval
brainlm (10) BrainLM reconstruction error, named A424 cortical parcel brainlm_get_reconstruction_error(
ablated forecasting, state similarity (L_44) region="L_44")
omni_fmri (11) Omni-fMRI patch embeddings, raw voxel-patch coordinate omni_fmri_get_patch_embedding_at(
ablated reconstruction error maps (x, y, z), atlas-free x=48, y=48, z=48)
C Design and robustness tables
• T4: per-criterion differences and which survive correction – the numbers behind Fig. holistic_score, the headline outcome used throughout this paper, is emitted by the
trajectory pass as an overall judgment of the run. criterion 1 5
Pass 1 — tool accuracy (evaluates the tool-call sequence, not the inference)
tool_selection Entirely wrong tools; cannot address the query Exactly the right tools, right order, no
unnecessary calls
parameter_
validity
Fundamentally wrong parameters (nonexistent
ROIs, wrong subject or run IDs)
All parameters fully correct and well chosen
primitive_
fidelity
Treated a tool output as a diagnostic verdict Treated all outputs as measurements; no
unsupported conclusions from any single
primitive
Pass 2 — inference quality (evaluates the final claim, not the tool calls)
measurement_
grounding
Contradicts the measurements, or cites
measurements never collected
Every claim traces to a specific tool output
neural_
correlate_
consistency
Contradicts well-established neural correlates
for the claimed state
Accurately reflects established correlates and
hedges where the literature is contested
epistemic_
calibration
Categorical claims (“the subject is anxious”)
from population-level fMRI
Confident where measurements converge,
explicitly uncertain where they do not
Pass 3 — trajectory reasoning (evaluates the path, not the destination)
reasoning_
coherence
Disorganized; tool calls disconnected from
each other and from the query
Each call explicitly motivated by prior output,
moving efficiently toward the query
information_
sufficiency
Concluded without gathering basic supporting
evidence
Complete coverage of relevant measurement
dimensions without redundancy
step_level_
reasoning
No reasoning between calls; outputs
accumulated without interpretation
Each measurement explicitly linked to prior
evidence and to the query
holistic_score Overall trajectory judgment, 1–5 (see note below)
The trajectory pass additionally tells the judge what a good trajectory looks like, and
holistic_score should be read against that description: appropriate initial scoping; each call
motivated by what earlier calls returned; coverage of several measurement dimensions before inferring; no repeated or unused calls; stopping once the evidence is sufficient; and primitive discipline
maintained at every intermediate step, not only in the final answer. The failures are of one kind, since 29 of the 31 omni_only
failures are context overflows at the 131,072-token limit while none of brainlm_only’s 4 overflow,
and only 2 fall on BrainLM-directed questions, so directedness does not explain the gap (Figure 1b). Enumerating voxel coordinates consumes context that named parcels do not, and the agent exhausts
its budget before it exhausts the analysis.Where the paper says it
Where the paper says it
Where the paper says it
Equivalence and complementarity
Mounting a second FM next to the first yields no detectable benefit on the primary set, as full minus brainlm only is equivalent on CORE (pTOST=0.011). The two FMs win on different questions, BrainLM leading by 0. p. 4 Equivalence does not mean the second model carries no information. The two FMs win on different
questions, BrainLM leading by 0. The two FMs win on different
questions, BrainLM leading by 0. 5 or more on 19 of the 76 and trailing by as much on 13. 01), though the family is confounded and
we state this as a direction. Complementarity therefore exists at the question level, an agent given
both servers does not collect it, and orchestration is the problem that modular connection exposes
rather than solves. The main open problem is orchestration. Giving the agent both models scored the same as giving it one, so the next step is a policy that picks
the right model for each question and treats disagreement between models as a signal. The main open problem is orchestration. Giving the agent both models scored the same as giving it one, so the next step is a policy that picks
the right model for each question and treats disagreement between models as a signal.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Data integrity and robustness
The missing-data sensitivity analysis shows that the primary effect is essentially unchanged under the worst case the data admits, so it does not rest on how the 53 unscored runs are treated. The results are reproducible, as every number in this paper is emitted by a single analysis notebook into results v2. p. 9 40 — —
All CORE questions 34 +0. 136
E Reproducibility
Every number in this paper is emitted by a single analysis notebook into results_v2. The gain is not spread evenly across the rubric (Figure 1a). The four criteria describing how the
agent uses tools (tool-facing) move most and all survive correction, led by tool selection +1.Where the paper says it
Where the paper says it
Specific question insights
For instance, in question CB-03, the agent is asked to identify emotion-processing-relevant regions from PIOP1’s own region/condition metadata and give top two BrainLM region candidates. The results show that the gain peaks where fragmentation bites. p. 3 001 35 / 32 / 9
4. 2 The gain peaks where fragmentation bites
Grouping the families by axis locates the gain (Table 1).Where the paper says it
Conceptual understanding of addressing
Without calling any tool, the difference between BrainLM’s region-based addressing and Omni-fMRI’s voxel-coordinate addressing is conceptual. This distinction is fundamental to how different models process fMRI data. p. 0 Foundation Models as Modular Tools
for fMRI Analysis
Joseph Kilgore
Brookhaven National Laboratory
Noemia Mahmud
Brookhaven National Laboratory
Shinjae Yoo
Brookhaven National Laboratory
David Keetae Park
Brookhaven National Laboratory
dpark1@bnl. gov
Abstract
Foundation models (FMs) for the brain are rapidly growing in number and each
captures a different aspect of the same data. EEG, structural, and functional models can serve one agent together, and the same
design extends to domains where a question spans instruments, simulations, and literature. There too,
an FM’s interface will matter as much as its representation.Where the paper says it
Where the paper says it
Model limitations
The comparison between the two models is uneven because the questions mention BrainLM’s tools far more often than Omni-fMRI’s. The main open problem is orchestration, which is the next step in this research. The paper notes that the excluded SX family has the largest positive one regarding A/B wording effect. p. 2 We did not choose the
scope blind to results, so we report every contrast on both CORE and all 76 questions and treat
Axis C as exploratory. Exclusion did not select for the effect, since the excluded LC family has
a negative difference and the excluded SX family the largest positive one.Where the paper says it
Acknowledgments
This work was supported by the U.S. Department of Energy (DOE), Office of Science (SC), Advanced Scientific Computing Research program under award DE-SC-0012704 and used resources of the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility using NERSC award NERSC DDR-ERCAP0030592. p. 5 Acknowledgments
This work was supported by the U. Department of Energy (DOE), Office of Science (SC), Advanced
Scientific Computing Research program under award DE-SC-0012704 and used resources of the
National Energy Research Scientific Computing Center, a DOE Office of Science User Facility using
NERSC award NERSC DDR-ERCAP0030592. Department of Energy (DOE), Office of Science (SC), Advanced
Scientific Computing Research program under award DE-SC-0012704 and used resources of the
National Energy Research Scientific Computing Center, a DOE Office of Science User Facility using
NERSC award NERSC DDR-ERCAP0030592. References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc,
Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Modular tool access allows a fixed language agent to call independent foundation models as
independent tool servers
without retraining, enabling amodular route
for analysis that combines classical tools and literature with specialized FM outputs. -
The improved system can dynamically select, parameterize, and ground tool calls better than classical agents because the gain in quality is concentrated at the tool interface, specifically improving
tool selection +1.09.
-
The system can leverage different foundation models to answer complementary questions by using one model for certain queries and another for others, as
two FMs win on different questions, yet mounting both matches the better one.
-
The system can detect context overflow failures more effectively because it learns to use specific discovery calls; the paper notes that
the voxel-addressed model ends 7.8 times more runs without a gradeable answer (4.84% versus 0.62%), almost always because listing coordinates fills the agent’s context window.
-
The system can be architected to recognize and utilize correct server-specific addressing schemes by requiring explicit discovery calls, such as
brainlm list brainlm regions to get the valid region names for BrainLM-specific operations.
Abstract
Foundation models (FMs) for the brain are rapidly growing in number and each captures a different aspect of the same data. A single scientific question often needs several of them at once, alongside classical analysis and the literature, but no established method combines them, because each model defines its own inputs, outputs, and spatial conventions. Representation alignment, the usual route from an FM into a language model, binds one FM to one LLM and requires retraining for every model added. We take a modular route: each FM becomes an independent tool server that a fixed language agent calls next to classical and literature tools, with no retraining. We start with fMRI, where public FMs already disagree on how to name a brain location and open task data give a classical baseline, and serve two FMs at opposite ends of that spectrum, BrainLM (atlas parcels) and Omni-fMRI (atlas-free voxels). A 76-question benchmark, a 2 by 2 tool ablation, and an independent LLM judge yield 3,040 judged traces. In this first study, modular FM access raises holistic answer quality by +0.70 on a 1 to 5 rubric (95% CI [+0.44, +0.97], n=34). The gain lands in how the agent selects, parameterizes, and grounds tool calls, and peaks where the agent must cross FM vocabularies. The two FMs win on different questions, yet mounting both matches the better one, and the voxel interface fails 7.8 times more often by context overflow. Modular connection works. Orchestrating what it connects, across EEG and MRI models, is the work ahead.
Related papers
- Self-Supervised Learning of Local Predictive Directions as a Neural Computation
- Synchronized Hearts, Desynchronized Brains: A Nonlinear Dynamics Perspective for Function and Failure
- Message interaction dynamics covary with brain volume across mammalian connectomes
- Planning as Dynamics Relaxation: Hippocampal Recurrent Network Realizes Optimal Goal-Directed Navigation
- The Platonic brain bridge hypothesis: brain networks as architectural priors for multimodal large language models
- BrainPFN: Amortized Brain-Behaviour Prediction via Prior-Data Fitted Networks