Foundation Models as Modular Tools for fMRI Analysis

arXiv:2611.06926 · q-bio.NC · Submitted 2026-09-19 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.

Ines: We're looking at the paper: "Foundation Models as Modular Tools for fMRI Analysis".

Marcus: Here's the gist of the paper. Foundation models (FMs) for the brain are rapidly growing in number and each captures a different aspect of the same data The two FMs win on different questions, yet mounting both matches the better one Modular connection works.

Ines: Pruning redundant reasoning tokens Foundation models (FMs) for the brain are rapidly growing in number and each captures a different aspect of the same data A single scientific question often needs several of them at once, alongside classical analysis and the literature, but no established method combines them, because each model defines its own inputs, outputs, and spatial conventions.

Marcus: Representation alignment binds one FM to one LLM and requires retraining for every model added.

Ines: How the training signal is built The authors propose a modular route where each FM becomes an independent tool server that a fixed language agent calls next to classical and literature tools, with no retraining.

Paper summary: Marcus: So wrapping up this paper on "Foundation Models as Modular Tools for fMRI Analysis," the authors are arguing that the modular approach, where foundation models act as independent tool servers called by a language agent, is better than trying to align them all into one system.

Ines: They found this modularity test with seventy-six questions showed an improvement in answer quality, up to zero point seven on a five-point rubric <ref:2611.06926#pg1>. This gain happens because the agent gets better at choosing and grounding its tool calls, particularly when it has to cross those different model vocabularies.

Yuki: It means that for researchers looking at brain data, we might not need one all-encompassing foundation model; we can leverage specialized tools that each understand a different way of describing the brain’s structure.

Marcus: The paper also points out the importance of using the correct discovery calls for each server, like making sure you call BrainLM's specific region listing tool instead of a general one, because that's where many errors happen when models use mismatched vocabularies.

Ines: So, in simple terms, this work suggests that as foundation models get more specialized for different aspects of brain data, the way we connect them through a modular agent is a viable path forward without needing massive retraining efforts.

Yuki: It shifts the focus from making one giant model perfect to building a smart system that knows how to use several specialized models effectively.

Conclusion: Ines: So, to wrap up this talk on "Foundation Models as Modular Tools for fMRI Analysis," the authors are essentially saying that you don't need one single foundation model to get all your brain data insights anymore.

Marcus: Right, they're proposing a modular system where different models handle different parts of the analysis—like structure versus function—and an agent picks the right tool for each question.

Ines: They found that this setup actually boosts the overall quality of the answers by about zero point seven on a five-point scale. That gain comes from how well the agent learns to choose and use those different tools when they don't all speak the same language.

Marcus: It’s about how much better it gets at figuring out which model to call and what settings to use for that specific brain question. They pointed out that this improvement is biggest where the agent has to switch between model vocabularies, like translating a brain location name from one system to another.

Ines: That’s actually interesting because it suggests we might not need one giant, all-encompassing model trained on everything if we can just connect these specialized tools together modularly.

Marcus: The main thing they flag is orchestration—figuring out the best policy for picking the right model for each question and how to use disagreements between models as a signal. That's the next big hurdle they see.

Ines: And it opens up possibilities for applying this idea beyond just fMRI, to EEG or structural data too, where having that interface property of a model matters just as much as what it knows about the data itself.

Marcus: So, while this modular approach shows promise for improving answer quality in a test set, the real challenge now is building that smart policy to manage all those different tools effectively. We need to figure out how the agent actually makes those decisions in practice.

Brookhaven National Laboratory

q-bio.NC

Submitted: 2026-09-19

Updated: 2026-09-19

Comments: 24 pages, 9 tables, 1 figure

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: Foundation models (FMs) for the brain are rapidly growing in number and each captures a different aspect of the same data The two FMs win on different questions…

Key concepts

Foundation Models (FMs) for the Brain
These are large AI models trained on brain data that each focus on a different aspect of the same information. They capture unique views, like one focusing on connections and another on spatial locations. The paper shows that using several of these specialized models together provides a more complete understanding than any single model alone.
Modular Tool Server Route
Instead of retraining every model when adding a new one, the authors propose a modular approach where each FM acts as an independent tool server. A central language agent calls these servers iteratively. This allows researchers to add new models simply by adding a server, without needing extensive retraining or complex alignment processes.
Interface Crossing
This refers to the moment when an AI agent has to switch between the different ways two or more models name or describe brain regions. When this happens, the agent's ability to select and use tools improves significantly. This highlights that how models communicate their findings is as important as what they know.
Orchestration
This is the main unsolved problem: creating a policy for an agent to decide which specific FM tool to use for each part of a question. The goal is to build a system that can intelligently switch between models and use their outputs effectively, treating disagreements between models as useful signals.

Terminology

Summary

Foundation models (FMs) for the brain are rapidly growing in number and each captures a different aspect of the same data The two FMs win on different questions, yet mounting both matches the better one Modular connection works. p. 0

Where the paper says it

Foundation Models as Modular Tools for fMRI Analysis Joseph Kilgore Brookhaven National Laboratory Noemia Mahmud Brookhaven National Laboratory Shinjae Yoo Brookhaven National Laboratory David Keetae Park Brookhaven National Laboratory dpark1@bnl. gov Abstract Foundation models (FMs) for the brain are rapidly growing in number and each captures a different aspect of the same data.

Pruning redundant reasoning tokens

Foundation models (FMs) for the brain are rapidly growing in number and each captures a different aspect of the same data A single scientific question often needs several of them at once, alongside classical analysis and the literature, but no established method combines them, because each model defines its own inputs, outputs, and spatial conventions. Representation alignment binds one FM to one LLM and requires retraining for every model added. p. 0

Where the paper says it

A single scientific question often needs several of them at once, alongside classical analysis and the literature, but no established method combines them, because each model defines its own inputs, outputs, and spatial conventions. Representation alignment, the usual route from an FM into a language model, binds one FM to one LLM and requires retraining for every model added.

How the training signal is built

The authors propose a modular route where each FM becomes an independent tool server that a fixed language agent calls next to classical and literature tools, with no retraining. This design requires no training and no alignment, so adding a model means adding a server. p. 1

Where the paper says it

We take a modular alternative in which each FM runs as an independent tool server under a standard schema, the Model Context Protocol (MCP) [2], alongside classical analysis tools and a literature server, and a fixed language agent calls whichever tools it judges relevant in a reason-and-act loop [13, 20]. The design requires no training and no alignment, so adding a model means adding a server.

The agent is a fixed LLM in an iterative tool-calling loop over four MCP servers: piop1 (54 classical tools for time-series, GLM contrasts, and graph metrics), pubmed (8 literature tools), brainlm (10 tools for reconstruction error, forecasting, and state similarity), and omni fmri (11 for patch embeddings and error maps). p. 1
Where the paper says it

2 A modular FM-as-a-tool system Registry of tool servers. The agent is a fixed LLM (mistral-24b-fp8, 131,072-token context) in an iterative tool-calling loop over four MCP servers: piop1 (54 classical tools for time-series, GLM contrasts, and graph metrics), pubmed (8 literature tools), brainlm (10 tools for reconstruction error, forecasting, and state similarity), and omni_fmri (11 for patch embeddings and error maps).

Where the gains come from

Modular FM access raises holistic answer quality by +0.70 on a 1 to 5 rubric (95% CI [+0. p. 0

Where the paper says it

In this first study, modular FM access raises holistic answer quality by +0. 70 on a 1 to 5 rubric (95% CI [+0.

44, +0.97], n = 34). The gain lands in how the agent selects, parameterizes, and grounds tool calls, and peaks where the agent must cross FM vocabularies. p. 0
Where the paper says it

97], n = 34). The gain lands in how the agent selects, parameterizes, and grounds tool calls, and peaks where the agent must cross FM vocabularies.

Specifically, tool selection improves by +1. p. 1
Where the paper says it

FMs with incompatible addressing schemes attach to one agent that never trains on them, and 76 designed questions with a ten-criterion judge measure what that access adds to an analysis. (2) Modular access improves analysis quality, and the improvement concentrates at the tool interface.

09. The improvement concentrates at the tool interface and is largest on questions that require translating a brain location from one model’s naming scheme to another (+1. p. 1
Where the paper says it

With an FM available, the agent selects and configures tools better (tool selection +1. 09) but reads the returned values no better, and the gain is largest on questions that require translating a brain location from one model’s naming scheme to another (+1.

04).

Interface properties and costs

Two failure modes in multi-FM use are identified, where the two models answer different questions well, but an agent with both scores the same as an agent with BrainLM alone (+0.04). Separately, the voxel-addressed model ends 7. p. 1

Where the paper says it

04), so the gain stops at whatever one model already provides. Separately, the voxel-addressed model ends 7.

8 times as many runs without a gradeable answer (4. p. 1
Where the paper says it

Separately, the voxel-addressed model ends 7. 8 times as many runs without a gradeable answer (4.

84% versus 0. p. 1
Where the paper says it

8 times as many runs without a gradeable answer (4. 84% versus 0.

62%) almost always because listing coordinates fills the agent’s context window. p. 1
Where the paper says it

84% versus 0. 62%), almost always because listing coordinates fills the agent’s context window.

This interface property is critical because an FM reaches the final answer only through the calls an agent can make and the tokens those calls consume. p. 1
Where the paper says it

An FM reaches the final answer only through the calls an agent can make and the tokens those calls consume, so two models that encode a scan equally well can differ sharply in how usable they are. fMRI makes this concrete, because its public models disagree on how to name a location in the brain.

Orchestration and future work

The main open problem is orchestration, which involves creating a policy that picks the right model for each question and treats disagreement between models as a signal. This modular approach can be extended to EEG, structural, and functional models where an FM’s interface will matter as much as its representation. p. 4

Where the paper says it

EEG, structural, and functional models can serve one agent together, and the same design extends to domains where a question spans instruments, simulations, and literature. There too, an FM’s interface will matter as much as its representation.

The paper concludes that an FM that provides all three necessary interface properties can work alongside other models as soon as it is released. p. 4
Where the paper says it

A discovery call lets the agent learn the model’s vocabulary at run time, a bounded output size keeps one call from filling the context that later calls need, and outputs stated as measurements keep the agent to what a value supports. An FM that provides all three can work alongside other models as soon as it is released.

Benchmark design and scoring

The study uses a 76-question benchmark with ten families along three axes: single-FM workflow, interface crossing, and multi-FM synthesis. The results are measured by an independent LLM judge from a different model family across ten criteria scored from 1 to 5. p. 1

Where the paper says it

We measure the design by ablation, holding the agent, dataset, 76 questions (Appendix G), and judge fixed and varying only which FM servers we mount. This gives four conditions and 3,040 judged traces, each scored on ten criteria (Appendix A) by an independent judge from a different model family [21].

The holistic score is the headline outcome used throughout this paper. p. 7
Where the paper says it

Each is an integer 1–5. holistic_score, the headline outcome used throughout this paper, is emitted by the trajectory pass as an overall judgment of the run.

Addressing scheme discrepancies

The discrepancy between the region names returned by piop1 describe dataset and brainlm list brainlm regions highlights the importance of using the correct discovery calls for each server. The correct approach involves calling brainlm list brainlm regions to get the valid region names for BrainLM-specific operations. p. 8

Where the paper says it

Server (tools) Provides Addressing scheme Example tool call piop1 (54) classical atlas time-series, MNI seed name, orientation piop1_extract_roi_timeseries() always on GLM contrasts, graph metrics only (frontal_inf_l) pubmed (8) literature search and — (no spatial addressing) pubmed_search_articles(query=. ) always on full-text retrieval brainlm (10) BrainLM reconstruction error, named A424 cortical parcel brainlm_get_reconstruction_error( ablated forecasting, state similarity (L_44) region="L_44") omni_fmri (11) Omni-fMRI patch embeddings, raw voxel-patch coordinate omni_fmri_get_patch_embedding_at( ablated reconstruction error maps (x, y, z), atlas-free x=48, y=48, z=48) C Design and robustness tables • T4: per-criterion differences and which survive correction – the numbers behind Fig.

This underscores the need for per-server discovery calls to ensure that the correct tools and region names are used. p. 7
Where the paper says it

holistic_score, the headline outcome used throughout this paper, is emitted by the trajectory pass as an overall judgment of the run. criterion 1 5 Pass 1 — tool accuracy (evaluates the tool-call sequence, not the inference) tool_selection Entirely wrong tools; cannot address the query Exactly the right tools, right order, no unnecessary calls parameter_ validity Fundamentally wrong parameters (nonexistent ROIs, wrong subject or run IDs) All parameters fully correct and well chosen primitive_ fidelity Treated a tool output as a diagnostic verdict Treated all outputs as measurements; no unsupported conclusions from any single primitive Pass 2 — inference quality (evaluates the final claim, not the tool calls) measurement_ grounding Contradicts the measurements, or cites measurements never collected Every claim traces to a specific tool output neural_ correlate_ consistency Contradicts well-established neural correlates for the claimed state Accurately reflects established correlates and hedges where the literature is contested epistemic_ calibration Categorical claims (“the subject is anxious”) from population-level fMRI Confident where measurements converge, explicitly uncertain where they do not Pass 3 — trajectory reasoning (evaluates the path, not the destination) reasoning_ coherence Disorganized; tool calls disconnected from each other and from the query Each call explicitly motivated by prior output, moving efficiently toward the query information_ sufficiency Concluded without gathering basic supporting evidence Complete coverage of relevant measurement dimensions without redundancy step_level_ reasoning No reasoning between calls; outputs accumulated without interpretation Each measurement explicitly linked to prior evidence and to the query holistic_score Overall trajectory judgment, 1–5 (see note below) The trajectory pass additionally tells the judge what a good trajectory looks like, and holistic_score should be read against that description: appropriate initial scoping; each call motivated by what earlier calls returned; coverage of several measurement dimensions before inferring; no repeated or unused calls; stopping once the evidence is sufficient; and primitive discipline maintained at every intermediate step, not only in the final answer.

The voxel-addressed model fails more often due to context overflow when listing coordinates compared to named parcels. p. 4
Where the paper says it

The failures are of one kind, since 29 of the 31 omni_only failures are context overflows at the 131,072-token limit while none of brainlm_only’s 4 overflow, and only 2 fall on BrainLM-directed questions, so directedness does not explain the gap (Figure 1b). Enumerating voxel coordinates consumes context that named parcels do not, and the agent exhausts its budget before it exhausts the analysis.

Equivalence and complementarity

Mounting a second FM next to the first yields no detectable benefit on the primary set, as full minus brainlm only is equivalent on CORE (pTOST=0.011). The two FMs win on different questions, BrainLM leading by 0. p. 4

Where the paper says it

Equivalence does not mean the second model carries no information. The two FMs win on different questions, BrainLM leading by 0.

5 or more on 19 of the 76 and trailing by as much on 13. p. 4
Where the paper says it

The two FMs win on different questions, BrainLM leading by 0. 5 or more on 19 of the 76 and trailing by as much on 13.

Complementarity exists at the question level, as an agent given both servers does not collect it. p. 4
Where the paper says it

01), though the family is confounded and we state this as a direction. Complementarity therefore exists at the question level, an agent given both servers does not collect it, and orchestration is the problem that modular connection exposes rather than solves.

The main open problem is orchestration, which is the next step in this research. p. 4
Where the paper says it

The main open problem is orchestration. Giving the agent both models scored the same as giving it one, so the next step is a policy that picks the right model for each question and treats disagreement between models as a signal.

p. 4
Where the paper says it

The main open problem is orchestration. Giving the agent both models scored the same as giving it one, so the next step is a policy that picks the right model for each question and treats disagreement between models as a signal.

Data integrity and robustness

The missing-data sensitivity analysis shows that the primary effect is essentially unchanged under the worst case the data admits, so it does not rest on how the 53 unscored runs are treated. The results are reproducible, as every number in this paper is emitted by a single analysis notebook into results v2. p. 9

Where the paper says it

40 — — All CORE questions 34 +0. 136 E Reproducibility Every number in this paper is emitted by a single analysis notebook into results_v2.

json. The final analysis shows that the four criteria describing how the agent uses tools all move under FM access. p. 3
Where the paper says it

The gain is not spread evenly across the rubric (Figure 1a). The four criteria describing how the agent uses tools (tool-facing) move most and all survive correction, led by tool selection +1.

Specific question insights

For instance, in question CB-03, the agent is asked to identify emotion-processing-relevant regions from PIOP1’s own region/condition metadata and give top two BrainLM region candidates. The results show that the gain peaks where fragmentation bites. p. 3

Where the paper says it

001 35 / 32 / 9 4. 2 The gain peaks where fragmentation bites Grouping the families by axis locates the gain (Table 1).

Conceptual understanding of addressing

Without calling any tool, the difference between BrainLM’s region-based addressing and Omni-fMRI’s voxel-coordinate addressing is conceptual. This distinction is fundamental to how different models process fMRI data. p. 0

Where the paper says it

Foundation Models as Modular Tools for fMRI Analysis Joseph Kilgore Brookhaven National Laboratory Noemia Mahmud Brookhaven National Laboratory Shinjae Yoo Brookhaven National Laboratory David Keetae Park Brookhaven National Laboratory dpark1@bnl. gov Abstract Foundation models (FMs) for the brain are rapidly growing in number and each captures a different aspect of the same data.

The paper suggests that the interface property of an FM will matter as much as its representation in future architectures. p. 4
Where the paper says it

EEG, structural, and functional models can serve one agent together, and the same design extends to domains where a question spans instruments, simulations, and literature. There too, an FM’s interface will matter as much as its representation.

Model limitations

The comparison between the two models is uneven because the questions mention BrainLM’s tools far more often than Omni-fMRI’s. The main open problem is orchestration, which is the next step in this research. The paper notes that the excluded SX family has the largest positive one regarding A/B wording effect. p. 2

Where the paper says it

We did not choose the scope blind to results, so we report every contrast on both CORE and all 76 questions and treat Axis C as exploratory. Exclusion did not select for the effect, since the excluded LC family has a negative difference and the excluded SX family the largest positive one.

Acknowledgments

This work was supported by the U.S. Department of Energy (DOE), Office of Science (SC), Advanced Scientific Computing Research program under award DE-SC-0012704 and used resources of the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility using NERSC award NERSC DDR-ERCAP0030592. p. 5

Where the paper says it

Acknowledgments This work was supported by the U. Department of Energy (DOE), Office of Science (SC), Advanced Scientific Computing Research program under award DE-SC-0012704 and used resources of the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility using NERSC award NERSC DDR-ERCAP0030592.

The references provided are listed at the end of the paper. p. 5
Where the paper says it

Department of Energy (DOE), Office of Science (SC), Advanced Scientific Computing Research program under award DE-SC-0012704 and used resources of the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility using NERSC award NERSC DDR-ERCAP0030592. References [1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Modular tool access allows a fixed language agent to call independent foundation models as independent tool servers without retraining, enabling a modular route for analysis that combines classical tools and literature with specialized FM outputs.

  2. The improved system can dynamically select, parameterize, and ground tool calls better than classical agents because the gain in quality is concentrated at the tool interface, specifically improving tool selection +1.09.

  3. The system can leverage different foundation models to answer complementary questions by using one model for certain queries and another for others, as two FMs win on different questions, yet mounting both matches the better one.

  4. The system can detect context overflow failures more effectively because it learns to use specific discovery calls; the paper notes that the voxel-addressed model ends 7.8 times more runs without a gradeable answer (4.84% versus 0.62%), almost always because listing coordinates fills the agent’s context window.

  5. The system can be architected to recognize and utilize correct server-specific addressing schemes by requiring explicit discovery calls, such as brainlm list brainlm regions to get the valid region names for BrainLM-specific operations.

Abstract

Foundation models (FMs) for the brain are rapidly growing in number and each captures a different aspect of the same data. A single scientific question often needs several of them at once, alongside classical analysis and the literature, but no established method combines them, because each model defines its own inputs, outputs, and spatial conventions. Representation alignment, the usual route from an FM into a language model, binds one FM to one LLM and requires retraining for every model added. We take a modular route: each FM becomes an independent tool server that a fixed language agent calls next to classical and literature tools, with no retraining. We start with fMRI, where public FMs already disagree on how to name a brain location and open task data give a classical baseline, and serve two FMs at opposite ends of that spectrum, BrainLM (atlas parcels) and Omni-fMRI (atlas-free voxels). A 76-question benchmark, a 2 by 2 tool ablation, and an independent LLM judge yield 3,040 judged traces. In this first study, modular FM access raises holistic answer quality by +0.70 on a 1 to 5 rubric (95% CI [+0.44, +0.97], n=34). The gain lands in how the agent selects, parameterizes, and grounds tool calls, and peaks where the agent must cross FM vocabularies. The two FMs win on different questions, yet mounting both matches the better one, and the voxel interface fails 7.8 times more often by context overflow. Modular connection works. Orchestrating what it connects, across EEG and MRI models, is the work ahead.

Related papers