Benchmarking large language models for parameter-level structural identifiability classification in epidemic ODE models
summary
The gist
Structural identifiability determines whether unknown parameters can be uniquely recovered from ideal model outputs, making it a prerequisite for interpreting parameter estimates on page 1.
In short
This study tested whether large language models (LLMs) can accurately classify structural identifiability for epidemic ODE models. Researchers used a benchmark of 88 scenarios derived from existing studies and symbolic analysis. GPT-5.5 Instant and Thinking Medium showed the highest accuracy, suggesting LLMs can provide preliminary hypotheses, but they should not be used to certify model identifiability.
Key concepts
- Structural Identifiability
- This is a mathematical property of a model that determines if all unknown parameters can be uniquely recovered solely by observing the model's output. If a model is structurally identifiable, it means there is only one possible set of parameters that could have generated the observed data.
- Epidemic ODE Models
- These are differential equations (Ordinary Differential Equations) used to mathematically describe how populations change over time in an epidemic setting. The paper focuses on these models because they are frequently used in public health to predict disease spread and evaluate interventions.
- Parameter-Level Classification
- This refers to the task of classifying each individual unknown parameter within the model as either identifiable or not. The LLMs were tested on their ability to make these specific decisions for each parameter based on the model structure and observed data.
Terminology used across episodes
This episode discusses
- Benchmarking large language models for parameter-level structural identifiability classification in epidemic ODE models · Paper Radio
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
The paper
Benchmarking large language models for parameter-level structural identifiability classification in epidemic ODE models · Read on arXiv
Geunsoo Jang, K. Selçuk Candan, Gerardo Chowell
School of Computing and Augmented Intelligence, Arizona State University · Department of Population Health Sciences, School of Public Health, Georgia State University
Transcript
Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.
Ines: I'm Ines.
Marcus: And I'm Marcus, here with Yuki.
Ines: We're looking at the paper: "Benchmarking large language models for parameter-level structural identifiability classification in epidemic ODE models".
Marcus: Here's the gist of the paper. Structural identifiability determines whether unknown parameters can be uniquely recovered from ideal model outputs, making it a prerequisite for interpreting parameter estimates on page 1.
Ines: This paper evaluates whether large language models (LLMs) can reliably provide preliminary structural identifiability classifications for epidemic models on page 2.
Marcus: Benchmark Construction and Methodology The researchers constructed a retrospective benchmark of 88 model scenarios from 14 studies, encompassing 696 parameter-level formal reference classifications on page 5.
Ines: Scenarios were included when sufficiently resolved structural identifiability classifications were available from published results or obtained through Julia-based symbolic analysis on page 5.
Paper summary: Ines: So this paper, "Benchmarking large language models for parameter-level structural identifiability classification in epidemic ODE models," it’s about whether these LLMs can actually tell us if we can figure out the unknown parts of an epidemic model from just looking at the model's outputs.
Marcus: Right. Basically, they set up a test where they compare what large language models say about structural identifiability against real results from published papers or from symbolic analysis done with Julia code.
Yuki: The main point is that fitting an epidemic model to data doesn't automatically mean the parameters are uniquely determined; different parameter settings can produce the same observed behavior, even if you see continuous data one <ref:2611.14184#pg1>. So this work checks if we can reliably use AI to flag those cases.
Ines: Exactly. They constructed a benchmark with eighty-eight scenarios from fourteen studies, looking at six hundred ninety-six formal classifications on page one of this paper <ref:2611.14184#pg2>. It matters because structural identifiability is the prerequisite for interpreting any parameter estimates we get from these models <ref:2611.14184#pg3>.
Marcus: What they claim is that they tested this using two different prompting styles, Domain-Aware and a canonically relabeled one called Domain-Anonymized <ref:2611.14184#pg2>. They also looked at how sensitive the results were to whether they used symbols versus actual names for the variables <ref:2611.14184#pg3>.
Yuki: It’s interesting that they found anonymization generally improved performance, especially when things got complicated in those scenarios <ref:2611.14184#pg2>. That suggests that the way we write things down matters a lot for the AI's judgment.
Ines: And they looked at how adding more unknown parameters or more observed outputs affects the AI’s accuracy <ref:2611.14184#pg9>. They found that each extra unknown parameter actually lowered the odds of a correct classification, with an odds ratio of zero point eight eight one <ref:2611.14184#pg9>.
Marcus: But they also saw that having more observed outputs increased those odds, with an odds ratio of one point six six six <ref:2611.14184#pg9>. That suggests more data helps the AI distinguish the parameters better in these scenarios.
Yuki: I wonder what this means for us in population genetics or epidemiology—if we can use this to filter out models that are structurally unidentifiable before we spend time fitting them to real data?
Ines: Well, they found that some parameters, the locally identifiable ones, were the hardest class for the AI to get right, with a strict recall of forty-two percent among their complete configurations <ref:2611.14184#pg9>. That’s a low number for something so specific.
Marcus: And they also found that in multivariable analysis, adding an extra unknown parameter reduced the odds of being right by that factor of zero point eight eight one <ref:2611.14184#pg9>. It's a small effect, but it points to how sensitive these models are to the number of variables we include.
Yuki: So, if we're looking at a complex disease model with lots of interacting factors, the paper suggests that as soon as you add another unknown parameter without enough data to break the symmetry, your chances of getting a correct structural classification go down.
Paper summary: Ines: Right. The researchers found that GPT-five point five Instant and GPT-five point five Thinking Medium had the highest accuracies among their complete runs, hitting eighty-eight percent and eighty-eight percent respectively <ref:2611.14184#pg9>. That’s a high bar for a preliminary classification task.
Marcus: And they also noted that canonical relabeling helped both of those configurations and actually made them better at handling scenarios with many unknown parameters or state variables <ref:2611.14184#pg2>. That's a useful piece of information about prompt engineering for this kind of task.
Yuki: So, the paper isn't saying the AI is perfect, but it’s showing us how it can act as a preliminary hypothesis generator or at least flag cases that need a deeper dive into formal symbolic methods <ref:2611.14184#pg2>. It’s not a final verdict.
Ines: That's the key caveat they put in <ref:2611.14184#pg2>. They explicitly state that LLM outputs should not be used to certify the structural identifiability of an entire model <ref:2611.14184#pg2>. You still need published results or formal symbolic tools to check their claims.
Marcus: So, if you're using this for cohort analysis, you use it as a starting point, not as proof that your specific set of parameters is uniquely determined <ref:2611.14184#pg9>. It’s a tool for flagging uncertainty before diving into the heavy lifting of formal analysis.
Yuki: For the listener, what this means is that when you see an epidemic model fit to data, don't just trust that it works; use these LLMs as a way to quickly ask, "Is there a mathematical reason why we can't find this parameter uniquely?"
Ines: Right. The paper also points out that formal paper-based analysis or symbolic computation is still the standard for proving structural identifiability <ref:2611.14184#pg9>. The AI isn't replacing that rigorous math yet; it’s just helping to narrow down what needs that math.
Marcus: And they made sure to make their methods available, you can find the prompts and the Julia scripts on GitHub and their model responses are archived on Zenodo <ref:2611.14184#pg26>. It’s good practice for researchers to see how these things are being tested.
Yuki: So, this work shows the capability of AI in hypothesis generation for complex biological systems, but it clearly defines its limits—it flags potential issues, but it doesn't provide the final certification <ref:2611.14184#pg2>.
Ines: That’s the core message of "Benchmarking large language models for parameter-level structural identifiability classification in epidemic ODE models" <ref:2611.14184#pg2>. It's about using AI to guide the search for unique parameters rather than replacing it entirely.
Marcus: And for those of us working on real data, it’s a good reminder that batch effects and model complexity still require solid statistical checks before we make claims about the underlying biology <ref:2611.14184#pg3>.
Yuki: We can use this framework to ask smarter questions about the models we're building, knowing where the AI is strong enough to give us a hint and where we still need to bring in the formal analysis <ref:2611.14184#pg9>.
Conclusion: Ines: So, we’ve been looking at how these large language models can guess if an epidemic model has unique parameters, and this paper is called "Benchmarking large language models for parameter-level structural identifiability classification in epidemic ODE models."
Marcus: Yeah, Ines. Basically it's testing whether these AIs can reliably sort out the structural identity of those complex epidemic differential equations.
Ines: Right. The authors took a big chunk of existing research and built their own test set with eighty-eight different scenarios from fourteen studies.
Yuki: I’m interested in how they defined that benchmark, because you need a solid way to measure if the AI is actually learning something useful about the biology behind the math.
Marcus: They used a mix of published results and some symbolic analysis done with Julia code to build this set, so it’s not just pure guesswork.
Ines: And they found that models like GPT-five point five Instant and GPT-five point five Thinking Medium actually performed the best on this test, hitting around eighty-eight percent accuracy on complete runs.
Yuki: Eighty-eight percent is high for this kind of task, so it shows there’s real potential for using these tools to quickly filter out models that are just structurally flawed before you get bogged down in fitting data.
Marcus: But the paper makes a big point about how sensitive the accuracy is to what you feed the AI, which means prompt engineering matters a lot when you're trying to get those right classifications.
Ines: Exactly. And they also highlighted that while these models are good at flagging possibilities, they absolutely should not be used to certify that a whole model is structurally identifiable on its own.
Yuki: So the implication for us is this: these AIs aren't replacing the formal math; they’re acting like a very fast first pass to point researchers toward where the real, rigorous analysis needs to happen.
Marcus: Right. It gives us a way to get ahead of that tedious part of model validation, letting us focus our time on the parts that truly matter for the cohort data we're looking at.
Ines: And it sets up a clear path forward for how we use these tools in computational biology, showing them as hypothesis generators rather than final decision-makers.
Yuki: It’s a useful tool for exploring which biological assumptions might be too restrictive or too loose based on the model's structure.
Marcus: So the big picture here is using this benchmarking to help us build better workflows for checking model assumptions before we start crunching massive datasets.
More episodes
- 2610.33881-Sensorimotor behavior from an untrained brain-and-nerve-cord connectome
- 2610.40344-Morphological Features of Murine Pulmonary Arterial Networks for Diagnosing Vascular Disease
- Daily Summary for 2026-11-19
- 2609.10193-Are You Learning Biological Signal or Shortcuts? Auditing and Mitigating Bias in Protein-Protein Interaction Datasets
- 2610.38236-Generative discovery of chemical reaction networks
- 2507.11176-An Interpretable AI framework Quantifying Traditional Chinese Medicine Principles Towards Enhancing and Integrating with Modern Biomedicine
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2504.03590-Towards a Unified Framework for Determining Conformational Ensembles of Disordered Proteins
- 2603.15217-A multiscale discrete-to-continuum framework for structured population models