Prospective multi-pathogen disease forecasting using autonomous LLM-guided tree search

Comprehensive Research Summary: Autonomous LLM-Guided Tree Search for Prospective Multi-Pathogen Disease Forecasting

This research presents a groundbreaking autonomous system, named Empirical Research Assistance (ERA), designed to overcome the significant labor bottleneck in developing expert-level epidemiological forecasting software. The core innovation lies in leveraging a Large Language Model (LLM) as an intelligent agent to iteratively generate, evaluate, and optimize executable forecasting models across multiple infectious disease pathogens—specifically influenza, COVID-19, and Respiratory Syncytial Virus (RSV)—in a fully prospective, real-time setting.

Core Methodology: ERA Framework

The ERA framework automates the development of empirical software for scorable scientific tasks by searching over the vast space of possible code implementations using a tree search algorithm. The LLM acts as the central agent within this search, responsible for generating, testing, and refining Python code based on natural language problem statements derived from epidemiological theory.

Key System Components:

1. LLM-Guided Tree Search: This is the engine of ERA. It explores a high-dimensional solution space by navigating a tree structure where each node represents a concrete implementation—encompassing specific combinations of feature engineering, data preprocessing techniques, and model architectures.

2. LLM Agent Role: The LLM agent is tasked with translating complex epidemiological theory into accurate, transparent code and iteratively refining it through testing against predefined objectives.

3. Model Generation Regimes: To ensure methodological diversity and robustness, the system employs three distinct prompt regimes for model generation:

• Single Model Adaptations: Leveraging existing methodologies.

• Recombinations: Synthesizing pairs of distinct methodological frameworks.

• Novel Methods: Engaging in unconstrained searches to discover entirely new approaches.

Experimental Setup and Evaluation

The study was conducted using a fully prospective, real-time evaluation setting spanning the 2025–2026 US respiratory season, utilizing data from three major CDC Forecast Hubs: FluSight (Influenza), COVIDHub (COVID-19), and RSVHub (RSV). The system generated weekly forecasts in real time with auditable timestamps.

Evaluation Metrics:

• Primary Metric: The Weighted Interval Score (WIS), a proper scoring rule that simultaneously penalizes both lack of precision and poor calibration.

• Model Comparison: Pairwise relative WIS is used as the primary metric in aggregated settings, calculated by taking the ratio of mean scores over the intersection of tasks submitted by two models and geometrically averaging this ratio across all opponents.

Key Findings and Contributions

The results demonstrate that ERA successfully generates models that are not only methodologically diverse but also competitively performant:

1. State-of-the-Art Performance: The aggregated ensemble of machine-generated models consistently matched or outperformed the gold-standard, human-curated CDC hub ensembles out-of-sample across all three pathogens. The Google AI (Gemini) ensembles achieved top performance across the board.

2. Robustness to Data Scarcity (Cold Start): ERA successfully navigated data-scarce cold start scenarios for RSV—a critical test demonstrating the system's ability to discover effective approaches without extensive domain-specific guidance, relying on unconstrained architectural searches.

3. Optimization Guidance and Reward Hacking Mitigation: Controlled retrospective ablations were crucial for understanding the search dynamics:

• Log-scale CRPS Optimization: This metric was identified as the most reliable optimization target. It successfully combined CRPS’s smooth reward landscape with superior validation/test agreement, avoiding the pitfalls of other metrics.

• Log Score Pathology: The Log Score metric was found to produce forecasts that were systematically too wide and biased downward, characterized by high between-replica variance rather than systematic overfitting.

4. Structural Fidelity via Judge-in-the-Loop: An automated judge implemented in the loop acts as a binary gate, zeroing out non-compliant branches during the search. This mechanism was highly effective at steering the search toward valid spatial time-series implementations, significantly improving both fidelity (from one to three matches) and mean WIS.

5. Methodological Discovery: The system demonstrated its capacity to generate models across a spectrum of complexity, from standard machine learning regressors (e.g., UMass-gbqr using LightGBM) to highly complex structures like stochastic, age-structured metapopulation models simulating disease spread via mobility networks.

6. Transparency and Reproducibility: A distinctive feature is the explicit re-writing of existing CDC hub models, allowing for a direct, real-time comparison between the prospective performance of ERA-generated adaptations and the original methods submitting to the same hub.

Conclusion and Practical Implications

This work fundamentally addresses the modeling labor bottleneck by automating model ideation, implementation, and evaluation. By autonomously translating epidemiological theory into accurate code optimized via sophisticated search heuristics (like Log-scale CRPS), ERA democratizes access to expert-level forecasting capabilities. The framework enables rapid deployment of highly optimized models at vastly larger scales, offering a powerful demonstration of the potential for LLM-guided agentic systems in real-world, resource-limited public good settings. While the results suggest that human expertise remains valuable for highlighting exploration opportunities, ERA proves its capacity to discover and implement state-of-the-art forecasting solutions autonomously.

1. textbfExtreme Scalability via Automation of Model Ideation and Implementation: The system can rapidly produce a large, methodologically diverse pool of competitive models whose ensemble matches or exceeds the gold-standard CDC hub ensembles. This overcomes the bottleneck where labor intensive manual model curation is required for granular geographic resolutions or emerging pathogens.

2. textbfRobustness Against Data Scarcity: The system can successfully navigate data-scarce cold start scenarios, as demonstrated by its performance on RSV, where it utilized unconstrained architectural search in which the system discovered effective approaches without domain-specific guidance.

3. textbfGuaranteed Code Transparency and Auditability: Every generated model is a complete, executable Python program whose source code is publicly available under the Apache 2.0 open-source license, ensuring that forecasts can be read, audited, and modified by any domain scientist.

4. textbfAdaptive Optimization Logic: The system can dynamically select the most reliable reward signal for search exploration; specifically, it identifies "Log-scale CRPS as the most reliable optimization target—it combines CRPS’s smooth reward landscape with closer validation/test agreement than CRPS itself, and avoids the wide, downward-biased forecasts induced by Log Score’s infinite-penalty structure."

5. textbfAdaptive Instruction Following Gate: The system can employ a judge-in-the-loop mechanism that acts as a binary gate that zeroes out non-compliant branches, ensuring structural fidelity to complex scientific theories even when the underlying LLM struggles with specific constraints, such as in the case of the Cornell JHU hierarchSIR method where it produced one fidelity match.

6. textbfGuaranteed Out-of-Sample Generalization: The selection scheme prioritizes generalization by using a compound selection score of Selection Score = Validation WIS + 2 × Retrospective Test WIS, which places greater emphasis on out-of-search generalization than on in-search fit.

Probabilistic forecasting of infectious diseases is crucial for public health monitoring and preparedness but has historically relied on labor-intensive manual model curation by expert modeling teams. Bespoke development is an obstacle to scaling models to granular geographic resolutions or emerging pathogens. Here, we prospectively evaluate Empirical Research Assistance (ERA), an autonomous system that uses Large Language Model-guided tree search to iteratively generate, evaluate, and optimize executable forecasting software. ERA was primarily prompted with methodological approaches to implement, while in some instances it was tasked with developing forecasting approaches from scratch. In a fully prospective, real-time evaluation during the 2025-26 United States respiratory season, ERA generated methodologically diverse models for influenza, COVID-19, and respiratory syncytial virus (RSV). Aggregating a selection of these automatically-generated models yielded an ensemble that ranked first overall in official Centers for Disease Control and Prevention (CDC) post-season evaluations and outperformed the gold-standard CDC forecast ensembles across all three pathogens in a real-time prediction challenge. Additionally, ERA successfully navigated data-scarce scenarios for RSV. Controlled retrospective ablations allowed for additional explorations of the impact of optimization target and instruction-following on forecast accuracy. Preliminary experiments indicate that ERA's search optimization metric influences behavior of output forecasts, and using higher-tier language models in-search improves predictive performance. By autonomously generating interpretable code that creates epidemiologically relevant predictions, this framework provides a path toward rapidly scaling high-quality disease forecasting.