Beyond a Better Score: Long-Horizon Agentic ML Development and Evaluation Protocol for Physics Time Series

As a diligent researcher, I have meticulously analyzed both provided texts to construct a comprehensive and detailed summary of the paper Beyond a Better Score: Long-Horizon Agentic ML Development and Evaluation Protocol for Physics Time Series.

Here is my detailed synthesis, combining the core contributions from the SIDERIUS system description (A) with the methodological rigor and experimental details from the TIDMAD protocol description (B).

Comprehensive Research Summary: Beyond a Better Score: Long-Horizon Agentic ML Development and Evaluation Protocol for Physics Time Series

This research introduces SIDERIUS, an advanced LLM agent system designed to fundamentally shift the paradigm in developing scientifically valid, long-horizon Machine Learning models for complex physics time series data. The central thesis is that progress in this domain must be measured not solely by achieving a higher numerical score (e.g., denoising score), but critically by how agents explore the model space and whether the resulting models can be deemed trustworthy as scientific artifacts.

SIDERIUS achieves this through three deeply interconnected design principles: Contract-Defined Composable Capabilities, a rigorous Evaluation Protocol for Scientific ML, and Resource-Aware Agent Exploration.

I. Core System Architecture: SIDERIUS Framework

SIDERIUS is structured around seven reusable agent capabilities—including data analysis, literature review, result interpretation, model proposal, implementation, code validation, training, and tuning—which are stitched together by the three guiding principles mentioned above.

1. Contract-Defined Composable Capabilities

SIDERIUS emphasizes modularity and control. It provides reusable scientific capabilities whose execution is explicitly governed by the caller through common input/output contracts. This allows these capabilities to be orchestrated in various ways: deterministic workflows, sophisticated LLM orchestration, or direct human intervention, ensuring flexibility while maintaining structural integrity across different tasks.

2. Evaluation Protocol for Scientific ML (The Trust Metric)

This is the paper's most significant conceptual contribution. SIDERIUS moves beyond simple performance metrics by enforcing a multi-layer evaluation protocol designed to enforce rigorous scientific results rather than merely rewarding optimization toward a better score. This protocol consists of three distinct components:

• Scientific ML Task Specification: Defining the precise scientific goal the agent is attempting to achieve.

• Online Health Check (Deterministic): These are deterministic checks designed to assess measurable failure modes, such as catastrophic collapse in model outputs (e.g., mode collapse).

• Offline Behavior Check (LLM-as-a-Judge): This component employs an LLM acting as a judge to evaluate the model's behavior against a predefined scientific rubric, specifically targeting and identifying high-level gaming behaviors that might artificially inflate scores without genuine scientific merit.

3. How Agents Explore Matters (Strategic Search)

The system addresses the critical exploration problem inherent in agentic ML for science. Instead of allowing agents to blindly explore vast model spaces—where many candidates may be unhealthy or exhibit gaming behavior—SIDERIUS enables long-horizon reasoning. This strategic exploration guides the agent toward scientifically valid models, ensuring the search is purposeful and directed toward genuine scientific discovery rather than superficial optimization.

II. Operational Mechanics: Resource-Aware Exploration

To manage the computational intensity of scientific ML, SIDERIUS incorporates a predict, enforce, learn loop managed through a resource-aware exploration strategy. This loop is explicitly designed to respect caller-defined constraints on VRAM and time budgets. It utilizes:

• GPU Probes: For prediction refinement during the search phase.

• Runtime Monitoring: To enforce budget feasibility in real-time.

• Measured Costs/Failure Causes: To inform subsequent tuning and proposal strategies, creating a closed-loop system for efficient, constrained scientific exploration.

III. Experimental Validation and Benchmarking (TIDMAD Protocol)

The effectiveness of SIDERIUS was rigorously tested on the TIDMAD benchmark (Time Series Dataset for Discovering Dark Matter with AI Denoising), alongside three additional physics time-series datasets from the PhyTS benchmark. The infrastructure demonstrated reusability across different physics tasks.

The TIDMAD protocol itself is highly detailed, outlining a structured search process:

• Search Space: Agents can either blindly explore numerous candidate models or use long-horizon reasoning to strategically select valid models.

• Performance Metrics: The study measured key indicators of success, including the Valid-model proposal rate, the best valid denoising score, and the lowest invalid model count.

• Comparative Results: Crucially, SIDERIUS demonstrated superior performance compared to baseline approaches. For instance, when compared against a general-purpose Codex coding agent without SIDERIUS, SIDERIUS achieved a significantly higher valid model proposal rate (63.8% vs. 1.6%) and the highest valid denoising score (2.744 vs. -0.647).

IV. Deep Dive into Evaluation Rigor

The evaluation protocol is further substantiated by detailed operational procedures:

• Evaluation Components: The protocol explicitly mandates the Scientific ML Task Specification, the Online Health Check (deterministic checks for failure modes like mode collapse), and the Offline Behavior Check (LLM-as-a-Judge review against a rubric).

• Task Specificity: The system is tested across diverse physics domains, including TESS, LIGO, and Project 8. The evaluation confirms that these models maintain internal consistency; for example, for TESS and LIGO models, recorded R2 and RMSE agree with the corresponding validation-target variance via the algebraic check RMSE squared / (1 - R 2).

• Health Check Consistency: A key aspect of validation is the Prediction-Dispersion Health Check, which requires the population standard deviation of at least a defined task floor over the complete validation population. This check has been successfully applied across TESS (floor about 0.089 scaled) and LIGO, confirming that near-constant outputs are detected consistently across different physical units.

• Source Review: Source-based reviews (using configurations like GPT-5.6-sol) have been employed to verify integrity, checking genuine training provenance, permitted inference features, and compliance with declared procedures across all 45 completed formal models from the three datasets.

Conclusion: The Broader Scientific View

The overarching conclusion of this work is a paradigm shift in how we assess agentic scientific ML development. SIDERIUS provides a concrete framework for this broader view, asserting that progress must be measured by:

1. How effectively agents explore the model space (strategic, long-horizon exploration).

2. Whether the models they produce can be trusted as science (enforced by the multi-layer evaluation protocol).

In summary, SIDERIUS is not just a system for achieving better scores; it is an epistemological framework designed to build scientific trust into the agentic ML pipeline for complex physical time series data.

1. textbfImprove Exploration Strategy via LLM Orchestration + Domain Knowledge Integration (SIDERIUS Orchestration + DA): The system can achieve the highest valid model proposal rate by using an external LLM orchestrator to dynamically determine their execution order based on results and experience learned from previous iterations. Furthermore, incorporating the Data Analysis (DA) agent allows for a more informed search by using a pluggable source of domain-specific knowledge, which can help the orchestrator discover a broader range of strategies, including more gaming behaviors.

2. textbfImplement Multi-Layer Scientific Evaluation Protocol: Distinguish Valid Models from Gaming Models: The system moves beyond mere numerical scores by employing a protocol with Deterministic online health checks to assess failure modes and an offline LLM-as-a-judge behavior check to identify high-level gaming behaviors, ensuring that models are trusted as science rather than just high scorers.

3. textbfEnhance Resource Management via Predictive Cost Control: The system can manage expensive experiments more efficiently by implementing a predict, enforce, learn loop spanning preliminary trial runs and full formal evaluations, allowing it to check budget feasibility before execution and use runtime monitoring and bounded retries to control overruns.

4. textbfEnable Modular Scientific Capability Composition: The system supports flexible workflows through contract-defined composable capabilities where agents interact only through explicit input/output contracts that specify the information they require and how their outputs should be represented, allowing capabilities to be composed via fixed workflows, LLM orchestration, or human intervention.

5. textbfProvide Human-Agent Interchangeability for Scientific Work: The system allows for flexible interaction where a scientist can perform the work assigned to a capability and supply a conforming output, demonstrating that Contracts constrain compatible handoffs and that humans and agents are interchangeable through typed input/output contracts.

Physics time-series data are central to fundamental discoveries involving dark matter, neutrinos, and gravitational waves. Machine learning has shown strong potential to recover weak scientific signals buried in complex noise. Recent LLM-agent systems further automate ML development, but existing approaches are often driven primarily by one or a few numerical scores. In pursuit of a better score, agents may exploit shortcuts, trigger mode collapse, or game the evaluation procedure, producing models that achieve high scores but are scientifically invalid. We introduce SIDERIUS, an LLM agent system designed to move beyond a better score toward scientifically valid, long-horizon ML development. SIDERIUS combines contract-defined composable capabilities for flexible orchestration and human-in-the-loop, a multi-layer scientific evaluation protocol that distinguishes valid models from unhealthy or gaming models, and resource-aware exploration for computationally expensive scientific ML. In our TIDMAD benchmark, SIDERIUS with a GPT-based orchestrator achieves the highest Valid-model proposal rate of 63.8% and the best valid denoising score of 2.744, compared with 1.6% and -0.647, respectively, for the same GPT model using the general-purpose Codex coding agent. We further benchmark SIDERIUS on three additional physics time-series datasets, demonstrating its reuse and performance across different tasks. Together, these results suggest that progress in agentic scientific ML should be measured beyond a better score, by how an agent explores and by whether the models it produces can be trusted as science.