Neurodata Without Boredom: Benchmarking Agentic AI for Data Reuse

Comprehensive Research Summary of Neurodata Without Boredom: Benchmarking Agentic AI for Data Reuse

This research paper addresses a critical bottleneck in neuroscience data science: the fragmentation, diverse formatting, and lack of descriptive documentation across laboratories when attempting to reuse large-scale neural population recordings. The core problem is the tension between creating formats flexible enough for diverse experimental paradigms and maintaining sufficient descriptive power without imposing an unsustainable burden of manual curation on researchers.

The authors propose Agentic AI as a powerful solution to this data integration challenge, leveraging the ability of Large Language Models (LLMs) to process code and text with sustained attention to low-level details that humans often overlook.

Methodology and Benchmark Framework

To rigorously evaluate agentic AI's performance on this task, the researchers constructed a realistic benchmark framework. This framework is based on eight recent papers detailing large-scale mouse neural population recordings. These selected datasets were deliberately chosen for their heterogeneity, spanning:

1. Diverse recording modalities (e.g., NWB formats).

2. Varied behavioral paradigms and experimental setups.

3. Heterogeneous data formats (including specialized APIs, general-purpose Python scripts, and MATLAB files).

The agents were tasked with a common downstream objective: training a decoder model designed to predict task or behavioral variables directly from the raw neural activity. The agents were provided with the raw data, associated analysis code, and the original research paper as context.

Evaluation Strategy

The evaluation was multi-faceted, employing both quantitative and qualitative measures:

1. Outcome-Level Metrics: These measure the final success of the agent's output (e.g., successful loading of data into a common format, successful training of a decoder model).

2. Process-Based Manual Rubric: This detailed rubric decomposed the complex data conversion process into numerous domain-relevant subtasks (e.g., data loading, neural signal processing, input/output variable construction). This approach was crucial because it revealed subtle failures and concerning decisions that aggregate output statistics alone would miss.

3. Agent-as-Judge Evaluation: The paper tested the reliability of agents to self-assess their work. They found that agents are unreliable judges of conversion quality, especially when no ground truth reference is provided, highlighting the necessity of human intervention.

Key Findings and Characterization of Failures

The research meticulously characterized the types of mistakes agents made and correlated these failures with specific properties of the input datasets:

• Subtask Success vs. End-to-End Failure: Agents performed well on individual subtasks but rarely managed to string together a fully error-free, end-to-end data conversion solution.

• Failure Modes Categorization: Failures were categorized into distinct types:

• FILTER: Inconsistent or unjustified choices during data filtering.

• TIME_RES: Incorrect decisions regarding temporal resolution (e.g., incorrect binning).

• PROCESS: Inconsistent implementation of complex data-processing pipelines.

• ASSUME/VARNAME: Incorrect assumptions about data semantics or variable naming conventions due to a lack of verification against raw data or metadata (e.g., incorrect variable mapping due to convention mismatch).

• MISC.

• Nature of Errors: Crucially, the majority of agent failures were not simple execution errors but rather reflected mistakes in scientific interpretation or processing decisions. Examples include incorrect temporal binning choices, improper handling of dropped frames, and misinterpreting variable mappings due to convention mismatches.

Proposed Solutions and Best Practices

The paper moves beyond simply identifying problems; it proposes actionable guidelines for the emerging agentic-AI era:

1. Data Sharing Best Practices: The most significant recommendation is that data releases must include working analysis code that directly interacts with the data and reproduces downstream variables end-to-end. This provides agents with an executable reference to interpret data structures correctly.

2. Documentation and Naming: Where detailed documentation is impractical, curators must at minimum ensure variable names are clear, unambiguous, and consistent.

3. Providing Processed Variables: Instead of forcing users or agents to reconstruct variables from raw data, datasets should provide the processed variables that downstream users are expected to use.

4. Sanity Checks: To improve reliability for iterative agent refinement, providing expected dataset statistics (e.g., expected number of sessions, trials, neurons) serves as a valuable sanity check target.

Conclusion: The Future Workflow

The paper concludes that current agentic AI tools can significantly lower the barrier to data reuse by automating tedious onboarding work when used within a human-in-the-loop workflow. While agents are powerful assistants, they are not yet reliable enough to serve as the primary quality control mechanism. Human review must be strategically focused on the subtasks where agents demonstrate the highest propensity for error—namely, scientific interpretation and complex processing decisions. The ultimate goal is an agentic system supported by robust data-sharing standards that make human oversight targeted and efficient.

1. Implement agentic AI workflows that strictly adhere to a sequential, iterative process by mandating Done when criteria at every step of data conversion to ensure error checking is continuous rather than episodic. This addresses the finding that agents rarely strung together a fully error-free end-to-end solution.

2. Develop agents capable of producing intermediate documentation, such as the required CONVERSION NOTES.md, at the start of every task and updating its status at each step to track progress. This enforces a rigorous self-review loop, mirroring the Check 1: Output log verification and Check 3: Reference code comparison requirements outlined in Step 10.

3. Enhance agent reliability by integrating external reference checks directly into the conversion script generation phase, ensuring agents use the same processing of the data described in the provided reference paper and code repository. This mitigates failures where agents bypassed the provided APIs and processed the raw data directly instead.

4. Improve agent judgment capability by refining their ability to distinguish between acceptable alternatives and incorrect solutions, potentially by training them on a finer-grained process-based rubric that captures subtle errors like those categorized as FILTER, TIME RES, or ASSUME.

5. Create an automated verification layer where agents are required to generate their own subtask decisions and ratings against the human rubric, allowing for a more reliable assessment of their judgment than relying solely on agent-as-judge results, which were found to be unreliable.

Neuroscience data are fragmented across labs, experimental paradigms, and storage formats, and data reuse often requires substantial manual effort. Standardized formats have been proposed in response, but a fundamental tension persists: formats flexible enough for diverse experiments are rarely self-explanatory, while fully descriptive datasets demand documentation and curation effort that few labs can sustain. Coding agents, which can read code and text quickly with apparent attention to details humans tend to skim, are a natural candidate to reduce this burden. To measure how well coding agents perform data reuse, we benchmark them on eight recent papers with large-scale mouse neural recordings spanning diverse recording modalities, behavioral paradigms, and formats. Given the data, code, and paper, the agents must reformat the data for a common downstream task: training a decoder from neural activity to task and behavioral variables. Evaluating the resulting solutions, however, is challenging, since many conversion decisions are under-specified and several choices are defensible. Yet a subtle but scientifically significant error can invalidate the dataset in ways aggregate metrics can miss. We therefore pair outcome-based metrics with a process-based rubric, scored by two human evaluators against manual reference solutions. We find that agents succeed on most subtasks but rarely produce a fully error-free end-to-end solution; we characterize their mistakes and suggest data-sharing practices for agent-assisted reuse. Finally, we find that agent judges are unreliable at catching errors on their own, but can automate process-based evaluation when given reference solutions.