Hard to Fool: A Bounded LLM Research Loop for Quantitative Finance, and What Two Dozen Campaigns Found

summary

The gist

Agentic research loops that let a language model propose a change, run a fixed experiment and keep whatever improves the metric work well in machine learning…

In short

Autonomous Quant Researcher uses a loop where an LLM proposes trading ideas, but only trusted code decides. It focuses on making it hard to report false discoveries by using strict boundaries for proposals and requiring multiple gates for acceptance, prioritizing well-documented 'no's over finding new signals.

Key concepts

Declarative Hypothesis Boundary
This restricts the LLM's proposals to a fixed vocabulary of ideas. Instead of letting the model generate executable code, it emits data points. This limits the attack surface and ensures every proposal is something the evaluator can actually price, making false positives less likely.
Staged Evidence Lifecycle
This is a structured process for testing ideas. It involves discovery, validation using a one-use holdout period enforced by a durable receipt, and admission control that prevents duplicate tests. This ensures each idea gets exactly one chance to be tested before being discarded.
Absolute Gates for Acceptance
Acceptance is not based on a single metric score. Instead, it requires passing several strict conditions simultaneously, such as achieving mean net P&L at elevated costs and meeting sample floors. This layered approach prevents reliance on any single, potentially misleading performance indicator.
Tamper-Evident Ledger
Every event in the research loop is recorded on a ledger that is tamper-evident. This provides a portable record of all proposals, evaluations, and decisions. It gives findings clear provenance and authority, making it easy to trace why a hypothesis was accepted or rejected.

Terminology used across episodes

This episode discusses

The paper

Hard to Fool: A Bounded LLM Research Loop for Quantitative Finance, and What Two Dozen Campaigns Found · Read on arXiv

Henry Zhang

Agentic research loops that let a language model propose a change, run a fixed experiment and keep whatever improves the metric work well in machine learning. In trading research the same loop is a machine for producing false discoveries: a backtest that "improves the metric" is, by default, overfit, look-ahead, a multiple-comparisons artifact, cost-free, or market drift wearing a strategy. We describe Autonomous Quant Researcher, an open-source loop built on the opposite premise: the model proposes, and only trusted code decides. Proposals are strict declarative JSON over a fixed vocabulary, so there is no model-written code to sandbox. Evaluation runs on immutable chronological stores split into discovery, validation and a final holdout that can be opened exactly once, after a durable receipt is written. Acceptance is a conjunction of absolute gates (positive edge and net P&L after costs, a positive weekly-cluster bootstrap lower bound, survival at elevated costs, per-year stability and sample floors), never a single score. Duplicates are refused before they spend a trial, and every formulation, refusal, crash and result is kept in a hash-chained, append-only ledger. We report every campaign run to date, across 0DTE options priced from traded option bars, overnight SPX verticals, daily ETF rules, technical-analysis premises, Kalshi crypto binaries, earnings options, off-exchange volume and a pre-registered alternative-data campaign. Most hypotheses fail, and we report them. Two results survive their pre-registered bars. The clearest example of the loop working is a failure: a language-model survivor that passed discovery and validation with Sharpe ratios near 3 and fell to 0.26 on its single holdout use. The value of such a loop lies less in what it finds than in how hard it is to make it report something that is not there.

Transcript

Introduction to the show: ident: Quantitative Finance. The day in quantitative finance: models of markets, risk and prices.

Ruth: We're looking at the paper: "Hard to Fool".

Warren: Here's the gist of the paper. Agentic research loops that let a language model propose a change, run a fixed experiment and keep whatever improves the metric work well in machine learning, where a lower validation loss is usually a better model In trading research the same loop is a machine for producing false discoveries The clearest example of the loop working is a failure: a language-model survivor that passed discovery and validation with Sharpe ratios near 3 and fell to 0.26 on its single holdout use This paper describes Autonomous Quant Researcher, an open-source loop built on the opposite premise where the model is allowed to propose, and only trusted code is allowed to decide The scientific value of such a loop lies less in what it finds than in how hard it is to make it report something that is not there Pruning redundant reasoning tokens The paper introduces several key contributions to this research loop, focusing on making the process robust against false positives and overfitting.

Ruth: These contributions include a declarative hypothesis boundary where the model emits data rather than code, fixing the attack surface.

Warren: It also establishes a staged evidence lifecycle featuring a one-use holdout enforced by a durable receipt and admission control that refuses duplicates before they spend a trial.

Ruth: Furthermore, acceptance is defined as a conjunction of absolute gates, such as positive edge after costs and survival at elevated costs, never relying on single scores.

Paper summary: Ruth: Thinking about the title "Hard to Fool: A Bounded LLM Research Loop for Quantitative Finance, and What Two Dozen Campaigns Found," it really captures the essence of what they built here. The authors are showing that by bounding the language model's proposal vocabulary and making sure there's this strict evidence lifecycle, you can build a system that resists simple overfitting.

Warren: I agree. It’s not about the AI being smarter; it’s about controlling the environment so it can’t just find noise as an answer. The two dozen campaigns they ran are less important than the fact that they showed how hard it is to get past those gates when you try to claim a result that isn't there >

Iris: From my econometric view, I see this as a very strong framework for interpretation. By forcing the acceptance criteria to be a conjunction of absolute gates—like having positive net P andL at elevated costs—it forces the model to survive real-world friction, not just look good on paper on some static metric >

Ruth: It matters because it changes the baseline for what we consider valid research output. If we can build a loop that systematically produces well-documented "no"s, then that documented refusal becomes the most reliable piece of data you have >

Warren: The implication is that for quantitative research, the goal should shift from chasing high validation loss reduction to building a process where reporting anything without absolute proof is actually more dangerous than saying nothing >

Conclusion: Ruth: So, we've seen how this autonomous research loop works—the AI proposes changes, runs tests, and keeps what works—and now we're looking at the conclusion of this paper on "Hard to Fool."

Warren: It really boils down to this idea that the scientific value isn't in finding something new; it’s in making it ridiculously hard for the AI to report something that isn't there.

Iris: That makes sense from a validation standpoint. They're not just looking for low error rates anymore; they’re building gates that check for survival at high costs, like surviving a big loss or hitting certain minimum profit levels after expenses.

Ruth: Exactly. The authors are showing how this structure creates these well-documented "no"s. It’s about producing evidence of failure more reliably than evidence of success.

Warren: The most telling example they show is with that QQQ 0DTE campaign where the AI proposed a rule that passed all the initial tests but then failed the final holdout because its expected return just dropped.

Iris: And what they found in that failure analysis is actually kind of important—it showed that those near-money options have this real premium because they absorb those unpredictable first-day shocks.

Ruth: So, for someone just listening to the show, it means we should judge a research loop first by how hard it is to trick and second by what it actually finds.

Warren: It shifts the focus away from chasing that perfect validation loss reduction and toward building a process where reporting anything without absolute proof is much riskier.

Iris: And they've actually shown how the system spots its own blind spots, like drift when alpha changes or when a simple discovery window gets too narrow and forces the AI to rethink what it’s seeing.

Ruth: The whole point seems to be that you need this kind of strict control over the proposal and evaluation phases to avoid those common pitfalls in finance research.

Warren: It’s about making sure the entire loop is tamper-evident, with a ledger tracking every single event so you can always see exactly why a decision was made.

Iris: It’s a very practical design because it forces you to be explicit about what costs are being measured versus just assumed.

Ruth: So the core message is that publishing the system's most common output, which is that well-documented "no," is actually the correct baseline for what we should rely on.

Warren: It’s about reproducibility and having a clear structure so other researchers can trust exactly how you got to your conclusion.

Iris: This structure seems like it could be a really useful framework for how we approach building more robust testing loops in this domain.

More episodes

← Home