Hard to Fool: A Bounded LLM Research Loop for Quantitative Finance, and What Two Dozen Campaigns Found

arXiv:2611.06607 · q-fin.CP, q-fin.TR · Submitted 2026-11-04 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Quantitative Finance. The day in quantitative finance: models of markets, risk and prices.

Ruth: We're looking at the paper: "Hard to Fool".

Warren: Here's the gist of the paper. Agentic research loops that let a language model propose a change, run a fixed experiment and keep whatever improves the metric work well in machine learning, where a lower validation loss is usually a better model In trading research the same loop is a machine for producing false discoveries The clearest example of the loop working is a failure: a language-model survivor that passed discovery and validation with Sharpe ratios near 3 and fell to 0.26 on its single holdout use This paper describes Autonomous Quant Researcher, an open-source loop built on the opposite premise where the model is allowed to propose, and only trusted code is allowed to decide The scientific value of such a loop lies less in what it finds than in how hard it is to make it report something that is not there Pruning redundant reasoning tokens The paper introduces several key contributions to this research loop, focusing on making the process robust against false positives and overfitting.

Ruth: These contributions include a declarative hypothesis boundary where the model emits data rather than code, fixing the attack surface.

Warren: It also establishes a staged evidence lifecycle featuring a one-use holdout enforced by a durable receipt and admission control that refuses duplicates before they spend a trial.

Ruth: Furthermore, acceptance is defined as a conjunction of absolute gates, such as positive edge after costs and survival at elevated costs, never relying on single scores.

Paper summary: Ruth: Thinking about the title "Hard to Fool: A Bounded LLM Research Loop for Quantitative Finance, and What Two Dozen Campaigns Found," it really captures the essence of what they built here. The authors are showing that by bounding the language model's proposal vocabulary and making sure there's this strict evidence lifecycle, you can build a system that resists simple overfitting.

Warren: I agree. It’s not about the AI being smarter; it’s about controlling the environment so it can’t just find noise as an answer. The two dozen campaigns they ran are less important than the fact that they showed how hard it is to get past those gates when you try to claim a result that isn't there >

Iris: From my econometric view, I see this as a very strong framework for interpretation. By forcing the acceptance criteria to be a conjunction of absolute gates—like having positive net P andL at elevated costs—it forces the model to survive real-world friction, not just look good on paper on some static metric >

Ruth: It matters because it changes the baseline for what we consider valid research output. If we can build a loop that systematically produces well-documented "no"s, then that documented refusal becomes the most reliable piece of data you have >

Warren: The implication is that for quantitative research, the goal should shift from chasing high validation loss reduction to building a process where reporting anything without absolute proof is actually more dangerous than saying nothing >

Conclusion: Ruth: So, we've seen how this autonomous research loop works—the AI proposes changes, runs tests, and keeps what works—and now we're looking at the conclusion of this paper on "Hard to Fool."

Warren: It really boils down to this idea that the scientific value isn't in finding something new; it’s in making it ridiculously hard for the AI to report something that isn't there.

Iris: That makes sense from a validation standpoint. They're not just looking for low error rates anymore; they’re building gates that check for survival at high costs, like surviving a big loss or hitting certain minimum profit levels after expenses.

Ruth: Exactly. The authors are showing how this structure creates these well-documented "no"s. It’s about producing evidence of failure more reliably than evidence of success.

Warren: The most telling example they show is with that QQQ 0DTE campaign where the AI proposed a rule that passed all the initial tests but then failed the final holdout because its expected return just dropped.

Iris: And what they found in that failure analysis is actually kind of important—it showed that those near-money options have this real premium because they absorb those unpredictable first-day shocks.

Ruth: So, for someone just listening to the show, it means we should judge a research loop first by how hard it is to trick and second by what it actually finds.

Warren: It shifts the focus away from chasing that perfect validation loss reduction and toward building a process where reporting anything without absolute proof is much riskier.

Iris: And they've actually shown how the system spots its own blind spots, like drift when alpha changes or when a simple discovery window gets too narrow and forces the AI to rethink what it’s seeing.

Ruth: The whole point seems to be that you need this kind of strict control over the proposal and evaluation phases to avoid those common pitfalls in finance research.

Warren: It’s about making sure the entire loop is tamper-evident, with a ledger tracking every single event so you can always see exactly why a decision was made.

Iris: It’s a very practical design because it forces you to be explicit about what costs are being measured versus just assumed.

Ruth: So the core message is that publishing the system's most common output, which is that well-documented "no," is actually the correct baseline for what we should rely on.

Warren: It’s about reproducibility and having a clear structure so other researchers can trust exactly how you got to your conclusion.

Iris: This structure seems like it could be a really useful framework for how we approach building more robust testing loops in this domain.

Henry Zhang

q-fin.CP, q-fin.TR

Submitted: 2026-11-04

Updated: 2026-11-04

Comments: 10 pages, 2 tables. Code: https://github.com/henryzhangpku/autonomous-quant-researcher

Code: https://github.com/henryzhangpku/autonomous-quant-researcher

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 90/100

The gist: Agentic research loops that let a language model propose a change, run a fixed experiment and keep whatever improves the metric work well in machine learning…

Key concepts

Declarative Hypothesis Boundary
This restricts the LLM's proposals to a fixed vocabulary of ideas. Instead of letting the model generate executable code, it emits data points. This limits the attack surface and ensures every proposal is something the evaluator can actually price, making false positives less likely.
Staged Evidence Lifecycle
This is a structured process for testing ideas. It involves discovery, validation using a one-use holdout period enforced by a durable receipt, and admission control that prevents duplicate tests. This ensures each idea gets exactly one chance to be tested before being discarded.
Absolute Gates for Acceptance
Acceptance is not based on a single metric score. Instead, it requires passing several strict conditions simultaneously, such as achieving mean net P&L at elevated costs and meeting sample floors. This layered approach prevents reliance on any single, potentially misleading performance indicator.
Tamper-Evident Ledger
Every event in the research loop is recorded on a ledger that is tamper-evident. This provides a portable record of all proposals, evaluations, and decisions. It gives findings clear provenance and authority, making it easy to trace why a hypothesis was accepted or rejected.

Terminology

Summary

Agentic research loops that let a language model propose a change, run a fixed experiment and keep whatever improves the metric work well in machine learning, where a lower validation loss is usually a better model In trading research the same loop is a machine for producing false discoveries The clearest example of the loop working is a failure: a language-model survivor that passed discovery and validation with Sharpe ratios near 3 and fell to 0. p. 0

Where the paper says it

com/henryzhangpku November 2026 Abstract Agentic research loops that let a language model propose a change, run a fixed experiment and keep whatever improves the metric work well in machine learning, where a lower validation loss is usually a better model. In trading research the same loop is a machine for producing false discoveries: a backtest that “improves the metric” is, by default, overfit, look-ahead, a multiplecomparisons artifact, cost-free, or market drift wearing a strategy.

26 on its single holdout use This paper describes Autonomous Quant Researcher, an open-source loop built on the opposite premise where the model is allowed to propose, and only trusted code is allowed to decide The scientific value of such a loop lies less in what it finds than in how hard it is to make it report something that is not there

Pruning redundant reasoning tokens

The paper introduces several key contributions to this research loop, focusing on making the process robust against false positives and overfitting. These contributions include a declarative hypothesis boundary where the model emits data rather than code, fixing the attack surface. p. 1

Where the paper says it

Concretely, this paper contributes: 1. A declarative hypothesis boundary: the model emits data, not code, so the attack surface and the space of searchable hypotheses are both fixed in advance (Section 2.

It also establishes a staged evidence lifecycle featuring a one-use holdout enforced by a durable receipt and admission control that refuses duplicates before they spend a trial. p. 1
Where the paper says it

A declarative hypothesis boundary: the model emits data, not code, so the attack surface and the space of searchable hypotheses are both fixed in advance (Section 2. A staged evidence lifecycle with a one-use holdout enforced by a durable receipt, admission control that refuses duplicates before they spend a trial, and acceptance by a conjunction of absolute gates (Section 2.

Furthermore, acceptance is defined as a conjunction of absolute gates, such as positive edge after costs and survival at elevated costs, never relying on single scores. p. 0
Where the paper says it

Evaluation runs on immutable chronological stores separated into discovery, validation and a final holdout that can be opened exactly once, after a durable receipt is written. Acceptance is a conjunction of absolute gates (positive edge after costs, positive net P&L, a positive weekly-cluster bootstrap lower bound, survival at elevated costs, per-year stability and sample floors), never a single score.

A tamper-evident ledger records every event, providing portable findings that carry their own provenance and authority. p. 1
Where the paper says it

A tamper-evident ledger and portable findings that carry their own provenance and authority (Section 2. A complete campaign report, including every rejection (Section 3).

How the training signal is built

The design of the loop emphasizes strict control over the proposal and evaluation phases to mitigate common pitfalls in quantitative finance research. The declarative hypothesis boundary restricts proposals to a fixed vocabulary, ensuring that every admissible hypothesis is one the evaluator knows how to price, removing the need for sandboxing executable code. p. 2

Where the paper says it

That design needed a sandbox (credential stripping, import and capability denial, a content-hashed evaluator) and it still left the candidate free to express things the evaluator did not anticipate. Removing the executable field removes the problem rather than containing it: there is nothing to sandbox, and every admissible hypothesis is, by construction, one the evaluator knows how to price.

Evaluation runs on immutable chronological stores separated into discovery, validation and a final holdout that can be opened exactly once after a durable receipt is written. p. 0
Where the paper says it

Proposals are strict declarative JSON over a fixed vocabulary, so there is no model-written code to sandbox. Evaluation runs on immutable chronological stores separated into discovery, validation and a final holdout that can be opened exactly once, after a durable receipt is written.

Acceptance requires passing multiple absolute gates, including mean net P&L at elevated costs and sample floors, rather than relying on a single metric score. p. 0
Where the paper says it

Evaluation runs on immutable chronological stores separated into discovery, validation and a final holdout that can be opened exactly once, after a durable receipt is written. Acceptance is a conjunction of absolute gates (positive edge after costs, positive net P&L, a positive weekly-cluster bootstrap lower bound, survival at elevated costs, per-year stability and sample floors), never a single score.

Where the gains come from

The results demonstrate that the system is designed to produce well-documented "no"s, making it hard to report something that is not there. The most instructive result involves a QQQ 0DTE campaign where a language-model proposer produced a rule that passed every discovery and validation gate but failed the holdout due to falling expectancy. p. 5

Where the paper says it

1 A language-model survivor meets its holdout The most instructive single result is a QQQ 0DTE campaign in which the language-model proposer, in its thirteenth run, produced a rule that sold the near-money put credit spread on Mondays and Tuesdays when the credit exceeded 10% of width. On the traded-bar windows above, and under that campaign’s own floors (40, 15 and 15 trades), it passed every discovery and validation gate (Table 2).

The analysis shows that the near-money 0DTE put premium is real, has the same shape on two underlyings, and earns its return by absorbing unforecastable first-day shocks. p. 6
Where the paper says it

The honest summary is that the near-money 0DTE put premium is real, has the same shape on two underlyings, is mostly not drift, and earns its return by absorbing unforecastable first-day shocks. The pre-registered bar, which asks for a positive fifth percentile of weekly means across a window containing three such shocks, is not met, and the bar was not moved.

The paper concludes that a research loop in quantitative finance should be judged first by how hard it is to fool and only second by what it finds. p. 7
Where the paper says it

Finally, this paper is written by the system’s author; the full campaign records behind each number are available from the author on request. 6 Conclusion A research loop in quantitative finance should be judged first by how hard it is to fool and only second by what it finds.

Failure modes the gates missed, and the change each forced

The system identifies specific failure modes that the initial gates did not catch, leading to necessary changes in campaign parameters. Examples of these failures include drift as alpha, where an unconditional long-delta credit spread collects market’s upward drift regardless of mispricing. p. 6

Where the paper says it

4 Failure modes the gates missed, and the change each forced Drift as alpha. An unconditional long-delta credit spread collects the market’s upward drift whether or not anything is mispriced.

Another failure mode was a one-direction discovery window, which the system corrected by recording that a passing pair is a long-side artifact rather than a signal. p. 6
Where the paper says it

Gates evaluated on one window cannot see that the window had one direction. The rule recorded with the grid is that a condition is read together with its opposite, and a passing pair is a long-side artifact, not a signal.

The paper also notes that costs assumed, not measured, led to two candidates being killed by a doubled-cost gate against an unmeasured cost. p. 6
Where the paper says it

Costs assumed, not measured. Two candidates were killed by a doubled-cost gate against a cost nobody had measured, and one holdout was spent before its cost premise was measured.

Limitations

The limitations of the loop include the fact that its vocabulary is deliberately small, meaning a hypothesis outside it cannot be tested until trusted code is extended. Several campaigns use short samples, and their nulls are statements about power as much as about markets. p. 7

Where the paper says it

Its vocabulary is deliberately small, and a hypothesis outside it cannot be tested until trusted code is extended. Several campaigns use short samples (for example 46 days of Kalshi data, 60 weekends of ETF flow), and their nulls are statements about power as much as about markets.

Option results from 2024 onward cover no bear market of 2022’s kind. p. 7
Where the paper says it

Several campaigns use short samples (for example 46 days of Kalshi data, 60 weekends of ETF flow), and their nulls are statements about power as much as about markets. Option results from 2024 onward cover no bear market of 2022’s kind.

Some modeled prices remain where actual quotes were not available, and costs in several campaigns are assumptions with stated breakevens rather than measurements. p. 7
Where the paper says it

Some modeled prices remain where actual quotes were not available; those campaigns say so. Costs in several campaigns are assumptions with stated breakevens rather than measurements.

Finally, this paper is written by the system’s author, and the full campaign records behind each number are available from the author on request. p. 7
Where the paper says it

Costs in several campaigns are assumptions with stated breakevens rather than measurements. Finally, this paper is written by the system’s author; the full campaign records behind each number are available from the author on request.

Conclusion

The engine code, a reference mission and the public pre-registration with its hashchained ledger are at https://github.com/henryzhangpku/autonomous-quant-researcher. p. 7

Where the paper says it

Engine code, a reference mission and the public pre-registration with its hashchained ledger are at https://github. com/henryzhangpku/autonomous-quant-researcher.

The test suite runs offline without keys, and campaign records are available from the author on request. p. 7
Where the paper says it

The test suite runs offline without keys. Campaign records are available from the author on request.

A browser demonstration runs the engine under Pyodide on a synthetic tape with a planted regime that discovery finds and validation refutes. p. 7
Where the paper says it

Campaign records are available from the author on request. A browser demonstration runs the engine under Pyodide on a synthetic tape with a planted regime that discovery finds and validation refutes.

The paper’s core message is that publishing the system’s most common output, which is a well-documented “no”, is the correct base rate. p. 7
Where the paper says it

Making the model propose data instead of code, separating stages physically, letting the holdout be spent once, accepting only on a conjunction of absolute gates, refusing duplicates before they cost a trial and keeping every failure in a chained ledger together produce a system whose most common output is a well-documented “no”. We think that is the correct base rate, and that publishing it is the point.

The reproducibility of the loop and its ledger structure are highlighted as key takeaways for future research in this domain. p. 7
Where the paper says it

6 Conclusion A research loop in quantitative finance should be judged first by how hard it is to fool and only second by what it finds. Making the model propose data instead of code, separating stages physically, letting the holdout be spent once, accepting only on a conjunction of absolute gates, refusing duplicates before they cost a trial and keeping every failure in a chained ledger together produce a system whose most common output is a well-documented “no”.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. A declarative hypothesis boundary limits model proposals to fixed vocabulary, ensuring the attack surface and the space of searchable hypotheses are both fixed in advance. This prevents models from proposing uninterpretable or executable code candidates.

  2. A staged evidence lifecycle enforces rigorous testing through a one-use holdout secured by a durable receipt, preventing strategies from surviving by luck or multiplecomparisons artifact. Acceptance requires a conjunction of absolute gates, such as positive edge after costs and a survival at elevated costs, rather than a single metric score.

  3. A tamper-evident ledger provides complete provenance, recording every event like formulation, refusal, crash and result in a hash-chained, append-only ledger. This allows downstream systems to know exactly what they are holding by exporting findings as portable JSON built only from frozen aggregate evidence.

  4. The system can conduct comprehensive campaign reporting across diverse financial instruments like SPY and QQQ zero-days-to-expiry (0DTE) options priced from actual traded option bars and report every rejection, including a language-model survivor that passed discovery and validation with Sharpe ratios near 3 and fell to 0.26 on its single holdout use.

  5. The system can identify failure modes the gates did not catch, such as when the control-relative metric is computed but is diagnostic only, forcing changes like adding a drift control (regress each trade on the underlying’s move over the same hold).

Abstract

Agentic research loops that let a language model propose a change, run a fixed experiment and keep whatever improves the metric work well in machine learning. In trading research the same loop is a machine for producing false discoveries: a backtest that "improves the metric" is, by default, overfit, look-ahead, a multiple-comparisons artifact, cost-free, or market drift wearing a strategy. We describe Autonomous Quant Researcher, an open-source loop built on the opposite premise: the model proposes, and only trusted code decides. Proposals are strict declarative JSON over a fixed vocabulary, so there is no model-written code to sandbox. Evaluation runs on immutable chronological stores split into discovery, validation and a final holdout that can be opened exactly once, after a durable receipt is written. Acceptance is a conjunction of absolute gates (positive edge and net P&L after costs, a positive weekly-cluster bootstrap lower bound, survival at elevated costs, per-year stability and sample floors), never a single score. Duplicates are refused before they spend a trial, and every formulation, refusal, crash and result is kept in a hash-chained, append-only ledger. We report every campaign run to date, across 0DTE options priced from traded option bars, overnight SPX verticals, daily ETF rules, technical-analysis premises, Kalshi crypto binaries, earnings options, off-exchange volume and a pre-registered alternative-data campaign. Most hypotheses fail, and we report them. Two results survive their pre-registered bars. The clearest example of the loop working is a failure: a language-model survivor that passed discovery and validation with Sharpe ratios near 3 and fell to 0.26 on its single holdout use. The value of such a loop lies less in what it finds than in how hard it is to make it report something that is not there.

Sources

Related papers