Hard to Fool: A Bounded LLM Research Loop for Quantitative Finance, and What Two Dozen Campaigns Found
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Quantitative Finance. The day in quantitative finance: models of markets, risk and prices.
Ruth: We're looking at the paper: "Hard to Fool".
Warren: Here's the gist of the paper. Agentic research loops that let a language model propose a change, run a fixed experiment and keep whatever improves the metric work well in machine learning, where a lower validation loss is usually a better model In trading research the same loop is a machine for producing false discoveries The clearest example of the loop working is a failure: a language-model survivor that passed discovery and validation with Sharpe ratios near 3 and fell to 0.26 on its single holdout use This paper describes Autonomous Quant Researcher, an open-source loop built on the opposite premise where the model is allowed to propose, and only trusted code is allowed to decide The scientific value of such a loop lies less in what it finds than in how hard it is to make it report something that is not there Pruning redundant reasoning tokens The paper introduces several key contributions to this research loop, focusing on making the process robust against false positives and overfitting.
Ruth: These contributions include a declarative hypothesis boundary where the model emits data rather than code, fixing the attack surface.
Warren: It also establishes a staged evidence lifecycle featuring a one-use holdout enforced by a durable receipt and admission control that refuses duplicates before they spend a trial.
Ruth: Furthermore, acceptance is defined as a conjunction of absolute gates, such as positive edge after costs and survival at elevated costs, never relying on single scores.
Paper summary: Ruth: Thinking about the title "Hard to Fool: A Bounded LLM Research Loop for Quantitative Finance, and What Two Dozen Campaigns Found," it really captures the essence of what they built here. The authors are showing that by bounding the language model's proposal vocabulary and making sure there's this strict evidence lifecycle, you can build a system that resists simple overfitting.
Warren: I agree. It’s not about the AI being smarter; it’s about controlling the environment so it can’t just find noise as an answer. The two dozen campaigns they ran are less important than the fact that they showed how hard it is to get past those gates when you try to claim a result that isn't there >
Iris: From my econometric view, I see this as a very strong framework for interpretation. By forcing the acceptance criteria to be a conjunction of absolute gates—like having positive net P andL at elevated costs—it forces the model to survive real-world friction, not just look good on paper on some static metric >
Ruth: It matters because it changes the baseline for what we consider valid research output. If we can build a loop that systematically produces well-documented "no"s, then that documented refusal becomes the most reliable piece of data you have >
Warren: The implication is that for quantitative research, the goal should shift from chasing high validation loss reduction to building a process where reporting anything without absolute proof is actually more dangerous than saying nothing >
Conclusion: Ruth: So, we've seen how this autonomous research loop works—the AI proposes changes, runs tests, and keeps what works—and now we're looking at the conclusion of this paper on "Hard to Fool."
Warren: It really boils down to this idea that the scientific value isn't in finding something new; it’s in making it ridiculously hard for the AI to report something that isn't there.
Iris: That makes sense from a validation standpoint. They're not just looking for low error rates anymore; they’re building gates that check for survival at high costs, like surviving a big loss or hitting certain minimum profit levels after expenses.
Ruth: Exactly. The authors are showing how this structure creates these well-documented "no"s. It’s about producing evidence of failure more reliably than evidence of success.
Warren: The most telling example they show is with that QQQ 0DTE campaign where the AI proposed a rule that passed all the initial tests but then failed the final holdout because its expected return just dropped.
Iris: And what they found in that failure analysis is actually kind of important—it showed that those near-money options have this real premium because they absorb those unpredictable first-day shocks.
Ruth: So, for someone just listening to the show, it means we should judge a research loop first by how hard it is to trick and second by what it actually finds.
Warren: It shifts the focus away from chasing that perfect validation loss reduction and toward building a process where reporting anything without absolute proof is much riskier.
Iris: And they've actually shown how the system spots its own blind spots, like drift when alpha changes or when a simple discovery window gets too narrow and forces the AI to rethink what it’s seeing.
Ruth: The whole point seems to be that you need this kind of strict control over the proposal and evaluation phases to avoid those common pitfalls in finance research.
Warren: It’s about making sure the entire loop is tamper-evident, with a ledger tracking every single event so you can always see exactly why a decision was made.
Iris: It’s a very practical design because it forces you to be explicit about what costs are being measured versus just assumed.
Ruth: So the core message is that publishing the system's most common output, which is that well-documented "no," is actually the correct baseline for what we should rely on.
Warren: It’s about reproducibility and having a clear structure so other researchers can trust exactly how you got to your conclusion.
Iris: This structure seems like it could be a really useful framework for how we approach building more robust testing loops in this domain.
Henry Zhang
q-fin.CP, q-fin.TR
Submitted: 2026-11-04
Updated: 2026-11-04
Comments: 10 pages, 2 tables. Code: https://github.com/henryzhangpku/autonomous-quant-researcher
Code: https://github.com/henryzhangpku/autonomous-quant-researcher
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 90/100
The gist: Agentic research loops that let a language model propose a change, run a fixed experiment and keep whatever improves the metric work well in machine learning…
Key concepts
- Declarative Hypothesis Boundary
- This restricts the LLM's proposals to a fixed vocabulary of ideas. Instead of letting the model generate executable code, it emits data points. This limits the attack surface and ensures every proposal is something the evaluator can actually price, making false positives less likely.
- Staged Evidence Lifecycle
- This is a structured process for testing ideas. It involves discovery, validation using a one-use holdout period enforced by a durable receipt, and admission control that prevents duplicate tests. This ensures each idea gets exactly one chance to be tested before being discarded.
- Absolute Gates for Acceptance
- Acceptance is not based on a single metric score. Instead, it requires passing several strict conditions simultaneously, such as achieving mean net P&L at elevated costs and meeting sample floors. This layered approach prevents reliance on any single, potentially misleading performance indicator.
- Tamper-Evident Ledger
- Every event in the research loop is recorded on a ledger that is tamper-evident. This provides a portable record of all proposals, evaluations, and decisions. It gives findings clear provenance and authority, making it easy to trace why a hypothesis was accepted or rejected.
Terminology
Summary
Agentic research loops that let a language model propose a change, run a fixed experiment and keep whatever improves the metric work well in machine learning, where a lower validation loss is usually a better model In trading research the same loop is a machine for producing false discoveries The clearest example of the loop working is a failure: a language-model survivor that passed discovery and validation with Sharpe ratios near 3 and fell to 0. p. 0 com/henryzhangpku
November 2026
Abstract
Agentic research loops that let a language model propose a change, run a fixed experiment
and keep whatever improves the metric work well in machine learning, where a lower validation
loss is usually a better model. In trading research the same loop is a machine for producing false
discoveries: a backtest that “improves the metric” is, by default, overfit, look-ahead, a multiplecomparisons artifact, cost-free, or market drift wearing a strategy.Where the paper says it
Pruning redundant reasoning tokens
The paper introduces several key contributions to this research loop, focusing on making the process robust against false positives and overfitting. These contributions include a declarative hypothesis boundary where the model emits data rather than code, fixing the attack surface. p. 1 Concretely, this paper contributes:
1. A declarative hypothesis boundary: the model emits data, not code, so the attack surface
and the space of searchable hypotheses are both fixed in advance (Section 2. A declarative hypothesis boundary: the model emits data, not code, so the attack surface
and the space of searchable hypotheses are both fixed in advance (Section 2. A staged evidence lifecycle with a one-use holdout enforced by a durable receipt, admission
control that refuses duplicates before they spend a trial, and acceptance by a conjunction of
absolute gates (Section 2. Evaluation runs on immutable
chronological stores separated into discovery, validation and a final holdout that can be opened
exactly once, after a durable receipt is written. Acceptance is a conjunction of absolute gates
(positive edge after costs, positive net P&L, a positive weekly-cluster bootstrap lower bound,
survival at elevated costs, per-year stability and sample floors), never a single score. A tamper-evident ledger and portable findings that carry their own provenance and authority
(Section 2. A complete campaign report, including every rejection (Section 3).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
How the training signal is built
The design of the loop emphasizes strict control over the proposal and evaluation phases to mitigate common pitfalls in quantitative finance research. The declarative hypothesis boundary restricts proposals to a fixed vocabulary, ensuring that every admissible hypothesis is one the evaluator knows how to price, removing the need for sandboxing executable code. p. 2 That design needed a sandbox (credential stripping, import and
capability denial, a content-hashed evaluator) and it still left the candidate free to express things
the evaluator did not anticipate. Removing the executable field removes the problem rather than
containing it: there is nothing to sandbox, and every admissible hypothesis is, by construction, one
the evaluator knows how to price. Proposals are strict declarative JSON over
a fixed vocabulary, so there is no model-written code to sandbox. Evaluation runs on immutable
chronological stores separated into discovery, validation and a final holdout that can be opened
exactly once, after a durable receipt is written. Evaluation runs on immutable
chronological stores separated into discovery, validation and a final holdout that can be opened
exactly once, after a durable receipt is written. Acceptance is a conjunction of absolute gates
(positive edge after costs, positive net P&L, a positive weekly-cluster bootstrap lower bound,
survival at elevated costs, per-year stability and sample floors), never a single score.Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
The results demonstrate that the system is designed to produce well-documented "no"s, making it hard to report something that is not there. The most instructive result involves a QQQ 0DTE campaign where a language-model proposer produced a rule that passed every discovery and validation gate but failed the holdout due to falling expectancy. p. 5 1 A language-model survivor meets its holdout
The most instructive single result is a QQQ 0DTE campaign in which the language-model proposer,
in its thirteenth run, produced a rule that sold the near-money put credit spread on Mondays and
Tuesdays when the credit exceeded 10% of width. On the traded-bar windows above, and under
that campaign’s own floors (40, 15 and 15 trades), it passed every discovery and validation gate
(Table 2). The honest summary is that the near-money 0DTE put premium is real, has the same shape
on two underlyings, is mostly not drift, and earns its return by absorbing unforecastable first-day
shocks. The pre-registered bar, which asks for a positive fifth percentile of weekly means across a
window containing three such shocks, is not met, and the bar was not moved. Finally, this paper is written by the
system’s author; the full campaign records behind each number are available from the author on
request. 6 Conclusion
A research loop in quantitative finance should be judged first by how hard it is to fool and only second
by what it finds.Where the paper says it
Where the paper says it
Where the paper says it
Failure modes the gates missed, and the change each forced
The system identifies specific failure modes that the initial gates did not catch, leading to necessary changes in campaign parameters. Examples of these failures include drift as alpha, where an unconditional long-delta credit spread collects market’s upward drift regardless of mispricing. p. 6 4 Failure modes the gates missed, and the change each forced
Drift as alpha. An unconditional long-delta credit spread collects the market’s upward drift
whether or not anything is mispriced. Gates evaluated on one window cannot see that the window
had one direction. The rule recorded with the grid is that a condition is read together with its
opposite, and a passing pair is a long-side artifact, not a signal. Costs assumed, not measured. Two candidates were killed by a doubled-cost gate against a
cost nobody had measured, and one holdout was spent before its cost premise was measured.Where the paper says it
Where the paper says it
Where the paper says it
Limitations
The limitations of the loop include the fact that its vocabulary is deliberately small, meaning a hypothesis outside it cannot be tested until trusted code is extended. Several campaigns use short samples, and their nulls are statements about power as much as about markets. p. 7 Its vocabulary is deliberately small, and a hypothesis outside it cannot be tested until
trusted code is extended. Several campaigns use short samples (for example 46 days of Kalshi data,
60 weekends of ETF flow), and their nulls are statements about power as much as about markets. Several campaigns use short samples (for example 46 days of Kalshi data,
60 weekends of ETF flow), and their nulls are statements about power as much as about markets. Option results from 2024 onward cover no bear market of 2022’s kind. Some modeled prices remain
where actual quotes were not available; those campaigns say so. Costs in several campaigns are
assumptions with stated breakevens rather than measurements. Costs in several campaigns are
assumptions with stated breakevens rather than measurements. Finally, this paper is written by the
system’s author; the full campaign records behind each number are available from the author on
request.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
The engine code, a reference mission and the public pre-registration with its hashchained ledger are at https://github.com/henryzhangpku/autonomous-quant-researcher. p. 7 Engine code, a reference mission and the public pre-registration with its hashchained ledger are at
https://github. com/henryzhangpku/autonomous-quant-researcher. The test suite runs offline without keys. Campaign records are available from the author on request. Campaign records are available from the author on request. A browser demonstration runs the engine under Pyodide on a synthetic tape with a planted regime
that discovery finds and validation refutes. Making the model propose data instead of code, separating stages physically, letting
the holdout be spent once, accepting only on a conjunction of absolute gates, refusing duplicates
before they cost a trial and keeping every failure in a chained ledger together produce a system
whose most common output is a well-documented “no”. We think that is the correct base rate, and
that publishing it is the point. 6 Conclusion
A research loop in quantitative finance should be judged first by how hard it is to fool and only second
by what it finds. Making the model propose data instead of code, separating stages physically, letting
the holdout be spent once, accepting only on a conjunction of absolute gates, refusing duplicates
before they cost a trial and keeping every failure in a chained ledger together produce a system
whose most common output is a well-documented “no”.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
A declarative hypothesis boundary limits model proposals to fixed vocabulary, ensuring
the attack surface and the space of searchable hypotheses are both fixed in advance.
This prevents models from proposing uninterpretable or executable code candidates. -
A staged evidence lifecycle enforces rigorous testing through a one-use holdout secured by a durable receipt, preventing strategies from surviving by luck or
multiplecomparisons artifact.
Acceptance requiresa conjunction of absolute gates,
such aspositive edge after costs
and asurvival at elevated costs,
rather than a single metric score. -
A tamper-evident ledger provides complete provenance, recording every event like
formulation, refusal, crash and result in a hash-chained, append-only ledger.
This allows downstream systems to know exactly what they are holding by exporting findings asportable JSON built only from frozen aggregate evidence.
-
The system can conduct comprehensive campaign reporting across diverse financial instruments like
SPY and QQQ zero-days-to-expiry (0DTE) options priced from actual traded option bars
and report every rejection, including alanguage-model survivor that passed discovery and validation with Sharpe ratios near 3 and fell to 0.26 on its single holdout use.
-
The system can identify failure modes the gates did not catch, such as when
the control-relative metric is computed but is diagnostic only,
forcing changes like adding adrift control (regress each trade on the underlying’s move over the same hold).
Abstract
Agentic research loops that let a language model propose a change, run a fixed experiment and keep whatever improves the metric work well in machine learning. In trading research the same loop is a machine for producing false discoveries: a backtest that "improves the metric" is, by default, overfit, look-ahead, a multiple-comparisons artifact, cost-free, or market drift wearing a strategy. We describe Autonomous Quant Researcher, an open-source loop built on the opposite premise: the model proposes, and only trusted code decides. Proposals are strict declarative JSON over a fixed vocabulary, so there is no model-written code to sandbox. Evaluation runs on immutable chronological stores split into discovery, validation and a final holdout that can be opened exactly once, after a durable receipt is written. Acceptance is a conjunction of absolute gates (positive edge and net P&L after costs, a positive weekly-cluster bootstrap lower bound, survival at elevated costs, per-year stability and sample floors), never a single score. Duplicates are refused before they spend a trial, and every formulation, refusal, crash and result is kept in a hash-chained, append-only ledger. We report every campaign run to date, across 0DTE options priced from traded option bars, overnight SPX verticals, daily ETF rules, technical-analysis premises, Kalshi crypto binaries, earnings options, off-exchange volume and a pre-registered alternative-data campaign. Most hypotheses fail, and we report them. Two results survive their pre-registered bars. The clearest example of the loop working is a failure: a language-model survivor that passed discovery and validation with Sharpe ratios near 3 and fell to 0.26 on its single holdout use. The value of such a loop lies less in what it finds than in how hard it is to make it report something that is not there.
Sources
- R&D-Agent-Quant: A Multi-Agent Framework for Data-Centric Factors and Model Joint Optimization
- AlphaAgent: LLM-Driven Alpha Mining with Regularized Exploration to Counteract Alpha Decay
- TradingAgents: Multi-Agents LLM Financial Trading Framework
- What LLM Trading Agents Actually Do in Production: A Six-Month, Population-Scale Record from Two Fleets
- Autonomous Market Intelligence: Agentic AI Nowcasting Predicts Stock Returns
- Interpretable Hypothesis-Driven Trading:A Rigorous Walk-Forward Validation Framework for Market Microstructure Signals
- Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarking
- Cognitive Alpha Mining via LLM-Driven Code-Based Evolution
- Navigating the Alpha Jungle: An LLM-Powered MCTS Framework for Formulaic Factor Mining
- Execution Realism and Reproducibility in LLM-Based Trading Systems: A Systematic Scoping Review and Evidence Audit
- AlphaAgents: Large Language Model based Multi-Agents for Equity Portfolio Constructions
- From Text to Alpha: Can LLMs Track Evolving Signals in Corporate Disclosures?
- FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Design
Related papers
- AFNO: Universal Certifiably-Arbitrage-Free Operator Learning Framework for European Call-Price Surfaces
- Diffusion models for dynamic volatility surface generation and data-driven hedging
- Rethinking Synthetic Scenario Realism: Compatibility, Not Fidelity, Drives Hedging Performance
- CosmosAlpha: Multi-Relational Graph Search with Pareto-Guided Exploration for LLM-Driven Alpha Mining
- Diversification Across Predictors: Heterogeneous Ensemble Learning for Cross-Sectional Return Prediction
- DSINC beyond Heston: the 3/2 and other non-affine stochastic-volatility models