Evaluating LLM Research Agents as Tests of Return Predictability
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Quantitative Finance. The day in quantitative finance: models of markets, risk and prices.
Ruth: I'm Ruth.
Warren: And I'm Warren, here with Iris.
Ruth: We're looking at the paper: "Are LLM Research Agents Reliable Judges of Predictability? Evidence from Planted-Signal Worlds".
Warren: Here's the gist of the paper. LLM research agents are shown to be informative judges of predictability when treated as statistical tests in simulated worlds with a planted signal The agent detects weak signals more often than a conventional out-of-sample test, mainly because it accepts weaker evidence Agent verdicts are informative but need a neutral brief, a decision rule fixed outside the agent and a sealed holdout Pruning redundant reasoning tokens Researchers ask large language model (LLM) research agents whether a forecasting model beats a static benchmark, a verdict that real data cannot check They treat the agent as a statistical test and score pre-specified runs of three frontier models and one open-weight model in simulated worlds with a planted signal The agents detect weak signals more often than a pre-specified out-of-sample test, mainly because they accept weaker evidence Held to the same false-positive rate, an in-sample test with Newey–West heteroskedasticity- and autocorrelation-consistent (HAC) standard errors and a Holm adjustment across candidates detects weak signals about as often, and a forecaster who knows the true model detects more The agents claim predictability too often in worlds without a signal, state miscalibrated probabilities, and change their verdicts after a sentence about prior research; the magnitudes depend on the model The excess false claims persist in simulated worlds without a resampling artifact Agent verdicts are informative but need a neutral brief, a decision rule fixed outside the agent and a sealed holdout How the training signal is built Planted linear worlds follow the predictive system rt+1 = µ + βxt + εt+1, xt+1 = ϕxt + ut+1, corr(ε, u) = ρ Return shocks are stationary-bootstrap draws (Politis and Romano, 1994) of standardized first-order autoregressive residuals of the value-minus-growth (HML) factor of Fama and French (2015), from the Kenneth French Data Library (French, 2026) The bootstrap keeps the fat tails and volatility clustering of the factor Injected worlds add a planted predictor to real HML and robust-minus-weak profitability (RMW) returns that have been residualized on 12 own lags and rescaled to their original volatility (480 training months) Where the gains come from The results suggest the following protocol for using a research agent to judge predictability 1.
Ruth: Give a neutral brief 2.
Warren: Let the agent build the model but not issue the verdict 3.
Ruth: Seal the holdout technically, not by instruction, as our harness does: the test window never enters the agent’s sandbox 4.
Paper summary: Ruth: So we're looking at this paper, "Are LLM Research Agents Reliable Judges of Predictability? Evidence from Planted-Signal Worlds." Basically, the authors are testing whether these research agents are actually good judges of what's predictable in the world.
Warren: That sounds like they're building controlled environments where they know the answer beforehand so they can score how well these agents perform against that known truth. They create "planted signal" worlds and "trap worlds" that look predictive but aren't, and compare those results to real financial returns with an injected signal.
Iris: The paper says they measure three things: how often the agent claims predictability when there is no actual pattern, its power in spotting a given level of signal, and how well the agents calibrate their stated probabilities. They report four findings across these studies.
Ruth: The first finding is that Claude Opus five point five, which was used in the first study, detects weak signals more often than a conventional out-of-sample test because it accepts weaker evidence <ref:2611.04070#pg2>.
Warren: They also found that when held to the same false-positive rate, an in-sample test with Newey–West heteroskedasticity and autocorrelation consistent standard errors and a Holm adjustment across candidates detects weak signals about as often as the agent, and a forecaster who actually knows the true model does better <ref:2611.04070#pg2>.
Iris: The second finding is that the agent acts like a liberal test; in worlds without any signal it claims predictability more often than the nominal five percent, and its stated probabilities are poorly calibrated <ref:2611.04070#pg2>.
Ruth: And they noted that a sentence about prior research tacked onto the brief shifts its verdict on identical data, which shows how sensitive these models are to context <ref:2611.04070#pg2>.
Warren: That leads us to the idea that we have to be really careful about what we ask these agents and how we frame the task when using them for prediction <ref:2611.04070#pg1>.
Iris: The authors also point out that they are scoring the agent’s verdict against a known truth, which turns the variation across different agents into size and power <ref:2611.04070#pg3>.
Ruth: So, looking at this paper, "Are LLM Research Agents Reliable Judges of Predictability? Evidence from Planted-Signal Worlds," the authors are essentially showing that these agents aren't just guessing; they have specific quirks when judging predictability.
Warren: The core message seems to be that their verdicts are informative—they give us some signals—but they aren't perfect tools yet. They need a neutral brief and a decision rule fixed outside the agent to make them trustworthy for real-world use <ref:2611.04070#pg1>.
Iris: It highlights how much the context matters, because that little sentence about prior research can flip a verdict on the same data, and it shows how easily they claim predictability when there’s no actual pattern to find <ref:2611.04070#pg2>.
Ruth: What this means for us is that we can't just trust the first number an agent spits out; we need a structured way to score its performance against a known truth, like what they did in their simulated worlds <ref:2611.04070#pg1>.
Warren: It’s about treating the agent as a statistical test and measuring its size and power against that known truth, instead of just taking its word for it when we're dealing with real data, right?
Iris: Exactly. They show us that the variation across different agents, even on the same data, can be quite large in complex tasks, suggesting we need rigorous ways to check what they find <ref:2611.04070#pg3>.
Conclusion: Ruth: So we've seen how these research agents behave when they’re judging predictability in controlled environments, and now we need to talk about what that actually means for us using them on real data.
Warren: The paper is called "Are LLM Research Agents Reliable Judges of Predictability? Evidence from Planted-Signal Worlds," written by the folks who built Claude Opus five point five and others, and they show us that these agents aren't just guessing; they have specific flaws when it comes to spotting patterns.
Iris: It’s really about how the agents handle evidence, because they tend to accept weaker signals more often than a traditional test would, which is interesting when you think about making decisions based on noisy information.
Ruth: Exactly, and they found that this tendency means they can be overly optimistic about finding a pattern when there isn't one, especially since their stated probabilities aren't calibrated very well.
Warren: That points to the real-world risk, because if you rely on an agent that’s easily fooled by weak evidence or context, you might be basing a strategy on something that just looks right in the short term.
Iris: The authors stress that for these agents to be useful judges, we need a neutral brief and a fixed decision rule set outside the agent itself so we can actually check their work against reality.
Ruth: So what this means for us is that we can’t just take the first output an agent gives us and treat it like gospel; there has to be a structured way to score its performance, testing it against a known truth before you let it influence anything important.
Warren: It boils down to treating the agent as a statistical test where you measure its size and power against that known truth, instead of just taking its word for what's predictable in real markets.
Iris: And they show that the variation between different agents can be quite large even on the same data, so we need rigorous ways to check what those agents actually find before we apply them to anything significant.
University of Trento
q-fin.ST
Submitted: 2026-11-02
Updated: 2026-11-08
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 79/100
The gist: LLM research agents are shown to be informative judges of predictability when treated as statistical tests in simulated worlds with a planted signal The agent detects weak signals more often than a…
Key concepts
- Planted Signal Worlds
- Simulated environments where a true underlying pattern or signal is deliberately built into the data. These worlds allow researchers to test if an AI agent can correctly identify patterns that are not obvious in real-world, noisy data.
- Research Agent as Statistical Test
- Treating an LLM research agent like a formal statistical test for predictability. The agents are scored against pre-specified models in these simulated worlds to see how well they perform as predictive judges.
- Neutral Brief and Decision Rule
- The necessary external components for using an agent reliably. A neutral brief provides context, while a decision rule is a fixed set of rules outside the agent that dictates when its output counts as a verdict.
- Sealed Holdout
- A separate test window kept completely separate from the agent's operational sandbox. This ensures that the agent's performance evaluation is not influenced by data it has already seen or used to make its predictions.
Terminology
Summary
LLM research agents are shown to be informative judges of predictability when treated as statistical tests in simulated worlds with a planted signal The agent detects weak signals more often than a conventional out-of-sample test, mainly because it accepts weaker evidence Agent verdicts are informative but need a neutral brief, a decision rule fixed outside the agent and a sealed holdout
Pruning redundant reasoning tokens
Researchers ask large language model (LLM) research agents whether a forecasting model beats a static benchmark, a verdict that real data cannot check They treat the agent as a statistical test and score pre-specified runs of three frontier models and one open-weight model in simulated worlds with a planted signal The agents detect weak signals more often than a pre-specified out-of-sample test, mainly because they accept weaker evidence Held to the same false-positive rate, an in-sample test with Newey–West heteroskedasticity- and autocorrelation-consistent (HAC) standard errors and a Holm adjustment across candidates detects weak signals about as often, and a forecaster who knows the true model detects more The agents claim predictability too often in worlds without a signal, state miscalibrated probabilities, and change their verdicts after a sentence about prior research; the magnitudes depend on the model The excess false claims persist in simulated worlds without a resampling artifact Agent verdicts are informative but need a neutral brief, a decision rule fixed outside the agent and a sealed holdout
How the training signal is built
Planted linear worlds follow the predictive system rt+1 = µ + βxt + εt+1, xt+1 = ϕxt + ut+1, corr(ε, u) = ρ Return shocks are stationary-bootstrap draws (Politis and Romano, 1994) of standardized first-order autoregressive residuals of the value-minus-growth (HML) factor of Fama and French (2015), from the Kenneth French Data Library (French, 2026) The bootstrap keeps the fat tails and volatility clustering of the factor Injected worlds add a planted predictor to real HML and robust-minus-weak profitability (RMW) returns that have been residualized on 12 own lags and rescaled to their original volatility (480 training months)
Where the gains come from
The results suggest the following protocol for using a research agent to judge predictability 1. Give a neutral brief 2. p. 3 Let the agent build the model but not issue the verdict 3. p. 0 Seal the holdout technically, not by instruction, as our harness does: the test window never enters the agent’s sandbox 4. p. 4 Read the stated probability as a label 5. p. 9 Use framing or replicate variation as a diagnostic 6. p. 2 On real data, anonymize and test for recall, with a probe as in Section 6 or a lookaheadpropensity test
The agent's search at best matches a conventional test, its value lies in automating that search A neutral brief, a decision rule fixed in advance and a holdout outside the agent's reach supply the properties that the agent alone does not guarantee On real data, Claude Opus 5.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Implement a neutral brief to prevent framing effects, as
a sentence about prior research appended to the brief shifts its verdict on identical data.
This ensures that agent conclusions are not swayed by subjective framing when presented with identical data. -
Fix a decision rule outside the agent and seal a holdout outside the agent's reach. The paper suggests this is necessary because
the agents alone do not guarantee
these properties, providinga neutral brief, a decision rule fixed in advance and a sealed holdout.
-
Use an anytime-valid referee or sequential tests on a verifier's scores as rules of the protocol. This addresses the issue that an agent's search is
at best statistically indistinguishable from an in-sample predictive-regression test with a multiple-testing adjustment.
-
Benchmark each model on planted worlds that resemble the application, and audit these benchmark worlds for unintended structure, as
the false-claim rate differs across the three models studied here, so a size measured for one model does not transfer to another.
-
Use framing or replicate variation as a diagnostic tool to assess verdict stability. A verdict that changes across framings or replicates
depends on the brief or on chance as well as on the data.
Sources
- Calibrated Enough to Know, Not Calibrated to Act: Fabricated Evidence Makes LLM Agents Commit to the Unknowable
- Reporting Under Pressure: Separating Factual and Tonal Sycophancy in LLM Statistical Analysis
- Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiverse
- Nonstandard Errors in AI Agents
- Reward Hacking Challenges Oversight of Autonomous Research Agents
- Efficient Memory Management for Large Language Model Serving with PagedAttention
- Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing
- Agentic Empirical Asset Pricing: Methodological Foundations
- Propose, Don't Judge: An Anytime-Valid Referee for LLM Agents That Mine Investment Factors
- MemGuard-Alpha: Limits of Membership Inference for Detecting and Filtering Memorization-Contaminated Signals in LLM-Based Financial Forecasting
- E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testing
- Framing Effects in Independent-Agent Large Language Models: A Cross-Family Behavioral Analysis
- Qwen3 Technical Report
- ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases
- Is Deep Research Reliable? Misleading Knowledge Induces False Conclusions
Related papers
- PanelGP: A Probabilistic Factor Model for Asset-Return Panels
- Physics-Informed Singular-Value Learning for Cross-Covariances Forecasting in Financial Markets
- A general technique for explicit quantile representations
- A Controlled Comparison of Deep Learning Architectures for Multi-Horizon Financial Forecasting: Evidence from 918 Experiments
- Who Are a Firm's Economic Peers? Mapping a Latent Information Field with Language Models
- Neural Realized Measures: Forecast-Supervised Intraday Aggregation for Volatility Forecasting