Are LLM Research Agents Reliable Judges of Predictability? Evidence from Planted-Signal Worlds
summary
The gist
LLM research agents are shown to be informative judges of predictability when treated as statistical tests in simulated worlds with a planted signal The agent detects weak signals more often than a…
In short
LLM research agents are tested as statistical tests in simulated worlds with hidden signals. They tend to detect weak signals more often than traditional out-of-sample tests because they accept weaker evidence. However, their verdicts require external rules and a sealed holdout to be reliable.
Key concepts
- Planted Signal Worlds
- Simulated environments where a true underlying pattern or signal is deliberately built into the data. These worlds allow researchers to test if an AI agent can correctly identify patterns that are not obvious in real-world, noisy data.
- Research Agent as Statistical Test
- Treating an LLM research agent like a formal statistical test for predictability. The agents are scored against pre-specified models in these simulated worlds to see how well they perform as predictive judges.
- Neutral Brief and Decision Rule
- The necessary external components for using an agent reliably. A neutral brief provides context, while a decision rule is a fixed set of rules outside the agent that dictates when its output counts as a verdict.
- Sealed Holdout
- A separate test window kept completely separate from the agent's operational sandbox. This ensures that the agent's performance evaluation is not influenced by data it has already seen or used to make its predictions.
Terminology used across episodes
This episode discusses
- Evaluating LLM Research Agents as Tests of Return Predictability · Paper Radio
- Calibrated Enough to Know, Not Calibrated to Act: Fabricated Evidence Makes LLM Agents Commit to the Unknowable · Paper Radio
- Reporting Under Pressure: Separating Factual and Tonal Sycophancy in LLM Statistical Analysis · Paper Radio
- Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiverse
- Nonstandard Errors in AI Agents
- Reward Hacking Challenges Oversight of Autonomous Research Agents · Paper Radio
- Efficient Memory Management for Large Language Model Serving with PagedAttention
- Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing · Paper Radio
- Agentic Empirical Asset Pricing: Methodological Foundations · Paper Radio
- Propose, Don't Judge: An Anytime-Valid Referee for LLM Agents That Mine Investment Factors · Paper Radio
- MemGuard-Alpha: Limits of Membership Inference for Detecting and Filtering Memorization-Contaminated Signals in LLM-Based Financial Forecasting · Paper Radio
- E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testing
- Framing Effects in Independent-Agent Large Language Models: A Cross-Family Behavioral Analysis
- Qwen3 Technical Report
- ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases
- Is Deep Research Reliable? Misleading Knowledge Induces False Conclusions
The paper
Evaluating LLM Research Agents as Tests of Return Predictability · Read on arXiv
University of Trento
Transcript
Introduction to the show: ident: Quantitative Finance. The day in quantitative finance: models of markets, risk and prices.
Ruth: I'm Ruth.
Warren: And I'm Warren, here with Iris.
Ruth: We're looking at the paper: "Are LLM Research Agents Reliable Judges of Predictability? Evidence from Planted-Signal Worlds".
Warren: Here's the gist of the paper. LLM research agents are shown to be informative judges of predictability when treated as statistical tests in simulated worlds with a planted signal The agent detects weak signals more often than a conventional out-of-sample test, mainly because it accepts weaker evidence Agent verdicts are informative but need a neutral brief, a decision rule fixed outside the agent and a sealed holdout Pruning redundant reasoning tokens Researchers ask large language model (LLM) research agents whether a forecasting model beats a static benchmark, a verdict that real data cannot check They treat the agent as a statistical test and score pre-specified runs of three frontier models and one open-weight model in simulated worlds with a planted signal The agents detect weak signals more often than a pre-specified out-of-sample test, mainly because they accept weaker evidence Held to the same false-positive rate, an in-sample test with Newey–West heteroskedasticity- and autocorrelation-consistent (HAC) standard errors and a Holm adjustment across candidates detects weak signals about as often, and a forecaster who knows the true model detects more The agents claim predictability too often in worlds without a signal, state miscalibrated probabilities, and change their verdicts after a sentence about prior research; the magnitudes depend on the model The excess false claims persist in simulated worlds without a resampling artifact Agent verdicts are informative but need a neutral brief, a decision rule fixed outside the agent and a sealed holdout How the training signal is built Planted linear worlds follow the predictive system rt+1 = µ + βxt + εt+1, xt+1 = ϕxt + ut+1, corr(ε, u) = ρ Return shocks are stationary-bootstrap draws (Politis and Romano, 1994) of standardized first-order autoregressive residuals of the value-minus-growth (HML) factor of Fama and French (2015), from the Kenneth French Data Library (French, 2026) The bootstrap keeps the fat tails and volatility clustering of the factor Injected worlds add a planted predictor to real HML and robust-minus-weak profitability (RMW) returns that have been residualized on 12 own lags and rescaled to their original volatility (480 training months) Where the gains come from The results suggest the following protocol for using a research agent to judge predictability 1.
Ruth: Give a neutral brief 2.
Warren: Let the agent build the model but not issue the verdict 3.
Ruth: Seal the holdout technically, not by instruction, as our harness does: the test window never enters the agent’s sandbox 4.
Paper summary: Ruth: So we're looking at this paper, "Are LLM Research Agents Reliable Judges of Predictability? Evidence from Planted-Signal Worlds." Basically, the authors are testing whether these research agents are actually good judges of what's predictable in the world.
Warren: That sounds like they're building controlled environments where they know the answer beforehand so they can score how well these agents perform against that known truth. They create "planted signal" worlds and "trap worlds" that look predictive but aren't, and compare those results to real financial returns with an injected signal.
Iris: The paper says they measure three things: how often the agent claims predictability when there is no actual pattern, its power in spotting a given level of signal, and how well the agents calibrate their stated probabilities. They report four findings across these studies.
Ruth: The first finding is that Claude Opus five point five, which was used in the first study, detects weak signals more often than a conventional out-of-sample test because it accepts weaker evidence <ref:2611.04070#pg2>.
Warren: They also found that when held to the same false-positive rate, an in-sample test with Newey–West heteroskedasticity and autocorrelation consistent standard errors and a Holm adjustment across candidates detects weak signals about as often as the agent, and a forecaster who actually knows the true model does better <ref:2611.04070#pg2>.
Iris: The second finding is that the agent acts like a liberal test; in worlds without any signal it claims predictability more often than the nominal five percent, and its stated probabilities are poorly calibrated <ref:2611.04070#pg2>.
Ruth: And they noted that a sentence about prior research tacked onto the brief shifts its verdict on identical data, which shows how sensitive these models are to context <ref:2611.04070#pg2>.
Warren: That leads us to the idea that we have to be really careful about what we ask these agents and how we frame the task when using them for prediction <ref:2611.04070#pg1>.
Iris: The authors also point out that they are scoring the agent’s verdict against a known truth, which turns the variation across different agents into size and power <ref:2611.04070#pg3>.
Ruth: So, looking at this paper, "Are LLM Research Agents Reliable Judges of Predictability? Evidence from Planted-Signal Worlds," the authors are essentially showing that these agents aren't just guessing; they have specific quirks when judging predictability.
Warren: The core message seems to be that their verdicts are informative—they give us some signals—but they aren't perfect tools yet. They need a neutral brief and a decision rule fixed outside the agent to make them trustworthy for real-world use <ref:2611.04070#pg1>.
Iris: It highlights how much the context matters, because that little sentence about prior research can flip a verdict on the same data, and it shows how easily they claim predictability when there’s no actual pattern to find <ref:2611.04070#pg2>.
Ruth: What this means for us is that we can't just trust the first number an agent spits out; we need a structured way to score its performance against a known truth, like what they did in their simulated worlds <ref:2611.04070#pg1>.
Warren: It’s about treating the agent as a statistical test and measuring its size and power against that known truth, instead of just taking its word for it when we're dealing with real data, right?
Iris: Exactly. They show us that the variation across different agents, even on the same data, can be quite large in complex tasks, suggesting we need rigorous ways to check what they find <ref:2611.04070#pg3>.
Conclusion: Ruth: So we've seen how these research agents behave when they’re judging predictability in controlled environments, and now we need to talk about what that actually means for us using them on real data.
Warren: The paper is called "Are LLM Research Agents Reliable Judges of Predictability? Evidence from Planted-Signal Worlds," written by the folks who built Claude Opus five point five and others, and they show us that these agents aren't just guessing; they have specific flaws when it comes to spotting patterns.
Iris: It’s really about how the agents handle evidence, because they tend to accept weaker signals more often than a traditional test would, which is interesting when you think about making decisions based on noisy information.
Ruth: Exactly, and they found that this tendency means they can be overly optimistic about finding a pattern when there isn't one, especially since their stated probabilities aren't calibrated very well.
Warren: That points to the real-world risk, because if you rely on an agent that’s easily fooled by weak evidence or context, you might be basing a strategy on something that just looks right in the short term.
Iris: The authors stress that for these agents to be useful judges, we need a neutral brief and a fixed decision rule set outside the agent itself so we can actually check their work against reality.
Ruth: So what this means for us is that we can’t just take the first output an agent gives us and treat it like gospel; there has to be a structured way to score its performance, testing it against a known truth before you let it influence anything important.
Warren: It boils down to treating the agent as a statistical test where you measure its size and power against that known truth, instead of just taking its word for what's predictable in real markets.
Iris: And they show that the variation between different agents can be quite large even on the same data, so we need rigorous ways to check what those agents actually find before we apply them to anything significant.
More episodes
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- The day in conversation for 2026-11-18
- Daily Summary for 2026-11-18
- 2601.05274-On the use of case estimate and transactional payment data in neural networks for individual loss reserving
- 2009.00868-Loss-Given-Default Modeling by Post-Last Passage Time Process
- 2606.30070-Financial Resilience Evaluation: From Conditional Expectations to Dynamic Convex Risk Measures
- 2212.12687-Measuring price impact and information content of trades in a time-varying setting
- Daily Summary for 2026-11-17
- 2401.08323-Dynamic portfolio selection under generalized disappointment aversion