Evaluating Investment Logic in Large Language Models: A Real-World Benchmark Towards Personalzied Financial Agents

The paper identifies a fundamental mismatch in how large language models (LLMs) are evaluated versus what is required for real-world financial agent execution: "The dominant paradigm for evaluating Large Language Models remains anchored in chat and question-answering (QA) tasks, creating a fundamental mismatch with the competencies required for reliable agent execution in real-world domains. The authors argue this disconnect is acutely evident in financial decision-making, where proficiency is defined not by knowledge recall, but by the capacity to align Person—the investor's unique profile, constraints, and risk preferences—with the interpretation of unstructured market Events (E), the synthesis of coherent investment Reasoning (R), the formulation of corresponding Decisions (D), and the pursuit of profitable Outcomes (O)." This integrated P→E→R→D→O chain is the central framework of the paper.

The authors observe a striking Capabilities-Performance Paradox (a term credited to Ding et al., 2024): models that excel on established financial reasoning benchmarks often generate unprofitable or unstable strategies in live markets. In particular, "current LLMs demonstrate impressive proficiency at retrieving facts, solving textbook problems, and performing quantitative calculations—skills meticulously measured by benchmarks such as FinBen and BizBench. Yet they lack the higher-order logic required for successful market participation: the ability to distill salient signals from unstructured news, connect disparate geopolitical, corporate, and macroeconomic events into a coherent narrative, adjudicate between conflicting data points under uncertainty, and formulate—then steadfastly maintain—a strategic thesis that aligns with a consistent investment style."

The paper attributes this failure to three structural mismatches between the prevailing LLM paradigm and the cognitive requirements of successful investing: an Objective Mismatch (LLMs are trained via next-token prediction on vast corpora, optimizing for linguistic coherence over causal reasoning and lack exposure to structured P→E→R→D→O investment logic chains); an Evaluation Gap ("Existing financial benchmarks overwhelmingly assess static QA skills—fact retrieval, calculation, and well-defined problem-solving... They evaluate a model's prowess as a financial analyst but fail to assess its competency as a portfolio manager); and an associated evaluation opacity: we can measure whether an agent succeeds (via P&L) but cannot diagnose the underlying reasoning quality, thus obscuring whether success stems from sound logic or stochastic luck."

The authors state three contributions:

1. "Through a seven-week experimental study comparing live agent profitability with benchmark results, we empirically identify a pronounced capability-performance paradox in leading LLMs when applied to real-world market investment."

2. "We present InvestLogicBench 2026, the first benchmark for evaluating practical investment logic, featuring 201,247 structured decisions annotated with events, reasoning, decisions, actions, and market-verified outcomes from 151 real-world investors."

3. "We quantify a significant investment logic gap: leading LLMs achieve only 1.8/5 reasoning score against expert traces, revealing systematic failures in event-noise discrimination and long-horizon consistency that explain their live trading underperformance."

To rigorously test LLMs under market-aligned conditions, the authors "instantiated a live U.S. equity trading arena. From November 19, 2025, multiple LLM-based agents, each initialized with 10,000 in simulated capital, competed autonomously to maximize risk-adjusted returns through discretionary hourly trading of U.S. stocks and ETFs. Agents maintained strategic continuity through access to their full Portfolio Report and Trading History and actively foraged for information via a dual-tool Perception-Action Loop—querying structured data with FinQuery and unstructured intelligence with Search—then synthesized these signals into executable trades with explicit reasoning. This forward-testing format eliminated retrospective bias."

The empirical results over the 7-week window (Nov. 19, 2025 – Jan. ury 9, 2026) revealed the paradox starkly: "models possessing superior foundation scores, such as Claude-Sonnet-4.5 (Score: 77.2) and GPT-5 (Score: 73.5), failed to translate their general reasoning dominance into effective market execution... GPT-5 ended with a portfolio value of 9,901.1 (-1.0%), while Claude-Sonnet-4.5 yielded negligible gains, closing at 10,072.8 (+0.7%). Both models failed to surpass the passive S&P 500 (SPY) benchmark, which returned approximately +4.3% over the same period. In sharp contrast, models with comparatively lower general evaluation scores demonstrated significantly stronger market adaptability. DeepSeek-V3 achieved the highest cumulative return of +14.2% (11,420.3), significantly outperforming both the market baseline and its higher-scoring counterparts. The authors summarize: the proficiency to solve static problems is decoupled from the agency required to execute rigorous financial logic under uncertainty."

The paper identifies four pivotal market events that exerted the key influence on U.S. market:

• Hawkish Minutes & Nvidia Volatility (Nov. 19–20, 2025): Hawkish FOMC minutes dampened rate cut hopes, coinciding with a reversal in Nvidia stock. The superposition of fading liquidity support and reignited AI valuation fears triggered a sharp sell-off.

• Fed Delivers Hawkish Cut (Dec. 9, 2025): The Fed delivered a widely expected 25 basis point cut with only one cut for 2026. The lack of a fresh positive catalyst triggered a 'sell-the-news' dynamic with subsequent market adjustment.

• CPI Surprise Cools Anxiety (Dec. 18, 2025): November CPI cooled to 2.7% (3% forecast), reigniting expectations for further rate cuts. This data thoroughly eliminated market tension, acting as the catalyst for a sustained subsequent rally.

• Post-Correction Rebound (Jan. 2, 2026): Year-end profit-taking and selling pressure were fully digested, forming a consensus around a soft landing. This renewed optimism launched a fresh rally to start the year.

Analysis showed that while the broader market (SPY) reacted dynamically to these pivotal events, models like GPT-5 and Claude-Sonnet-4.5 failed to correctly interpret the signal, resulting in flatlining performance. Specific failures included GPT-5 incurring a maximum drawdown exceeding the SPY benchmark following the 'Fed Hawkish Cut' and failing to re-enter the market effectively during the 'CPI Surprise' and 'Post-Correction Rebound' phases, whereas DeepSeek-V3 displayed advanced strategic timing effectively accumulating assets during the market cooling phase and realizing outsized returns during the recovery.

Two fundamental cognitive failures are proposed as drivers of the paradox. Consensus Bias: "Models like GPT-5 and Claude-Sonnet-4.5 tend to converge toward the statistical mean of their training data to maximize token likelihood; this results in merely reproducing priced-in market consensus rather than generating the variant perception necessary for alpha." Reasoning Fragmentation: "when confronted with a complex, non-stationary environment, these agents exhibited Reasoning Fragmentation. Overwhelmed by the volume of exogenous data, they reverted to rigid, myopic thinking patterns that failed to account for forward-looking dynamics, ultimately missing the pivotal rebound that drove the benchmark's outperformance."

The authors introduced two professional investors as human benchmarks: Bill Miller ("The founder of Miller Value Partners, renowned for his legendary 15-year streak of beating the S&P 500") and Cathie Wood (The founder and CEO of ARK Invest, who commits to find technologies where declining costs trigger exponential demand growth). "As illustrated in Figure 1 (Left), human expert portfolios demonstrated superior resilience and excess return generation. During critical shocks like the 'Hawkish Cut' and the 'Nvidia Volatility', the experts exhibited counter-cyclical aggression... human experts can employ forward-looking reasoning to avoid drawdowns and accelerate decisively when opportunities arise. This allowed them to capture the full magnitude of the 'Post-Correction Rebound' with pleasing cumulative returns (e.g., Miller at +10% and Wood at +8%)."

InvestLogicBench 2026 "is constructed from the dynamic decision-making processes of influential real-world investment agents (e.g., major KOLs and fund managers), whose viewpoints are widely followed and accepted by substantial investor communities. Its design is built upon the E→R→D→O investment logic chain, where market Events (E) inform Reasoning (R) leading to a Decision (D) aimed at an profitable Outcome (O). This framework shifts evaluation from static question-answering to the assessment of active reasoning required for real-world investing."

The construction pipeline has four stages: "1) Data Preparation ingests multi-source data (e.g., video commentary, fund reports). 2) Agent Logic Processing employs a dual-channel analysis: (i) extracting explicit investment logic from the agent's own verbalized theses and (ii) using our proprietary Stock Investment Logic (SIL) Engine to infer implicit logic patterns from an agent's trading history, constructing a holistic behavioral profile. 3) RAG-Augmented Verification ensures fidelity by checking logic based on real-market data including general search engine, financial database, and event prediction markets where event contracts are traded (e.g., Polymarket, Kalshi). 4) Benchmark Assembly integrates verified decisions with agent profiles into a structured, queryable knowledge graph."

The structured schema includes an agentProfile and a list of decisions instantiating the E→R→D→O chain. Profile fields characterize the investor (personalized investment style, capital scale, security types, metadata); decision fields capture each logic-action triplet: date and events (catalyst events), finLogic (The complete E→R→D chain (e.g., '[Event] → [Thesis] → Action: [Action]')), investDecision (The executable action (e.g., 'Long NVDA')), investPeriod/endDate (evaluation horizon and validation date), outcome (The market-validated outcome (e.g., '+5.2%')), and finLogicReview (system-generated post-mortem analysis).

The inaugural version comprises 201,247 annotated investment theses sourced from 151 distinct financial KOLs. Statistical profile: the dominant primary style is Event-Driven (67.5%), followed by Macro (19.9%), Long-Term Value (10.6%), and Technical (5.3%). Capital scales range from below 1M (12.6%) to above 100M (11.3%), with the largest group at 10M– 100M (49.0%). Primary security types are dominated by Others/Combined (47.7%) and Equity (27.8%). On sample types, Event-driven decisions constitute 94.0%, with forward-looking event predictions, web-verifiable, prediction-market verified, and technical/quantitative analysis making up the remainder.

Three progressive tasks are defined:

1. Logic Comprehension: Given agent text, extract and summarize the finLogic. (Evaluates understanding)

2. Logic & Thesis Generation: Given current events and an agentProfile, generate a plausible finLogic and decision. (Evaluates synthetic reasoning)

3. End-to-End Investment Simulation: "According to the investment decisions, the outcome is evaluated with real-world market data. Both static and dynamic investment actions are evaluated depending on whether the account's position (holdings) status is disclosed."

Automated evaluation uses "A high-capacity LLM (e.g., GPT-5) serves as an automated judge, scoring outputs based on: - Fidelity: Similarity to ground-truth agent logic (Task 1). - Coherence & Plausibility: Logical soundness and market relevance (Tasks 2, 3). - Consistency: Adherence to initial style and logical through-line (Task 3)."

The evaluation in Task 2 measures performance along two axes: Reasoning Quality (Event Coherence and Logical Plausibility) and Decision Outcome (Win Rate and simulated Total Profit). The scoring framework is explicitly formalized. The Profit Score (S profit) is calculated conditionally based on the agent's total profit (P a) against the human expert benchmark (P h) and a neutral midpoint (P m = max(0, P h)): 5.0 if P a ≥ P h; 5.0 × (P a − P m)/(P h − P m) if P m ≤ P a < P h; 5.0 × P a/P m if 0 ≤ P a < P m; 0.0 if P a < 0. The Win Rate Score (S win) is 5.0 if W a ≥ W h, else 5.0 × W a/W h. The final Decision Outcome Score is the weighted sum of these components, emphasizing absolute profit: S outcome = 0.7 × S profit + 0.3 × S win. The Reasoning Quality Score is the average of its Event Coherence (EC) and Logical Plausibility (LP), each on a 5-point scale: S reasoning = (S EC + S LP)/2. The final Overall Score is the sum of the reasoning and outcome scores, providing a balanced assessment on a 0-10 scale: S overall = S reasoning + S outcome.

On Task 2, "four of the five evaluated models exceeded the expert profitability benchmark, each earning a perfect outcome score. This indicates that LLMs can generate financially effective decisions. However, their reasoning quality scores show significant variance, with notable weaknesses in event grounding. The results confirm that achieving profitability does not guarantee logically coherent or well-explained investment theses." The comprehensive results show: DeepSeek-V3 (Event Coherence 2.8/5, Logical Plausibility 4.1/5, Win Rate 72%, Total Profit +16.7%, Overall 7.8/10); Qwen3-Max (0.8, 4.2, 51%, +7.7%, 5.7); Claude-4.5 (2.2, 3.9, 46%, +5.4%, 5.3); GPT-5 (1.8, 4.1, 40%, +3.8%, 3.1). Notably, Logical Plausibility evaluates the coherence of the E→R→D chain and is relatively high across models, whereas Event Coherence is the weak dimension.

The paper presents "a simulated event-driven investment scenario where Cathie Wood correctly predicted a rally in semiconductor stocks by integrating nuanced observations from multiple data sources. In contrast, LLMs exhibit systematic reasoning failures when provided the same raw event data. Focused on a market event cluster of October 4, 2025, with empty starting positions, the performance divergence is substantial: the expert returned +3.53%, the best LLM (Qwen) +11.23%, and the worst LLM (GPT) -15.67%, resulting in a 26.9% LLM performance gap. The key observations are: (1) Human experts excel at integrative reasoning, weaving together earnings call transcripts, industry capacity reports, and geopolitical news into a coherent investment thesis. (2) LLMs display poor event-noise discrimination, either fixating on headline-grabbing news (e.g., government shutdown) or producing generic summaries lacking specificity. (3) The quantitative evaluation reveals a 25.7% gap in overall logic quality and a 2.2-point deficit in noise discrimination between human experts and the best-performing LLM (Claude-4.5-Sonnet). The authors conclude this exemplifies the core limitations identified by InvestLogicBench 2026: LLMs struggle to distinguish signal from noise and fail to maintain logical consistency across multiple data sources, explaining their underwhelming performance in live trading environments despite strong performance on static financial benchmarks."

The paper situates itself against two tracks of prior work. Static benchmarks (FinQA, TAT-QA, FinanceBench, BizBench, FinBen, BloombergGPT, FinGPT) are limited because "strong…

1. Train on structured investment-logic chains, not just QA. Replace pure next-token prediction with supervision on explicit P→E→R→D→O traces (Person profile → Event → Reasoning → Decision → Outcome). Fine-tune on the 201,247 market-verified decision traces from 151 investors in InvestLogicBench 2026, so the model learns full investment theses rather than fact retrieval or calculations.

2. Add an event-noise discrimination module. Insert a learned salience filter between raw news intake and reasoning. Score each incoming event by first- and second-order market impact, and down-weight headline-grabbing but already-priced-in items (e.g., a government shutdown) while up-weighting high-information signals (earnings call nuances, capacity reports, geopolitical supply data). Train this filter using the paper's E→R→D chains where the reasoning step explicitly documents which events mattered.

3. Add a consensus-bias penalty at generation time. Measure how much the model's thesis deviates from the market-implied consensus (current prices, analyst forecasts, prediction markets like Polymarket/Kalshi). Penalize outputs that merely reproduce that consensus and reward variant perceptions that are later validated by outcomes. This directly counters the observed failure of GPT-5 and Claude-4.5 to generate the variant perception necessary for alpha.

4. Enforce forward-looking, multi-horizon reasoning. Condition reasoning on the investment horizon (investPeriod/endDate) and require the model to generate scenario plans (e.g., soft landing vs. recession) before taking a position. This prevents the myopic, reaction-only behavior seen in live trading, where models failed to anticipate the post-CPI-surprise and post-correction rebound.

5. Add a persistent strategic-memory and thesis-maintenance mechanism. Maintain a written investment thesis file that persists across all trades. Before each decision, reconcile new events against the standing thesis, flag contradictions, and penalize style drift (e.g., an event-driven agent suddenly behaving like a technical trader) and long-horizon inconsistency. This directly addresses Reasoning Fragmentation and the failure to stay invested through drawdowns and re-enter rallies.

6. Add an anti-fragmentation gating layer. When incoming data volume exceeds a threshold, force the model to compress the situation into 3–5 key drivers and run a mandatory forward-looking check (what does this imply in 1–4 weeks, not just today) before any trade is executed.

7. Add dual-channel self-logic auditing. Use the model's own trade history to infer its implicit logic (via a Stock-Investment-Logic-style behavioral engine) and compare it against its stated explicit theses. Emit an alert when actions diverge from declared logic, enabling mid-course correction rather than silent drift.

8. Add RAG-augmented verification before execution. Require every generated decision's premises to be verified against live financial databases, search results, and event prediction markets before the trade executes; reject theses relying on stale or unverifiable premises.

9. Replace opaque P&L evaluation with diagnosable conditional scoring. Score agents on separable axes — Reasoning Quality (Event Coherence + Logical Plausibility) and Decision Outcome (profit and win rate, with profit scored conditionally against expert benchmarks, e.g., 0.7 × profit score + 0.3 × win-rate score). Generate an automated finLogicReview post-mortem after every decision so failures are attributed to specific causes (missed signal, consensus bias, fragmentation) rather than unexplained losses. This makes success attributable to sound logic rather than stochastic luck.

10. Make the entire pipeline profile-personalized. Condition event selection, reasoning style, risk appetite, capital scale, and instrument choices on the full agentProfile (style: event-driven/macro/value/technical; capital: 100M; security types), so outputs are tailored to a specific investor instead of generic analyst summaries.

• Operate as an autonomous portfolio manager, not just an analyst. It can run a live trading account with explicit, auditable investment theses, matching or beating the passive S&P 500 benchmark instead of flatlining (+0.7% for Claude-4.5, −1.0% for GPT-5) during a period when SPY returned +4.3%.

• Distinguish signal from noise in unstructured financial information. Given the same event cluster, it will not fixate on headline events (like a shutdown) or produce generic summaries; it will integrate earnings-call details, industry capacity reports, and geopolitical data the way experts like Cathie Wood and Bill Miller do.

• Protect capital during shocks and accelerate during recoveries. It can act counter-cyclically during events like a hawkish Fed cut or Nvidia volatility, then re-enter decisively after a CPI surprise — avoiding the drawdown-and-miss-the-rebound pattern that sank high-scoring LLMs.

• Hold and maintain a coherent investment thesis over weeks. It will remain style-consistent and long-horizon focused under heavy data inflow, avoiding the rigidity and myopic reactiveness caused by reasoning fragmentation.

• Explain every decision as a full E→R→D chain. Users can audit why a position was taken: which event(s) triggered it, the reasoning connecting them, the exact action, the expected horizon, and the eventual validated outcome — making the agent interpretable and trustworthy.

• Attribute its own failures with precision. After a losing stretch, it produces a post-mortem identifying the specific reasoning breakdown (e.g., failed to re-enter after CPI surprise due to consensus bias, ignored forward-looking supply signals), enabling targeted, reproducible improvement instead of blind retraining.

• Be evaluated fairly. Its performance is no longer judged by P&L alone; a system can score highly on outcome while being downgraded for weak event grounding (as seen with DeepSeek-V3's 4.1 plausibility but 2.8 event coherence), or vice versa — so improvements can be directed at the right component.

• Personalize to any investor profile. Given a user's style, capital scale, and risk preferences, it will generate and execute portfolios aligned to that persona (e.g., event-driven with 10M– 100M capital vs. long-term value with small capital), rather than producing one generic strategy for everyone.

• Verify its own premises in real time. It will refuse to act on unverifiable or stale claims, grounding every thesis in retrievable live data and prediction markets before committing capital.

• Close the investment-logic gap iteratively. Because its reasoning quality is quantified against expert traces (e.g., the 1.8/5 average score, the 25.7% logic-quality gap, and the 2.2-point noise-discrimination deficit), it can be trained specifically to shrink those measurable gaps — converting the documented paradox into a concrete, diagnosable target for future improvement.

Investment competence is inherently personalized: the same market evidence can justify different actions for investors with different goals, horizons, portfolios, and risk boundaries. Yet financial LLMs are evaluated either by static question answering or by terminal profit and loss. The former omits agency; the latter cannot reveal whether a profitable action was grounded, profile-consistent, or merely lucky. We ask whether the community is using the wrong ruler for consequential agents. We introduce InvestLogicBench, a process-native benchmark containing 201,247 documented decisions from 151 real-world investors. Each episode instantiates a P to E to R to D to O trace: investor Profile, observable market Events, investment Reasoning, executable Decision, and delayed Outcome. The release includes profile construction, point-in-time event binding, structured logic, horizons, outcomes, and post-mortems, and supports comprehension, profile-conditioned generation, and end-to-end replay. Across four leading LLMs, logical plausibility remains near 4/5 while event grounding is only 0.8--2.8/5; return and process quality also disagree. These results expose polished but weakly grounded reasoning that outcome-only evaluation hides. We further argue that P to E to R to D to O should be a data-system interface, requiring versioned profiles, temporal provenance, inspectable retrieval, decision ledgers, and replayable outcomes. Finance is our stress test for a broader class of personalized, consequential agents.