QuantEval: A Benchmark for Financial Quantitative Tasks in Large Language Models

arXiv:2601.08689 · cs.CL · Submitted 2026-01-13 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "QuantEval: A Benchmark for Financial Quantitative Tasks in Large Language Models".

Jane: The gist: QuantEval introduces a benchmark for evaluating Large Language Models across three quantitative finance dimensions—knowledge-based QA, quantitative mathematical reasoning,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're talking about this paper today called "QuantEval: A Benchmark for Financial Quantitative Tasks in Large Language Models." It basically says that while large language models are getting really good at answering simple finance questions, evaluating them on the actual work quant finance professionals do is still a mess.

Jane: That sounds like it. So what's the main point of this paper? What is QuantEval actually trying to prove or measure?

Tom: The thesis of QuantEval is introducing a benchmark to evaluate whether these large language models can actually complete the whole loop that quantitative finance needs: compute, implement, and validate. It goes beyond just checking if the model knows a fact or can pull something from a document.

Lu: What's really interesting about this is that they are testing three specific dimensions of quantitative finance workflows: knowledge-based question answering, which is understanding the basic financial stuff, quantitative mathematical reasoning, and then strategy coding.

Meng: Strategy coding sounds like the hardest part to get right in practice. So how do they measure if an AI can actually write a working trading strategy?

Tom: They use something called a CTA-style backtesting framework for the coding task. This means they aren't just checking if the code looks syntactically correct or matches some template, which is what you often see with simpler tests.

Jane: But how does that framework work for measuring performance? What are they comparing the model's strategy against?

Tom: They execute the model-generated strategies and then compare the resulting risk and return metrics against references from actual human experts. This moves it way past just checking if a program compiles or matches a pattern.

Lu: The construction of this benchmark is pretty detailed, involving three stages: data preparation, data construction, and validation. They pull materials from financial textbooks, academic papers, regulatory filings, and real market data like Yahoo Finance and Stooq for the reasoning part.

Meng: I wonder about the validation process. How do they make sure those expert-validated samples are actually reliable?

Tom: They have a multi-tiered validation system. Automated agents do an initial screen to catch duplicates or easy items, but then domain experts perform thorough reviews focusing on factual accuracy and mathematical soundness for the reasoning and coding parts.

Jane: So it’s not just one person saying "yes" or "no," but a process involving both automated checks and human experts reviewing things like code validity.

Lu: For the strategy coding part, they specifically require detailed explanations of the program logic in Chinese and using type hinting in the code requirements, which adds a layer of practical constraint to what an LLM has to generate.

Tom: And this is where I saw something that got me excited: they evaluate thirteen open-source and proprietary models <ref:2601.08689#pg2>. They report prompt and reliability analyses, and they actually observed substantial gaps relative to human experts in multi-step reasoning and framework-constrained coding.

Paper summary: Jane: That means even the best current models aren't hitting the mark when it comes to complex, multi-step problems or when they have to stick to a specific trading framework.

Meng: I've seen this before; models struggle with that level of constraint. So, what about training them? Did they try anything to fix these gaps?

Tom: They did supervised fine-tuning and reinforcement learning experiments using domain-aligned data. The results show that combining domain-aligned supervised fine-tuning with structured reinforcement learning for finance reasoning tasks does improve the reasoning performance.

Lu: But they also found a trade-off. Training can improve quantitative reasoning, but it also exposes gaps in factual recall, which is something you have to watch closely when you're tuning models for finance.

Jane: So training helps with the math and logic, but maybe it makes them less reliable when it comes to recalling specific financial definitions or data points?

Tom: Exactly. And they did release the dataset, the evaluation scripts, and even a full backtesting configuration so other researchers can do reproducible work on this paper.

Meng: That reproducibility is crucial for me. If I want to test my own models against QuantEval results, I need to know exactly how those backtests were run.

Lu: They released the full deterministic backtesting configuration, including the asset universe and cost model assumptions, so you can replicate the exact strategy coding results under identical assumptions.

Jane: It sounds like this benchmark is a very solid step toward understanding what it actually takes for an AI to be useful in real quantitative finance workflows. So, what does this paper mean for us as users or researchers?

Tom: It means we need to stop looking at models that just pass a simple knowledge test and start demanding proof that they can execute the full process—from idea to code execution and actual backtesting results.

Jane: I think the real implication is shifting the focus from just general intelligence in finance to proving specific, rigorous quantitative skills in AI. This paper gives us a much clearer way to measure that depth of skill.

Lu: It really highlights that while LLMs can handle the knowledge part well, getting them into the actual execution and validation loop requires specialized training and very specific constraints on how they operate.

Meng: From my point of view, this benchmark shows me exactly where the current production-ready models fall short when you try to deploy them in a real trading environment without massive oversight.

Tom: So, to wrap up this segment, QuantEval is a structured way to test if LLMs can handle knowledge, complex math, and actual strategy coding using a rigorous backtesting setup. Next up we're going to talk about what the authors are saying about the overall implications of this work and its future direction.

Conclusion: Tom: So, QuantEval is here to show us how models can actually do the work in quantitative finance, not just talk about it.

Jane: Right, it’s this benchmark that tries to measure if an AI can actually follow the whole process from idea to actual backtesting results.

Tom: The paper's called "QuantEval: A Benchmark for Financial Quantitative Tasks in Large Language Models," and it’s trying to solve that gap between theoretical knowledge and real-world execution.

Jane: It sets up three specific challenges for these models—knowledge questions, math reasoning, and strategy coding—and they use a deterministic backtesting framework to check the results against expert benchmarks.

Tom: The authors found that even state-of-the-art models have significant gaps when it comes to multi-step reasoning and sticking to those strict coding rules.

Jane: They also showed that fine-tuning with domain data helps improve the reasoning part, but there’s still a real gap compared to actual human experts.

Tom: The authors are providing the data and the scripts so other researchers can actually run these tests themselves, which is important for getting reproducible results on this stuff.

Jane: It really forces us to think about what it means for these models when we try to use them in high-stakes financial environments.

Tom: And that leads us to a big question about where this benchmark fits into the future of AI finance research and deployment.

Zhaolu Kang, Junhao Gong, Wenqing Hu, Shuo Yin, Kehan Jiang, Zhicheng Fang, Yingjie He, Chunlei Meng, Rong Fu, Dongyang Chen, Leqi Zheng, §eric Hanchen Jiang 5, §yunfei Feng 6, §yitong Leng 7, §junfan Zhu 8, Ɣxyaoyou Chen 9, ᘾ xi Yang 10, Richeng Xuan ▲†

Peking University · Tsinghua University · Fudan University · University of Macau · University of California, Los Angeles · Shanghai Jiao Tong University · §Shanghai Weina Software Technology Technology Company · ƔBeijing Academy of Artificial Intelligence

cs.CL

Submitted: 2026-01-13

Updated: 2026-10-14

Comments: https://github.com/Zhaolu-K/QuantEval

Code: https://github.com/Zhaolu-K/QuantEval

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 76/100

The gist: The gist: QuantEval introduces a benchmark for evaluating Large Language Models across three quantitative finance dimensions—knowledge-based QA, quantitative mathematical reasoning, and

Key concepts

Knowledge-based QA: This tests an LLM's conceptual understanding of finance. It involves answering questions about financial definitions, terminology, and theories found in textbooks or academic papers. It measures whether the model has grasped the underlying financial concepts.
Quantitative Mathematical Reasoning: This assesses an LLM's ability to solve complex quantitative problems using real market data and financial formulas. It requires multi-step problem-solving grounded in actual numbers, moving beyond simple recall to demonstrate genuine mathematical computation.
Quantitative Strategy Coding: This evaluates the model's skill in writing executable trading strategies within a specific framework. The code is tested by running it through a deterministic backtesting system that compares its risk and return metrics against expert benchmarks.

Terminology

Benchmark — 3× in this paper · explained in 2 episodes
Fine-Tuning — 1× in this paper · explained in 5 episodes
Large Language Models (LLMs) — 1× in this paper · explained in 8 episodes
Reinforcement Learning — 1× in this paper · explained in 4 episodes

Summary

The gist: QuantEval introduces a benchmark for evaluating Large Language Models across three quantitative finance dimensions—knowledge-based QA, quantitative mathematical reasoning, and quantitative strategy coding—integrating a CTA-style backtesting framework for realistic assessment.

QuantEval Overview

QuantEval is designed to evaluate Large Language Models (LLMs) on core financial quantitative competencies by explicitly targeting three dimensions that define quantitative finance workflows: (1) Knowledge-based QA, (2) Quantitative Mathematical Reasoning, and (3) Quantitative Strategy Coding QuantEval is a benchmark designed to evaluate Large Language Models (LLMs) on core financial quantitative competencies by explicitly targeting three dimensions that define quantitative finance workflows: (1) Knowledge-based QA, (2) Quantitative Mathematical Reasoning, and (3) Quantitative Strategy Coding Compared with prior financial benchmarks that primarily emphasize knowledge-centric QA, QuantEval explicitly targets three dimensions that collectively define quantitative finance workflows: 1. Knowledge-based QA: conceptual understanding of financial definitions, terminology, and theory; 2. Quantitative Mathematical Reasoning: multi-step quantitative problem solving grounded in real market data and financial formulas; 3. Quantitative Strategy Coding: generating executable trading strategies under a unified framework.

Data Preparation and Construction

The construction of the benchmark samples involves a three-stage pipeline: Data Preparation, Data Construction, and Validation. For Knowledge-based QA, authoritative textual materials are sourced from financial textbooks, peer-reviewed academic papers, regulatory filings, and curated glossaries of financial terminology. Quantitative Mathematical Reasoning requires datasets rooted in realistic financial computations and problem-solving scenarios by gathering extensive historical market data and macro indicators alongside problem templates extracted from quantitative finance coursework. For Quantitative Strategy Coding, expert curation ensures that code snippets are annotated to verify correctness, readability, and alignment with realistic trading scenarios. The data construction stage employs a hybrid approach combining expert annotation with automated multi-agent generation. This framework includes specialized roles such as a summarizer agent, a problem extractor agent, and a question and code generator agent.

Validation and Evaluation Protocol

Ensuring the reliability of QuantEval samples involves a multi-tiered validation process involving both automated agents and domain experts. Automated validation agents conduct initial screening to detect duplicates or trivial items, while domain experts perform thorough reviews focusing on factual accuracy, mathematical soundness, and code validity. For strategy coding tasks, the evaluation is executed in an integrated CTA-style backtesting framework to measure execution success rate and deviations in key financial metrics. The deterministic backtesting configuration specifies a fixed asset universe, data range from 2010-01-01 to 2025-01-01, and specific cost model assumptions.

Evaluation Results and Analysis

Evaluations on 13 state-of-the-art LLMs reveal substantial performance gaps relative to human experts, particularly in reasoning and strategy coding. Human experts achieve strong performance on both Knowledge QA and Reasoning, while the best-performing LLMs remain behind, especially on quantitative reasoning. CoT prompting helps reasoning but is not always beneficial and can even reduce accuracy. Strategy coding remains the hardest dimension, with most open-source models failing to generate executable strategies under strict CTA framework constraints. Proprietary models substantially improve executability, leading them to dominate across all tasks.

Training Strategies and Limitations

Improvements in performance are demonstrated through large-scale supervised fine-tuning (SFT) and reinforcement learning (RL) experiments on domain-aligned data. The results show that combining domain-aligned SFT with structured RL for finance reasoning tasks provides further gains in reasoning performance. However, a substantial gap to human experts remains, indicating that domain-aligned training improves reasoning performance but the remaining gap is significant. Limitations include the benchmark focusing on English-only instances and the strategy coding task being evaluated under fixed backtesting assumptions. The strategy coding task currently has 60 instances, which is smaller than QA and reasoning subsets due to the high cost of expert validation.

Reproducibility and Ethics

The paper releases the full deterministic backtesting configuration, including asset universe, cost model, and metric definitions to ensure strict reproducibility. The evaluation harness is provided so that exact replication of strategy coding results under identical assumptions can be achieved. Ethics statement confirms that QuantEval is constructed from publicly available financial texts and data sources, ensuring no proprietary trading signals or confidential strategies are included. Strategy coding tasks are evaluated under offline historical backtesting and are intended solely for research purposes, not for direct trading decisions. The dataset includes expert review to filter sensitive content and reduce potential harms from misuse.

Conclusion

QuantEval introduces a benchmark for evaluating Large Language Models across three quantitative finance dimensions—knowledge-based QA, quantitative mathematical reasoning, and quantitative strategy coding—integrating a CTA-style backtesting framework for realistic assessment. Evaluations on 13 open-source and proprietary models reveal large gaps to human experts, especially in multi-step reasoning and framework-constrained coding. Domain-aligned SFT and RL yield consistent improvements. The paper releases the dataset, evaluation scripts, and a standardized CTA backtesting harness to enable reproducible research on LLMs for quantitative finance. The benchmark provides rigorous assessment beyond pass/fail compilation by measuring execution success rate and deviations in key financial metrics. The paper concludes that QuantEval requires not only surface-level financial familiarity but also precise multi-step quantitative computation and robust reasoning. The benchmark is a significant step toward facilitating research on LLMs’ quantitative finance capabilities and accelerating their practical adoption in real-world trading workflows. The paper provides a detailed breakdown of the QuantEval data construction pipeline, which consists of three main stages: Data Preparation, Data Construction, and Validation. The final dataset comprises 1,575 samples: 660 Knowledge QA, 855 Reasoning, and 60 Strategy Coding instances. The benchmark is built to cover a broad range of quantitative finance concepts with controlled difficulty across the three task categories. The paper emphasizes that execution-based evaluation for strategy coding measures execution success rate and deviations in key financial metrics. The benchmark is a distinctive feature because it integrates a CTA-style backtesting framework to assess generated strategies using standardized risk–return metrics. The paper concludes that the gap between existing benchmarks and QuantEval is particularly pronounced on quantitative reasoning due to reasoning depth and execution constraints. The paper highlights that most failures in opensource models for strategy coding are dominated by interface mismatch and framework misuse. The benchmark provides a detailed analysis of failure modes across all three tasks, identifying common error types such as conceptual confusion in QA and arithmetic mistakes in reasoning. The paper provides a comprehensive overview of the data sources used for QuantEval, which includes public datasets like Yahoo Finance and Stooq. The paper provides a complete specification of the deterministic backtesting configuration used in QuantEval to ensure strict reproducibility. The benchmark is designed to cover a broad range of quantitative finance concepts with controlled difficulty across the three task categories. The paper provides detailed information about human performance, establishing an approximate upper bound for the accuracy of Knowledge QA and Reasoning tasks.

Improvements for AI systems

Improved LLMs can perform execution-based strategy validation by integrating a CTA-style backtesting framework, enabling a more realistic assessment of quantitative coding ability as described in the abstract.
AI systems can be trained to improve quantitative finance competency through large-scale supervised fine-tuning and reinforcement learning experiments on domainaligned data, leading to demonstrated improvements in reasoning and strategy coding performance.
Models will be equipped with a structured evaluation protocol that measures four key metrics for strategy coding: Executable Rate, Return MAE, Drawdown MAE, and Sharpe MAE using an integrated backtesting harness.
The system can utilize Chain-of-Thought (CoT) prompting specifically for quantitative tasks, as CoT is found to yield the largest improvements on quantitative reasoning across models, although its benefit is not uniform.
AI systems can be designed to generate executable trading strategies that adhere strictly to defined backtesting constraints, such as fixed asset universe, execution Next-bar open market orders, and specific risk limits like Max leverage 2.0.
The system can incorporate robust data construction pipelines combining expert annotation with multi-agent generation to scale the creation of high-quality benchmark samples for knowledge QA, reasoning, and coding tasks.
LLMs will be better at generating code that matches expert validation benchmarks by incorporating detailed explanations of the program logic in Chinese and ensuring full PEP 8 compliance.
The system can utilize advanced de-duplication techniques for both questions and code, such as AST-level similarity checks, to prevent training-data leakage from existing benchmarks.