Backtrader-Bench: Benchmarking LLM Agents on Algorithmic Trading with Self-Generated MCQs
Ruoxi Zhao, Maziar Raissi
cs.CL, cs.AI
Submitted: 2026-07-31
Updated: 2026-08-13
Comments: Accepted to the FinLLM Workshop at IJCAI 2026. Code and data: https://github.com/rzhao999/Backtrader-Bench
Code: https://github.com/rzhao999/Backtrader-Bench
License: http://creativecommons.org/licenses/by/4.0/
The gist: Evaluating LLM coding agents in algorithmic trading is difficult because static benchmarks risk data contamination and numerical backtest outputs require ground truth from actual code execution.
Terminology
Abstract
Evaluating LLM coding agents in algorithmic trading is difficult because static benchmarks risk data contamination and numerical backtest outputs require ground truth from actual code execution. We present Backtrader-Bench, a framework with two complementary pipelines. A deterministic multiple-choice question (MCQ) pipeline generates questions from backtest configurations across five trading strategies, 33 templates, and three difficulty tiers, with an independent checker that re-derives every answer. A generator-solver filtering pipeline autonomously mines harder questions: a generator writes questions verified by executable code, converts them to MCQs, and discards any that a no-tool solver can answer without code execution. We evaluate 11 models without tools (10 runs each) and four with-tools configurations on a 30-question curated set. Tool-augmented agents reach 90.0% accuracy in a single pass (GPT-5.5 and Opus 4.7), outperforming the best no-tools baselines (73.0%, averaged over 10 runs) by 17 percentage points. On 38 separately mined questions, no-tools accuracy drops further, with half the models falling to roughly random-chance level (25%). Beyond evaluation, the scalable MCQ infrastructure is designed to produce a training corpus for reinforcement learning, with the ultimate goal of building a specialized agent for quantitative trading workflows.
Sources
- Program Synthesis with Large Language Models
- Evaluating Large Language Models Trained on Code
- QuantCode-Bench: A Benchmark for Evaluating the Ability of Large Language Models to Generate Executable Algorithmic Trading Strategies
- INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agent
- FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance
- BloombergGPT: A Large Language Model for Finance
- TradingAgents: Multi-Agents LLM Financial Trading Framework
- PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance
- FinBen: A Holistic Financial Benchmark for Large Language Models
- FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making
- A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering