Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents
cs.AI, cs.LG
Submitted: 2026-09-24
Updated: 2026-09-24
License: http://creativecommons.org/licenses/by/4.0/
The gist: We introduce Forecast-Dojo, a replayable environment for benchmarking and training LLM forecasting agents.
Terminology
Abstract
We introduce Forecast-Dojo, a replayable environment for benchmarking and training LLM forecasting agents. It combines resolved prediction-market questions with dated news, allowing agents to research an event and revisit their predictions at successive historical dates. The same tasks and tools support repeated evaluation, collection of training interactions, and feedback from recorded outcomes without waiting for new events to resolve. Forecast-Dojo contains 1,568 Polymarket events, split by time into training and evaluation periods, and 18.8M dated news articles. In an evaluation of 12 models, research tools lower Brier score for all 12. Forecasts also improve as events unfold, with the largest gains at steps where more newly dated evidence is recorded. Every model still trails historical market forecasts in both Brier score and accuracy. A belief notebook carried between dates lowers research cost but does not consistently improve forecast quality. Beyond evaluation, Forecast-Dojo provides interaction trajectories and outcome feedback for agent learning, with supervised fine-tuning as a proof of concept.
Sources
- Scaling Open-Ended Reasoning to Predict the Future
- PolyBench: Benchmarking LLM Forecasting and Trading Capabilities on Live Prediction Market Data
- WorldReasoner: Evaluating Whether Language Model Agents Forecast Events with Valid Reasoning
- FutureSim: Replaying World Events to Evaluate Adaptive Agents
- FastText.zip: Compressing text classification models
- Reinforcement Learning for LLM-based Event Forecasting
- Evaluating Strategic Reasoning in Forecasting Agents
- Agentic Forecasting with Structured Linguistic Beliefs
- Consistency Checks for Language Model Forecasters
- MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering
- Qwen3 Technical Report
- Scaling Language Models: Methods, Analysis & Insights from Training Gopher
- FutureX-Pro: Extending Future Prediction to High-Value Vertical Domains
- Bench to the Future: A Pastcasting Benchmark for Forecasting Agents
- LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena
- MIRAI: Evaluating LLM Agents for Event Forecasting
- Do Language Models Update their Forecasts with New Information?
- FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction
- Prediction Arena: Benchmarking AI Models on Real-World Prediction Markets
- Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems