Programming Prescience: Issues in Human Construction of Forecasting Bots
cs.HC, cs.AI
Submitted: 2026-11-09
Updated: 2026-11-09
License: http://creativecommons.org/licenses/by/4.0/
The gist: AI models' growing capacity for complex reasoning has made them attractive tools for forecasting, or the systematic prediction of the future based on past and present data.
Abstract
AI models' growing capacity for complex reasoning has made them attractive tools for forecasting, or the systematic prediction of the future based on past and present data. However, should AI forecasting systems be adopted, human builders will shoulder the work of building and evaluating them. In this paper, we explore what this work looks like with a qualitative interview study with 15 builders of forecasting bots, sampled from among successful participants in a recent AI forecasting competition. Our analysis surfaces the challenges around (1) tuning a bot to acquire factual research, (2) implementing coherent reasoning, and (3) understanding how well one's bot actually does. During their work, builders frequently introduced complexity that did not pay off, and tested in ways that were opportunistic and piecemeal. Our detailed account of their challenges helps clarify what problems should be targeted in upcoming innovations in process, tools, and human-AI teaming for forecasting.
Sources
- AIA Forecaster: Technical Report
- Challenges in Human-Agent Communication
- Pitfalls in Evaluating Language Model Forecasters
- Can Language Models Use Forecasting Strategies?
- Large Language Model Prediction Capabilities: Evidence from a Real-World Forecasting Tournament
- LLMs Can Teach Themselves to Better Predict the Future
- Future-as-Label: Scalable Supervision from Real-World Outcomes
Related papers
- EmoPhone: A Multi-Wave Dataset for In-the-Wild Mobile and Wearable Affect Sensing
- Tasteply: Externalizing Stable and Situational Preference as Manipulable Spatial Objects
- Shielded Trajectory Sampling for Personalized Relational Safety in Conversational AI
- From Tasks to Turns: Measuring Human Labor Contributions in Human-Agent Joint Production
- Reconstruction and Reflection of Positive Experiences through Resurfacing Laughter-indexed Everyday Moments
- LLM-Assisted Reranking to Operationalize Nuanced Objectives in Recommender Systems