Calibrated Forecasting and Persuasion
summary
The gist
We study a dynamic game where an expert sends probabilistic forecasts to a decision-maker and we characterize the optimal forecasting strategy by reducing the dynamic game to a static persuasion…
This episode discusses
- Calibrated Forecasting and Persuasion · Paper Radio
- Generalized Principal-Agent Problem with a Learning Agent
The paper
Calibrated Forecasting and Persuasion · Read on arXiv
ATULYA JAIN, VIANNEY PERCHET
University of Bonn · ENSAE & Criteo AI Lab
We study a dynamic game where an expert sends probabilistic forecasts to a decision-maker. The decision-maker verifies these forecasts using a calibration test based on past data. How should the expert send forecasts to maximize her payoff while passing the test? For a stationary ergodic process, we characterize the optimal forecasting strategy by reducing the dynamic game to a static persuasion problem. Calibration restricts long-run forecast distributions to mean-preserving contractions of the distribution of truthful forecasts. We compare the payoffs attainable by an informed and uninformed expert, providing a benchmark for the value of information. Finally, we consider a decision-maker who uses a mean-based learning strategy and show that the expert can guarantee at least the calibration benchmark and sometimes strictly more.
Transcript
Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.
Ruth: We're looking at the paper: "Calibrated Forecasting and Persuasion".
Warren: Here's the gist of the paper. We study a dynamic game where an expert sends probabilistic forecasts to a decision-maker and we characterize the optimal forecasting strategy by reducing the dynamic game to a static persuasion problem.
Ruth: The Dynamic Game Setup The paper introduces a dynamic game between a sender (expert) and a receiver (decision-maker) where the state of the world evolves over time.
Warren: In each period, the sender provides a probabilistic forecast about that period’s unknown state to the receiver, who then chooses an action.
Ruth: The sender is assumed to be informed and knows the distribution of states, while the receiver is uninformed.
Title and authors: Ruth: So we're looking at this paper by Atulya Jain and Vianney Perchet called "Calibrated Forecasting and Persuasion". The core idea is a dynamic game between an expert sending probabilistic forecasts and a decision-maker who tests those forecasts using a calibration test based on past data.
Warren: It sounds like we're trying to figure out how an expert should forecast over time to maximize her payoff while passing this credibility check, right? What does the paper actually do with this setup?
Iris: They reduce the dynamic game down to a static persuasion problem by showing that the distributions of forecasts that can pass calibration tests are exactly what we call mean-preserving contractions of the distribution of conditionals.
Ruth: Mean-preserving contractions, that's a heavy term. So, instead of trying to figure out the whole path over time, they simplify it to finding a good static policy based on those conditional distributions. But they also compare what an informed expert can achieve versus what an uninformed one can do.
Warren: And that comparison is key for us traders, because we're always dealing with imperfect information about future states. Does this mean the paper gives us a way to quantify the value of knowing the true underlying process?
Iris: It does, by characterizing the attainable payoff for both experts. For an informed expert, they find that calibrated strategies induce mean-preserving contractions of the distribution of conditionals and her maximum payoff is tied to Per(Cµ, uˆS) where Cµ is the distribution of conditionals and uˆS is the indirect utility.
Ruth: So we have a benchmark for how much value information provides. But what about when the expert doesn't know the data-generating process at all? What are her limits there?
Warren: The paper characterizes her attainable payoff function for an uninformed sender by looking at the closed convex hull of her indirect utility, CoˆuS(f*)(p) in an adversarial setting, and if it’s an ergodic Markov chain, she can attain uˆS(f*)(p) where p is the long-run empirical distribution of states.
Iris: That's a bit different because for an uninformed sender, they look at the set of possible strategies rather than just one optimal path. It suggests that even without knowing the true process, you can still find a strategy that performs well based on what you observe over time.
Ruth: So it’s like saying if you don't know the rules of the game perfectly, you can still construct a strategy that gives you a decent return by playing optimally against the worst possible version of those rules. But then they bring in regret minimization as an alternative approach to look at how this plays out in practice.
Warren: Regret minimization is interesting because it suggests that if the receiver just reacts myopically best to calibrated forecasts, he incurs no regret in the long run. But wait, the paper also says you can do better than that against a mean-based receiver. That's what I want to hear for running a strategy.
Title and authors: Iris: That means if you use those calibrated forecasts, even against someone who just tries to follow the average outcome every period, you can still get a strictly better payoff than the benchmark Per(Cµ, uˆS). It exploits how mean-based algorithms adapt gradually over time.
Ruth: That’s a big implication for us in markets. It suggests that being calibrated isn't just about hitting a minimum bar; it's an active way to beat simpler learning strategies if you can structure your forecasts correctly. So, what are the practical applications they show?
Warren: They apply this to financial platforms where the optimal calibrated strategy achieves Per(Cµ, uˆS) when the state space support is affinely independent. In that case, it shows that even though we don't always forecast the absolute truth of the conditional probability, announcing accurate but coarser forecasts can be better than trying to be perfectly truthful with every single piece of information.
Iris: That makes sense for real-world data where perfect knowledge isn't available, so coarse forecasting is a practical trade-off. It also points toward the fact that calibration and regret minimization are closely related ideas in online learning literature.
Ruth: So to wrap up, the main point of this paper by Atulya Jain and Vianney Perchet is that calibration provides a solid micro-foundation for commitment assumptions in Bayesian persuasion by linking it to mean-preserving contractions. They show what an informed expert can guarantee, what an uninformed one can attain, and how you can beat simple learning strategies using regret minimization heuristics.
Warren: It seems like the ultimate message is that even when the expert is not perfectly informed or when facing a less sophisticated decision-maker, there's a way to structure your forecasting strategy—through calibration—to guarantee at least this benchmark and sometimes do better. That means calibration isn't just academic; it’s a tool for maximizing expected payoff under uncertainty.
Iris: I just want to emphasize that the model assumes the expert is informed when they are trying to pass any complex statistical test like calibration, which is an important assumption we should keep in mind when applying this to real markets.
Ruth: Exactly, because if the expert isn't informed about the process, her ability to pass that calibration test becomes much more constrained. The paper sets a high bar for what's possible under those assumptions.
Warren: So, moving forward, it seems like the next step is figuring out exactly how to implement this in a trading system where we have continuous data streams and need to make these dynamic adjustments based on the state evolution.
Iris: And maybe exploring extensions, like allowing the receiver’s actions to actually affect the state transitions in a Markov Decision Process. That would add another layer of complexity that's worth investigating after we digest this paper.
The paper's summary: Ruth: So we're looking at "Calibrated Forecasting and Persuasion," and what they're doing is taking this dynamic game—the expert sending forecasts over time—and turning it into a static persuasion problem.
Warren: They’re saying that the way you can pass the credibility test, which they call calibration, is mathematically equivalent to finding mean-preserving contractions of the distribution of conditionals.
Iris: So instead of tracking every single state evolution period by period, they boil it down to this static distribution relationship between what we expect and what we actually see.
Ruth: That’s right. The main idea is that calibration isn't just some arbitrary rule for experts; it’s a precise mathematical structure—the mean-preserving contraction—that describes the set of forecasts that are credible over time.
Warren: And they use this to set up the value of the game. For an informed expert, her maximum payoff is determined by Per(Cµ, uˆS).
Iris: That means she’s not just aiming for any high probability; she’s aiming for a specific utility measure derived from that distribution of conditionals and her indirect utility.
Ruth: It connects the dynamic prediction problem directly to a static one, which is a big simplification. But they also look at what happens when the expert doesn't know the underlying process perfectly.
Warren: For an uninformed sender, they characterize her possible payoffs using the closed convex hull of her indirect utility, CoˆuS(f*)(p). That’s her attainable payoff boundary in a worst-case scenario.
Iris: If it’s an ergodic Markov chain, she can actually reach the utility uˆS(f* (p)), where p is just the long-run average empirical distribution of states.
Ruth: So the paper shows that even without full knowledge, there are specific achievable payoffs based on how you structure your strategy against whatever process you observe.
Warren: And they bring in regret minimization as a practical tool here, suggesting that if the receiver just plays myopically best against calibrated forecasts, he incurs no regret in the long run.
Iris: But they also show you can get a strictly better payoff than that benchmark against a mean-based receiver by exploiting how those mean-based algorithms adapt over time.
Ruth: So what this means for us is that calibration is not just about meeting some minimum standard; it’s an active way to structure forecasts that can actually beat simpler learning strategies if you're willing to play them correctly.
Warren: The application in finance shows that when the state space support has a certain property, like being affinely independent, the optimal calibrated strategy ends up announcing accurate but coarser forecasts instead of trying to be perfectly truthful every single time.
Iris: That’s a practical trade-off for real-world data where perfect knowledge isn't available, so coarse forecasting can be better than constant high-frequency adjustments.
Ruth: It also points out that calibration and regret minimization are fundamentally related ideas in online learning literature, which is an important connection to keep in mind when designing systems that learn.
Warren: The paper sets a benchmark for what an informed system can guarantee, and it gives us the tools to see how to improve upon those benchmarks when we introduce heuristics like regret minimization.
Iris: But they do flag that they’re working under assumptions about the expert being informed when she's trying to pass calibration, so we have to be careful applying this directly if our model assumptions are wrong.
Ruth: Exactly. The next step is seeing how this framework handles more complex scenarios, like when the receiver’s actions actually change the future state transitions in a Markov Decision Process.
The paper's improvements: Tom: So we're looking at how they improve on that initial framework for "Calibrated Forecasting and Persuasion," focusing on how you can actually get better results than just sticking to the baseline benchmark.
Ruth: They suggest using regret minimization as a heuristic, which is pretty practical. The idea is if the receiver just plays myopically best against calibrated forecasts, he incurs no regret over time.
Warren: That’s interesting because we know that if you use those calibrated forecasts, even against someone who just tries to follow the average outcome every period, you can get a strictly better payoff than that Per(Cµ, uˆS) benchmark.
Iris: They show this happens by exploiting how mean-based algorithms adapt gradually over time; they exploit the cumulative payoffs from action up to period t.
Ruth: So it’s not just about being calibrated right now; it’s about structuring your forecasts so that you capitalize on the receiver's slow learning process.
Warren: That means if we can model that adaptation, we can design a strategy that outperforms the static benchmark in a dynamic setting.
Iris: They also explore extensions, like allowing the receiver’s actions to actually affect how the state transitions in a Markov Decision Process. That adds another layer of complexity to what they modeled.
Ruth: That would mean moving from a simple game where you just report probabilities, to something where your own forecasts influence the environment itself.
Warren: If that were feasible, the whole payoff structure would change because the expert’s signal isn't just being tested; it’s actively shaping the distribution of future states.
Iris: That would require a completely new way to look at identification, because you’d be tracking how your forecast affects the evolution of that distribution.
Ruth: It gets complicated fast, but the implication is that this research opens up ways to model more complex interactions in strategic environments.
Warren: So while they give us a solid baseline for informed experts and show how heuristics can beat simple learning, the real challenge now is modeling these dynamic feedback loops where actions matter.
Conclusion: Tom: So we're wrapping up on "Calibrated Forecasting and Persuasion," which basically shows how you can turn this dynamic expert forecasting game into a static persuasion problem based on mean-preserving contractions.
Ruth: It really boils down to calibration being a test of credibility, and they provide a solid mathematical foundation for what that credibility means in terms of forecasts.
Warren: For the quantitative side, it gives us the payoff bound Per(Cµ, uˆS) for informed experts and even characterizes the attainable set for uninformed ones as CoˆuS(f* (p)).
Iris: The identification part is solid because they link these dynamic games to static persuasion problems, which helps us isolate exactly what variation in the process is doing the work.
Ruth: It means we have a clear benchmark for performance—that Per value—and we know how much better you can get if you introduce regret minimization heuristics.
Warren: The implication is that even against a simple learner, if you structure your forecasts correctly, you can beat that benchmark by exploiting adaptation over time.
Iris: And they do flag the limitation that the expert being informed when passing calibration is a necessary assumption we should keep in mind for real-world applications.
Ruth: It’s a solid piece of work because it connects market incentives with rigorous statistical concepts like Markov chains and concentration inequalities.
Warren: So, while this paper gives us tools to structure better forecasts, the next logical step is seeing how these ideas play out when the receiver's actions actually change the future state transitions in a Markov Decision Process.
Iris: That’s where we should look next to see if this framework can handle those more interactive scenarios.
More episodes
- Daily Summary for 2026-11-19
- 2608.21274-Recommendation Quality and the Concentration of Consumption: Experimental Evidence from Netflix
- 2408.02757-A nonparametric test for diurnal variation in spot correlation processes
- 2306.16393-High-Dimensional Canonical Correlation Analysis
- 2605.17299-Geometric Brownian motion with intermittent entries and exits
- 2607.04753-Fooling Yourself: how narratives shape beliefs
- 2408.14671-Inference on Treatment Effects with Repeatedly Mismeasured High-Dimensional Controls
- 2201.06898-Difference-in-Differences Estimators for Treatments Continuously Distributed at Every Period
- 2501.15307-Influence Functions: A Unified Differentiation-Based Framework
- The day in conversation for 2026-11-19