Calibrated Forecasting and Persuasion

arXiv:2406.15680 · econ.TH, cs.GT · Submitted 2024-06-21 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.

Ruth: We're looking at the paper: "Calibrated Forecasting and Persuasion".

Warren: Here's the gist of the paper. We study a dynamic game where an expert sends probabilistic forecasts to a decision-maker and we characterize the optimal forecasting strategy by reducing the dynamic game to a static persuasion problem.

Ruth: The Dynamic Game Setup The paper introduces a dynamic game between a sender (expert) and a receiver (decision-maker) where the state of the world evolves over time.

Warren: In each period, the sender provides a probabilistic forecast about that period’s unknown state to the receiver, who then chooses an action.

Ruth: The sender is assumed to be informed and knows the distribution of states, while the receiver is uninformed.

Title and authors: Ruth: So we're looking at this paper by Atulya Jain and Vianney Perchet called "Calibrated Forecasting and Persuasion". The core idea is a dynamic game between an expert sending probabilistic forecasts and a decision-maker who tests those forecasts using a calibration test based on past data.

Warren: It sounds like we're trying to figure out how an expert should forecast over time to maximize her payoff while passing this credibility check, right? What does the paper actually do with this setup?

Iris: They reduce the dynamic game down to a static persuasion problem by showing that the distributions of forecasts that can pass calibration tests are exactly what we call mean-preserving contractions of the distribution of conditionals.

Ruth: Mean-preserving contractions, that's a heavy term. So, instead of trying to figure out the whole path over time, they simplify it to finding a good static policy based on those conditional distributions. But they also compare what an informed expert can achieve versus what an uninformed one can do.

Warren: And that comparison is key for us traders, because we're always dealing with imperfect information about future states. Does this mean the paper gives us a way to quantify the value of knowing the true underlying process?

Iris: It does, by characterizing the attainable payoff for both experts. For an informed expert, they find that calibrated strategies induce mean-preserving contractions of the distribution of conditionals and her maximum payoff is tied to Per(Cµ, uˆS) where Cµ is the distribution of conditionals and uˆS is the indirect utility.

Ruth: So we have a benchmark for how much value information provides. But what about when the expert doesn't know the data-generating process at all? What are her limits there?

Warren: The paper characterizes her attainable payoff function for an uninformed sender by looking at the closed convex hull of her indirect utility, CoˆuS(f*)(p) in an adversarial setting, and if it’s an ergodic Markov chain, she can attain uˆS(f*)(p) where p is the long-run empirical distribution of states.

Iris: That's a bit different because for an uninformed sender, they look at the set of possible strategies rather than just one optimal path. It suggests that even without knowing the true process, you can still find a strategy that performs well based on what you observe over time.

Ruth: So it’s like saying if you don't know the rules of the game perfectly, you can still construct a strategy that gives you a decent return by playing optimally against the worst possible version of those rules. But then they bring in regret minimization as an alternative approach to look at how this plays out in practice.

Warren: Regret minimization is interesting because it suggests that if the receiver just reacts myopically best to calibrated forecasts, he incurs no regret in the long run. But wait, the paper also says you can do better than that against a mean-based receiver. That's what I want to hear for running a strategy.

Title and authors: Iris: That means if you use those calibrated forecasts, even against someone who just tries to follow the average outcome every period, you can still get a strictly better payoff than the benchmark Per(Cµ, uˆS). It exploits how mean-based algorithms adapt gradually over time.

Ruth: That’s a big implication for us in markets. It suggests that being calibrated isn't just about hitting a minimum bar; it's an active way to beat simpler learning strategies if you can structure your forecasts correctly. So, what are the practical applications they show?

Warren: They apply this to financial platforms where the optimal calibrated strategy achieves Per(Cµ, uˆS) when the state space support is affinely independent. In that case, it shows that even though we don't always forecast the absolute truth of the conditional probability, announcing accurate but coarser forecasts can be better than trying to be perfectly truthful with every single piece of information.

Iris: That makes sense for real-world data where perfect knowledge isn't available, so coarse forecasting is a practical trade-off. It also points toward the fact that calibration and regret minimization are closely related ideas in online learning literature.

Ruth: So to wrap up, the main point of this paper by Atulya Jain and Vianney Perchet is that calibration provides a solid micro-foundation for commitment assumptions in Bayesian persuasion by linking it to mean-preserving contractions. They show what an informed expert can guarantee, what an uninformed one can attain, and how you can beat simple learning strategies using regret minimization heuristics.

Warren: It seems like the ultimate message is that even when the expert is not perfectly informed or when facing a less sophisticated decision-maker, there's a way to structure your forecasting strategy—through calibration—to guarantee at least this benchmark and sometimes do better. That means calibration isn't just academic; it’s a tool for maximizing expected payoff under uncertainty.

Iris: I just want to emphasize that the model assumes the expert is informed when they are trying to pass any complex statistical test like calibration, which is an important assumption we should keep in mind when applying this to real markets.

Ruth: Exactly, because if the expert isn't informed about the process, her ability to pass that calibration test becomes much more constrained. The paper sets a high bar for what's possible under those assumptions.

Warren: So, moving forward, it seems like the next step is figuring out exactly how to implement this in a trading system where we have continuous data streams and need to make these dynamic adjustments based on the state evolution.

Iris: And maybe exploring extensions, like allowing the receiver’s actions to actually affect the state transitions in a Markov Decision Process. That would add another layer of complexity that's worth investigating after we digest this paper.

The paper's summary: Ruth: So we're looking at "Calibrated Forecasting and Persuasion," and what they're doing is taking this dynamic game—the expert sending forecasts over time—and turning it into a static persuasion problem.

Warren: They’re saying that the way you can pass the credibility test, which they call calibration, is mathematically equivalent to finding mean-preserving contractions of the distribution of conditionals.

Iris: So instead of tracking every single state evolution period by period, they boil it down to this static distribution relationship between what we expect and what we actually see.

Ruth: That’s right. The main idea is that calibration isn't just some arbitrary rule for experts; it’s a precise mathematical structure—the mean-preserving contraction—that describes the set of forecasts that are credible over time.

Warren: And they use this to set up the value of the game. For an informed expert, her maximum payoff is determined by Per(Cµ, uˆS).

Iris: That means she’s not just aiming for any high probability; she’s aiming for a specific utility measure derived from that distribution of conditionals and her indirect utility.

Ruth: It connects the dynamic prediction problem directly to a static one, which is a big simplification. But they also look at what happens when the expert doesn't know the underlying process perfectly.

Warren: For an uninformed sender, they characterize her possible payoffs using the closed convex hull of her indirect utility, CoˆuS(f*)(p). That’s her attainable payoff boundary in a worst-case scenario.

Iris: If it’s an ergodic Markov chain, she can actually reach the utility uˆS(f* (p)), where p is just the long-run average empirical distribution of states.

Ruth: So the paper shows that even without full knowledge, there are specific achievable payoffs based on how you structure your strategy against whatever process you observe.

Warren: And they bring in regret minimization as a practical tool here, suggesting that if the receiver just plays myopically best against calibrated forecasts, he incurs no regret in the long run.

Iris: But they also show you can get a strictly better payoff than that benchmark against a mean-based receiver by exploiting how those mean-based algorithms adapt over time.

Ruth: So what this means for us is that calibration is not just about meeting some minimum standard; it’s an active way to structure forecasts that can actually beat simpler learning strategies if you're willing to play them correctly.

Warren: The application in finance shows that when the state space support has a certain property, like being affinely independent, the optimal calibrated strategy ends up announcing accurate but coarser forecasts instead of trying to be perfectly truthful every single time.

Iris: That’s a practical trade-off for real-world data where perfect knowledge isn't available, so coarse forecasting can be better than constant high-frequency adjustments.

Ruth: It also points out that calibration and regret minimization are fundamentally related ideas in online learning literature, which is an important connection to keep in mind when designing systems that learn.

Warren: The paper sets a benchmark for what an informed system can guarantee, and it gives us the tools to see how to improve upon those benchmarks when we introduce heuristics like regret minimization.

Iris: But they do flag that they’re working under assumptions about the expert being informed when she's trying to pass calibration, so we have to be careful applying this directly if our model assumptions are wrong.

Ruth: Exactly. The next step is seeing how this framework handles more complex scenarios, like when the receiver’s actions actually change the future state transitions in a Markov Decision Process.

The paper's improvements: Tom: So we're looking at how they improve on that initial framework for "Calibrated Forecasting and Persuasion," focusing on how you can actually get better results than just sticking to the baseline benchmark.

Ruth: They suggest using regret minimization as a heuristic, which is pretty practical. The idea is if the receiver just plays myopically best against calibrated forecasts, he incurs no regret over time.

Warren: That’s interesting because we know that if you use those calibrated forecasts, even against someone who just tries to follow the average outcome every period, you can get a strictly better payoff than that Per(Cµ, uˆS) benchmark.

Iris: They show this happens by exploiting how mean-based algorithms adapt gradually over time; they exploit the cumulative payoffs from action up to period t.

Ruth: So it’s not just about being calibrated right now; it’s about structuring your forecasts so that you capitalize on the receiver's slow learning process.

Warren: That means if we can model that adaptation, we can design a strategy that outperforms the static benchmark in a dynamic setting.

Iris: They also explore extensions, like allowing the receiver’s actions to actually affect how the state transitions in a Markov Decision Process. That adds another layer of complexity to what they modeled.

Ruth: That would mean moving from a simple game where you just report probabilities, to something where your own forecasts influence the environment itself.

Warren: If that were feasible, the whole payoff structure would change because the expert’s signal isn't just being tested; it’s actively shaping the distribution of future states.

Iris: That would require a completely new way to look at identification, because you’d be tracking how your forecast affects the evolution of that distribution.

Ruth: It gets complicated fast, but the implication is that this research opens up ways to model more complex interactions in strategic environments.

Warren: So while they give us a solid baseline for informed experts and show how heuristics can beat simple learning, the real challenge now is modeling these dynamic feedback loops where actions matter.

Conclusion: Tom: So we're wrapping up on "Calibrated Forecasting and Persuasion," which basically shows how you can turn this dynamic expert forecasting game into a static persuasion problem based on mean-preserving contractions.

Ruth: It really boils down to calibration being a test of credibility, and they provide a solid mathematical foundation for what that credibility means in terms of forecasts.

Warren: For the quantitative side, it gives us the payoff bound Per(Cµ, uˆS) for informed experts and even characterizes the attainable set for uninformed ones as CoˆuS(f* (p)).

Iris: The identification part is solid because they link these dynamic games to static persuasion problems, which helps us isolate exactly what variation in the process is doing the work.

Ruth: It means we have a clear benchmark for performance—that Per value—and we know how much better you can get if you introduce regret minimization heuristics.

Warren: The implication is that even against a simple learner, if you structure your forecasts correctly, you can beat that benchmark by exploiting adaptation over time.

Iris: And they do flag the limitation that the expert being informed when passing calibration is a necessary assumption we should keep in mind for real-world applications.

Ruth: It’s a solid piece of work because it connects market incentives with rigorous statistical concepts like Markov chains and concentration inequalities.

Warren: So, while this paper gives us tools to structure better forecasts, the next logical step is seeing how these ideas play out when the receiver's actions actually change the future state transitions in a Markov Decision Process.

Iris: That’s where we should look next to see if this framework can handle those more interactive scenarios.

ATULYA JAIN, VIANNEY PERCHET

University of Bonn · ENSAE & Criteo AI Lab

econ.TH, cs.GT

Submitted: 2024-06-21

Updated: 2026-10-29

Comments: The conference version of this work has been accepted to the Twenty-Fifth ACM Conference on Economics and Computation (EC'24)

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: We study a dynamic game where an expert sends probabilistic forecasts to a decision-maker and we characterize the optimal forecasting strategy by reducing the dynamic game to a static persuasion…

Terminology

Summary

We study a dynamic game where an expert sends probabilistic forecasts to a decision-maker and we characterize the optimal forecasting strategy by reducing the dynamic game to a static persuasion problem. p. 0

Where the paper says it

How should the expert send forecasts to maximize her payoff while passing the test? For a stationary ergodic process, we characterize the optimal forecasting strategy by reducing the dynamic game to a static persuasion problem.

The Dynamic Game Setup

The paper introduces a dynamic game between a sender (expert) and a receiver (decision-maker) where the state of the world evolves over time. In each period, the sender provides a probabilistic forecast about that period’s unknown state to the receiver, who then chooses an action. p. 5

Where the paper says it

ATULYA JAIN AND VIANNEY PERCHET that period’s unknown state. The receiver observes the forecast and then chooses an action.

The sender is assumed to be informed and knows the distribution of states, while the receiver is uninformed. p. 5
Where the paper says it

, ωt−1) the history of states up to period t. We assume the sender is informed and knows the distribution µ, whereas the receiver is uninformed.

A forecasting strategy is truthful if, in every period t, the sender reports the conditional probability that depends on the history of states. p. 5
Where the paper says it

The state and the receiver’s action in that period determine the sender’s and receiver’s payoffs, denoted by uS(ωt , at) and uR(ωt , at). A forecasting strategy is truthful if, in every period t, the sender reports the conditional, that is, ft = pt .

Calibration as a Credibility Test

The decision-maker assesses credibility using a calibration test based on past data. This test checks whether the forecast of an outcome is close to the actual proportion of times the outcome occurred when the forecast was made. p. 0

Where the paper says it

The idea is to check whether the forecast of an outcome is close to the actual proportion of times the outcome occurred when the forecast was made. For example, the investor checks whether an asset outperformed others on 70% of the days when the analyst predicted a 0.

A sequence of forecasts is defined as T-calibrated if its frequency-weighted average distance between forecasts and the empirical distributions of states does not exceed a tolerance margin epsilon T. p. 6
Where the paper says it

ATULYA JAIN AND VIANNEY PERCHET A T-sequence of forecasts is ϵT -calibrated if the frequency-weighted average distance between forecasts and the empirical distributions of states does not exceed the error margin ϵT . 6 We illustrate the calibration test using a rain forecasting example.

The sender incurs a punishment cost if she fails this test in that period. p. 7
Where the paper says it

ATULYA JAIN AND VIANNEY PERCHET Fail: If the sequence of past forecasts is not ϵt−1-calibrated, the sender fails the calibration test in period t. In this case, she incurs a punishment cost −c, where c > 0.

Reduction to Static Persuasion

The main result establishes that the sender’s maximum payoff in the dynamic forecasting game equals the value of a static persuasion problem. This equivalence shows that the calibration test provides a micro-foundation for the commitment assumption in Bayesian persuasion. p. 2

Where the paper says it

Thus, the sender’s dynamic problem reduces to a static persuasion problem. This equivalence shows that the calibration test provides a micro-foundation for the commitment assumption in Bayesian persuasion.

The state, signal, and prior in the persuasion problem correspond to the conditional probability, the forecast, and the distribution of conditionals in the dynamic game. p. 1
Where the paper says it

Our main result (Theorem 1) establishes that the sender’s maximum payoff in the dynamic forecasting game equals the value of a static persuasion problem. The state, signal, and prior in the persuasion problem correspond to the conditional probability, the forecast, and the distribution of conditionals in the dynamic game.

Informed Sender Results

For a stationary ergodic process, calibrated strategies induce precisely the mean-preserving contractions of the distribution of conditionals. The sender’s maximum payoff equals Per(Cµ, uˆS) where Cµ is the distribution of conditionals and uˆS is the indirect utility. p. 9

Where the paper says it

In the dynamic game, uˆS(f) equals the sender’s expected payoff when the receiver best responds to belief f. The sender chooses a distribution Q ∈ M(P) to maximize her expected indirect utility.

This result relies on Lemma 3, which shows that for a stationary ergodic process µ, the distribution of conditionals Cµ exists and is constant almost surely. p. 0
Where the paper says it

For a stationary ergodic process, we characterize the optimal forecasting strategy by reducing the dynamic game to a static persuasion problem. The distributions of forecasts that can arise under calibration are precisely the mean-preserving contractions of the distribution of conditionals.

Uninformed Sender Results

For an uninformed sender, the attainable payoff function is characterized by the closed convex hull of her indirect utility CoˆuS(f∗(p)) in the adversarial case. When the unknown process is an ergodic Markov chain, she can attain uˆS(f∗(p)) where p is the long-run empirical distribution of states. p. 2

Where the paper says it

Second, when the unknown process is an ergodic Markov chain, she can do better. In particular, she can approximately attain the payoff corresponding to the no-disclosure policy in the persuasion problem with prior given by the distribution of conditionals—the same problem as in the informed sender case (Proposition 3).

Regret Minimization and Heuristics

The paper examines regret minimization as an alternative heuristic for decision-making. It shows that if the receiver myopically best responds to calibrated forecasts, he incurs no regret in the long run. p. 2

Where the paper says it

Regret minimization is closely linked to calibration (see Perchet, 2014), and we build on this connection to obtain new results. We show that if the receiver myopically best responds to the forecasts of any calibrated strategy, then he incurs no regret, thereby justifying the use of calibration as a decision-making heuristic (Proposition 4).

Furthermore, against a mean-based receiver, the sender can attain a payoff strictly greater than Per(Cµ, uˆS). p. 4
Where the paper says it

A common theme in the literature is that the sender can obtain higher payoff than the rational benchmark when facing such algorithms. Similarly, we show the sender can obtain a higher utility than the calibration benchmark against a mean-based learner.

Application to Financial Platforms

The application to a financial platform shows that the optimal calibrated strategy achieves Per(Cµ, uˆS) when the state space support is affinely independent. The optimal strategy announces accurate but coarser forecasts rather than the truthful conditionals. p. 2

Where the paper says it

We characterize the platform’s optimal forecasting strategy using the concavification approach and illustrate it graphically. We show that it may be optimal for the platform to send coarse forecasts rather than truthful ones.

Conclusion and Future Work

The paper concludes that calibration and regret minimization are closely related notions in the online learning literature. Many problems remain open, such as characterizing the optimal calibrated strategy for any stochastic process. p. 7

Where the paper says it

Unlike the finite test, the asymptotic notion does not depend on the chosen sequence of error margins. The truthful forecasting strategy is calibrated for any stochastic process [Dawid, 1982].

The model admits natural extensions, including allowing the receiver’s actions to affect state transitions in a Markov Decision Process. p. 1
Where the paper says it

Given that she is tested by a calibration test, how should the expert send forecasts to maximize her payoff? To do this, we develop a dynamic sender-receiver model where the state of the world evolves according to a stochastic process.

Appendix B Details

Appendix B provides an equivalent formulation of the persuasion problem in terms of Blackwell experiments rather than mean-preserving contractions. It shows that a probability distribution Q is a simple mean-preserving contraction if and only if E(Q) is a garbling of E(P). p. 9

Where the paper says it

Definition 3. A probability distribution Q = (µ, r) is a simple mean-preserving contraction of P = (λ, q), if there exists a row-stochastic matrix G ∈ R n×m satisfying (3.

p. 9
Where the paper says it

Definition 3. A probability distribution Q = (µ, r) is a simple mean-preserving contraction of P = (λ, q), if there exists a row-stochastic matrix G ∈ R n×m satisfying (3.

This equivalence links the two ways of describing the persuasion problem. p. 2
Where the paper says it

Thus, the sender’s dynamic problem reduces to a static persuasion problem. This equivalence shows that the calibration test provides a micro-foundation for the commitment assumption in Bayesian persuasion.

p. 2
Where the paper says it

Thus, the sender’s dynamic problem reduces to a static persuasion problem. This equivalence shows that the calibration test provides a micro-foundation for the commitment assumption in Bayesian persuasion.

The optimization problem (B. p. 0
Where the paper says it

How should the expert send forecasts to maximize her payoff while passing the test? For a stationary ergodic process, we characterize the optimal forecasting strategy by reducing the dynamic game to a static persuasion problem.

p. 0
Where the paper says it

How should the expert send forecasts to maximize her payoff while passing the test? For a stationary ergodic process, we characterize the optimal forecasting strategy by reducing the dynamic game to a static persuasion problem.

1) is in terms of constrained Blackwell experiments while the equivalent problem (3. p. 4
Where the paper says it

All omitted proofs are in Appendix A. Appendix B contains a reformulation of the persuasion problem in terms of Blackwell experiments.

p. 4
Where the paper says it

All omitted proofs are in Appendix A. Appendix B contains a reformulation of the persuasion problem in terms of Blackwell experiments.

3) is in terms of mean-preserving contractions. p. 0
Where the paper says it

For a stationary ergodic process, we characterize the optimal forecasting strategy by reducing the dynamic game to a static persuasion problem. The distributions of forecasts that can arise under calibration are precisely the mean-preserving contractions of the distribution of conditionals.

p. 0
Where the paper says it

For a stationary ergodic process, we characterize the optimal forecasting strategy by reducing the dynamic game to a static persuasion problem. The distributions of forecasts that can arise under calibration are precisely the mean-preserving contractions of the distribution of conditionals.

The optimal garbling F≾E(P) maximizes EQ[ˆuS]. p. 9
Where the paper says it

3) Per(P, uˆS) := max Q∈M(P) EQ[ˆuS]. Let Q∗ denote an optimal distribution of forecasts solving (3.

p. 9
Where the paper says it

3) Per(P, uˆS) := max Q∈M(P) EQ[ˆuS]. Let Q∗ denote an optimal distribution of forecasts solving (3.

The special case for a perfectly informed sender reduces the problem to the canonical Bayesian persuasion setting. p. 2
Where the paper says it

First, in an adversarial setting where nature can adapt its strategy, there is no fixed prior, and the attainable payoff is evaluated as a function of the realized long-run empirical distribution of states. In this case, the sender can approximately guarantee the minimal payoff in the canonical persuasion problem, where the mean of the prior equals the realized empirical distribution (Theorem 2).

p. 2
Where the paper says it

First, in an adversarial setting where nature can adapt its strategy, there is no fixed prior, and the attainable payoff is evaluated as a function of the realized long-run empirical distribution of states. In this case, the sender can approximately guarantee the minimal payoff in the canonical persuasion problem, where the mean of the prior equals the realized empirical distribution (Theorem 2).

The proof of Proposition 3 uses opportunistic approachability to show that under certain restrictions on nature’s play, the sender can approach R+(Q). p. 2
Where the paper says it

We show that if the receiver myopically best responds to the forecasts of any calibrated strategy, then he incurs no regret, thereby justifying the use of calibration as a decision-making heuristic (Proposition 4). Conversely, when the informed sender faces a regret-minimizing receiver, she can guarantee at least the calibration benchmark—namely, the persuasion value—and, for a natural class of regret-minimizing algorithms, sometimes strictly more (Proposition 5 and Theorem 3).

p. 2
Where the paper says it

We show that if the receiver myopically best responds to the forecasts of any calibrated strategy, then he incurs no regret, thereby justifying the use of calibration as a decision-making heuristic (Proposition 4). Conversely, when the informed sender faces a regret-minimizing receiver, she can guarantee at least the calibration benchmark—namely, the persuasion value—and, for a natural class of regret-minimizing algorithms, sometimes strictly more (Proposition 5 and Theorem 3).

The final result shows that the sender can guarantee at least Per(Cµ, uˆS) against a regret-minimizing receiver. p. 2
Where the paper says it

We show that if the receiver myopically best responds to the forecasts of any calibrated strategy, then he incurs no regret, thereby justifying the use of calibration as a decision-making heuristic (Proposition 4). Conversely, when the informed sender faces a regret-minimizing receiver, she can guarantee at least the calibration benchmark—namely, the persuasion value—and, for a natural class of regret-minimizing algorithms, sometimes strictly more (Proposition 5 and Theorem 3).

p. 2
Where the paper says it

We show that if the receiver myopically best responds to the forecasts of any calibrated strategy, then he incurs no regret, thereby justifying the use of calibration as a decision-making heuristic (Proposition 4). Conversely, when the informed sender faces a regret-minimizing receiver, she can guarantee at least the calibration benchmark—namely, the persuasion value—and, for a natural class of regret-minimizing algorithms, sometimes strictly more (Proposition 5 and Theorem 3).

This is achieved by showing that for every forecast f sent with positive frequency, any no-regret learning strategy plays a best response to f with long-run frequency one. p. 7
Where the paper says it

A forecasting strategy σ is calibrated if it is ϵ-calibrated, for every ϵ > 0. Intuitively, a forecasting strategy is calibrated if, for every forecast used with positive long-run frequency, the empirical distribution of states given that forecast equals the forecast itself.

The construction of the optimal strategy exploits the fact that mean-based algorithms adapt gradually over time. p. 2
Where the paper says it

First, in an adversarial setting where nature can adapt its strategy, there is no fixed prior, and the attainable payoff is evaluated as a function of the realized long-run empirical distribution of states. In this case, the sender can approximately guarantee the minimal payoff in the canonical persuasion problem, where the mean of the prior equals the realized empirical distribution (Theorem 2).

The overall average payoff in this example is approximately 28 > 2. p. 1
Where the paper says it

Intuitively, this corresponds to merging low-value and high-value truthful forecasts with appropriate weights so that the overall forecast matches the true frequencies. Thus, while the forecasts must be correct on average, they can be less informative than truthful forecasts.

p. 1
Where the paper says it

Intuitively, this corresponds to merging low-value and high-value truthful forecasts with appropriate weights so that the overall forecast matches the true frequencies. Thus, while the forecasts must be correct on average, they can be less informative than truthful forecasts.

References

The paper cites a wide range of literature, including works by Kamenica and Gentzkow on Bayesian persuasion, Foster and Vohra on asymptotic calibration, and studies on regret minimization in strategic environments. The references cover topics such as optimal persuasion via bi-pooling, and the relationship between calibration and online learning heuristics. p. 2

Where the paper says it

As shown by Foster and Vohra [1998], even an uninformed sender can pass the calibration test for any stochastic process. As before, we characterize the sender’s attainable payoff via a signaling policy in a persuasion problem.

p. 2
Where the paper says it

As shown by Foster and Vohra [1998], even an uninformed sender can pass the calibration test for any stochastic process. As before, we characterize the sender’s attainable payoff via a signaling policy in a persuasion problem.

p. 2
Where the paper says it

As shown by Foster and Vohra [1998], even an uninformed sender can pass the calibration test for any stochastic process. As before, we characterize the sender’s attainable payoff via a signaling policy in a persuasion problem.

The work also relates to strategic information transmission in dynamic environments. p. 3
Where the paper says it

ATULYA JAIN AND VIANNEY PERCHET 1. Related Literature Strategic Information Transmission: Our work contributes to the literature on communication between an informed sender and an uninformed receiver.

p. 3
Where the paper says it

ATULYA JAIN AND VIANNEY PERCHET 1. Related Literature Strategic Information Transmission: Our work contributes to the literature on communication between an informed sender and an uninformed receiver.

p. 3
Where the paper says it

ATULYA JAIN AND VIANNEY PERCHET 1. Related Literature Strategic Information Transmission: Our work contributes to the literature on communication between an informed sender and an uninformed receiver.

The authors reference specific results concerning concentration inequalities for Markov chains. p. 5
Where the paper says it

3 considers the case where the sender is uninformed. 4This holds in natural environments, such as finite Markov chains.

p. 5
Where the paper says it

3 considers the case where the sender is uninformed. 4This holds in natural environments, such as finite Markov chains.

p. 5
Where the paper says it

3 considers the case where the sender is uninformed. 4This holds in natural environments, such as finite Markov chains.

Appendix A Details

Appendix A contains omitted results and proofs, including Proposition 7 which states that if the punishment cost satisfies c > (1 + √2)L, then any optimal forecasting strategy passes the calibration test with long-run frequency one in expectation. The proof of Proposition 7 demonstrates that the sender’s average payoff is bounded by Per(Cµ, uˆS) under these conditions. p. 9

Where the paper says it

Given prior P ∈ ∆(∆Ω) and indirect utility uˆS, define the persuasion value by (3. 3) Per(P, uˆS) := max Q∈M(P) EQ[ˆuS].

The proof of Proposition 2 shows that a calibrated strategy is equivalent to condition (3. p. 2
Where the paper says it

Regret minimization is closely linked to calibration (see Perchet, 2014), and we build on this connection to obtain new results. We show that if the receiver myopically best responds to the forecasts of any calibrated strategy, then he incurs no regret, thereby justifying the use of calibration as a decision-making heuristic (Proposition 4).

7) in terms of empirical frequencies. p. 6
Where the paper says it

The truthful forecasts F1 pass the calibration test in this example. This reflects the fact that, in the long run, empirical frequencies converge to the true conditional probabilities.

The proof of Lemma 3 establishes the existence and constancy of the distribution of conditionals Cµ for stationary ergodic processes. p. 0
Where the paper says it

For a stationary ergodic process, we characterize the optimal forecasting strategy by reducing the dynamic game to a static persuasion problem. The distributions of forecasts that can arise under calibration are precisely the mean-preserving contractions of the distribution of conditionals.

The proof of Lemma 1 characterizes the set M(Cµ) and proves that every finitely supported Q in M(Cµ) can be induced by a calibrated strategy. p. 7
Where the paper says it

The truthful forecasting strategy is calibrated for any stochastic process [Dawid, 1982]. Moreover, one can choose the sequence ϵt → 0 slowly enough so that truthful forecasting fails the finite calibration test only finitely often (see Proposition 6 in Appendix A).

The proof of Theorem 2 shows that the function p→CoˆuS(f∗(p)) is attainable and is the maximum attainable function. p. 2
Where the paper says it

First, in an adversarial setting where nature can adapt its strategy, there is no fixed prior, and the attainable payoff is evaluated as a function of the realized long-run empirical distribution of states. In this case, the sender can approximately guarantee the minimal payoff in the canonical persuasion problem, where the mean of the prior equals the realized empirical distribution (Theorem 2).

The proof of Proposition 4 confirms that a receiver who best responds to each forecast incurs no regret if a forecast is sent only with vanishing frequency. p. 2
Where the paper says it

Regret minimization is closely linked to calibration (see Perchet, 2014), and we build on this connection to obtain new results. We show that if the receiver myopically best responds to the forecasts of any calibrated strategy, then he incurs no regret, thereby justifying the use of calibration as a decision-making heuristic (Proposition 4).

The proof of Proposition 5 shows that the sender can guarantee Per(Cµ, uˆS) against a regret-minimizing receiver. p. 2
Where the paper says it

We show that if the receiver myopically best responds to the forecasts of any calibrated strategy, then he incurs no regret, thereby justifying the use of calibration as a decision-making heuristic (Proposition 4). Conversely, when the informed sender faces a regret-minimizing receiver, she can guarantee at least the calibration benchmark—namely, the persuasion value—and, for a natural class of regret-minimizing algorithms, sometimes strictly more (Proposition 5 and Theorem 3).

The construction of the strategy exploits the fact that mean-based algorithms adapt gradually over time to achieve a strictly higher payoff than the calibration benchmark. p. 4
Where the paper says it

A common theme in the literature is that the sender can obtain higher payoff than the rational benchmark when facing such algorithms. Similarly, we show the sender can obtain a higher utility than the calibration benchmark against a mean-based learner.

p. 4
Where the paper says it

A common theme in the literature is that the sender can obtain higher payoff than the rational benchmark when facing such algorithms. Similarly, we show the sender can obtain a higher utility than the calibration benchmark against a mean-based learner.

The final result shows that for any stochastic process, an informed platform can pass the calibration test by forecasting truthfully when Cµ is well defined. p. 1
Where the paper says it

We first analyze the case of an informed sender who knows the data-generating process. She can pass the calibration test by forecasting truthfully.

The paper cites a wide range of literature, including works by Kamenica and Gentzkow on Bayesian persuasion, Foster and Vohra on asymptotic calibration, and studies on regret minimization in strategic environments. The references cover topics such as optimal persuasion via bi-pooling, and the relationship between calibration and online learning heuristics. The work also relates to strategic information transmission in dynamic environments. The authors reference specific results concerning concentration inequalities for Markov chains. The references cover topics such as optimal persuasion via bi-pooling, and the relationship between calibration and online learning heuristics. The work also relates to strategic information transmission in dynamic environments. The authors reference specific results concerning concentration inequalities for Markov chains.

Appendix B provides an equivalent formulation of the persuasion problem in terms of Blackwell experiments rather than mean-preserving contractions. It shows that a probability distribution Q is a simple mean-preserving contraction if and only if E(Q) is a garbling of E(P). This equivalence links the two ways of describing the persuasion problem. The optimization problem (B.1) is in terms of constrained Blackwell experiments while the equivalent problem (3.3) is in terms of mean-preserving contractions. The optimal garbling F≾E(P) maximizes EQ[ˆuS]. The special case for a perfectly informed sender reduces the problem to the canonical Bayesian persuasion setting. The proof of Proposition 3 uses opportunistic approachability to show that under certain restrictions on nature’s play, the sender can approach R+(Q). The final result shows that the sender can guarantee at least Per(Cµ, uˆS) against a regret-minimizing receiver. The construction of the strategy exploits the fact that mean-based algorithms adapt gradually over time to achieve a strictly higher payoff than the calibration benchmark. The overall average payoff in this example is approximately 28 > 2.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Dynamic forecasting strategy optimization for informed experts: The system can determine that the distributions of forecasts that can arise under calibration are precisely the mean-preserving contractions of the distribution of conditionals. This allows an AI to select forecasts that maximize payoff while satisfying a calibration test, effectively finding strategies where the distributions that can arise under calibration are precisely the mean-preserving contractions of the distribution of conditionals.

  2. Financial platform strategic forecasting: The AI can optimize a financial platform's strategy by choosing forecasts based on the concave envelope of indirect utility: The platform’s optimal calibrated strategy achieves Per(Cµ, uˆS). This means the system will issue coarse forecasts rather than truthful ones when it maximizes its expected long-run average payoff.

  3. Uninformed sender performance benchmarking: The system can provide upper bounds for uninformed experts' payoffs by characterizing the closed convex hull of her indirect utility, which is Co ˆuS(f∗(p)) in the adversarial case and uˆS(f∗(p)) when the process is an ergodic Markov chain.

  4. Regret minimization heuristic integration: The AI can utilize regret minimization as a decision-making heuristic, knowing that if the receiver myopically best responds to the forecasts of any calibrated strategy, then he incurs no regret, thus justifying calibration as a practical tool.

  5. No-regret learner interaction modeling: When interacting with mean-based learning algorithms, the system can predict superior payoffs by exploiting adaptation: the sender can attain a payoff strictly greater than Per(Cµ, uˆS) by swapping forecasts to exploit cumulative payoffs from action up to period t.

Abstract

We study a dynamic game where an expert sends probabilistic forecasts to a decision-maker. The decision-maker verifies these forecasts using a calibration test based on past data. How should the expert send forecasts to maximize her payoff while passing the test? For a stationary ergodic process, we characterize the optimal forecasting strategy by reducing the dynamic game to a static persuasion problem. Calibration restricts long-run forecast distributions to mean-preserving contractions of the distribution of truthful forecasts. We compare the payoffs attainable by an informed and uninformed expert, providing a benchmark for the value of information. Finally, we consider a decision-maker who uses a mean-based learning strategy and show that the expert can guarantee at least the calibration benchmark and sometimes strictly more.

Sources

Related papers