Learning controls in mean field games: the fictitious play and related gradient descent
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Mathematics. The day in mathematics: new theorems, new proofs, and what they open.
Ada: I'm Ada.
Emmy: And I'm Emmy, here with Kolya.
Ada: We're looking at the paper: "Learning controls in mean field games".
Emmy: Here's the gist of the paper. Mean field games are investigated through two learning procedures and their relationship to the formation of equilibrium in potential mean field games The first one is a version of the fictitious play in which players observe directly the actions (controls) of other players and learn the distribution of the MFG through an update rules on controls On the other hand we also consider the learning procedure of a principal agent trying to solve a mean field control problem with only local gradient information Learning Procedures and Equilibrium Convergence The paper investigates two learning procedures and their relationship to the formation of equilibrium in potential mean field games The first is fictitious play where players learn the distribution of the MFG through an update rules on controls This leads to a new iterative procedure which converges for a wide class of potential MFGs of controls, even in the presence of common noise On the other hand we also consider the learning procedure of a principal agent trying to solve a mean field control problem with only local gradient information We prove that under very general assumption, the gradient descent of such agent converges to a solution of the associated potential MFG Fictitious Play Mechanism The fictitious play procedure involves an iterative scheme where players choose the best response to their belief on the distribution of other players and then update their belief on the control played by the population This new update rule for the fictitious play converges under a different set of assumptions and can, in particular, handle MFGs of controls The interpretation is that at each iteration players choose the best response αn to their belief (¯mnt)t∈
0,T: on the distribution of other players Gradient Descent in Mean Field Control The second part of this article is also dedicated to explaining how gradient descent in mean field control (MFC) problems converges under very general assumptions to a MFG equilibrium of the associated potential game A principal agent looking to solve the MFC problem with only local information will eventually converge to a control solution of the potential MFG Convergence Results and Noise Effects The main result concerns the convergence of fictitious play (1.4) to the set of MFG equilibria satisfying (1.2) in an appropriate weak sense due to the presence of common noise The fact that two possible learning rules coexists is in our opinion, another manifestation of the two known approaches in mean field games: the probabilistic theory
14: relying on stochastic processes and a more analytic point of view which consists in working directly at the level of probability measures
12: Gradient Descent Analysis The gradient descent algorithm on the space of controls yields an iterative procedure which converges to an equilibrium of the mean field game problem The sequence (αn)n∈N generated by (3.2) is bounded in HT and limn→∞∥∇J (αn)∥T = 0 Strong Convergence in Convex Setting Under Hypothesis 2.14, if the sequence of step sizes (ηn)n∈N satisfies X+∞ n=1 ηn = +∞, X+∞ n=1 η2 n < +∞, then the sequence of controls (αn)n∈N generated by the fictitious play converges strongly in HTd to the optimal control α∗ solution of the mean field control problem (2.6) Convergence with Common Noise Under Hypotheses 3.2 and 3.1, if σ > 0, then for any sequence (ηn)n∈N satisfying (3.4), the sequence (αn)n∈N generated by (3.2) is precompact in HTd and any limit point α∗ is a solution of the mean field game problem The convergence to a MFG equilibrium is demonstrated by showing that the associated cost converges to the equilibrium cost Numerical Implementation The iterative scheme (3.9) can be implemented numerically, although there are technical details regarding computing conditional expectation with respect to the filtration The trade-off in terms of speed for each iteration is considered worth it Final Conclusion on Convergence In conclusion, the sequence (αn)n∈N generated by the fictitious play converges strongly in HTd to the optimal control α∗ solution of the mean field control problem when appropriate convexity assumptions are met This result holds even in the presence of common noise under certain conditions The representative player process is exactly the process which to each player ω maps its optimal control and state References
1: Christian Bender and Jianfeng Zhang.
Ada: Time discretization and markovian iteration for coupled fbsdes
2: Alain Bensoussan, Jens Frehse, and Phillip Yam.
Emmy: Mean Field Games and Mean Field Type Control Theory
3: Charles Bertucci.
Ada: Monotone solutions for mean field games master equations: continuous state space and common noise
4: Charles Bertucci.
Paper summary: Ada: Wrapping up this paper "Learning controls in mean field games: the fictitious play and related gradient descent", it really boils down to showing that two very different ways of learning—watching the crowd versus just taking local steps based on gradients—both work to find an equilibrium in potential mean field games <ref:2610.06292#pg1>.
Emmy: The authors are connecting the practical idea of fictitious play, where agents try to infer others' actions from past experiences, with the more analytic approach of gradient descent for a principal agent solving a control problem <ref:2610.06292#pg3>. It shows these methods aren't just theoretical exercises; they lead to actual solutions for mean field games of controls.
Kolya: From my view, what this means is that we have more reliable ways to understand how complex systems, like large populations making decisions based on local rules, settle down into a stable collective behavior <ref:2610.06292#pg3>. The convergence results under noise are particularly relevant because real-world data is rarely perfectly clean.
Ada: So the implication is that whether you model the learning through direct observation of controls or through a gradient descent process, you can find a meaningful result for the mean field game equilibrium <ref:2610.06292#pg3>. The work by Meynard and others on this convergence is quite thorough thirty-one <ref:2610.06292#pg3>.
Emmy: It provides a solid foundation for applying these ideas to actual control problems where the cost function depends on the distribution of controls, not just individual actions <ref:2610.06292#pg2>. The fictitious play mechanism helps us understand how that distribution itself evolves.
Kolya: I think the paper points toward a more analytic viewpoint for mean field games, working directly with probability measures instead of just simulations thirty-three <ref:2610.06292#pg3>. That shift in perspective is important for getting tighter bounds on the convergence rates <ref:2610.06292#pg3>.
Ada: So to sum up, this paper gives us convergence results for both learning procedures in potential mean field games, showing how they lead to solutions even with common noise, and how that relates back to the underlying control problem structure <ref:2610.06292#pg1>.
Emmy: It’s a deep dive into the structure of these systems, showing that the fictitious play approach is as robust as it gets for this class of problems thirty-two <ref:2610.06292#pg3>.
Kolya: And for those of us interested in the probabilistic side, it connects simulation ideas with rigorous convergence theorems, which is pretty useful when you're trying to build something that actually runs fifteen.
Ada: That’s the gist of "Learning controls in mean field games: the fictitious play and related gradient descent". It’s about showing that learning mechanisms are powerful tools for finding equilibrium in these large-scale systems <ref:2610.06292#pg1>.
Conclusion: Ada: So, we've seen how two different learning methods, fictitious play and gradient descent, both lead to solutions for these mean field games of controls <ref:2610.06292#pg1>.
Emmy: That’s right. The paper is about these two ways of learning—one where players watch what others do, and the other where a principal agent just takes local steps based on gradients—and how they connect to the actual equilibrium we're looking for <ref:2610.06292#pg3>.
Kolya: What I find interesting is that these methods coexist. It suggests there are different structural ways to approach these mean field games, like using stochastic processes versus working directly with probability measures <ref:2610.06292#pg1>.
Ada: Exactly. The authors show that the fictitious play procedure converges under certain conditions, even when there's common noise in the system <ref:2610.06292#pg3>.
Emmy: And they also prove that for gradient descent, if you have very general assumptions, it will settle down to a solution of the potential mean field game <ref:2610.06292#pg3>.
Kolya: The convergence results with common noise are pretty solid under those specific hypotheses <ref:2610.06292#pg3>. It shows that even if things are noisy, the system still finds a stable point <ref:fifteen.
Ada: So, in conclusion, these learning procedures aren't just theoretical exercises anymore; they are concrete ways to find the optimal control solutions for these complex systems <ref:2610.06292#pg1>.
Emmy: It really highlights how important it is to have different lenses—the observational one and the analytic one—to understand these kinds of control problems <ref:thirty-three.
Kolya: It makes me wonder what happens when we apply these results to real-world scenarios where the noise isn't just a simple constant, but something more complex like in financial modeling or climate science.
math.OC, math.PR
Submitted: 2026-10-05
Updated: 2026-11-12
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 80/100
The gist: Mean field games are investigated through two learning procedures and their relationship to the formation of equilibrium in potential mean field games The first one is a version of the fictitious…
Key concepts
- Fictitious Play Mechanism
- This is an iterative process where players repeatedly choose the best response based on their current belief about what other players are doing. They then update this belief based on the actual controls played by the entire population in previous steps. This mechanism allows players to learn the underlying distribution of actions.
- Gradient Descent in Mean Field Control
- This procedure involves a principal agent trying to solve a mean field control problem using only local gradient information. The paper proves that this gradient descent algorithm converges to a solution of the associated potential mean field game under very general assumptions, meaning the agent eventually finds the correct control strategy.
- Potential Mean Field Game (MFG)
- An MFG is a mathematical model describing how many interacting agents behave in a large population. A potential MFG adds an extra layer where the system's behavior is linked to a potential function, which helps define and find stable equilibrium states for the control problem.
Terminology
Summary
Mean field games are investigated through two learning procedures and their relationship to the formation of equilibrium in potential mean field games The first one is a version of the fictitious play in which players observe directly the actions (controls) of other players and learn the distribution of the MFG through an update rules on controls On the other hand we also consider the learning procedure of a principal agent trying to solve a mean field control problem with only local gradient information
Learning Procedures and Equilibrium Convergence
The paper investigates two learning procedures and their relationship to the formation of equilibrium in potential mean field games The first is fictitious play where players learn the distribution of the MFG through an update rules on controls This leads to a new iterative procedure which converges for a wide class of potential MFGs of controls, even in the presence of common noise On the other hand we also consider the learning procedure of a principal agent trying to solve a mean field control problem with only local gradient information We prove that under very general assumption, the gradient descent of such agent converges to a solution of the associated potential MFG
Fictitious Play Mechanism
The fictitious play procedure involves an iterative scheme where players choose the best response to their belief on the distribution of other players and then update their belief on the control played by the population This new update rule for the fictitious play converges under a different set of assumptions and can, in particular, handle MFGs of controls The interpretation is that at each iteration players choose the best response αn to their belief (¯mnt)t∈[0,T] on the distribution of other players
Gradient Descent in Mean Field Control
The second part of this article is also dedicated to explaining how gradient descent in mean field control (MFC) problems converges under very general assumptions to a MFG equilibrium of the associated potential game A principal agent looking to solve the MFC problem with only local information will eventually converge to a control solution of the potential MFG
Convergence Results and Noise Effects
The main result concerns the convergence of fictitious play (1.4) to the set of MFG equilibria satisfying (1. p. 12) in an appropriate weak sense due to the presence of common noise The fact that two possible learning rules coexists is in our opinion, another manifestation of the two known approaches in mean field games: the probabilistic theory [14] relying on stochastic processes and a more analytic point of view which consists in working directly at the level of probability measures [12]
Gradient Descent Analysis
The gradient descent algorithm on the space of controls yields an iterative procedure which converges to an equilibrium of the mean field game problem The sequence (αn)n∈N generated by (3.2) is bounded in HT and limn→∞∥∇J (αn)∥T = 0
Strong Convergence in Convex Setting
Under Hypothesis 2.14, if the sequence of step sizes (ηn)n∈N satisfies X+∞ n=1 ηn = +∞, X+∞ n=1 η2 n < +∞, then the sequence of controls (αn)n∈N generated by the fictitious play converges strongly in HTd to the optimal control α∗ solution of the mean field control problem (2. p. 66)
Convergence with Common Noise
Under Hypotheses 3.2 and 3.1, if σ > 0, then for any sequence (ηn)n∈N satisfying (3. p. 7 p. 74), the sequence (αn)n∈N generated by (3. p. 1 p. 12) is precompact in HTd and any limit point α∗ is a solution of the mean field game problem The convergence to a MFG equilibrium is demonstrated by showing that the associated cost converges to the equilibrium cost
Numerical Implementation
The iterative scheme (3.9) can be implemented numerically, although there are technical details regarding computing conditional expectation with respect to the filtration The trade-off in terms of speed for each iteration is considered worth it
Final Conclusion on Convergence
In conclusion, the sequence (αn)n∈N generated by the fictitious play converges strongly in HTd to the optimal control α∗ solution of the mean field control problem when appropriate convexity assumptions are met This result holds even in the presence of common noise under certain conditions The representative player process is exactly the process which to each player ω maps its optimal control and state
References
[1] Christian Bender and Jianfeng Zhang. Time discretization and markovian iteration for coupled fbsdes
[2] Alain Bensoussan, Jens Frehse, and Phillip Yam. Mean Field Games and Mean Field Type Control Theory
[3] Charles Bertucci. Monotone solutions for mean field games master equations: continuous state space and common noise
[4] Charles Bertucci. Analysis on spaces of measures
[5] Charles Bertucci, Jean-Michel Lasry, and Pierre-Louis Lions. On lipschitz solutions of mean field games master equations
[6] Charles Bertucci and Charles Meynard. A study of common noise in mean field games
[7] Charles Bertucci and Charles Meynard. Noise through an additional variable for mean field games master equation on finite state space
[8] Philippe Briand and Céline Labart. Simulation of bsdes by wiener chaos expansion
[9] George W. Brown. Iterative solution of games by fictitious play
[10] Peter Caines, Minyi Huang, and Roland Malhame. Large population stochastic dynamic games: closed-loop mckean-vlasov systems and the nash certainty equivalence principle
[11] Pierre Cardaliaguet, François Delarue, Jean-Michel Lasry, and Pierre-Louis Lions. The Master Equation and the Convergence Problem in Mean Field Games: (AMS-201)
[12] Pierre Cardaliaguet and Saeed Hadikhanloo. Learning in mean field games: The fictitious play
[13] Pierre Cardaliaguet and Charles-Albert Lehalle. Mean field game of controls and an application to trade crowding
[14] René Carmona and François Delarue. Probabilistic Theory of Mean Field Games with Applications I–II, volume 83–84 of Probability Theory and Stochastic Modelling
[15] René Carmona, François Delarue, and Daniel Lacker. Mean field games with common noise
[16] Alekos Cecchin and François Delarue. Weak Solutions to the Master Equation of Potential Mean Field Games
[17] Mao Fabrice Djete. Extended mean field control problem: a propagation of chaos result
[18] Mao Fabrice Djete. Large population games with interactions through controls and common noise: convergence results and equivalence between open-loop and closed-loop controls
[19] Romuald Elie, Julien Pérolat, Mathieu Laurière, Matthieu Geist, and Olivier Pietquin. On the convergence of model free learning in mean field games
[20] E. A. Feinberg, P. O. Kasyanov, and Y. Liang. Fatou’s lemma for weakly converging measures under the uniform integrability condition
[21] Drew Fudenberg and David K. Levine. The Theory of Learning in Games, volume 2 of MIT Press Series on Economic Learning and Social Evolution
[22] Saeed Hadikhanloo, Rida Laraki, Panayotis Mertikopoulos, and Sylvain Sorin. Learning in nonatomic games, part i: Finite action spaces and population games
[23] Saeed Hadikhanloo and Francisco J. Silva. Finite mean field games: Fictitious play and convergence to a first order continuous mean field game
[24] Daniel Lacker. Limit theory for controlled mckean–vlasov dynamics
[25] Jean-Michel Lasry and Pierre-Louis Lions. Mean field games
[26] Pierre Lavigne and Laurent Pfeiffer. Generalized conditional gradient and learning in potential mean field games
[27] P.-L. Lions. Lectures at collège de france, 2007-2011
[28] Alpár R. Mészáros and Chenchen Mou. Mean field games systems under displacement monotonicity
[29] Charles Meynard. Monotone solutions to mean field games master equation in the l2-monotone setting
[30] Charles Meynard. Extragradient methods for mean field games of controls and mean field type fbsdes
[31] Dov Monderer and Lloyd S. Shapley. Fictitious play property for games with identical interests
[32] Sarah Perrin, Julien Perolat, Mathieu Laurière, Matthieu Geist, Romuald Elie, and Olivier Pietquin. Fictitious play for mean field games: Continuous time analysis and applications
[33] Julia Robinson. An iterative method of solving a game
[34] Rubio, Gerardo. Existence and uniqueness to the cauchy problem for linear and semilinear parabolic equations with local conditions
[35] Jianfeng Zhang. Backward Stochastic Differential Equations: From Linear to Fully Nonlinear Theory, volume 86 of Probability Theory and Stochastic Modelling
--- Page 1 ---
Mean field games are investigated through two learning procedures and their relationship to the formation of equilibrium in potential mean field games The first one is a version of the fictitious play in which players observe directly the actions (controls) of other players and learn the distribution of the MFG through an update rules on controls On the other hand we also consider the learning procedure of a principal agent trying to solve a mean field control problem with only local gradient information
The first one is a version of the fictitious play in which players observe directly the actions (controls) of other players and learn the distribution of the MFG through an update rules on controls This leads to a new iterative procedure which converges for a wide class of potential MFGs of controls, even in the presence of common noise On the other hand we also consider the learning procedure of a principal agent trying to solve a mean field control problem with only local gradient information We prove that under very general assumption, the gradient descent of such agent converges to a solution of the associated potential MFG
The fictitious play procedure involves an iterative scheme where players choose the best response to their belief on the distribution of other players and then update their belief on the control played by the population This new update rule for the fictitious play converges under a different set of assumptions and can, in particular, handle MFGs of controls [13]
Under Hypotheses 3.2 and 3.1, if σ > 0, then for any sequence (ηn)n∈N satisfying (3.4), the sequence (αn)n∈N generated by (3.2) is precompact in HTd and any limit point α∗ is a solution of the mean field game problem
The iterative scheme (3.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved Fictitious Play Learning for Controls: The system can now learn optimal controls by directly observing other players' actions instead of just learning from a distribution, as described in equation (1.4). This allows for a more intuitive learning procedure where agents update their belief on the control played by the population through an explicit update rule on controls.
-
Gradient Descent for Mean Field Control: The AI can perform gradient descent directly on the space of stochastic processes using the gradient of the potential J, defined in Lemma 3.3 and equation (3.9). This iterative procedure converges to a solution of the mean field game problem, connecting agent learning to control optimization.
-
Convergence to Optimal Controls under Potential Convexity: When Hypothesis 2.14 holds (convexity assumption), the fictitious play sequence converges strongly in HTd to the unique optimal control alpha star of the mean field control problem (2.6). This provides a strong convergence guarantee for agents learning in a potential MFG setting with common noise.
-
Robustness to Common Noise: The system can handle common noise by relying on relaxed controls and measure valued MFG equilibrium definitions, leading to Theorem 2.13 showing precompactness of the sequence Qn and its limit points are measure valued MFG equilibria. This extends learning results beyond the lack of compactness caused by common noise.
-
Learning from Black Box Algorithms: The system can learn Nash equilibria even without knowing the coefficients L and G, provided it has a block box algorithm that yields optimal trajectories for a given prior distribution. This capability is useful for machine learning where real-world trajectories are available as data without prior knowledge of the optimization problem.
Related papers
- An Accelerated Proximal Gradient Method for Bures-Wasserstein Space
- FOMTools: Development and AI-Assisted Code Generation for First-Order Methods
- Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise
- Dynamic Service Recommendation with Congestion-Dependent Joining: Near-Optimal and Constant-Factor Approximation Algorithms
- Integrating Learning and Optimization for Open Transport Capacity Pooling under Disruption
- Zeroth-Order Gradient Estimators under Decision-Dependent Noise