From Individual Learning to Market Equilibrium: Correcting Structural and Parametric Biases in RL Simulations of Economic Models

summary

The gist

The application of Reinforcement Learning (RL) to economic modeling reveals a fundamental conflict between the assumptions of equilibrium theory and the emergent behavior of learning…

In short

Applying Reinforcement Learning (RL) to economic models creates simulation biases: structural bias makes agents act like market manipulators, and parametric bias misinterprets time costs. The solution is Calibrated Mean-Field RL, which embeds an agent in a macroeconomic field and adjusts the cost function to align agent behavior with theoretical competitive equilibrium.

Key concepts

Structural Bias (Agency-Structure Dilemma)
This occurs when economic models assume agents are passive 'takers' of market conditions, but a naive single-agent RL setup incentivizes the agent to become an active 'manipulator.' The RL agent learns to strategically alter market variables for its own gain rather than following equilibrium rules.
Parametric Bias
This bias arises from misaligned interpretations of time preferences and costs. Economic models use parameters like the interest rate (r) to capture opportunity costs, whereas standard RL often uses a fixed discount factor ($eta$) and treats costs as simple per-period penalties, leading to inaccurate cost evaluations.
Calibrated Mean-Field Reinforcement Learning (Calibrated MF-RL)
This proposed framework embeds the agent within a larger macroeconomic field (MFG formulation) where aggregate variables are treated as fixed. It corrects the parametric bias by introducing an effective cost ($c_{eff}$) that internalizes both job duration and capital opportunity costs, ensuring the agent's hiring decisions match economic theory.
Contraction Mapping (Theorem 1)
This mathematical proof justifies the convergence of the learning algorithm. It shows that under certain conditions, the iterative update process for market variables ($ heta_{k+1} = ext{Ψ}( heta_k)$) is a contraction mapping, guaranteeing that it converges exponentially fast to a unique fixed point representing the true economic equilibrium.

Terminology used across episodes

This episode discusses

The paper

From Individual Learning to Market Equilibrium: Correcting Structural and Parametric Biases in RL Simulations of Economic Models · Read on arXiv

Nagoya University · Ulm University

DOI: 10.1007/s10614-026-11464-y

Transcript

Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.

Ruth: I'm Ruth.

Warren: And I'm Warren, here with Iris.

Ruth: We're looking at the paper: "From Individual Learning to Market Equilibrium".

Warren: Here's the gist of the paper. The application of Reinforcement Learning (RL) to economic modeling reveals a fundamental conflict between the assumptions of equilibrium theory and the emergent behavior of learning agents This paper first demonstrates this discrepancy within a search-and-matching model with concave production, showing that a standard RL agent learns a non-equilibrium, monopsonistic policy Additionally, we identify a parametric bias arising from the mismatch between economic discounting and RL’s treatment of intertemporal costs To address both issues, we propose a calibrated Mean-Field Reinforcement Learning framework that embeds a representative agent in a fixed macroeconomic field and adjusts the cost function to reflect economic opportunity costs Our iterative algorithm converges to a self-consistent fixed point where the agent’s policy aligns with the competitive equilibrium The Core Discrepancy in Simulation The paper identifies two distinct sources of bias that arise when naively translating economic models into RL settings The structural bias reflects what we term an agency-structure dilemma, where many economic models assume atomistic agents act as ‘takers’ of aggregate market conditions, but a naive single-agent RL simulation incentivizes the agent to become a ‘manipulator’ of its environment This leads an RL agent tasked with optimizing firm behavior to learn a policy that strategically manipulates market variables to its advantage, behaving effectively as a monopsonist The parametric bias stems from misaligned interpretations of costs and discounting, where economic models incorporate time preferences and opportunity costs through parameters like the interest rate r, while RL models represent time preferences via a fixed discount factor β and often interpret costs as per-period penalties Diagnosing Simulation Errors The authors diagnose the dual simulation bias problem by comparing the theoretical steady-state equilibrium with a naive RL simulation The structural bias is demonstrated by showing that the RL agent learns a policy that results in a market tightness far below the equilibrium level, raising the question why This is explained by analyzing the RL agent’s objective, which shows it will deliberately reduce v to suppress θ and lower w if it internalizes that θ depends on v The parametric bias is identified through a mismatch in cost specification, where the expected discounted cost of hiring in the economic model is (r + λ)c q(θ), but the RL simulation uses an effective cost of λc q(θ), leading to an underestimation of true economic costs The Proposed Unified Solution To correct both sources of bias, the paper proposes a unified framework called Calibrated Mean-Field Reinforcement Learning (Calibrated MF-RL) This approach addresses the structural bias by embedding the agent within a MFG formulation, ensuring that aggregate variables like θ are treated as exogenous during optimization but endogenously updated across iterations The parametric bias is corrected by introducing an effective cost parameter ceff, defined as ceff:= 1 + rλ/c, which internalizes both the job’s expected duration and the capital opportunity cost This calibration ensures that the RL agent evaluates its hiring decision in a way equivalent to the firm in the economic model Validation and Convergence The effectiveness of this framework is validated through simulation results, which demonstrate convergence toward the theoretical economic equilibrium The authors show that only when both corrections are applied does the learned agent behavior align with the competitive equilibrium, effectively simulating a “price-taking” firm The convergence of the RL-based mean field learning algorithm is theoretically justified by Theorem 1, which proves that under certain Lipschitz continuity assumptions on the policy map and environment response map, the composite mapping Ψ is a contraction mapping This guarantees that the iterates θk+1 = Ψ(θk) converge exponentially fast to a unique fixed point θ∗, corresponding to the equilibrium of the corrected system The ablation studies confirm that correcting only one bias is insufficient for equilibrium alignment This paper demonstrates that naively applying RL to economic models can produce systematically biased outcomes due to a mismatch between the agent’s optimization structure and the assumptions embedded in economic theory The authors conclude that correcting both structural and parametric biases is necessary for faithful computational modeling of equilibrium behavior This work offers a blueprint for more reliable application of RL in economic analysis by demanding careful alignment between the agent’s learning objective and the theoretical underpinnings of the economic environment Appendix A Equations for Steady State The equations for the steady state are presented, including several key conditions derived from the firm's optimality and market equilibrium Table 2 summarizes these theoretical steady state values for variables such as l, u, q, w, v, and θ Appendix B RL Simulation settings The simulation details specify the use of a Deep Deterministic Policy Gradient (DDPG) algorithm with an actor-critic architecture for the learning agent The state space is defined by ot = (lt, ut), and the action space consists of a single continuous action vt representing vacancy posting intensity Appendix C Ablation Study The ablation study shows that only structural correction leads to over-optimistic hiring, while only parametric correction causes the agent to behave as a market manipulator Figure 3 compares the learned θ across these two settings, showing convergence to the correct value only when both corrections are applied Appendix D Convergence of the RL-based Mean Field Fixed-Point Iteration Theorem 1 formalizes the convergence proof, stating that if Assumptions (A1) and (A2) hold and L = L1L2 < 1, then the composite mapping Ψ is a contraction mapping This leads to exponential convergence of the mean field sequence θk+1 = Ψ(θk) to a unique fixed point θ∗, which corresponds to the equilibrium of the corrected system References The paper cites numerous foundational and related works in economics and RL, including Smith

1999: , Pissarides

2000: , Sutton and Barto

2018: , and various papers on Mean Field Games The references listed include those for MF-RL, economic modeling, and deep reinforcement learning applications The reference list is extensive, covering topics from search-and-matching models to agent-based macroeconomic modeling This paper demonstrates that naively applying RL to economic models can produce systematically biased outcomes due to a mismatch between the agent’s optimization structure and the assumptions embedded in economic theory The authors conclude that correcting both structural and parametric biases is necessary for faithful computational modeling of equilibrium behavior This work offers a blueprint for more reliable application of RL in economic analysis by demanding careful alignment between the agent’s learning objective and the theoretical underpinnings of the economic environment Appendix A Equations for Steady State Equations for Steady State are provided, including several key conditions derived from the firm's optimality and market equilibrium Table 2 summarizes these theoretical steady state values for variables such as l, u, q, w, v, and θ References

Paper summary: Iris: So, to recap, this paper "From Individual Learning to Market Equilibrium: Correcting Structural and Parametric Biases in RL Simulations of Economic Models" points out two main problems with naive RL applications: the structural bias where agents become manipulators because the simulation incentivizes them that way, and the parametric bias from misaligning how we treat time preferences through discounting versus actual economic opportunity costs.

Ruth: That's right. The paper first demonstrates this discrepancy within a search-and-matching model with concave production, showing that a standard reinforcement learning agent learns a non-equilibrium, monopsonistic policy.

Warren: And then they identify that parametric bias arising from the mismatch between economic discounting and the reinforcement learning treatment of intertemporal costs. That means the way time preference is modeled doesn't match how it works in reality.

Iris: To address both issues, they propose a calibrated Mean-Field Reinforcement Learning framework. This approach embeds a representative agent in a fixed macroeconomic field and adjusts the cost function to reflect economic opportunity costs.

Ruth: It’s about ensuring that when you have many agents, like in the aggregate market, the individual agent learns its role within that larger structure correctly.

Warren: They claim this unified framework addresses both sources of bias simultaneously. It resolves the structural issue by ensuring agents internalize their atomistic role in aggregate dynamics and fixes the parametric issue by calibrating costs to reflect the total intertemporal cost of vacancy creation implied by the economic model.

Iris: The authors show that when you apply this calibrated Mean-Field Reinforcement Learning framework, their iterative algorithm converges to a self-consistent fixed point where the agent’s policy aligns with the competitive equilibrium.

Ruth: That convergence is key because it means the simulation settles on a state that matches the theoretical economic prediction, even though we started with standard RL setups that were off course.

Warren: It suggests that instead of seeing biased results in these simulations, you need to apply this calibration work to make sure they model something real, like market equilibrium.

Iris: But the paper is careful about what it doesn't do. They note that as the number of agents increases, the joint state-action space expands exponentially, which leads to scalability issues that motivate this search for a more parsimonious framework.

Ruth: So they’re not just fixing one error; they’re building a whole new way to model these interactions that handles both the structural and parametric problems at once.

Conclusion: Warren: So, to wrap up this piece on "From Individual Learning to Market Equilibrium: Correcting Structural and Parametric Biases in RL Simulations of Economic Models," it really boils down to fixing two big errors when we use reinforcement learning for economics. One is the structural bias, where the agent acts like a manipulator instead of just a price taker.

Iris: And the other is parametric bias, which comes from misaligning how we handle time preferences and opportunity costs between our economic model and the reinforcement learning setup.

Ruth: To get to that real market equilibrium, you need both corrections applied simultaneously. The authors show that fixing only one of those biases isn't enough to get the simulation right.

Warren: They use a framework called Calibrated Mean-Field Reinforcement Learning to embed the agent in a field and calibrate its costs to match economic reality.

Iris: And they prove mathematically, using something called Theorem One, that this iterative process converges exponentially fast to that correct equilibrium point under specific mathematical conditions.

Ruth: It’s a blueprint for applying reinforcement learning reliably in economics because it shows exactly how to align the agent's learning goal with the underlying economic theory.

Warren: If you want your RL simulations to model something real, like market equilibrium, you can't just use a standard setup; you have to do this calibration work.

Iris: It highlights that for anyone building economic models with reinforcement learning, setting up the objective function and defining time preferences is absolutely critical.

Ruth: So the main thing here is making sure we move past just getting a simulation running and actually modeling what markets settle into when you use these techniques.

More episodes

← Home